第34期 | Quantifying infrastructure noise in agentic coding evals
今日摘要
X Andrej Karpathy：Farzapedia, personal wikipedia of Farza, good example following my Wiki LLM tweet. I really like this approach to personalization…
X Andrej Karpathy：Something I've been thinking about - I am bullish on people (empowered by AI) increasing the visibility, legibility and accountabi…
X Andrej Karpathy：Wow, this tweet went very viral! I wanted share a possibly slightly improved version of the tweet in an "idea file". The idea of t…
X Andrej Karpathy：LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of…
OpenAI Blog：OpenAI acquires TBPN to accelerate global conversations around AI and support independent media, expanding dialogue with builders,…
总结 + 观点：Codex now includes pay-as-you-go pricing for Cha…｜中文观点：围绕 Codex now offers more flexible pricing for…
总结 + 观点：OpenAI engineering 列表显示，Responses API agent comp…｜中文观点：From model to agent: Equipping the Responses…
总结 + 观点：OpenAI RSS Model Spec agent｜中文观点：从 Inside our approach to the Model Spec 看，后续更…
总结 + 观点：Anthropic agentic coding benchmark，波动甚至可能超过榜单模型之…｜中文观点：比起表面参数，Quantifying infrastructure noise in ag…
总结 + 观点：Anthropic harness agent runtime、上下文和安全边界设计问题。｜中文观点：从 Harness design for long-running application…
Farzapedia, personal wikipedia of Farza, good example following my Wiki LLM tweet.
标签：#x_profiles #extended
作者：
原文：Farzapedia, personal wikipedia of Farza, good example following my Wiki LLM tweet. I really like this approach to personalization in a number of ways, compared to "status quo" of an AI that allegedly gets better the more you use it or something: 1. Explicit. The memory artifact is explicit and navigable (the wiki), you can see exactly what the AI does and does not know and you can inspect and manage this artifact, even if you don't do the direct text writing (the LLM does). The knowledge of you is not implicit and unknown, it's explicit and viewable. 2. Yours. Your data is yours, on your local computer, it's not in some particular AI provider's system without the ability to extract it. You're in control of your information. 3. File over app. The memory here is a simple collection of files in universal formats (images, markdown). This means the data is interoperable: you can use a very large collection of tools/CLIs or whatever you want over this information because it's just files. The agents can apply the entire Unix toolkit over them. They can natively read and understand them. Any kind of data can be imported into files as input, and any kind of interface can be used to view them as the output. E.g. you can use Obsidian to view them or vibe code something of your own. Search "File over app" for an article on this philosophy. 4. BYOAI. You can use whatever AI you want to "plug into" this information - Claude, Codex, OpenCode, whatever. You can even think about taking an open source AI and finetuning it on your wiki - in principle, this AI could "know" you in its weights, not just attend over your data. So this approach to personalization puts *you* in full control. The data is yours. In Universal formats. Explicit and inspectable. Use whatever AI you want over it, keep the AI companies on their toes! Certainly this is not the simplest way to get an AI to know you - it does require you to manage file directories and so on, but agents also make it quite simple and they can help you a lot. I imagine a number of products might come out to make this all easier, but imo "agent proficiency" is a CORE SKILL of the 21st century. These are extremely powerful tools - they speak English and they do all the computer stuff for you. Try this opportunity to play with one. Farza (@FarzaTV) This is Farzapedia. I had an LLM take 2,500 entries from my diary, Apple Notes, and some iMessage convos to create a personal Wikipedia for me. It made 400 detailed articles for my friends, my startups, research areas, and even my favorite animes and their impact on me complete with backlinks. But, this Wiki was not built for me! I built it for my agent! The structure of the wiki files and how it's all backlinked is very easily crawlable by any agent makes it a truly useful knowledge base. I can spin up Claude Code on the wiki and starting at index.md (a catalog of all my articles) the agent does a really good job at drilling into the specific pages on my wiki it needs context on when I have a query. For example, when trying to cook up a new landing page I may ask: "I'm trying to design this landing page for a new idea I have. Please look into the images and films that inspired me recently and give me ideas for new copy and aesthetics". In my diary I kept track of everything from: learnings, people, inspo, interesting links, images. So the agent reads my wiki and pulls up my "Philosophy" articles from notes on a Studio Ghibli documentary, "Competitor" articles with YC companies whose landing pages I screenshotted, and pics of 1970s Beatles merch I saved years ago. And it delivers a great answer. I built a similar system to this a year ago with RAG but it was ass. A knowledge base that lets an agent find what it needs via a file system it actually understands just works better. The most magical thing now is as I add new things to my wiki (articles, images of inspo, meeting notes) the system will likely update 2-3 different articles where it feels that context belongs, or, just creates a new article. It's like this super genius librarian for your brain that's always filing stuff for your perfectly and also let's you easily query the knowledge for tasks useful to you (ex. design, product, writing, etc) and it never gets tired. I might spend next week productizing this, if that's of interest to you DM me tell me your usecase! Video https://nitter.net/FarzaTV/status/2040563939797504467#m
Something I've been thinking about - I am bullish on people (empowered by AI) increasing the visibility, legibility and accountability of th...
标签：#x_profiles #extended
作者：
原文：Something I've been thinking about - I am bullish on people (empowered by AI) increasing the visibility, legibility and accountability of their governments. Historically, it is the governments that act to make society legible (e.g. "Seeing like a state" is the common reference), but with AI, society can dramatically improve its ability to do this in reverse. Government accountability has not been constrained by access (the various branches of government publish an enormous amount of data), it has been constrained by intelligence - the ability to process a lot of raw data, combine it with domain expertise and derive insights. As an example, the 4000-page omnibus bill is "transparent" in principle and in a legal sense, but certainly not in a practical sense for most people. There's a lot more like it: laws, spending bills, federal budgets, freedom of information act responses, lobbying disclosures... Only a few highly trained professionals (investigative journalists) could historically process this information. This bottleneck might dissolve - not only are the professionals further empowered, but a lot more people can participate. Some examples to be precise: Detailed accounting of spending and budgets, diff tracking of legislation, individual voting trends w.r.t. stated positions or speeches, lobbying and influence (e.g. graph of lobbyist - firm - client - legislator - committee - vote - regulation), procurement and contracting, regulatory capture warning lights, judicial and legal patterns, campaign finance... Local governments might be even more interesting because the governed population is smaller so there is less national coverage: city council meetings, decisions around zoning, policing, schools, utilities... Certainly, the same tools can easily cut the other way and it's worth being very mindful of that, but I lean optimistic overall that added participation, transparency and accountability will improve democratic, free societies. (the quoted tweet is half-ish related, but inspired me to post some recent thoughts) Harry Rushworth (@Hrushworth) The British Government is a complicated beast. Dozens of departments, hundreds of public bodies, more corporations than one can count... Such is its complexity that there isn't an org chart for it. Well, there wasn't... Introducing ⚙️Machinery of Government⚙️ https://nitter.net/Hrushworth/status/2040406616806179001#m
Wow, this tweet went very viral! I wanted share a possibly slightly improved version of the tweet in an "idea file".
标签：#x_profiles #extended
作者：
原文：Wow, this tweet went very viral! I wanted share a possibly slightly improved version of the tweet in an "idea file". The idea of the idea file is that in this era of LLM agents, there is less of a point/need of sharing the specific code/app, you just share the idea, then the other person's agent customizes builds it for your specific needs. So here's the idea in a gist format: gist.github.com/karpathy/442… You can give this to your agent and it can build you your own LLM wiki and guide you on how to use it etc. It's intentionally kept a little bit abstract/vague because there are so many directions to take this in. And ofc, people can adjust the idea or contribute their own in the Discussion which is cool. Andrej Karpathy (@karpathy) LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge (stored as markdown and images). The latest LLMs are quite good at it. So: Data ingest: I index source documents (articles, papers, repos, datasets, images, etc.) into a raw/ directory, then I use an LLM to incrementally "compile" a wiki, which is just a collection of .md files in a directory structure. The wiki includes summaries of all the data in raw/, backlinks, and then it categorizes data into concepts, writes articles for them, and links them all. To convert web articles into .md files I like to use the Obsidian Web Clipper extension, and then I also use a hotkey to download all the related images to local so that my LLM can easily reference them. IDE: I use Obsidian as the IDE "frontend" where I can view the raw data, the the compiled wiki, and the derived visualizations. Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly. I've played with a few Obsidian plugins to render and view data in other ways (e.g. Marp for slides). Q&A: Where things get interesting is that once your wiki is big enough (e.g. mine on some recent research is ~100 articles and ~400K words), you can ask your LLM agent all kinds of complex questions against the wiki, and it will go off, research the answers, etc. I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents and it reads all the important related data fairly easily at this ~small scale. Output: Instead of getting answers in text/terminal, I like to have it render markdown files for me, or slide shows (Marp format), or matplotlib images, all of which I then view again in Obsidian. You can imagine many other visual output formats depending on the query. Often, I end up "filing" the outputs back into the wiki to enhance it for further queries. So my own explorations and queries always "add up" in the knowledge base. Linting: I've run some LLM "health checks" over the wiki to e.g. find inconsistent data, impute missing data (with web searchers), find interesting connections for new article candidates, etc., to incrementally clean up the wiki and enhance its overall data integrity. The LLMs are quite good at suggesting further questions to ask and look into. Extra tools: I find myself developing additional tools to process the data, e.g. I vibe coded a small and naive search engine over the wiki, which I both use directly (in a web ui), but more often I want to hand it off to an LLM via CLI as a tool for larger queries. Further explorations: As the repo grows, the natural desire is to also think about synthetic data generation finetuning to have your LLM "know" the data in its weights instead of just context windows. TLDR: raw data from a given number of sources is collected, then compiled by an LLM into a .md wiki, then operated on by various CLIs by the LLM to do Q&A and to incrementally enhance the wiki, and all of it viewable in Obsidian. You rarely ever write or edit the wiki manually, it's the domain of the LLM. I think there is room here for an incredible new product instead of a hacky collection of scripts. https://nitter.net/karpathy/status/2039805659525644595#m
LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research...
标签：#x_profiles #extended
作者：
原文：LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge (stored as markdown and images). The latest LLMs are quite good at it. So: Data ingest: I index source documents (articles, papers, repos, datasets, images, etc.) into a raw/ directory, then I use an LLM to incrementally "compile" a wiki, which is just a collection of .md files in a directory structure. The wiki includes summaries of all the data in raw/, backlinks, and then it categorizes data into concepts, writes articles for them, and links them all. To convert web articles into .md files I like to use the Obsidian Web Clipper extension, and then I also use a hotkey to download all the related images to local so that my LLM can easily reference them. IDE: I use Obsidian as the IDE "frontend" where I can view the raw data, the the compiled wiki, and the derived visualizations. Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly. I've played with a few Obsidian plugins to render and view data in other ways (e.g. Marp for slides). Q&A: Where things get interesting is that once your wiki is big enough (e.g. mine on some recent research is ~100 articles and ~400K words), you can ask your LLM agent all kinds of complex questions against the wiki, and it will go off, research the answers, etc. I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents and it reads all the important related data fairly easily at this ~small scale. Output: Instead of getting answers in text/terminal, I like to have it render markdown files for me, or slide shows (Marp format), or matplotlib images, all of which I then view again in Obsidian. You can imagine many other visual output formats depending on the query. Often, I end up "filing" the outputs back into the wiki to enhance it for further queries. So my own explorations and queries always "add up" in the knowledge base. Linting: I've run some LLM "health checks" over the wiki to e.g. find inconsistent data, impute missing data (with web searchers), find interesting connections for new article candidates, etc., to incrementally clean up the wiki and enhance its overall data integrity. The LLMs are quite good at suggesting further questions to ask and look into. Extra tools: I find myself developing additional tools to process the data, e.g. I vibe coded a small and naive search engine over the wiki, which I both use directly (in a web ui), but more often I want to hand it off to an LLM via CLI as a tool for larger queries. Further explorations: As the repo grows, the natural desire is to also think about synthetic data generation finetuning to have your LLM "know" the data in its weights instead of just context windows. TLDR: raw data from a given number of sources is collected, then compiled by an LLM into a .md wiki, then operated on by various CLIs by the LLM to do Q&A and to incrementally enhance the wiki, and all of it viewable in Obsidian. You rarely ever write or edit the wiki manually, it's the domain of the LLM. I think there is room here for an incredible new product instead of a hacky collection of scripts.
OpenAI acquires TBPN
标签：#ai_engineering_blogs #core
作者：
原文：OpenAI acquires TBPN to accelerate global conversations around AI and support independent media, expanding dialogue with builders, businesses, and the broader tech community.
Codex now offers more flexible pricing for teams
标签：#ai_engineering_blogs #core
作者：
原文：Codex now includes pay-as-you-go pricing for ChatGPT Business and Enterprise, providing teams a more flexible option to start and scale adoption.
链接：https://openai.com/index/codex-flexible-pricing-for-teams
From model to agent: Equipping the Responses API with a computer environment
标签：#uncategorized #core
作者：
原文：OpenAI engineering 列表显示，Responses API agent computer environment，这意味着模型调用正在往更完整的 agent runtime
链接：https://openai.com/index/equip-responses-api-computer-environment/
Inside our approach to the Model Spec
标签：#uncategorized #core
作者：
原文：OpenAI RSS Model Spec agent
Quantifying infrastructure noise in agentic coding evals
标签：#uncategorized #core
作者：
原文：Anthropic agentic coding benchmark，波动甚至可能超过榜单模型之间的差距。这对 agent eval
链接：https://www.anthropic.com/engineering/infrastructure-noise
Harness design for long-running application development
标签：#uncategorized #core
作者：
原文：Anthropic harness agent runtime、上下文和安全边界设计问题。
链接：https://www.anthropic.com/engineering/harness-design-long-running-apps
2025 LLM Year in Review
标签：#uncategorized #core
作者：
原文：Karpathy 2025 LLM RLVR reasoning test-time compute
Gradient Labs gives every bank customer an AI account manager
标签：#ai_engineering_blogs #core
作者：
原文：Gradient Labs uses GPT-4.1 and GPT-5.4 mini and nano to power AI agents that automate banking support workflows with low latency and high reliability.
Announcing the OpenAI Safety Fellowship
标签：#ai_engineering_blogs #core
作者：
原文：A pilot program to support independent safety and alignment research and develop the next generation of talent
链接：https://openai.com/index/introducing-openai-safety-fellowship
Industrial policy for the Intelligence Age
标签：#ai_engineering_blogs #core
作者：
原文：Explore our ambitious, people-first industrial policy ideas for the AI era—focused on expanding opportunity, sharing prosperity, and building resilient institutions as advanced intelligence evolves.
链接：https://openai.com/index/industrial-policy-for-the-intelligence-age
Accelerating the next phase of AI
标签：#ai_engineering_blogs #core
作者：
原文：OpenAI raises $122 billion in new funding to expand frontier AI globally, invest in next-generation compute, and meet growing demand for ChatGPT, Codex, and enterprise AI.
HookProbe Open-source AI IDs that runs on a $75 Raspberry Pi
标签：#research_community #extended
作者：
原文：在 75 美元树莓派上跑的开源 AI 入侵检测系统，把端侧 ML 安全这条路径开放出来。
Context Engineering for AI Coding Agents
标签：#research_community #extended
作者：
原文：讲 Claude Code 里 subagent 模式的上下文工程实践，关注怎么把大任务拆给不同 agent 又不让上下文爆掉。
huggingface/smolagents
标签：#github_orgs #extended
作者：
原文：Hugging Face 的小型 agent 框架，强调代码极简和可 hack，与 LangChain 等重型栈形成明显对照。
Guinndex. AI-agent called 3k Irish pubs to map the price of a pint of Guinness
标签：#research_community #extended
作者：
原文：用 AI agent 电话打给 3000 家爱尔兰酒吧，采集并可视化了当地黑啤价格，既是段子也是真实 agent 产品原型。
guide.world: A compendium of travel guides
标签：#research_community #extended
作者：
原文：一个旅行攻略聚合站，定位在把零散 UGC 内容整理成结构化 itinerary，以 AI 辅助编辑为卖点。