A processor compiled into model weights, twenty hostile scripts, and a quiet hour treated as an outage: four agent-builder signals from October 24-31
Four in-window builds where the verdict on the agent's work came from outside the agent: an NES processor compiled into 4B model weights and checked against a test log from 2011, an agent-written evaluation of a 5 MB browser runner, a game company that treats an hour without a ship as an outage, and a coding agent run with the network switched off.
Four pieces of work landed between October 24 and October 31, and each one had to answer the same question: how do you know the agent got it right?
Drew Breunig compiled a Nintendo processor into the weights of a small language model, then checked it against a test log committed in 2011. Simon Willison had an agent take a 5 MB browser runner apart and write down what broke.
Jacob Sansbury runs a game company where ten people ship 108 changes an hour, and an hour without a ship counts as an outage. Rittika Jindal ran a coding agent from the foothills of the Himalayas with her network switched off.
This issue is about what each of them accepted as the verdict, and what that verdict cost.
The comparison map
| Signal | Builder or project | In-window date | Artifact to open | First test |
|---|---|---|---|---|
| A processor compiled into model weights | Drew Breunig / nesformer | Oct 30 | nesformer | Run the repository's check against a test ROM. |
| An agent's written verdict on a browser runner | Jeff Lindsay / webframe, reviewed by Simon Willison | Oct 24-30 | webframe: evaluated as a scraper and a sandbox | Ask an agent to break one dependency you already use. |
| A company that treats a quiet hour as an outage | Jacob Sansbury / Spawn | Oct 29 | How I Built the Most AI-Pilled Company | Count how long your tests take to notice a break. |
| A coding agent with the network switched off | Rittika Jindal / OpenCode, Ollama, Qwen | Oct 27 | Coding From the Himalayas | Pull your own network and fix one failing test. |
Four signals to inspect
1. A processor compiled into model weights
On October 30, Drew Breunig published nesformer, a repository that carries a working 6502 CPU, its 2 KB of work RAM and a cartridge's program ROM inside a stock Qwen3.5 checkpoint 1.
Nobody trained those weights. A compiler computed every one of them, and the model predicts the CPU's execution log one token at a time 1. A small host program around it draws the picture, reads the controller and fires the vblank interrupt.
The technique comes from Qwenputer, Divyansh Agrawal's experiment eight days earlier that compiled a RISC-V CPU into Qwen3.5-4B and used the model's Gated DeltaNet layers as its memory 2. Breunig reused that compiler and that one trick, and pointed them at a console people already have ROMs for.
The verification is the part to copy. It starts with a test ROM called nestest, whose golden log by Christopher Pow was committed to a public repository in August 2011 3.
Breunig wrote a reference 6502 in Python and matched it against that log for 3,348 instructions, then against the py65 emulator across 5.7 million instructions of game code 1.
With the reference agreed, the weights had to reproduce it. Run on their own, they generated traces identical to the reference CPU: 4,586 tokens for nestest ending in the expected HALT, and 700,000 tokens for 271 frames of Thwaite 1.
The longer runs hold up too. Six hundred frames of Thwaite took 1.9 million tokens and all 300 saved frames matched; a second game took 2.6 million tokens for 420 frames with all 210 matching 1.
The repository ships that comparison as a command.
python -m nesformer check ROM runs the weights and the reference CPU side by side and stops at the first difference 1.
What it costs is speed and scope. The engine runs at about 1,650 tokens per second on one CPU core, and a real NES would need roughly 700,000, which puts the model 150 to 290 times slower than the console 1. A frame takes one to six seconds, so ten seconds of game time takes about 25 minutes.
The README lists what the model cannot do. Only NROM cartridges work, because a bank-switching mapper would need the whole ROM inside the model's state; unofficial opcodes and jumps into RAM emit a HALT 1. The picture and sound chips sit outside the weights, so there is no audio and mid-scanline effects do not survive.
Breunig's own account of how it got built is in a Hacker News comment: one prompt asking Opus 5.5 to read the Qwenputer write-up, explore whether the NES chip and RAM could be emulated the same way, and pick a model size if so. 4
Try it. Take one deterministic piece of your own system, write the plain implementation of it yourself, and make the agent's version match yours instruction by instruction.
2. An agent's written verdict on a 5 MB browser runner
webframe is Jeff Lindsay's project, published under the tractordev account and released twice this week: v0.1 on October 24 and v0.2 on October 30 67.
The idea is to run a web page the way you run a program. You point it at a URL or a file, its
console output goes to stdout, exceptions to stderr, and you can evaluate JavaScript in the page from stdin 6.It uses the webview the operating system already ships, through the glaze library, so there is no bundled browser engine and no cgo on most platforms 6. The whole thing builds to a 5.3 MB binary from about 460 lines of Go and JavaScript 8.
Simon Willison read webframe as a smaller alternative to his own
shot-scraper javascript command and handed the evaluation to an agent 9.The prompt was one paragraph asking Claude Opus 5.5 to try webframe, build a few exploratory prototypes, and evaluate it as a sandbox and as a scraper using the patterns from
shot-scraper javascript 910.The result is a published research report, labelled at the top as generated by an LLM, with a verdict and a table of findings 8. Out of the box, it says, evaluations fail on any page whose content-security policy blocks
eval, and the viewport is 0×0.A failed page load can hang forever, storage persists between runs, and the WebKit renderer's sandbox is switched off 8. A hostile page can overwrite the bridge the tool uses and answer every evaluation with a lie.
The report also credits what the tool gets right. WebRTC is absent,
window.open is dropped, downloads are cancelled, permissions are denied, and a file:// page cannot read other file:// URLs 8.The report's sandbox verdict is direct: webframe is not a sandbox by itself. The page has the full network including localhost, storage persists, and there are no time or memory limits 8.
The report then built the sandbox around it and attacked it with 20 hostile snippets. The combination that held was an empty network namespace, an ephemeral profile with bubblewrap, a timeout that kills the process group, a fresh process per call, and a nonce on every returned result 8.
Scraping is where it is nearly ready. The report measured speed comparable to
shot-scraper and startup around half a second, and wrote a 97-line patch that adds an ephemeral mode, a bubblewrap flag and a fallback for pages that block eval 8.
One finding stays open. Because the page and the tool's own hooks share a JavaScript world, a page that replaces
JSON.stringify can still silence the bridge 8. The report names the fix it could not reach, a WebKit isolated script world, which the library underneath does not expose.Try it. Give an agent one dependency you already ship, ask for a table of what failed and the commands that produced each failure, and read it before you upgrade.
3. A company that treats a quiet hour as an outage
Jacob Sansbury runs Spawn, a multiplayer game-creation engine, with ten people and about $3.1 million a month of model tokens, roughly thirty times his payroll 11.
Their agents shipped 77,000 substantial changes to players in thirty days, which he puts at 108 an hour around the clock 11. A substantial change means engine code or game logic: moving GPU buffers, changing memory layouts, adding API surface.
The agents are permanent. Each owns an area of the company, keeps a memory of what it has done, and holds real permissions 11.
Most of the architecture exists to keep that fleet from waiting on other agents. They removed pull requests, on the argument that agents each understand the whole system and can own a task end to end 11.
A change that does not fit goes in and breaks the thing that depended on the old shape, and the agent that wanted the change fixes both. Everything is required to line up once an hour 11.
The test suite is what makes that tolerable. A change is checked by running the code and using the product; the whole suite then runs against the latest code every 15 minutes, so the signal an agent receives is "it broke in the last 15 minutes" 11.
The bill for that cadence runs on about 5,000 vCPUs, 17 TB of RAM and 150 GPUs on AWS, with another 2,700 vCPUs and 90 GPUs parked 11.
Review moved too. Helper agents that did not write a change try to break it before it lands, and the only hard-coded gates left are the actions that touch the outside world and cannot be undone, such as taking a payment 11.
The number that decides whether any of this works is the revert rate: 1,055 of 123,548 changes over the past year, about 0.9 percent 11.
Faults reach the agents without a person relaying them. About 60 percent of work begins with a production error, a bug report from the in-game assistant Savi, something odd in the logs, or a feature request 11.
Savi files bugs herself, and a report usually becomes a change delivered to that player within the hour, in the same chat 11. When one of their Anthropic API keys leaked, the agents caught the spend anomaly in 15 minutes and replaced every key 11.
The discipline that follows from it is a cadence rule: an hour without a ship is a SEV-0, their most severe class of incident 11. A broken change rolls back in 30 seconds, in the place it broke.
His reason for the hourly rule is that an agent can hold only a few hours of work in its head, so shipping faster than that keeps it coherent 11.
The bottleneck that eventually surfaced was taste. One morning he woke up to 700 decisions waiting on him, so they turned two years of recorded calls into principles and built judges to apply them 11.
A judge gets a question and does one of three things: decides it and cites the earlier decision, sends it back to the agent that filed it, or holds it for a human 11.
What all of it depends on is a specific model and a tolerance for breakage. He tried frontier models from several labs and says only Anthropic's Fable line held together; asked to fix a security review, GPT-6 Astra made the product secure by switching every feature off 11.
The other precondition is letting agents touch production. Reverts help, he writes, but the loop only closes if an agent sees the failure itself 11.
Try it. Put your full test suite on a timer against the latest code and let the first failure wake an agent with the diff, instead of a person.
4. A coding agent with the network switched off
Rittika Jindal, a principal engineer at Thomson Reuters, moved to the foothills of the Himalayas a year ago and works around network and power cuts 12. She wanted to know how much of her agent workflow survives without a connection.
The stack is OpenCode 1.19.4 as the agent, Ollama as the model server, and Qwen3.5 9B as the model, a download of 6.6 to 7.6 GB 12.
Her
opencode.json points both the main and the helper model at the local server on 127.0.0.1:11434, enables only the local provider, blocks the web tools, and asks before editing a file or running a command 12.She started Ollama with cloud features disabled and a 64k context, pulled the model, and ran the agent once while still online, so every piece it needs was already on the laptop 12.
Then she switched off Wi-Fi and checked both directions: the local API still answered, and an external request failed with "Could not resolve host" 12.
The task was small on purpose. A sample program adding five expenses, including a refund, should print 25.50 and prints 8.00, and three of its five tests fail 12.
The failing tests came first, before the agent saw the bug. They gave it a target and gave her a way to check the result independently of the agent's explanation 12.
One hiccup came from the test runner: the agent proposed pytest in a project that uses
unittest, and the permission prompt is what caught it 12.The fix was a single line,
total = Decimal(amount) changed to total += Decimal(amount), and all five tests passed with the test file untouched 12. She then ran the tests herself, outside the agent.
Most of her advice is about what has to be on the laptop first. Download the dependencies and the docs you will need, launch the exact command once while online, and tell the agent the project's test command 12.
The costs are memory and tokens. A 6.6 GB model build wants 12 to 16 GB of memory before a large context, and a preprint she cites measured OpenCode using about 40 times Goose's tokens per solved task 12.
What the test does not show is scale. A one-line fix in a standard-library Python project says nothing about a large repository or an unfamiliar framework, and she says so before offering two harder exercises she has not yet run against the agent 12.
Try it. Put a failing test in front of your agent, tell it to leave the test file alone, and run the tests yourself when it says it is done.
What to try this week
- Write the reference first. Take one deterministic piece of your system and implement it plainly yourself, so the agent's version has something exact to match.
- Ask for a verdict, not a demo. Point an agent at one dependency and require a table of failures with the commands that produced them.
- Put the suite on a clock. Run the full suite against latest every 15 minutes and send the first failure straight to an agent with the diff.
- Measure your revert rate. Track how many of last month's changes you had to undo; that number sets how fast you can safely ship.
- Pull the cable once. Try one real task with networking off and find out which of your tools quietly needs the internet.
The pattern
Breunig's verdict is a log written in 2011 by someone who never met the model. Willison's is a table of twenty attacks and the failures they produced. Sansbury's is a test suite on a timer and a revert count. Jindal's is a test file the agent was told to leave alone.
All four were written down before or outside the run, and each one reports a yes or no that a program can act on.
The rule the week supports is short. Before you hand an agent work, write the pass condition, keep it where the agent cannot edit it, and give it a path back to the agent without a person in between.
This issue covers October 24 through October 31; the next one arrives next Saturday.
References
- 1dbreunig/nesformer
github.com
- 2
- 3nestest.txt
github.com
- 4a comment on Hacker News
news.ycombinator.com
- 5dbreunig/nesformer
simonwillison.net
- 6tractordev/webframe
github.com
- 7Releases
github.com
- 8
- 9webframe: evaluated as a scraper and a sandbox
simonwillison.net
- 10shot-scraper javascript
shot-scraper.datasette.io
- 11
- 12
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.