Levie's 14 points; Rauch's durable agent; Collison's acquisition a year on
This week's founder writing sits on the layer a demo hides: Aaron Levie's tests of whether an agent finishes the job, Guillermo Rauch's machinery for keeping a run alive and pricing it, and Patrick Collison's check on an acquisition thesis a year after the deal.
On October 13, Aaron Levie published the results of a test he runs inside Box, the company he co-founded and leads. The test gives an AI agent hard enterprise tasks built from real documents and scores what comes back. Levie reports a 14-point gain across the suite 1. In the same post he describes what the losing model did: it answered the question, then stopped before the job was done.
Three founders wrote between October 11 and 18 about the work between a demo and a product that holds. Levie measured whether an agent finishes. Guillermo Rauch, co-founder and chief executive of Vercel, described what keeps a run alive and what it costs. Patrick Collison, co-founder and chief executive of Stripe, checked an acquisition thesis against its first year.
Aaron Levie: the eval scores whether the agent finished
Box's Complex Work eval hands an agent a set of enterprise documents and one hard task. The tasks resemble an afternoon of knowledge work: a funding-gap analysis, an intensive-care plan, a decision about which region to optimize first. Levie ran the Box Agent on Gemini 4 Argon, the Google model announced that morning 2, against Gemini 3.1 Pro.
The five examples he chose to publish run above that average, because they are the wins:
- Financial services, +18. The brief spelled out every line of the projected statements, down to the loan interest and the overdraft for each year. Argon built all of them; 3.1 Pro folded the interest into a single figure 1.
- Healthcare, +25. Both models spotted a dangerous potassium level and a drug interaction. Only Argon went back and pulled the offending drugs out of the final regimen, so its plan matched its own diagnosis.
- Technology, +37. The task was to decide which of two regions should have its cloud infrastructure optimized first. Argon pulled the right month's user, retention and per-user figures; 3.1 Pro pulled wrong rows, once reporting retention above 100%, and pointed at the wrong region.
- Energy, +16. The report had to flag gaps in the meter data. Argon caught that one facility's smart meter log stops in November; 3.1 Pro often missed it and at times called the logs complete.
- Consumer products, +33. The decision was which production expansion to fund. Argon saw that existing inventory could bridge an early capacity gap, which made the smaller expansion viable sooner; 3.1 Pro defaulted to the larger one.
Read down that list and the errors cluster at the end of the task. The weaker model folds steps together, leaves a finding out of its own recommendation, and calls incomplete data complete. Its two other failures sit earlier, in the rows it pulled and the option it settled on.
Loading content card…
Levie's summary of the change describes that habit. The model "also works far harder to complete tasks fully without stopping short of the desired result," he wrote 1.
The numbers come from Box's own tasks, run for a week on two models, and Levie is reporting his own benchmark. He gives no task count, no scoring rubric and no repeated runs, so the size of the gain cannot be checked from outside. He also notes that Gemini 4 Argon will be sold through Box's AI Studio, so the report arrives alongside a commercial relationship.
The test design travels further than the numbers. An eval that scores answers rewards the model that stopped early, because a stopped answer still reads well. An eval that scores finishing has to check whether the output agrees with itself: whether the plan uses the diagnosis, whether the parts add up to the total, whether the data called complete is complete.
Guillermo Rauch: an agent that survives a restart
Rauch's three posts this week were product announcements from Vercel, and together they cover what happens to an agent after the model has answered. On October 16, Vercel made its agent harness durable: the same
createFxAgent() call now keeps its state on disk, in Postgres or in Vercel Workflows, on Node.js or Bun 3.He states the requirement plainly. "Agents need long-running compute. But they also need to tolerate restarts, redeploys, timeouts, crashes…"
The promise is specific enough to test. If a user sends a prompt and then throws their phone into a black hole, Rauch writes, the turn still runs to completion; the black hole is his image for a client that vanishes mid-request. If the process crashes halfway through a tool call, it resumes from there.
In an October 15 post he had put the architecture in three words: the modern scalable agent, in his phrase, is "brain, hands, files" 4. The model reasons, the tools act, and the files are where the work survives between one step and the next.
On October 18 he added the bill. Vercel's AI Gateway now groups every request into a session, with its cost, its cache hit rate and the agent that ran it, and holds each session on the provider that carries its prompt cache 56. Rauch's framing is about what a founder owns: "You own the model choice, your data (ZDR), and crucially the metadata to understand RoI."
Session affinity is a cost decision. A session that drifts to a second provider pays to rebuild the same prompt cache, so identical work costs more without producing more. Once sessions are grouped, the number a founder can see is cost per completed session, which is the figure you can compare across models.
Rauch is describing a framework he sells, and he publishes no failure counts. The argument for durability rests on what happens to a customer whose run dies mid-task.
Patrick Collison: the thesis, one year later
Stripe bought Privy, a wallet infrastructure company, a little over a year ago. Collison marked the anniversary on October 15 with what the purchase was for, what it has produced, and what remains unfinished 7.
The reasons were written down as three named factors. "Our acquisition was predicated on three factors: stablecoin growth, self-custody as a key enabler of broader global coverage, and rapidly increasing demand for programmable financial infrastructure." He quoted the sentence he had used at the time: "money has to reside somewhere, and Privy builds the world's best programmable vaults."
Naming factors instead of intentions is what makes the anniversary checkable. Each factor can be right or wrong on its own, and a year later the reader can ask which one paid.
The update: assets held in Privy wallets have grown about 25 times, monthly active developers about 3 times, and revenue about 9 times, with nearly $25 billion of monthly transaction volume. He gives the multiples without the figures they multiply, so absolute size and the starting point stay out of reach.
Integration is the part he is most candid about. Privy is being built into Stripe's Treasury, Link and Connect products, with wallet balances coming to Link. "There's been so much feature and functionality expansion that we haven't yet had much time to reconcile everything."
A fourth demand shows up in the update. Privy's wallet infrastructure, he writes, "is the ideal substrate for agents to transact and earn," and he lists companies already building on it. Then the sentence that separates the two buyers: "Developers prefer programmable financial infrastructure, but agents need it."
Agents appear here as a direction. He gives no agent-specific revenue or volume, so the size of that demand is still a claim about where things are heading.
For a founder weighing a small acquisition, the post leaves two usable pieces. Naming the factors at the start gives the buyer something to check later, in place of "strategic fit," which gives nobody anything. Reconciliation also lags feature growth even inside Stripe, which makes it a phase to staff rather than a task that closes.
Four decisions this week
- Score whether the task finished. Write the pass condition as a property of the finished output: the figures in the answer total correctly, and the recommendation follows from the diagnosis. Levie's five cases are all versions of that check 1.
- Make one run survive a restart before you add a second agent. If a deploy ends a turn that was halfway done, the customer loses work they paid for, and a second agent only doubles the exposure 3.
- Put a cost and a cache figure on the session. Cost per token tells you what a call costs; cost per completed session tells you what your product costs. The gap between them is where retries, cold caches and abandoned runs hide 5.
- Name a default model and leave the door open. Amjad Masad, co-founder and chief executive of Replit, said on October 17 that Replit stays model-agnostic while OpenAI is now its preferred partner on performance, cost and partnership, with users still choosing 8. Levie's reading of Meta's open-weights Muse model points the same way: frontier models for orchestration, cheaper tuned models for repeatable workflows 9.
A decision to run now
Take the hardest task your product performs end to end. Write the pass condition as two checks the finished output must satisfy: the numbers reconcile, and the recommendation follows from the evidence in the document. Run the task five times on your default model and count the runs that pass both.
Then read the failures and note where each run stopped. A run that stops after the diagnosis, a table that is missing a row, a conclusion that ignores the gap it found — each of those is a part of your product that is unfinished, and each is fixable this week by tightening the output contract or by routing that task to a stronger model.
Keep those five runs as your baseline. Re-run them after the next model change, and the count becomes the one number that tells you whether a new model moved your product.
References
- 1
- 2
- 3
- 4
- 5
- 6Vercel changelog, October 18, 2026
vercel.com
- 7
- 8
- 9
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.