5,500 lines of generated world, 100,000× scaling, and a prompt that stopped helping
Four original posts from Simon Willison, Andrej Karpathy, François Chollet, and Ethan Mollick show how hidden features, inspection limits, test-time compute, and prompting shape AI results.
The short read
Four original posts in the 24-hour window point to a less glamorous AI frontier: models can produce more, but the surrounding controls still decide whether the output can be used. Simon Willison found more capability hidden inside ChatGPT Work than its product naming suggests. Andrej Karpathy spent about $10 to get a custom world from Opus 5, then ran into the model's difficulty inspecting that world. François Chollet argued that test-time compute changed the scaling story, while Ethan Mollick pointed to evidence that generic chain-of-thought prompts now add little to reasoning models. 1 2 3 4
Scope: This edition covers original posts published from Aug 1, 18:00 through Aug 2, 18:00. The X account connector is not linked, so the source pool is the configured public AI and tech accounts rather than a personal following list.
AI tools and developer ecosystem
ChatGPT Work is doing more than its name makes obvious
- What happened: Simon Willison says the mobile and web version of ChatGPT Work can browse, take screenshots, and deploy web apps to Cloudflare Workers through a feature called "ChatGPT Sites." 1
- Why it matters: The same name refers to a different desktop mode, so product-surface clarity is now part of capability: a feature people cannot find is effectively unavailable to them. 1
- Constraint: The post is a firsthand feature discovery, not evidence of general availability, eligibility, or rollout scope beyond what Willison encountered. 1
Loading content card…
The interesting signal is not another model score. It is how much of the product is still hidden behind naming, mode confusion, and accidental discovery.
A custom world is cheap enough to be disposable
- What happened: Karpathy gave Opus 5 the opening paragraph of The Lord of the Rings, a 1-million-token budget costing about $10, and asked for a Three.js rendering; he says the model worked for about two hours and wrote 5,500 lines of code. 2
- Why it matters: This is a different kind of capability demo: an artifact that would be too custom and time-consuming for a person to build on a whim becomes cheap enough to try. 2
- Constraint: Karpathy also reports that Opus 5 could not efficiently perceive the rendered result or play through it; it relied on slow screenshots, made mistakes, and left visible roughness. 2
Loading content card…
Generation is getting cheap faster than inspection is getting native. That gap matters whenever the output is a world, a video, or anything else that cannot be checked from a text diff.
Research
Chollet's scaling argument starts with the base model
- What happened: François Chollet says base LLMs without test-time compute still perform poorly on the 2019 ARC 1 benchmark's unseen tasks, despite what he describes as roughly 100,000× scaling since 2019. 3
- Why it matters: His claim is that scaling the single-pass, static next-token paradigm from the GPT-2 through GPT-4 era was approaching a capability asymptote, and that test-time adaptation was the necessary way around it. 3
- Constraint: This is Chollet's interpretation in an X thread, not an independently audited comparison; the post does not provide the model list, scores, or evaluation table needed to test the causal claim here. 3
Loading content card…
The practical takeaway is narrower than "scaling stopped." When a benchmark asks a model to adapt at inference time, the model's runtime procedure becomes part of what is being measured.
"Think step by step" is no longer a safe default
- What happened: Ethan Mollick linked a Wharton Generative AI Labs report testing chain-of-thought prompts on 198 PhD-level GPQA Diamond questions, with 25 trials per condition. The report was published in June 2025; the new signal is Mollick's in-window post resurfacing it. 5 4
- Why it matters: The report found modest average gains for some non-reasoning models, but only small gains for reasoning models: 2.9% for o3-mini, 3.1% for o4-mini, and a 3.3% decrease for Gemini Flash 2.5 on its reported comparison. 5
- Cost: For reasoning models, the report says chain-of-thought requests took 20–80% longer for often marginal accuracy gains; prompt advice should therefore be tested against the specific model and task, not copied as a universal recipe. 5
Loading content card…
Today's four signals fit together without requiring a grand prediction. The feature surface changes what users can reach; inspection determines whether generated artifacts can be trusted; test-time compute changes the effective model; and prompting changes the cost and variance of the answer. Those variables now belong in the description of an AI system, not in the footnotes.
Related content
- Sign in to comment.