Amazon blocks Muse; Opus 5.5 meets GPT-6 Sol; the eval step most teams skip

AI models: two reviews of one morning's launches

"I left Claude for months. Opus 5.5 is why I'm back" — Lenny's Newsletter (How I AI)

1. The complaint was tone, and Vo says it has been fixed. She left Claude's harness for months and moved most of her work to Codex because the models rambled and hedged. Anthropic's pitch for Opus 5.5 is Fable-5.1-class work at roughly 40% less than Opus 5 and about 30% faster, and her test is blunter than any benchmark: she asked whether the model was annoying, and reports it answering in short, plain bullets. The cost of that fix is a quieter model — a long task can run eight or nine minutes without narrating, which she says makes it feel slower than it is.
2. Long agentic tasks held up, and front-end design is the strength. Four long-running tasks finished, covering inbox triage, a backend feature, long research and computer use, at 25 to 82 steps per prompt. Inbox triage ignored a prompt injection, and the research task applied a different ranking to a strategic memo after finding that 41 of 44 Confluence tickets came from one customer. Front-end results are where she plans to act: it redesigned the ChatPRD homepage well enough that she intends to ship it, and it produced the best SVG illustrations of the models she compared. Consumer app design remains its weak spot.
3. Where it lands in her stack, and the two things to weigh. Opus 5.5 is back in her daily work for pull-request review, architecture questions and front-end design, while computer use and video editing still go to Codex. The friction: it refused her instruction to push straight to production with tests failing, and it ships with the same cyber and bio safeguards as Anthropic's Fable models, so security-adjacent requests get bumped down to Opus 4.8. Anthropic says this is the first model released since it called for pacing the frontier, and that outside evaluators vetted it. 1

"Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test?" — Lenny's Newsletter (How I AI)

1. A wider bench, scored blind. Three models arrived that morning: Opus 5.5, GPT-6 Sol and GPT-6 Luna, all cheaper and faster than what they replace, with Opus 5.5 priced at about twice GPT-6 Sol. Vo expanded the How I AI bench past PRDs and prototypes to personal productivity, front-end and back-end coding, agent personality, long-running research, computer use, character SVGs and video editing, then scored anonymous outputs, leaving the coding and research sections to an LLM judge.
2. The verdict splits three ways. The GPT-6 models — "Astra and Sol win my heart" — suit her taste best. Opus 5.5 "wins my week", the model she rated highly across the widest range of work, including the B2B renewals dashboard, agentic voice and long-running tasks. Sol stays uneven, and its price is the draw. Her own LLM judge disagreed with her rankings outright, putting Fable first and Sol much lower.
3. Two surprises and one unsolved test. Character SVG illustrations, which she expected Anthropic to sweep, went to the OpenAI models. Every model failed at cutting short-form video from a selfie clip, with poor overlays and too few cuts, a gap she attributes to a missing skill rather than a model limit. Barbie Bench, her test that asks for a playable 3D Barbie fashion game, is still unpassed: far better than earlier attempts, hands and feet still bad.

Building AI products: finding the failures worth measuring

"Advanced evals: How to find (and fix) hidden AI failures in your product" — Lenny's Newsletter

1. Error discovery is the step most teams skip, and the authors say to prioritise it. Teams jump to writing metrics because metrics are concrete and easy to automate, and then measure the wrong things. "If you only have time for one part of the eval process, you should prioritize it." Error discovery does for evals what product discovery does for a roadmap: it decides which failures are worth measuring in the first place.
2. A folder of traces handed to an agent will miss the failures that matter most. Reviewing examples is what changes your definition of a good experience — the paper term is criteria drift. Their example: an AI leasing assistant answered a prospect who said "this is out of my budget" with a polite goodbye, when the product's job was to offer cheaper units, and agents reviewing that trace marked it a success. In a study of 100 production traces from the same assistant, coding agents missed product-judgment failures like that one, along with missed human handoffs and stray Markdown in text messages, while reliably catching answers that contradicted tool output — and flagging some good responses as failures.
3. The workflow is half an hour once learned, and the payoffs are cited. Log one trace per user session — input, system prompt, tool calls, retrieved context, model calls and final output. Then review at least 10 traces by hand before letting the agent suggest anything, and keep going toward 100; the authors' free plugin builds a review app for your data, clusters your annotations into failure modes and counts them. The results they cite from other teams: Shopify's AI workflow builder came out 2.2 times faster and 68% cheaper than the frontier-model system it replaced, Cursor cut costs 41% while raising user satisfaction, Ramp took receipt-matching precision from 35% to 83%, and Harvey nearly doubled its internal quality score. Their third step turns failure modes into product priorities, and that part is for subscribers.

Platforms: the moat a Muse deal would trade on

"Amazon Blocks Muse, Amazon's Moat, Aggregator v Aggregator" — Stratechery

1. Amazon blocked Muse, and Thompson treats the block as expected rather than final. His public line: "Amazon predictably blocked Muse, but there is room for a deal". Meta's Muse is the personal agent Thompson called the best he had tried in his free essay of September 21, so the block lands on a product he had just praised. 46
2. The deal he sketches turns on Amazon's physical-world investments, which he calls an AI moat. The second marker names the claim, and it is the strongest statement the public page makes: warehouses, logistics and the storefront are the assets a consumer agent would have to reach, so the terms are a trade over that footing. What Amazon objected to, and what a deal would involve, sits in the subscriber text.
3. The third marker sets two aggregators against each other. "Aggregator" is Thompson's own term for the companies that own demand and leave their suppliers to compete on price. A title naming two of them, on a story about a consumer agent meeting a storefront, reads as a contest over which side holds the customer — an inference from the marker, since the argument underneath it is behind the subscription.

Where to start