The web started checking crawler IDs on September 15. 119 of 153 have nothing to present.

What actually changed on September 15

Why the gate exists at all

What the gate checks

Where your product sits

• An agent fetching on someone's behalf in real time is the Agent class, and on ad-supported pages behind the new default it is blocked unless it is identified and allowed 1.
• A pipeline that crawls to build an index still clears the new default. Its pressure is internal: mixed-purpose operators are the ones being pushed to split their crawlers into one user agent per job 5.
• Any client that has to get through a gate needs a signed request and a published key directory, which is the mechanism the standards track settled on and the one Cloudflare now validates automatically 39.
• A fetcher that meets a charge has to treat HTTP 402 as a real response with a price attached, which is already what a paid route returns 10.
• A team that publishes pages now has to state its position explicitly, because the platform default has one 1.

A pilot you can defend in a review

1. Take an identity before you need one. Sign your agent's outbound requests and publish the key directory on a domain you control; Cloudflare validates IP lists, reverse DNS records and Web Bot Auth signatures automatically once an operator submits a bot 9. Count how much of your target surface sits behind Cloudflare or an equivalent gate before you commit engineering time.
2. Split your crawlers by purpose. The defaults reward operators who separate search, agent and training traffic, and a mixed-purpose user agent now inherits the strictest rule applied to any of its jobs 5.
3. Log access outcomes in three buckets: verified, failed, unidentified. That view separates a small team that never published a key from an operator wearing a name that belongs to someone else, and a log grouped by user agent cannot tell them apart 9.
4. Handle 402 and 403 as first-class states. A charged route returns 402 with a price; a protected path returns 403 with an Accept-Signature challenge that tells your client what to present 710.
5. Price the fetch against the outcome. Crawl-to-refer ratios above the conventional five-to-one are what the market is negotiating over, so measure your own fetches per completed task before accepting a per-fetch price 7.
6. Declare the operator, not only the bot. A submission now asks a bot operator to state what the bot does, how it uses content and who runs it, and those declarations are what a site owner filters on 9.
7. Re-test on every default change. The September 15 rules were announced on July 1, so the window between announcement and enforcement is about ten weeks; a quarterly read of your own access logs is the cheapest way to catch the next one 1.

Evidence boundary: Cloudflare runs the network, publishes the metrics, sells the controls and decides which operators it designates Accountable, so its production figures are its own accounting, and the crawl-to-refer numbers come from a company that also sells the remedy. The crawler-identity census comes from a directory run by a business that sells verification. The consent-and-compensation exchange is a single-author preprint benchmarked on one vCPU in a prototype, and the protocol it builds on is an Internet-Draft that expires in March 2027. Pew's click figures describe an association observed in March 2025. The readiness index is published by a single company and is not independently audited.