Your bot rules may be blocking Googlebot — check that before the September spam update takes the blame
Google's September 2026 spam update is still rolling out, so this issue sets the update aside for a moment and rules out a self-inflicted cause of the same symptom: bot protection that turns Googlebot away, and the Search Console check that shows whether it is happening.
Google released the September 2026 spam update on Thursday, 24 September at about 12 pm ET. Google's note on the Search status dashboard says the update applies to all languages and locations, and that "the rollout may take up to two weeks to complete." 12
The length is the unusual part. The three spam updates before this one finished inside roughly two days each, and Search Engine Roundtable asked John Mueller whether the two-week window was a typo; Mueller confirmed the longer rollout was intended. 1 Google's own page on spam updates is blunt about the timescale afterwards: a site that violates the spam policies may rank lower or drop out of results, and changes help only if Google's automated systems "learn over a period of months" that the site complies. 3
The first heavy movement landed across the weekend of 25–27 September. Search Engine Roundtable reported large drops starting Friday, more on Saturday, and a calmer Sunday, and noted that with five days gone there was more of the rollout still to come. Glenn Gabe described drops across a number of sites, spanning "verticals and countries." 4
So a dip that arrives this week has almost two weeks of possible explanation in front of it, and for most small sites the verdict will arrive in October. There is a faster check worth running first, because the same week's practitioner threads show a cause of lost traffic that sits on your own side of the connection and that an update diagnosis will hide.
On 27 September, a WebmasterWorld member posting as Whitey described a fortnight in which sales fell hard while his "traffic or position" held steady, and said he had put visible Cloudflare security verification in place around the middle of September and was having the effect investigated. 5 Chris travel 30 gave the thread its method: read your own logs and your CDN's records for false positives — real visitors who were blocked — and keep the protection in place if that count is small. 5
The same week, a separate WebmasterWorld thread collected the rules its members use to hold bot traffic down. The author, gatormark, runs challenge rules against anything that is not a verified good bot or crawler, and his second rule keeps the crawlers out of that: "Skip verified Cloudflare bots. Only allow verified search engine and crawlers." 6 Juniya replied that he was weighing a paid Cloudflare plan for the same reason. 6 In the observations thread, RedBar had lost about 60% of his traffic on 22 September when a Cloudflare connectivity problem took his sites down for most of a day. 7
The scope for the check below is any change in the last six weeks to the way your site is served or protected: a CDN or firewall rule set, a country block, a rate limit, a login wall, an "under attack" or staging mode, an edited robots.txt, or a DNS move.
The one action: check that Googlebot can still get through
Search Console's Crawl Stats report is Google's own record of what it asked your server for and what came back. The report needs a root-level property — a Domain property such as example.com, or a URL-prefix property at the root — and Google points out that it is aimed at advanced users, with sites under a thousand pages unlikely to need it. 8 The whole check runs on Search Console and your own server logs.
- Open Settings → Crawl stats and read Host status first. Green means Google hit no significant crawl availability problem on your site in the past 90 days. A recent-issue status means it hit at least one in the last week, and the report tells you which of the three categories failed: robots.txt fetching, DNS resolution, or server connectivity. 8
- Open the Crawl responses table. Google groups every crawl request by the response it received. "OK (200)" should dominate. The rows to hunt are the ones Google marks as bad and tells you to fix: "robots.txt not available", "Unauthorized (401/407)", "Server error (5XX)", and "Other client error (4XX)" — the row a 403 from a challenge page lands in. 8
- Check the Googlebot type table on the same page. It breaks the requests down by the crawler that made them — smartphone, desktop, image, video, page resource load, AdsBot — so you can confirm that the requests coming back as client errors belong to the crawler you care about. 8
- Verify in your logs that a blocked request really came from Google. Take the IP from your access log and run a reverse DNS lookup on it with the
hostcommand. The hostname must end ingooglebot.com,google.comorgoogleusercontent.com, and a forward lookup on that hostname must return the same IP. 9
What to change, and how you know it worked
If those tables show Googlebot being turned away, build the exemption that gatormark's thread already describes: keep the challenge and the rate limit for everything you have not verified, and let verified Google crawlers through. 6 Google publishes its crawler address ranges as JSON files, one for common crawlers such as Googlebot, so the rule can match an address range instead of a user agent string. 910
One page deserves a check of its own, because it governs everything else. Google treats a robots.txt request that returns 429 or a 5XX as a failed request, where 403, 404 and 410 all count as success. 8 After a day of failed robots.txt requests Google stops crawling the site; between 12 hours and 30 days it works from the last copy it fetched successfully, and after 30 days it either crawls without restraint or gives up on the site altogether. Google adds that returning 503 or 429 for more than two or three days teaches it to crawl your site less often over the long term. 8
Then read the same three places again:
- Host status returns to green and stays there.
- Crawl responses shows 200s where the error rows were, and the share of those rows falls over the following days.
- URL Inspection → Test live URL fetches a page in real time and reports "URL is available to Google" with Page fetch reading "Successful" once nothing is turning the crawler away. Site-wide failures such as "Robots.txt unreachable" and "Server connection error" appear in the same panel. 11
Two limits belong with the reading. Crawl Stats counts most crawl requests but not all of them — Google lists that as a known issue and says the report and your server logs can differ slightly — and the data is limited to the domain you have selected, so resources served from elsewhere will not appear. 8 A small, healthy site will usually find its host status green and its responses in the 200s, which leaves the spam update as the likelier explanation for a September dip.
The forum evidence behind this check is one publisher's correlation plus a method the rest of the thread agreed on, and the threads carry no count of how many sites were hit. With the rollout still running into October, a recovery that arrives right after you clear a block also cannot be credited to the fix on its own — the update will still be moving rankings underneath you. Write down the day you changed the rule, and compare against the same tables next Monday.
References
- 1Google September 2026 Spam Update Has Been Unleashed
seroundtable.com
- 2Google Search Status Dashboard
status.search.google.com
- 3Google Search spam updates and your site
developers.google.com
- 4Google September 2026 Spam Update Has Big Weekend Impact
seroundtable.com
- 5September 2026 Google Search Observations
webmasterworld.com
- 6Helpful Cloudflare Security Rules Setup to Limit Bots & Help Adsense
webmasterworld.com
- 7September 2026 Google Search Observations
webmasterworld.com
- 8Crawl Stats report
support.google.com
- 9Verify requests from Google crawlers and fetchers
developers.google.com
- 10common-crawlers.json
developers.google.com
- 11URL Inspection tool
support.google.com
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.