When Scale Runs Out, Choices Get Sharper

When Scale Runs Out, Choices Get Sharper

In a study of two hundred and sixty-three South African news sites, only about three in ten explicitly blocked at least one artificial-intelligence crawler. That is less a technology story than a business question: who can afford to say no?

0:00 / 8:17
This week’s briefing asks what small and regional publishers can do when they lack the scale, capital, and technical capacity that larger media companies use to navigate platform change.

The operating question

Two fresh reports point to the same choice from different directions. Digital Content Next’s September 3 review of New Zealand publishing shows how a small market can turn constraints into sharper audience, product, and workflow decisions. A September 1 report on South African news sites shows the cost of leaving AI access as an unowned technical detail.
The lesson is not to copy New Zealand or South Africa. The lesson is to make four decisions explicit: which audiences matter, what the newsroom can provide that generic systems cannot, which narrow workflow problem AI should solve, and what each crawler is allowed to do.

Audience intelligence before audience scale

New Zealand has just over five million people, roughly the size of the Boston metropolitan area. NZ On Air’s 2026 Where Are The Audiences? study surveyed 1,701 people and found that media-use changes have stabilized after the rapid shifts of 2016 to 2022. Local television and broadcaster video-on-demand together reached 63% of the population daily, while global video-sharing platforms reached 62%. The study also found that 81% of New Zealanders like seeing New Zealand faces and places on television. 1
For a local newsroom, the useful point is methodological. A single blended audience number can hide the groups that need different products, formats, and distribution. First-party registration, newsletter, membership, and subscription data should be cut by a small number of meaningful needs rather than by every available platform label.
Digital Content Next’s account of The Spinoff gives that principle a product shape. The Spinoff spent ten months building a custom audience platform and added a free-account tier between an anonymous visitor and a paying member. Google News Initiative’s case study says the soft launch produced more than 2,500 organic free accounts and made three audience states trackable: logged out, logged-in, and logged-in member. 2
The immediate opportunity is not a large data warehouse. It is one useful middle step. A regional publisher might start with a free account that saves articles, manages newsletter preferences, or unlocks a practical local tool. The test is whether that state improves the next action and the next return visit.

Distinctiveness and narrow AI pilots

The New Zealand examples also separate two ideas that are often bundled together. Distinctiveness is the value readers cannot get from a generic answer engine. Automation is the work a newsroom can make faster without handing over editorial judgment.
Google News Initiative reported that Pacific Media Network built a translation tool that reduced a bulletin workflow from as much as three hours to about 15 minutes. The Central App used a tool to identify angles in council-meeting transcripts, and the organization reported that two reporters doubled their story output. Those are publisher and program reports, not independent causal studies. 3
The practical takeaway is to start with the bottleneck. Name the workflow owner, define the source data, keep a human review step, and measure time saved alongside error rate, rework, and the reader or business outcome. A translation assistant for a known bulletin is a different risk from a chatbot that answers open-ended civic questions.
That makes this a trend to watch with an immediate test attached. The trend is small publishers using narrow tools to extend local relevance. The test is one workflow, one owner, one baseline, and one stop condition.

AI access is a governance choice

The South African report, The Protocol Gap: South Africa, analyzed the robots.txt files of 263 news websites. The report found that 74.1% had a robots.txt file, but only 30.4% explicitly blocked at least one AI crawler. Among sites that blocked a crawler, the average was six AI agents. The report says adoption is concentrated among larger, better-resourced publishers, while smaller and community outlets face greater dependence on visibility and referral traffic. 4
Daily Maverick’s report on the study adds the business pressure. It says popular chatbots referred just under two million visits to 222 South African news sites between January and May 2025, less than 1% of the 256 million chatbot visits recorded in the same period. The article also notes that robots.txt is a signal, not a technical lock, because compliance is voluntary. 5
A small publisher should not treat this as a binary argument between opening everything and blocking everything. Build a simple crawler inventory. Separate search, training, and agent access where the platform allows it. Record the business reason for each rule, the owner who reviews it, and the traffic or licensing assumption that could change the rule.
The warning is important: the report does not prove that AI caused every traffic decline, and robots.txt cannot enforce itself. The value of the exercise is control and documentation. A dated policy gives the newsroom a position to test, negotiate, and revise.

A 30-day operator test

For the next month, choose one audience need that your newsroom can serve unusually well. Define one direct action, such as a free account, a service newsletter, or a paid membership. Pick one narrow AI workflow that supports the product or reduces a known bottleneck. Then write a crawler policy that distinguishes the access you want from the access you merely tolerate.
Track repeat visits, the direct action, workflow time, error and rework, and any measurable downstream value. Review the test with the editor, audience lead, product owner, and technical owner together.
The industry noise is the promise that one platform button, one chatbot, or one robots file will repair the business. The actionable signal is smaller: publishers with limited resources are making sharper choices about audience, distinctiveness, workflow, and access.
The opening number was 30.4%: the share of South African news sites in the study that explicitly blocked at least one AI crawler. The decision behind that number is the real story. A small newsroom may not control the platforms, but it can control whether its audience, workflow, and content permissions have an owner.

This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.

Related content

More from this channel