Cloudflare AI Search gets custom domains and Access: productionize the endpoint, not the host
Cloudflare’s Aug. 6 AI Search update adds custom-domain endpoints, Cloudflare Access, namespace routing, and sitemap-free discovery; adopt it to productionize search, not to justify moving the rest of your stack.
Cloudflare’s August 6 AI Search release removes two awkward steps in taking a generated search endpoint public: it can now live on a domain you own and sit behind Cloudflare Access. The same changelog entry adds one endpoint for several search instances and a crawler mode for sites without a complete sitemap. 1
Decision: adopt this if AI Search is already close to a production use case and the remaining friction is endpoint ownership, access control, or routing. It is a reason to productionize the search boundary; it is not, by itself, a reason to move the rest of your application to Cloudflare.
What changed
| Area | New option | Why it changes the rollout |
|---|---|---|
| Endpoint identity | Serve the public endpoint from a hostname in a zone you own, such as https://search.example.com/search, instead of only the generated search.ai.cloudflare.com hostname. 1 | The search URL can now fit the application’s domain, naming, and certificate strategy. |
| Access control | Put Cloudflare Access in front of the custom-domain endpoint. Agents can use an Access service token; people opening it in a browser authenticate through the configured identity provider. 1 | The endpoint no longer has to be public merely because an application needs a stable URL. |
| Multiple instances | A namespace can expose /search, /chat/completions, and /mcp, then fan out across selected instances with instance_ids. 1 | One front door can route requests across separate indexes, such as documentation and support content. |
| Website ingestion | The new discover parse type starts from a source URL and collects pages from both sitemaps and links found while crawling. 1 | A missing or incomplete sitemap is less likely to leave holes in the index. |
All four are opt-in. Cloudflare says the previous behavior remains the default, so existing endpoints do not change until you select the new options. 1
The real change is the boundary around search
A generated hostname is fine for a prototype or an internal experiment. It becomes a less comfortable product boundary when the URL appears in customer-facing code, an MCP client, a support portal, or a set of agent credentials. The custom-domain option gives the endpoint a place in the same naming scheme as the rest of the application. Access gives the team a choice between authenticated agents and authenticated people instead of an all-or-nothing public URL. 1
That does not make AI Search a complete application gateway. Your application still has to decide who gets which content, how credentials are issued and rotated, and what the client is allowed to ask. Access handles the identity boundary described in the release; it does not define your product’s authorization model. Treat that distinction seriously if one namespace contains both public documentation and private operational material.
The namespace endpoint is useful when the index is a set of deliberately separate sources rather than one undifferentiated corpus. A request can point at selected instances, while the same namespace exposes search, chat-completions, and MCP paths. That creates a cleaner integration surface for clients, but it also makes instance selection part of your routing policy. Decide whether clients may select instances directly or whether your own API should choose them. The changelog documents the capability, not a safe default for every data boundary. 1
The
discover mode solves a different problem: coverage. It follows links as well as sitemap entries, which is useful for documentation sites whose navigation is more complete than their sitemap. It also changes the operational question from "did the sitemap include this page?" to "which links can this crawler reach?" Start with a small source and inspect the resulting index before increasing crawl depth or limits. That is rollout advice, not a promise from Cloudflare that discovery will match your intended content boundary.Adopt, watch, or migrate?
Adopt the endpoint changes now when
- You already use Cloudflare AI Search and have a real consumer for the index: a product search box, documentation assistant, support workflow, or controlled MCP client.
- The generated hostname is now leaking into application configuration or user-facing integrations.
- You need agent access and human access to follow different authentication paths.
- Separate indexes must sit behind one stable route, but the client should not know how the underlying instances are arranged.
Use the custom-domain documentation and Cloudflare Access integration as the implementation starting points. The release is additive, so the low-risk sequence is to create the new endpoint, test it with a small client set, and keep the existing endpoint available during the switch. 1
Watch when
- Your current generated endpoint is internal and already meets the access requirements.
- The search corpus is small, stable, and covered reliably by its existing sitemap.
- You would need to redesign application authorization before exposing a namespace publicly.
- You are considering the feature only because it is new, not because a current integration has a concrete boundary problem.
The comparison with Vercel, Render, Fly.io, or Railway is narrower than a hosting bake-off. A normal application deployment on any of those platforms can put an API behind a custom domain and an authentication layer, but your team would own the glue that connects the search index, client routes, agent credentials, and multi-index policy. Cloudflare’s update packages those controls inside the AI Search endpoint. If you already operate that boundary elsewhere, the feature is parity; if you do not, it is a reason to evaluate AI Search without moving the frontend, API, or worker that surrounds it.
A safe rollout sequence
- Map the consumers. List browser clients, server-side callers, MCP clients, and agents that will use the endpoint. Give each one an explicit authentication path.
- Separate the instances. Decide which indexes may be addressed together and which must stay behind an application-owned routing decision. Test
instance_idswith the smallest useful set first. 1 - Move the name before moving traffic. Create the custom-domain endpoint, validate TLS and Access behavior, then switch one client. Keep the generated endpoint as a rollback path until logs show the new route is stable.
- Audit discovery. If you enable
discover, review what the crawler finds before treating the index as production data. The website parse-type documentation is the place to check the available crawl modes. - Test the three interfaces. Exercise
/search,/chat/completions, and/mcpseparately. They may share a namespace, but they serve different client contracts. 1
Verdict: productionize the search endpoint, keep the host decision separate
This is a meaningful production-readiness update for teams already near an AI Search launch. Custom domains remove an awkward public URL, Access supplies an identity gate for agents and people, namespaces reduce the number of integration endpoints, and
discover addresses incomplete sitemap coverage. None of those changes requires a wholesale hosting migration, and Cloudflare explicitly keeps the old behavior as the default. 1Adopt the pieces that solve a boundary you currently operate by hand. Watch the rest until a real consumer, authorization rule, or indexing gap makes the change worth carrying into production.
References
- 1AI Search makes it easier to build a search engine for your data
developers.cloudflare.com
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.