Cloudflare's CEO, Matthew Prince, told analysts on the company's third-quarter earnings call that helping customers get more traffic from AI agents is "sort of the new version of SEO," as reported by Search Engine Journal.

Two details from the call matter to anyone who runs a website:

  • OpenAI pays Cloudflare for change signals. Under a pilot that started in July, Cloudflare tells OpenAI's crawler when a page has been updated, so it doesn't have to fetch pages that haven't changed. Prince said "OpenAI is paying for that." He didn't say how much, and it isn't clear whether the sites involved get any share.
  • Sites may charge AI agents, and charge less for explaining themselves. Prince expects some form of payment per agent visit, and said the more an agent explains why it's visiting, the more a site may be willing to "charge it less to access the page."

What the pilot actually did

Cloudflare's own write-up (Oct. 15) says the pilot covered about 1% of its network and helped OpenAI discover "hundreds of millions of new web pages." Besides change signals, it filters out addresses that don't exist or that redirect before the crawler fetches them.

Two points in it are easy to miss:

  • A signal "doesn't grant permission to access" a page. Site owners still decide who can crawl.
  • Cloudflare says the pilot uses page metadata for search and discovery, not page content for training models.

It doesn't describe a way for site owners to sign up.

The setting that matters today: AI Crawl Control

Whatever happens with paid crawling, every Cloudflare site already has AI Crawl Control, "available on all plans." It shows which AI crawlers visit your site and lets you allow or block each one individually. Pay per crawl, where sites set a price for crawler access, is still a private beta.

Cloudflare's documentation doesn't say whether AI crawlers are blocked by default when you add a new domain. So don't assume: check the setting when you set the domain up.

What we chose for this site

We are moving The Search Post onto Cloudflare this week, so we had to make the call ourselves. Our reasoning:

  • We allow AI search crawlers. A news site wants to be cited when someone asks ChatGPT, Perplexity or Google's AI Mode about a search change. Blocking the crawlers that power those answers would make us invisible there.
  • We allow the rest, for now, including crawlers used to train models. For an SEO publication, being known to AI systems is part of being found. A publisher whose content is its product (recipes, reviews, paid research) could reasonably decide the opposite.
  • Our robots.txt allows everything, and we'll check AI Crawl Control after the first month to see who actually visits and whether any crawler ignores robots.txt.

The distinction to make for your own site: AI companies increasingly use separate crawlers for search and for training. OpenAI, for example, documents one for ChatGPT search and another for model training. You can allow the first and block the second if being cited matters to you but training use doesn't.

What to do

  1. If your site is on Cloudflare: open AI Crawl Control and look at which AI crawlers visit and how often. Make a deliberate choice per crawler instead of living with whatever default you got.
  2. If you want AI citations: make sure the search crawlers of the assistants you care about are allowed, in Cloudflare and in robots.txt.
  3. If you want to block training but keep citations: block the training crawlers by name, not every AI crawler.
  4. Watch the payment experiments, but don't plan around them yet. Pay per crawl is a private beta, and no one has said what sites in the OpenAI pilot earn.

We'll report what AI Crawl Control shows for this site after its first month.