The Great Divergence:
State of the Programmable Web

A new economic model is taking shape

Cloudflare sees over 200 trillion Internet interactions every single day. The data from Q1 2026 tells a story of rapid transition — from a human-centric Open Web to a machine-centric Programmable Web — and the emergence of a new content economy built on permission, attribution, and value exchange. AI is not the enemy of the web. It is the forcing function for a better deal.

AI crawler share grew 4.2 percentage points quarter-over-quarter, from 17.5% in Q4 2025 to 21.8% in Q1 2026. That growth represents enormous demand for publisher content — demand that is increasingly being channelled into formal licensing agreements. The question is no longer whether AI companies will pay for content. The question is whether publishers have the tools to set the terms. That infrastructure exists now.

The Great Divergence

The fundamental collapse is not hard to find in the data. The bot category breakdown tells the story of a web that has pivoted away from human utility:

Bot Traffic by Category — Q1 2026 vs Q4 2025

The Top Bot Operators

Among individual bot operators, Google still commands 35.5% of all verified bot traffic — but its share dropped 6.2 percentage points QoQ. OpenAI is now the #3 bot operator on the entire Internet, at 9.5%, and growing.

AI Services Are Now Internet Infrastructure

One measure of how deeply AI has embedded itself in the Internet: ChatGPT is the #8 most-trafficked service globally, ahead of Amazon Shopping, TikTok, and Netflix. It is the #12 most-visited domain in the world.

The Asymmetry of Extraction

The core dysfunction of the current AI web is not that bots exist — it's that they consume without reciprocating. Among all AI bot crawl activity in Q1 2026:

89.3% of all AI crawler requests are extractive, consuming content to build models that may eventually route around the source entirely. Only 8.1% powers a search product that sends users back to original content.

The Individual Crawler Pecking Order

The individual crawler ranking reveals the precise hierarchy of extraction. These are the ten most active crawlers on the web, ranked by share of all crawler traffic:

The Industry Heatmap: What AI Is Actually Consuming

Cloudflare Radar data on AI bot vertical distribution reveals which industries are most exposed — and what kind of content they should be protecting.

What Each Vertical Should Protect

Shopping and retail is the most-crawled category at 31.1% — not because consumers are using AI to shop (yet), but because product data is rich training corpus for AI models that aim to assist with purchasing decisions. The crawl happens long before any agentic commerce product launches.

News and Media accounts for 9.2% of AI crawl activity — a figure that understates the impact. A retail product description is one sentence. A news article is thousands of words of structured, fact-checked prose. The training value per crawl request is not equal.

From Robots.txt to Honest Bot Detection

Publishers are responding with the only tool most of them have: a 47-year-old text file. GPTBot and ClaudeBot are the most restricted crawlers in the robots.txt ecosystem. In Cloudflare's verified bot catalog, 37 of 200 tracked bots are now AI crawlers or AI assistants — a category that barely existed two years ago.

Why Robots.txt Is Not Enough

First: compliance is voluntary. Our AI crawl timeseries shows that AI bot traffic hit its single-day peak of Q1 2026 on February 9 — even as blocking rates have been rising throughout the quarter. Volume is growing faster than restrictions can contain it.

Second: shadow scrapers exist. Our data shows 0.44% of AI bot crawl activity comes from bots declaring no purpose — Undeclared crawlers operating without transparency. These are bots that deliberately misidentify themselves, wearing a Googlebot mask to crawl as if they have permission they have not sought.

Third: the past is already gone. Models trained on data crawled before publishers erected their defenses have already consumed that content. The window of prevention is partly closed.

Web Bot Auth: Cryptographic Verification

This is why Cloudflare built Web Bot Auth — a new standard using cryptographic signatures to verify a bot's identity at the protocol level. If a bot claims to be Googlebot, it signs its requests with a key only Google holds. If the signature fails, the claim is false. Web Bot Auth does not rely on bots choosing to be honest; it makes dishonesty technically detectable.

AI Search: A New Referral Ecosystem

The AI Search category — crawlers attached to AI-powered search products that do return traffic to original sources — is one of the most strategically important in the ecosystem. It grew 1.0 percentage point QoQ — the fastest-growing bot subcategory this quarter.

Apple is the dominant AI search crawler, driven by Applebot's role in powering Siri and Apple Intelligence. Apple's crawl activity was highly volatile in Q1 2026 — near zero on some days, spiking by late March — suggesting active product development. A company with 2.35 billion active devices and a first-party AI search product is a fundamentally different referral engine than anything that preceded it.

OAI-SearchBot spiked noticeably during peak AI news periods (Jan 22–24, Feb 9, Mar 10–11), suggesting its crawl intensity tracks closely with ChatGPT usage surges. As AI search products mature and publishers begin demanding referral as a condition of access, this is the category that matters most.

The AI Agent Layer: From Crawlers to Action

The most nascent but strategically significant development is the emergence of AI assistants as a distinct traffic category. These are not crawlers passively indexing content — they are agents actively executing tasks: browsing pages, filling forms, clicking buttons, completing purchases.

The 21 verified AI assistant bots in our catalog include Amazon Bedrock AgentCore Browser (deployed across 9 AWS regions), Amazon's "Buy For Me" agent, Devin AI, CartAI, Apify, and Browserbase.

The AI Inference Economy

Beyond what bots consume, Cloudflare's AI inference data reveals what developers are building on top of AI infrastructure in Q1 2026.

Top Models by Deployment

Text generation dominates at 62.9% of all AI inference, confirming that the primary use of AI infrastructure is language. Meta's Llama 3 is the most widely deployed model on Cloudflare's network at 40.6% of all inference accounts — Meta's open-weight strategy has produced the Internet's most-used inference model. The newly released Llama 4 Scout already accounts for 7.0% of inference accounts despite launching during the quarter — the fastest adoption curve we have observed for any new model.

Bot Management: Cloudflare's Defense Layer

The data in this report — crawl volumes, intent classification, shadow scrapers, verified identities — doesn't surface without infrastructure. Cloudflare's Bot Management platform is the layer between publishers and the automated web, scoring every request with a bot probability, a purpose classification, and an enforcement decision. At the scale of 200+ trillion signals per day, this is not a feature. It is the operating system of the Programmable Web.

How Bot Management Works

Every request that passes through Cloudflare receives a bot score from 1 to 99. Scores near 1 indicate automated traffic; scores near 99 indicate human. This score is derived from a multi-layered detection stack that runs at the edge without adding latency:

AI Crawl Control: Publisher Enforcement at the Edge

The most consequential Bot Management capability for publishers in Q1 2026 is AI Crawl Control — purpose-built tooling for per-operator access policy on AI bots. Unlike robots.txt, which is a voluntary convention any bad actor can ignore, AI Crawl Control enforces access decisions at the network edge before content is ever served.

AI Crawl Control lets publishers set distinct rules per crawler identity — allowing OAI-SearchBot (which drives referral traffic back to the source) while blocking GPTBot (which trains models without reciprocation) on the same domain. Publishers can say, for the first time: you may index, but you may not train.

Bot Score Distribution — Q1 2026

Cloudflare classifies 30.5% of all web traffic as automated in Q1 2026. Within that automated layer, the distribution polarizes: most bot traffic is either clearly machine (score 1–29) or clearly human (score 80–99). The ambiguous middle — scores 30–79 — is where sophisticated evasion operates and where ML models are most actively contested.

Crawler Hints: Telling Good Bots When to Come

Bot Management is not only about blocking. The other half of the problem is waste. Cloudflare's analysis of search crawler traffic found that 53% of good bot visits are wasted — crawlers revisiting pages that haven't changed since their last visit, burning compute and energy for no new information. For a category that represents over 5% of all global Internet traffic, that inefficiency has material scale: Cloudflare estimates that reducing excessive crawl could eliminate as much as 26 million tonnes of carbon equivalent per year — the equivalent of shutting down six coal-fired power plants permanently.

The root cause is structural: crawlers have no reliable signal for when content changes. Without that signal, they default to visiting pages on a fixed schedule regardless of whether anything is new. Most of the time, nothing is.

The mechanism works through Cloudflare's edge cache layer. When a page's content changes — detected by hash or timestamp — Cloudflare records the event and notifies participating crawlers via IndexNow. Each search engine gets its own tailored feed of what changed since its last visit, ordered by page importance (inferred from human visitor frequency). The result: crawlers spend their budget on pages that are actually fresh, and publishers get their new content indexed faster.

Crawler Hints connects directly to the economics of the AI content market. When a licensed AI crawler has access to a publisher's content, both parties benefit if that crawler indexes fresh content efficiently — the AI company's product improves, and the publisher's content is correctly represented. Excessive recrawl of stale content wastes the licensed crawler's budget on pages it has already seen, while new articles sit unindexed. Crawler Hints ensures the deal works as intended: the crawler arrives when there is something new to read, not on an arbitrary schedule.

The /crawl Endpoint: Compliant Crawling by Design

On March 10, 2026, Cloudflare launched a new Browser Rendering /crawl endpoint — a developer API that submits a starting URL and returns a full website's content as HTML, Markdown, or structured JSON, via a headless browser that automatically discovers and renders pages. It is designed for building RAG pipelines, training datasets, and content monitoring — and critically, it is built from the ground up to be a compliant, permission-respecting crawler.

Key capabilities include incremental crawling via modifiedSince and maxAge parameters — meaning it skips pages that haven't changed, saving cost and reducing unnecessary load. Configurable crawl depth, URL patterns, and page limits give operators precise control over scope. Static mode (render: false) fetches HTML without a headless browser for faster crawls of non-JS sites. Available on both Free and Paid plans.

Some publishers raised concerns in the days following launch — specifically that the crawler's user agent was not yet listed in Cloudflare's Verified Bot catalog, meaning bot management rules keyed to that catalog required a manual update to explicitly reference it. Cloudflare clarified the behavior in an edited changelog post and the path to blocking it for any publisher who chooses to is straightforward: the crawler identifies itself honestly, which is precisely what makes it possible to block with a simple firewall rule. That is the opposite of the problem posed by shadow scrapers that rotate identities to evade detection.

The broader point is significant: the /crawl endpoint represents what compliant AI crawling should look like. It is honest about its identity. It respects publisher signals. It cannot impersonate other bots. Every AI company operating a crawler should be held to the same standard — and Cloudflare's infrastructure is what makes that standard enforceable at scale.

From Defense to Access Control Infrastructure

The strategic shift underway is this: Bot Management is no longer purely a security product. It is the access control layer for a new content economy. Block the bad actors. Challenge the ambiguous. Signal the verified good ones to crawl smarter. The /crawl endpoint is a blueprint for what the well-behaved automated web looks like: identified, transparent, and operating within publisher-defined terms. Publishers who use Cloudflare's bot infrastructure are not just protecting their content — they are creating the conditions for productive partnership. Granular identity enforcement at the edge is what makes tiered access possible: free for search, licensed for training, auditable for both. The deal is only as good as the infrastructure beneath it — and that infrastructure now exists.

The Attribution Dashboard: Your Negotiating Brief

The existing Cloudflare AI bots dashboard tells you which bots are hitting your site and whether they are being blocked. The new Attribution Overview goes further — it turns bot management data into a deal-ready evidence base. These are the capabilities that are new, and why each one matters for licensing leverage.

Bot View — Operator-Level Attribution

Articles View — Content-Level Attribution

Monetization: The Publisher-AI Deal Economy

The data has produced a market. Since the first publisher-AI deal in July 2023, a licensing economy has grown from a handful of bilateral experiments into 50+ documented agreements across OpenAI, Google, Perplexity, Microsoft, Meta, Amazon, Mistral, and Prorata — spanning wire services, national newspapers, magazines, specialist publishers, and collective licensing pools. The market is moving quickly, and the commercial models that will define it are being shaped right now.

Complete Deal Timeline: 2023–2026

Every confirmed publisher-AI content licensing deal, in chronological order. Covers training data licences, RAG/retrieval integrations, revenue-share programs, and platform-specific content marketplaces.

Sources: Press Gazette, Digiday, BuzzStream, company announcements. 2023–March 2026. Orange rows = 2025 deals. Green rows = 2026 deals.

Google's Structural Exception

Google's approach stands apart from every other AI platform: the Associated Press is the only publisher with a confirmed deal covering LLM training data for AI Overviews, AI Mode, and Gemini. The December 2025 pilot program with The Guardian, Washington Post, and others is a non-training arrangement covering AI article overviews in Google News. Google extracts from virtually every publisher via Googlebot — and licenses from almost none. This is the structural gap at the centre of the CMA's investigation.

Two Deal Models — Different Risk Profiles

Four Industry Initiatives Forcing Payment

Beyond bilateral deals, publishers are forming collective structures to create negotiating leverage — and to address the transparency gap that makes individual licensing unenforceable at scale.

How Bot Management Makes Deals Enforceable

Commercial deals are only worth the paper they're printed on if the publisher can control who accesses their content. Cloudflare's Bot Management infrastructure is what converts a licensing agreement from a legal document into an enforced reality:

The Revenue Lag — and What Comes Next

Despite the deal velocity, publisher revenue from AI licensing remains far below what search advertising once delivered at its peak. People Inc.'s Jon Roberts described the state of the market at the Digiday Publishing Summit in March 2026 as an "AI licensing boom — and a revenue lag." Deals are being signed faster than revenue is materializing.

The pay-per-use marketplace model is most likely to close this gap, because it ties compensation to actual consumption at scale. But the market infrastructure — verified identity, consumption metering, access enforcement — must exist before that pricing can be trusted. Meanwhile, publishers are running out of time to wait for legislation. The UK stepped back from an opt-out approach in March 2026, leaving space for industry-led solutions — but also leaving the field open. Publishers will have to choose a route. The clear risk is fragmentation: different collective initiatives pulling in different directions, weakening the position of all of them.

The convergence of Web Bot Auth, AI Crawl Control, and pay-per-use marketplace models represents the technical and commercial stack needed for a functioning content economy on the Programmable Web. The infrastructure is in place. The deals are being signed. The window to shape the terms of this market — as a publisher, a developer, or an AI company — is open now.

The New Deal Is Being Written Now

The data from Q1 2026 tells a story of transition — and of opportunity. AI crawler volume is growing. So is the number of licensing deals. So is the sophistication of the tools publishers have to set terms, verify identity, attribute consumption, and enforce access. The infrastructure for a new content economy is not theoretical. It is operational.

Developers building on AI are not the enemy. They are the market. The question is whether publishers can participate in that market on fair terms — and the answer, for the first time, is yes.

Cloudflare's role in this transition is to be the infrastructure layer that makes the new model work — for both sides. Bot Management and AI Crawl Control give publishers the ability to define who accesses their content and on what terms. Crawler Hints and the /crawl endpoint give developers efficient, compliant access to the web's content. Web Bot Auth gives everyone a cryptographic foundation for identity. The Attribution Dashboard gives publishers the evidence base to negotiate from a position of knowledge rather than assumption.

The Programmable Web is being built. Cloudflare is building the access layer that determines whether it is built on extraction or on agreement. The terms are being written now — by publishers who block, by developers who comply, and by AI companies who are discovering that a web full of licensed, attributed, fairly compensated content is more valuable than a web of dark patterns and silent scraping.

Block strategically. License intentionally. Build on agreement. The new deal is better for everyone.