Bots Are Taking Over Web Traffic. What Does That Mean for Publishers?

Why Are Bots Generating So Much Traffic?

Bots on the web aren't new.

Googlebot has been crawling websites for decades. SEO tools crawl websites. Monitoring tools crawl websites. And, of course, scrapers have always existed.

What's changing is the scale of AI and agentic browsing.

Think about how a person researches something online.

You want to buy a new camera. You search Google, open five or six websites, read a few reviews and make a decision.

An AI agent can potentially visit hundreds or even thousands of pages while trying to complete the same task.

One human request can therefore trigger a huge number of web requests in the background. Multiply that across millions of people using AI assistants and agents, and you can see why bot traffic can grow incredibly quickly.

Publishers Are Getting Scraped a Lot. But What Do They Get Back?

This is where TollBit's latest report gets interesting.

For European publishers, it found that it takes around:

179 AI bot visits → 1 human referral

And the trend appears to be getting worse.

In Q1 2026, the ratio was 150:1. In Q2, it reached 227:1.

In other words, AI systems are consuming more publisher content, while the amount of traffic being sent back isn't growing at the same rate.

The old web deal was pretty straightforward:

Crawl my content → index it → send me traffic.

Publishers gave search engines access to their content. Search engines gave publishers distribution. That exchange wasn't perfect, but there was an exchange.

With AI crawlers, the value exchange is much less clear.

And robots.txt Isn't Stopping It

We already have a way of telling bots not to crawl parts of a website. It's called robots.txt.

robots.txt was created back in 1994. It's essentially a public instruction file that sits on a website and tells crawlers which parts of the site they are allowed to access. A well-behaved crawler is supposed to visit the file, read the rules and respect them. But robots.txt isn't a security barrier. A bot can simply ignore those instructions and crawl the site anyway.

TollBit found that robots.txt instructions were regularly ignored. Its report also describes bots that:

• impersonate Googlebot
• rotate IP addresses
• use residential proxies
• execute JavaScript
• scroll pages to behave more like human visitors

It also found that 95% of the top UK websites it tested could be scraped by at least one of the 14 scrapers tested.

We're basically trying to control increasingly sophisticated AI systems with a text file designed for the web of 1994. That's probably not going to be enough. If you want a more concrete way to verify and control what's actually hitting your servers, I wrote a separate guide on securing your content in the AI era , covering server-side access control and crawler verification.

There's an Interesting SEO Question Here Too

How much of this bot activity can we actually see?

Sometimes you can spot some pretty strange things in Google Search Console.

For example, look at the queries below.

These don't really look like something a normal person would search for. They look more like automated searches generated by a bot or another system that is scraping Google results.

Of course, we can't say for sure that these searches came from AI agents. But they are a good example of how difficult it is becoming to tell whether there is actually a human behind a search.

If AI crawlers are visiting your website directly, you need to look at things like server logs, CDN data, Cloudflare or other bot-monitoring tools to see what's really happening.

For SEOs, understanding who, or what, is visiting our websites could soon become much more important.

The Revenue Math

Let's say you run a content-slop website.

You're generating thousands of cheap AI articles or programmatic pages and flooding the web with them.

Maybe you don't care that much if bots are your biggest audience. Your cost of producing that content is tiny anyway, so even the minimum return is okay for you.

But now take a publisher paying real people to create content.

• Journalists.
• Editors.
• Researchers.
• Photographers.
• Experts.
• Developers.

That content costs real money to produce.

Now imagine a growing percentage of requests to that publisher's website are machines. Those machines consume the content and potentially use it to answer questions inside someone else's product.

The publisher might get a few users back. But according to TollBit, we're talking about roughly 179 AI visits for every one referral for European publishers.

Let me simplify this: 179 bots are visiting, getting your content for free, and they are sending only ONE user back (eventually).

That's a very different economic model from traditional search.

And in some cases, the companies building these AI products are raising millions from investors while building on top of existing frontier models and content created by other businesses.

That's where the economics start to look strange.

The expensive part of the information ecosystem is still being funded by publishers. But increasingly, someone else owns the interface where that information is consumed.

We Need New Rules for the Agentic Web

I'm not arguing that we should block AI.

AI agents are going to become a normal way people use the internet.

But the rules need to catch up.

Bots should be able to reliably identify themselves.

Publishers should be able to decide who can access their content, what they can use it for and under what terms (I know, it sounds easier than it actually is).

And we probably need better ways for AI companies to pay for high-quality content when they're consuming it at scale. Because there's a basic problem here: if you need better context for your AI application, you have to pay. Simple.

AI needs good content. But good content isn't free to produce.

If publishers stop getting enough value from creating it, eventually they'll produce less of it. And then AI systems have worse information to work with too.

In late May, bots were around 60% of traffic in Cloudflare's data. Less than three months later, they're at 65%.

Whatever the final solution looks like, we probably don't have until 2027 to figure it out. I've written more on how sites and publishers need to adapt for AI agents in my guide on Agentic Engine Optimisation (AEO) .