← Guides

AI bots are hammering stores: hundreds of thousands of requests

Updated 2026-08-03

How much AI bots really cost you

Numbers from our AI Traffic Index #1 (period 13 Jul – 1 Aug 2026, a panel of three MD/RO stores):

• ~1,005,146 verified AI-bot requests in two weeks;
• the top vendor alone: 606,017 hits — about 34,000 requests a day from a single source;
• the most-read product was fetched 73,715 times.

A million requests across three small stores in two weeks sounds alarming. Before you start cutting bots off, let's see what it actually means for your server.

Is that a lot for your server?

Convert it to requests per second. 34,000 hits a day from the top vendor averages under 0.4 req/s. That's nothing for any host — on average. The catch is bursts: crawlers don't spread load evenly, they come in waves — dozens of requests in a second, then silence.

What happens next depends on what your server returns:

• Static and cached. If product pages and images come from cache or a CDN, even hundreds of thousands of hits barely touch the origin.
• Dynamic. If every bot visit hits PHP and the database (cold cache, personalization, heavy filters), the same traffic starts to bite.

One number stands out: 73,715 requests to a single product. If that page is rendered fresh on every visit, you're paying for it 73,000 times.

When to actually worry

Watch the symptoms, not the raw hit count:

• origin response time climbs, especially on product pages;
• 5xx errors and timeouts show up during off-peak, low-human hours;
• CPU and database max out without any rise in human traffic;
• product images return 403/406 to bots.

That last one is sneaky. Across our rating of 63 MD/RO stores it's problem #1: a bot gets a 403 on an image, treats it as a failure, and comes back to retry. You lose product visibility in AI answers and gain extra load from retries at the same time. Fix the images and part of the "aggression" disappears on its own.

How to see your own load

1. Open your access logs and filter by known bot User-Agents: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bytespider, CCBot.
2. Measure the bot share of total requests and the distribution by hour — that's where the bursts show.
3. Verify by IP, not just by name. A User-Agent is trivial to fake; plenty of random scrapers hide behind GPTBot. Botmetria checks every bot against the vendors' published IP ranges and drops the impostors.

No time to grep logs? The free Botmetria audit — its "Hygiene" category surfaces response codes and speed — plus the AI Traffic Index give you a quick baseline.

How to throttle the aggressive ones gently

The goal is to shed load without losing the bots that put you into AI answers. From safest to strictest:

1. Cache. The cheapest win. Serve pages and images via a CDN and full-page cache. Then even 70,000 hits on one product cost the origin almost nothing.
2. Rate-limit. Cap requests per IP/subnet — say N per second with a graceful 429 slowdown rather than a hard block. Well-behaved bots respect 429 and back off.
3. Firewall by IP range. If one vendor is genuinely too aggressive, block it surgically by its verified ranges, not by User-Agent (or you'll catch fakes and miss the real ones).

One safety rule above all: never block Googlebot or Bingbot. Those are classic search engines; lose them and your regular SEO dies. Botmetria's firewall has that fuse built in — it filters AI crawlers while leaving search engines untouched.

What not to do

• Nuke everything. Block GPTBot and OAI-SearchBot and you vanish from ChatGPT answers — and 90% of AI referral visits in the index came from ChatGPT.
• Block by User-Agent blindly. Real bots suffer; fake ones just change their name.
• Confuse crawlers with people. A bot that runs JS is a human inside an assistant; one that doesn't is a crawler. Only throttle the latter, and gently.

Good news: AI orders in the index are still 0, but the traffic is real. Your job isn't to evict the bots — it's to let them read your products cheaply. Then the load becomes manageable and your store stays visible to AI.

FAQ