Skip to content
Blog / 6 min read

Fake Googlebot: how to spot one in your logs

A fake Googlebot is a bot that copies the user agent of Google's crawler but runs on someone else's server. Your access log cannot tell the two apart by name, so you check the IP address instead.

This guide shows why scrapers do it, how to test an IP in two commands, which patterns give a fake away, and how to block it without ever touching the real crawler.

The Pagereckon team Updated
Fake googlebot at a glance, from Pagereckon

Why scrapers pose as Googlebot

Site owners want Google to crawl their pages. So many firewalls, rate limits and security plugins wave Googlebot through, while they slow down or block unknown bots. A scraper that claims to be Googlebot hopes to get the same free pass.

The trick costs nothing. A user agent is a plain text header, and any script can send any value. Google itself warns that other crawlers often spoof the Googlebot user agent. For that reason, you should never trust the name alone.

The visitors behind the mask vary. Some copy your content, some test for weak spots, and some are SEO tools that crawl harder than you would like.

How to spot a fake Googlebot in your logs

Google gives two ways to check a request: a DNS test and a list of IP ranges. Both look at the network the request came from, not at the user agent. You need shell access to your raw access log for the steps below.

Step 1: list the IPs that claim to be Googlebot

First, pull every IP that used the Googlebot user agent, with a count per IP.

Top IPs that claim to be Googlebot
grep "Googlebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

Step 2: run a reverse DNS lookup

Next, ask DNS which host name belongs to each IP. A real Googlebot IP resolves to a name that ends in googlebot.com or google.com. Google also lists googleusercontent.com for some fetchers that users trigger, but not for Googlebot itself.

Reverse lookup of an IP from Google's own example
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

Step 3: confirm with a forward lookup

Then look up that host name and make sure it points back to the same IP. This step matters, because anyone who controls the reverse DNS of their own IP block can set a name that says googlebot.com. Only Google controls the forward record.

Forward lookup must return the same IP
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

If the reverse lookup fails, returns another domain, or the forward lookup gives a different IP, you have a fake Googlebot.

Or check the IP against Google's list

For many IPs at once, DNS gets slow. Instead, Google publishes its crawler ranges as JSON files. Googlebot falls under the common crawlers file at https://developers.google.com/static/crawling/ipranges/common-crawlers.json. Check the verification page for the current file names, since Google updates them.

Patterns that give a fake Googlebot away

The IP test is the proof. Still, some patterns tell you where to look first.

Real Googlebot versus a typical fake
SignalReal GooglebotTypical fake
Reverse DNSgooglebot.com or google.com, confirmed forwardHosting company, cloud provider, or none
SpeedOn average no more than once every few secondsBursts of many pages per second
RenderingRenders pages in headless Chromium, so it fetches scriptsOften HTML only, no CSS, scripts or images
PathsPages in your sitemap and linksOdd paths such as /.env, /wp-config.php.bak, /xmlrpc.php

None of these alone proves a fake, because real crawls can spike too. However, a Googlebot user agent that probes /.env from a cloud server is a strong hint.

What to do about a fake Googlebot

Block by IP, never by user agent. A rule that matches the word Googlebot also stops the real crawler, and that can drop your pages from Google Search. So the safe rule is simple: verify, then block the exact IP or range that failed the test.

Apache 2.4 (.htaccess or server config)
<RequireAll>
    Require all granted
    Require not ip 203.0.113.7
</RequireAll>
Nginx (server or location block)
deny 203.0.113.7;

The address 203.0.113.7 belongs to a range set aside for documentation, so replace it with the IP you found. Also keep the list short and review it now and then. Cloud IPs change hands, and a blocked address may later belong to a real visitor.

What Pagereckon shows, and what it does not

Pagereckon reads your access log and sorts every request into people and bots. Its Google watch view shows Googlebot crawls per page, and whether Google crawled a page after you edited it. That helps you see if Google has picked up a change yet.

Pagereckon also flags disguised bots: bots that pose as a normal browser. It spots them by behavior, such as outdated browser versions, bursts, HEAD requests, and pages opened without CSS or images. Those visits never count as people.

However, be clear on one limit. Pagereckon goes by the user agent for known bots and does not run DNS checks. So a fake Googlebot lands in the Googlebot row, next to the real one. If a page shows far more Googlebot crawls than you expect, use the steps above on your raw log. Pagereckon cannot list the IPs for you, since it keeps no visitor IP addresses in its database.

The one-click bot blocking in Pagereckon Pro works by user agent too. For that reason it always keeps Googlebot and other good bots out of the rule, so it cannot block the real crawler by mistake. A fake Googlebot needs an IP rule like the ones above. For more on Google's crawler, see the Googlebot page.

Fake Googlebot: questions

How can I tell if a Googlebot visit is a fake Googlebot?

Run a reverse DNS lookup on the IP. A real Googlebot resolves to googlebot.com or google.com, and a forward lookup of that name returns the same IP. Anything else is a fake.

Is a fake Googlebot dangerous?

Often it just copies content or wastes server time. Some fakes also probe for weak spots, so treat them like any other unknown bot.

Can I block a fake Googlebot by user agent?

No. The fake and the real crawler send the same user agent, so a rule on the name blocks both. Block the IP that failed the DNS test instead.

Does Google publish a list of Googlebot IPs?

Yes. Google publishes its crawler IP ranges as JSON files and links them from its verification page. Check that page for the current file names.

Does Pagereckon verify Googlebot by IP?

No. Pagereckon identifies Googlebot by its user agent and shows its crawls per page. To confirm a single IP, use the host commands on your raw access log.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.