Fake Googlebot: how to spot one in your logs
A fake Googlebot is a bot that copies the user agent of Google's crawler but runs on someone else's server. Your access log cannot tell the two apart by name, so you check the IP address instead.
This guide shows why scrapers do it, how to test an IP in two commands, which patterns give a fake away, and how to block it without ever touching the real crawler.
Why scrapers pose as Googlebot
Site owners want Google to crawl their pages. So many firewalls, rate limits and security plugins wave Googlebot through, while they slow down or block unknown bots. A scraper that claims to be Googlebot hopes to get the same free pass.
The trick costs nothing. A user agent is a plain text header, and any script can send any value. Google itself warns that other crawlers often spoof the Googlebot user agent. For that reason, you should never trust the name alone.
The visitors behind the mask vary. Some copy your content, some test for weak spots, and some are SEO tools that crawl harder than you would like.
How to spot a fake Googlebot in your logs
Google gives two ways to check a request: a DNS test and a list of IP ranges. Both look at the network the request came from, not at the user agent. You need shell access to your raw access log for the steps below.
Step 1: list the IPs that claim to be Googlebot
First, pull every IP that used the Googlebot user agent, with a count per IP.
grep "Googlebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20Step 2: run a reverse DNS lookup
Next, ask DNS which host name belongs to each IP. A real Googlebot IP resolves to a name that ends in googlebot.com or google.com. Google also lists googleusercontent.com for some fetchers that users trigger, but not for Googlebot itself.
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.Step 3: confirm with a forward lookup
Then look up that host name and make sure it points back to the same IP. This step matters, because anyone who controls the reverse DNS of their own IP block can set a name that says googlebot.com. Only Google controls the forward record.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1If the reverse lookup fails, returns another domain, or the forward lookup gives a different IP, you have a fake Googlebot.
Or check the IP against Google's list
For many IPs at once, DNS gets slow. Instead, Google publishes its crawler ranges as JSON files. Googlebot falls under the common crawlers file at https://developers.google.com/static/crawling/ipranges/common-crawlers.json. Check the verification page for the current file names, since Google updates them.
Patterns that give a fake Googlebot away
The IP test is the proof. Still, some patterns tell you where to look first.
| Signal | Real Googlebot | Typical fake |
|---|---|---|
| Reverse DNS | googlebot.com or google.com, confirmed forward | Hosting company, cloud provider, or none |
| Speed | On average no more than once every few seconds | Bursts of many pages per second |
| Rendering | Renders pages in headless Chromium, so it fetches scripts | Often HTML only, no CSS, scripts or images |
| Paths | Pages in your sitemap and links | Odd paths such as /.env, /wp-config.php.bak, /xmlrpc.php |
None of these alone proves a fake, because real crawls can spike too. However, a Googlebot user agent that probes /.env from a cloud server is a strong hint.
What to do about a fake Googlebot
Block by IP, never by user agent. A rule that matches the word Googlebot also stops the real crawler, and that can drop your pages from Google Search. So the safe rule is simple: verify, then block the exact IP or range that failed the test.
<RequireAll>
Require all granted
Require not ip 203.0.113.7
</RequireAll>deny 203.0.113.7;The address 203.0.113.7 belongs to a range set aside for documentation, so replace it with the IP you found. Also keep the list short and review it now and then. Cloud IPs change hands, and a blocked address may later belong to a real visitor.
What Pagereckon shows, and what it does not
Pagereckon reads your access log and sorts every request into people and bots. Its Google watch view shows Googlebot crawls per page, and whether Google crawled a page after you edited it. That helps you see if Google has picked up a change yet.
Pagereckon also flags disguised bots: bots that pose as a normal browser. It spots them by behavior, such as outdated browser versions, bursts, HEAD requests, and pages opened without CSS or images. Those visits never count as people.
However, be clear on one limit. Pagereckon goes by the user agent for known bots and does not run DNS checks. So a fake Googlebot lands in the Googlebot row, next to the real one. If a page shows far more Googlebot crawls than you expect, use the steps above on your raw log. Pagereckon cannot list the IPs for you, since it keeps no visitor IP addresses in its database.
The one-click bot blocking in Pagereckon Pro works by user agent too. For that reason it always keeps Googlebot and other good bots out of the rule, so it cannot block the real crawler by mistake. A fake Googlebot needs an IP rule like the ones above. For more on Google's crawler, see the Googlebot page.
Fake Googlebot: questions
How can I tell if a Googlebot visit is a fake Googlebot?
Run a reverse DNS lookup on the IP. A real Googlebot resolves to googlebot.com or google.com, and a forward lookup of that name returns the same IP. Anything else is a fake.
Is a fake Googlebot dangerous?
Often it just copies content or wastes server time. Some fakes also probe for weak spots, so treat them like any other unknown bot.
Can I block a fake Googlebot by user agent?
No. The fake and the real crawler send the same user agent, so a rule on the name blocks both. Block the IP that failed the DNS test instead.
Does Google publish a list of Googlebot IPs?
Yes. Google publishes its crawler IP ranges as JSON files and links them from its verification page. Check that page for the current file names.
Does Pagereckon verify Googlebot by IP?
No. Pagereckon identifies Googlebot by its user agent and shows its crawls per page. To confirm a single IP, use the host commands on your raw access log.