Googlebot: the crawler that feeds Google Search
Googlebot is Google's main crawler, the one that fetches pages for Google Search. For most WordPress sites it is the single most important bot in the log, because nothing reaches Google's index without it.
This page is not about blocking it. Instead, it covers the two questions that matter in practice: is that request really from Google, and did Google come back after you changed a page?
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- Operator
- Type
- Search engine crawler
- robots.txt token
Googlebot- Obeys robots.txt
- Yes, by the operator's documentation. Google says its common crawlers always respect robots.txt for automatic crawls
- Official docs
- developers.google.com
Two user agents, one token
Google runs a smartphone crawler and a desktop crawler. Both share the token, so robots.txt rules apply to both. The smartphone string imitates a Nexus 5X running Chrome. The desktop string puts the token inside a plain Chrome-like line.
In both cases, W.X.Y.Z stands for the current Chrome version, so the exact string changes over time. Match on the token, not on the full line.
How to verify a real Googlebot
Fake crawlers love this name, because many sites let it through without limits. Luckily, Google offers two ways to check.
- Run a reverse DNS lookup on the IP from your log.
- Check that the host name ends in googlebot.com, google.com or googleusercontent.com.
- Run a forward DNS lookup on that host name.
- Confirm it returns the same IP you started with.
Instead of DNS, you can also compare the IP with the ranges Google publishes in JSON. The file for common crawlers lives at developers.google.com/static/crawling/ipranges/common-crawlers.json. Our post on fake Googlebot traffic walks through what impostors look like in a real log.
Should you ever block Googlebot?
Almost never on a public site, because a block keeps your pages out of Google Search, and for most blogs that is the main source of readers.
Pagereckon Pro's one-click blocking reflects this. It only offers a Block button for bots that crawl hard and give nothing back, and it never blocks a user agent that also names a bot it keeps, such as Googlebot. If you want AI opt-outs from Google, use Google-Extended instead, which Google says has no effect on Search.
Googlebot in Pagereckon: Google watch
Pagereckon files the crawler under search crawlers and gives it a view of its own. Google watch lists the crawls per page and shows whether Googlebot fetched a page again after your last edit.
That answers a common question after an update. If you changed a post on Monday and the crawler has not returned, Google has not fetched the new version yet. In Pagereckon Pro, the before and after report warns you when Google has not re-crawled a page you marked, so you do not judge a change too early.
Also, because Pagereckon reads the server log, it sees every crawl as it happens. See SEO testing for how the control-day report uses this, or the AI crawler checker for the AI side.
Googlebot questions
How do I know if Googlebot is fake?
Do a reverse DNS lookup on the IP, check that the host ends in googlebot.com, google.com or googleusercontent.com, then confirm the forward lookup returns the same IP.
Does Google's crawler obey robots.txt?
Yes. Google says its common crawlers always respect robots.txt rules for automatic crawls.
Does blocking Google-Extended stop Googlebot?
No. Google-Extended only controls Gemini training and grounding. The crawler keeps fetching pages for Search.
Can Pagereckon tell me when Googlebot re-crawled a page?
Yes. Google watch in the free plugin shows Googlebot crawls per page and whether Google crawled after an edit.