Bingbot: Microsoft's crawler for Bing
Bingbot is Microsoft's standard crawler, and Bing says it handles most of its crawling needs each day. It fetches your pages so they can appear in Bing.
Unlike most crawlers, it gives you a dial, not just a switch. Bing Webmaster Tools lets you set its crawl rate hour by hour, so you rarely need to block it at all.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
- Operator
- Microsoft
- Type
- Search engine crawler
- robots.txt token
bingbot- Obeys robots.txt
- Yes, by the operator's documentation. Bing points site owners to robots.txt and its Crawl control tool
- Official docs
- www.bing.com
Bingbot user agents and their siblings
Bing lists three strings for the crawler: an evergreen desktop one, a short classic one, and a smartphone one. In each, W.X.Y.Z stands for the latest Microsoft Edge version Bing uses, because Bing keeps its rendering engine current.
Bing also runs other agents that you may see in the same log. Knowing them helps you read your numbers.
| Crawler | Role |
|---|---|
| Bingbot | Standard crawler, most of the daily crawling |
| AdIdxBot | Crawls ads for Bing Ads quality control |
| BingPreview | Makes page snapshots for Bing |
| MicrosoftPreview | Makes page snapshots for Microsoft products |
| BingVideoPreview | Makes video previews in Bing |
Note that BingPreview, per Bing's list, can also carry the same token in its string. So a snapshot request may look like a crawl in a simple user agent count.
Control Bingbot instead of blocking it
Bing names two tools for this. First, robots.txt tells its crawlers which paths they may fetch. Second, the Crawl control setting in Bing Webmaster Tools sets a crawl rate for each hour of the day.
As a result, a site on a small server can ask for light crawling during its busy hours and more at night. That keeps you in the index without the load spikes.
A full block removes your pages from Bing. Pagereckon Pro's one-click blocking reflects that: it only offers a Block button for bots that crawl hard and give nothing back, such as Bytespider, and leaves search crawlers alone.
Verify a Bingbot request
Bing warns that its user agent strings are easy to fake. So it offers the Verify Bingbot tool, both inside Webmaster Tools and as a public page at bing.com/toolbox/verify-bingbot. You enter an IP and get a verdict.
For bulk checks, Bing also publishes its crawler IP list in JSON at bing.com/toolbox/bingbot.json. Then you can match every log line at once instead of pasting IPs by hand. The same idea works for Google; see our post on fake Googlebot traffic.
Bingbot in Pagereckon
Pagereckon files the crawler under search crawlers, next to Googlebot and Applebot. You see the pages it reads and how often it comes back.
With Pagereckon Pro, you can also connect Bing Webmaster Tools, so Bing's clicks and impressions sit next to its crawls. Then a page that Bing crawls often but never shows is easy to spot. Compare it with the AI crawlers in the AI crawler checker, and with Googlebot for Google's side.
Bingbot questions
How do I verify Bingbot?
Use Bing's Verify Bingbot tool with the IP from your log, or match the IP against the JSON list Bing publishes.
Can I slow the crawler down?
Yes. Bing Webmaster Tools has a Crawl control setting that sets the crawl rate by the hour.
Why do I see bingbot in snapshot requests?
Bing lists BingPreview strings that also carry the crawler's token. They make page snapshots, not regular crawls.
Does the free plugin show Bing's crawler?
Yes. Search crawlers per page are free. Download the free plugin; Bing Webmaster Tools data needs Pro.