Diffbot: the crawler behind a web Knowledge Graph
Diffbot runs a general, proactive web crawler to build a search engine. The company says this lets websites be discovered and cited from its Knowledge Graph and web search services. It also states that it does not crawl pages to train generative AI foundation models.
The company also sells crawl tools to its customers. That part matters for your logs, because those crawls do not always carry the Diffbot name.
Diffbot
- Operator
- Diffbot
- Type
- Search engine crawler
- robots.txt token
Diffbot- Obeys robots.txt
- Partly. By default, including Crawl-delay; Diffbot says robots.txt can be overridden under an agreement with the site
- Official docs
- www.diffbot.com
Two Diffbot tokens
The company lists two user agent tokens, but no full user agent string. The main token covers proactive crawling. A second one, Diffbot-User, covers requests that come from a person browsing a URL through Diffbot software.
| Token | Purpose | Compared to |
|---|---|---|
| Diffbot | Proactive crawling for search and the Knowledge Graph | Googlebot, Bing |
| Diffbot-User | Requests on behalf of a person browsing a URL | A regular browser |
Customer crawls may not say Diffbot
Here is the part most bot pages skip. With the Extract and Crawlbot APIs, customers define their own crawl settings on hosted software. The company advises them to set a user agent that names their own organization.
So a crawl that runs on that software may show up in your log under a company name you have never heard of. A rule for the Diffbot token will not match it. Its docs also say Crawlbot's robots.txt setting is on by default, which suggests a customer can switch it off.
Does Diffbot respect robots.txt?
By default, yes. The docs say its crawls follow a site's robots.txt, including both disallow and crawl-delay. However, it adds that in specific cases, usually because of a partnership or agreement with the site, the rules can be overridden.
For a normal WordPress site with no such agreement, a Disallow group is the standard opt-out:
User-agent: Diffbot
Disallow: /
User-agent: Diffbot-User
Disallow: /Since the crawler supports Crawl-delay, you can also slow it down instead of blocking it. Pagereckon Pro's one-click blocking targets bots that crawl hard and give nothing back, such as Bytespider, and does not cover this one.
What blocking Diffbot costs
The company describes its crawler as the way sites get discovered and cited from its Knowledge Graph and web search. A block removes that chance. Because it says it does no generative AI training, a block does not change anything on the training side.
There is also no IP list or reverse DNS rule in the docs we read. So you cannot verify a request beyond its user agent.
Diffbot in Pagereckon
Pagereckon matches the token and lists the hits with the AI crawlers, outside the per-assistant split. You see which pages it reads and how often, straight from your server log.
Remember the customer crawls above, though. A crawl that uses its own user agent lands wherever that name fits, often in the general bot list. For the full view, open the AI crawler checker, or compare with another open crawler, CCBot, on the AI bots list.
Diffbot questions
Does Diffbot train AI models?
The company says it does not crawl pages for training generative AI foundation models.
Does Diffbot obey robots.txt?
By default, yes, including Crawl-delay. The company says the rules can be overridden in specific cases, usually under an agreement with the site.
What is Diffbot-User?
A second token for requests made on behalf of a person browsing a URL with the company's software.
Can I track it for free?
Yes. Download the free plugin to see its visits per page.