Skip to content
AI bots / Diffbot

Diffbot: the crawler behind a web Knowledge Graph

Diffbot runs a general, proactive web crawler to build a search engine. The company says this lets websites be discovered and cited from its Knowledge Graph and web search services. It also states that it does not crawl pages to train generative AI foundation models.

The company also sells crawl tools to its customers. That part matters for your logs, because those crawls do not always carry the Diffbot name.

The Pagereckon team Updated
Diffbot at a glance, from Pagereckon
User-agent string
Diffbot
Operator
Diffbot
Type
Search engine crawler
robots.txt token
Diffbot
Obeys robots.txt
Partly. By default, including Crawl-delay; Diffbot says robots.txt can be overridden under an agreement with the site
Official docs
www.diffbot.com

Two Diffbot tokens

The company lists two user agent tokens, but no full user agent string. The main token covers proactive crawling. A second one, Diffbot-User, covers requests that come from a person browsing a URL through Diffbot software.

User agents, per the company's docs
TokenPurposeCompared to
DiffbotProactive crawling for search and the Knowledge GraphGooglebot, Bing
Diffbot-UserRequests on behalf of a person browsing a URLA regular browser

Customer crawls may not say Diffbot

Here is the part most bot pages skip. With the Extract and Crawlbot APIs, customers define their own crawl settings on hosted software. The company advises them to set a user agent that names their own organization.

So a crawl that runs on that software may show up in your log under a company name you have never heard of. A rule for the Diffbot token will not match it. Its docs also say Crawlbot's robots.txt setting is on by default, which suggests a customer can switch it off.

Does Diffbot respect robots.txt?

By default, yes. The docs say its crawls follow a site's robots.txt, including both disallow and crawl-delay. However, it adds that in specific cases, usually because of a partnership or agreement with the site, the rules can be overridden.

For a normal WordPress site with no such agreement, a Disallow group is the standard opt-out:

robots.txt: both Diffbot tokens
User-agent: Diffbot
Disallow: /

User-agent: Diffbot-User
Disallow: /

Since the crawler supports Crawl-delay, you can also slow it down instead of blocking it. Pagereckon Pro's one-click blocking targets bots that crawl hard and give nothing back, such as Bytespider, and does not cover this one.

What blocking Diffbot costs

The company describes its crawler as the way sites get discovered and cited from its Knowledge Graph and web search. A block removes that chance. Because it says it does no generative AI training, a block does not change anything on the training side.

There is also no IP list or reverse DNS rule in the docs we read. So you cannot verify a request beyond its user agent.

Diffbot in Pagereckon

Pagereckon matches the token and lists the hits with the AI crawlers, outside the per-assistant split. You see which pages it reads and how often, straight from your server log.

Remember the customer crawls above, though. A crawl that uses its own user agent lands wherever that name fits, often in the general bot list. For the full view, open the AI crawler checker, or compare with another open crawler, CCBot, on the AI bots list.

Diffbot questions

Does Diffbot train AI models?

The company says it does not crawl pages for training generative AI foundation models.

Does Diffbot obey robots.txt?

By default, yes, including Crawl-delay. The company says the rules can be overridden in specific cases, usually under an agreement with the site.

What is Diffbot-User?

A second token for requests made on behalf of a person browsing a URL with the company's software.

Can I track it for free?

Yes. Download the free plugin to see its visits per page.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.