Skip to content
AI bots / Cohere

Cohere AI token: what Cohere does not document

Many robots.txt blocklists include a line for the cohere-ai token, and some log tools label requests with it. Yet the crawler page from Cohere AI does not mention that token at all. This page sticks to what Cohere publishes, and says clearly where the record ends.

In short, Cohere says it does not run crawlers to collect web content for training its generative models at this time. It still tells site owners how to opt out, just in case.

The Pagereckon team Updated
Cohere ai at a glance, from Pagereckon
User-agent string
cohere-ai
Operator
Cohere
Type
Several purposes
robots.txt token
cohere-ai
Obeys robots.txt
Not stated by the operator
Official docs
docs.cohere.com

What Cohere AI actually says

Cohere's page on web crawlers is short. Its table of bots lists none: the bot name and the user agent both read N/A. Then it makes three points.

  • Cohere does not use bots or user agents to crawl or scrape web content for training its generative AI models, at this time.
  • Its policy is to require that crawlers respect robots.txt.
  • Blocking IP addresses may not give a reliable or lasting opt-out, because it stops a crawler from reading your robots.txt.

Notably, the example on that page uses the token Coherebot, not cohere-ai. We found no Cohere document that defines the cohere-ai token, its user agent string or its purpose.

Where the cohere-ai name comes from

Third-party bot directories list the token, and they disagree about it. Some call it a training crawler, while others describe it as a fetcher for user prompts. Because none of them cite Cohere, we do not repeat their claims as fact.

What you can check is your own log. If requests carry the token, you can see how often they come and which pages they read. That is real data, whoever sends it.

How to block the Cohere AI tokens

Since the record is unclear, cover both names. A robots.txt group for each costs nothing:

robots.txt: both Cohere tokens
User-agent: cohere-ai
Disallow: /

User-agent: Coherebot
Disallow: /

If you see traffic under the token that keeps coming after the rule, a server rule answers it with 403:

.htaccess (Apache, LiteSpeed)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} cohere-ai [NC]
RewriteRule .* - [F,L]
</IfModule>
Nginx server block
if ($http_user_agent ~* "cohere-ai") {
    return 403;
}

Pagereckon Pro's one-click blocking covers bots on its list of heavy crawlers that give nothing back, such as Bytespider. The cohere-ai token is not on that list, so add the rule above by hand if you need it.

What a block costs, and how to verify

As far as Cohere's docs show, nothing. Cohere describes no product that crawls your pages, so there is no documented benefit to lose.

Cohere also publishes no IP list or reverse DNS rule. Without one, nobody can prove that a request with this name came from Cohere. Treat it as an unverified label.

Cohere AI in Pagereckon

Pagereckon matches the word cohere in the user agent and lists any hits with the AI crawlers, outside the per-assistant split. So if requests with this name reach your site, you will see the pages and the volume.

If the row stays empty, that matches Cohere's statement that it runs no training crawlers right now. Check your own numbers in the AI crawler checker, or compare documented crawlers like CCBot on the AI bots list.

Cohere AI crawler questions

Does Cohere AI crawl my website?

Cohere says it does not use bots or user agents to crawl web content for training its generative models, at this time.

Is cohere-ai an official Cohere token?

We could not confirm it. Cohere's crawler page does not mention it and uses Coherebot in its example instead.

Should I still block it?

It costs nothing. Add Disallow groups for both cohere-ai and Coherebot if you want to be sure.

Can I see if anything uses the name?

Yes. Download the free plugin and any requests that name Cohere appear with the AI crawlers.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.