Skip to content
AI bots / Meta

Meta externalagent: the crawler behind Meta AI

Meta externalagent is the crawler Meta uses for cases such as training foundation AI models or improving products by indexing content directly. In logs it appears as meta-externalagent/1.1, sometimes with a link to Meta's crawler docs after it.

Meta runs several agents with similar names. They do very different jobs, and two of them follow different robots.txt rules. So it pays to know which one you are looking at.

The Pagereckon team Updated
Meta externalagent at a glance, from Pagereckon
User-agent string
meta-externalagent/1.1
Operator
Meta
Type
Several purposes
robots.txt token
meta-externalagent
Obeys robots.txt
Yes, by the operator's documentation. Meta's docs give a Disallow for it and name robots.txt bypass only for other Meta agents; robots.txt may stay cached for 24 hours

Meta's agents, side by side

Meta documents five agents on one page. Here is what each does, in Meta's own framing.

Meta web agents and their jobs
AgentJobrobots.txt note
meta-externalagentAI training, or indexing content to improve productsDisallow shown by Meta
meta-webindexerImproves Meta AI search; helps Meta cite and link youNo bypass stated
meta-externalfetcherFetches a link at a user's requestMay bypass robots.txt
facebookexternalhitTitle, description and image for shared linksMay bypass for security checks
meta-externaladsAdvertising and business productsNo bypass stated

The practical upshot: a rule for the Meta externalagent crawler does not touch link previews. People can still share your posts on Facebook and get a proper card.

How to block Meta externalagent

Meta's docs show a robots.txt group for the token as the way to keep the crawler out. Meta also notes that its crawlers may cache robots.txt for up to 24 hours, so give a change a day.

robots.txt: keep Meta's AI crawler out
User-agent: meta-externalagent
Disallow: /

You can also block a single folder instead of the whole site, for example Disallow: /members/. That way your public posts stay readable while private areas stay out.

A server rule, if you want a hard stop

If you prefer a 403 at the server, match the token in the user agent. This catches copycats that use the name as well.

.htaccess (Apache, LiteSpeed)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} meta-externalagent [NC]
RewriteRule .* - [F,L]
</IfModule>
Nginx server block
if ($http_user_agent ~* "meta-externalagent") {
    return 403;
}

Pagereckon Pro's one-click blocking is reserved for bots that crawl hard and give nothing back, such as Bytespider. Meta's crawler is not on that list, so you add the rule yourself if you want one.

What blocking it costs

Blocking the Meta externalagent crawler keeps your pages out of its training and product indexing. It does not stop link previews, which come from a different agent.

However, Meta says that allowing meta-webindexer helps Meta AI cite and link your content. So if you want citations in Meta AI answers, leave that indexer allowed even when you block the training crawler.

Verifying Meta's crawler

Meta's crawler page does not list IP ranges or a reverse DNS rule for this agent. It suggests allow-listing by IP as the more secure option, but leaves the addresses out. Without that, the user agent is the only label, and anyone can copy it.

As a result, a server rule on the token is the simplest way to deal with both the real crawler and impostors.

Meta externalagent in Pagereckon

Pagereckon files the crawler under training in the Meta AI row. Its live fetcher, meta-externalfetcher, goes under live reads, because a person asked Meta AI about the page.

So you can tell a training crawl from a real question at a glance. See the split for your own site in the AI crawler checker, and compare with other operators on the AI bots list.

Meta externalagent questions

What is meta-externalagent?

It is a Meta crawler for uses such as training foundation AI models or improving products by indexing content directly.

Will blocking Meta externalagent break Facebook link previews?

No. Link previews come from facebookexternalhit, a separate agent.

Does it respect robots.txt?

Meta's docs show a robots.txt Disallow for it and flag robots.txt bypass only for other Meta agents. Changes may take up to 24 hours because of caching.

Can I see Meta's crawlers for free?

Yes. Download the free plugin to see Meta's crawler and fetcher per page.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.