Meta externalagent: the crawler behind Meta AI
Meta externalagent is the crawler Meta uses for cases such as training foundation AI models or improving products by indexing content directly. In logs it appears as meta-externalagent/1.1, sometimes with a link to Meta's crawler docs after it.
Meta runs several agents with similar names. They do very different jobs, and two of them follow different robots.txt rules. So it pays to know which one you are looking at.
meta-externalagent/1.1
- Operator
- Meta
- Type
- Several purposes
- robots.txt token
meta-externalagent- Obeys robots.txt
- Yes, by the operator's documentation. Meta's docs give a Disallow for it and name robots.txt bypass only for other Meta agents; robots.txt may stay cached for 24 hours
- Official docs
- developers.facebook.com
Meta's agents, side by side
Meta documents five agents on one page. Here is what each does, in Meta's own framing.
| Agent | Job | robots.txt note |
|---|---|---|
| meta-externalagent | AI training, or indexing content to improve products | Disallow shown by Meta |
| meta-webindexer | Improves Meta AI search; helps Meta cite and link you | No bypass stated |
| meta-externalfetcher | Fetches a link at a user's request | May bypass robots.txt |
| facebookexternalhit | Title, description and image for shared links | May bypass for security checks |
| meta-externalads | Advertising and business products | No bypass stated |
The practical upshot: a rule for the Meta externalagent crawler does not touch link previews. People can still share your posts on Facebook and get a proper card.
How to block Meta externalagent
Meta's docs show a robots.txt group for the token as the way to keep the crawler out. Meta also notes that its crawlers may cache robots.txt for up to 24 hours, so give a change a day.
User-agent: meta-externalagent
Disallow: /You can also block a single folder instead of the whole site, for example Disallow: /members/. That way your public posts stay readable while private areas stay out.
A server rule, if you want a hard stop
If you prefer a 403 at the server, match the token in the user agent. This catches copycats that use the name as well.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} meta-externalagent [NC]
RewriteRule .* - [F,L]
</IfModule>if ($http_user_agent ~* "meta-externalagent") {
return 403;
}Pagereckon Pro's one-click blocking is reserved for bots that crawl hard and give nothing back, such as Bytespider. Meta's crawler is not on that list, so you add the rule yourself if you want one.
What blocking it costs
Blocking the Meta externalagent crawler keeps your pages out of its training and product indexing. It does not stop link previews, which come from a different agent.
However, Meta says that allowing meta-webindexer helps Meta AI cite and link your content. So if you want citations in Meta AI answers, leave that indexer allowed even when you block the training crawler.
Verifying Meta's crawler
Meta's crawler page does not list IP ranges or a reverse DNS rule for this agent. It suggests allow-listing by IP as the more secure option, but leaves the addresses out. Without that, the user agent is the only label, and anyone can copy it.
As a result, a server rule on the token is the simplest way to deal with both the real crawler and impostors.
Meta externalagent in Pagereckon
Pagereckon files the crawler under training in the Meta AI row. Its live fetcher, meta-externalfetcher, goes under live reads, because a person asked Meta AI about the page.
So you can tell a training crawl from a real question at a glance. See the split for your own site in the AI crawler checker, and compare with other operators on the AI bots list.
Meta externalagent questions
What is meta-externalagent?
It is a Meta crawler for uses such as training foundation AI models or improving products by indexing content directly.
Will blocking Meta externalagent break Facebook link previews?
No. Link previews come from facebookexternalhit, a separate agent.
Does it respect robots.txt?
Meta's docs show a robots.txt Disallow for it and flag robots.txt bypass only for other Meta agents. Changes may take up to 24 hours because of caching.
Can I see Meta's crawlers for free?
Yes. Download the free plugin to see Meta's crawler and fetcher per page.