Amazonbot: Amazon's web crawler, explained
Amazonbot is the crawler Amazon uses to improve its products and services. Amazon says the data helps it give customers more accurate information and may be used to train Amazon AI models.
Amazon also offers a control most crawlers lack: a page-level tag that keeps one page out of model training while the crawler can still read it.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
- Operator
- Amazon
- Type
- Several purposes
- robots.txt token
Amazonbot- Obeys robots.txt
- Yes, by the operator's documentation. No Crawl-delay support; Amazon may use a cached robots.txt up to 30 days old
- How to verify
- Match the IP against Amazon's published list
- Official docs
- developer.amazon.com
Opt out of training with noarchive, page by page
Amazon's agents honor the noarchive robots meta tag, and Amazon defines it as do not use the page for model training. So you can keep a single post out of training without blocking the whole crawler.
<meta name="robots" content="noarchive">In addition, the crawler honors noindex and none at page level and rel=nofollow on links. Most WordPress SEO plugins let you add robots meta values per post, so no code is needed.
Block Amazonbot with robots.txt
Amazon says its agents follow the Robots Exclusion Protocol, with the user agent and the allow and disallow lines. A plain group does the job:
User-agent: Amazonbot
Disallow: /Two Amazon quirks are worth knowing. First, the crawler does not support Crawl-delay, so that line will not slow it. Second, Amazon may use a cached robots.txt up to 30 days old, even though it says settings take about 24 hours to show.
Pagereckon Pro's one-click blocking targets bots that crawl hard and give nothing back, such as Bytespider. Amazon's crawler follows robots.txt, so it does not get that button.
Amazonbot and its two siblings
Amazon documents two more agents on the same page, each with its own token and IP list. Each setting is independent of the others.
| Agent | Job | AI training |
|---|---|---|
| Amazonbot | Improves Amazon products and services | May be used |
| Amzn-SearchBot | Search experiences such as Alexa | No, per Amazon |
| Amzn-User | Live fetches for a user, such as an Alexa question | No, per Amazon |
Note two rules here. If your robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the rules you gave them. And Amzn-User may not follow all robots.txt directives, because a user can start its actions.
What blocking it costs
Amazon ties search visibility to Amzn-SearchBot, not to the main crawler. So blocking Amazonbot alone mainly opts you out of its product work and possible AI training, while Alexa search experiences depend on the search agent.
If you only want out of training, the noarchive tag is the lighter tool. It leaves the crawler working for everything else.
Verify an Amazonbot request
Amazon publishes the crawler's IP addresses at developer.amazon.com/amazonbot/ip-addresses/. Check the IP from your log against that list. The search agent and the user agent have their own lists on the same site.
When an address is missing from the list, the request only borrowed Amazon's name.
Amazonbot in Pagereckon
Pagereckon lists the crawler with the AI crawlers, outside the per-assistant split, and shows the pages it reads straight from your server log. Since it sends no visitors by itself, a high count is a load signal, not a traffic signal.
Compare it with the other agents in the AI crawler checker, or browse the AI bots list.
Amazonbot questions
Does Amazonbot train AI models?
Amazon says its data may be used to train Amazon AI models. A noarchive robots meta tag keeps a page out of model training.
Does Amazonbot support Crawl-delay?
No. Amazon says its agents do not support the crawl-delay directive.
Why does my robots.txt change not work yet?
Amazon may use a cached robots.txt copy from the last 30 days, so a change can take a while.
Is tracking Amazon's crawler free?
Yes. Download the free plugin to see it per page.