Skip to content
AI bots / Amazon

Amazonbot: Amazon's web crawler, explained

Amazonbot is the crawler Amazon uses to improve its products and services. Amazon says the data helps it give customers more accurate information and may be used to train Amazon AI models.

Amazon also offers a control most crawlers lack: a page-level tag that keeps one page out of model training while the crawler can still read it.

The Pagereckon team Updated
Amazonbot at a glance, from Pagereckon
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
Operator
Amazon
Type
Several purposes
robots.txt token
Amazonbot
Obeys robots.txt
Yes, by the operator's documentation. No Crawl-delay support; Amazon may use a cached robots.txt up to 30 days old

Opt out of training with noarchive, page by page

Amazon's agents honor the noarchive robots meta tag, and Amazon defines it as do not use the page for model training. So you can keep a single post out of training without blocking the whole crawler.

In the page head: no model training for this page
<meta name="robots" content="noarchive">

In addition, the crawler honors noindex and none at page level and rel=nofollow on links. Most WordPress SEO plugins let you add robots meta values per post, so no code is needed.

Block Amazonbot with robots.txt

Amazon says its agents follow the Robots Exclusion Protocol, with the user agent and the allow and disallow lines. A plain group does the job:

robots.txt: keep Amazon's crawler out
User-agent: Amazonbot
Disallow: /

Two Amazon quirks are worth knowing. First, the crawler does not support Crawl-delay, so that line will not slow it. Second, Amazon may use a cached robots.txt up to 30 days old, even though it says settings take about 24 hours to show.

Pagereckon Pro's one-click blocking targets bots that crawl hard and give nothing back, such as Bytespider. Amazon's crawler follows robots.txt, so it does not get that button.

Amazonbot and its two siblings

Amazon documents two more agents on the same page, each with its own token and IP list. Each setting is independent of the others.

Amazon's web agents
AgentJobAI training
AmazonbotImproves Amazon products and servicesMay be used
Amzn-SearchBotSearch experiences such as AlexaNo, per Amazon
Amzn-UserLive fetches for a user, such as an Alexa questionNo, per Amazon

Note two rules here. If your robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the rules you gave them. And Amzn-User may not follow all robots.txt directives, because a user can start its actions.

What blocking it costs

Amazon ties search visibility to Amzn-SearchBot, not to the main crawler. So blocking Amazonbot alone mainly opts you out of its product work and possible AI training, while Alexa search experiences depend on the search agent.

If you only want out of training, the noarchive tag is the lighter tool. It leaves the crawler working for everything else.

Verify an Amazonbot request

Amazon publishes the crawler's IP addresses at developer.amazon.com/amazonbot/ip-addresses/. Check the IP from your log against that list. The search agent and the user agent have their own lists on the same site.

When an address is missing from the list, the request only borrowed Amazon's name.

Amazonbot in Pagereckon

Pagereckon lists the crawler with the AI crawlers, outside the per-assistant split, and shows the pages it reads straight from your server log. Since it sends no visitors by itself, a high count is a load signal, not a traffic signal.

Compare it with the other agents in the AI crawler checker, or browse the AI bots list.

Amazonbot questions

Does Amazonbot train AI models?

Amazon says its data may be used to train Amazon AI models. A noarchive robots meta tag keeps a page out of model training.

Does Amazonbot support Crawl-delay?

No. Amazon says its agents do not support the crawl-delay directive.

Why does my robots.txt change not work yet?

Amazon may use a cached robots.txt copy from the last 30 days, so a change can take a while.

Is tracking Amazon's crawler free?

Yes. Download the free plugin to see it per page.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.