Skip to content
AI bots / OpenAI

OAI SearchBot: the crawler behind ChatGPT search

OAI SearchBot is the OpenAI crawler that finds and indexes pages so ChatGPT search can show and link them. If you want your posts cited in ChatGPT answers with a link, this is the agent you should let in.

It is easy to confuse with GPTBot. The two share a company and a log format, but they answer different questions: one is about search visibility, the other about model training.

The Pagereckon team Updated
Oai searchbot at a glance, from Pagereckon
User-agent string
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Operator
OpenAI
Type
AI search indexer
robots.txt token
OAI-SearchBot
Obeys robots.txt
Yes, by the operator's documentation. OpenAI says changes can take about 24 hours to show in search

Why OAI SearchBot matters more than GPTBot

OpenAI states that sites which opt out of this crawler will not appear in ChatGPT search answers. They can still show up as plain navigational links, but not as cited sources inside an answer.

In contrast, blocking GPTBot only affects training. OpenAI treats the two settings as independent. So a common setup is to disallow training and allow search:

robots.txt: no training, yes to ChatGPT search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

When you allow both crawlers, OpenAI says it may use one crawl for both jobs. As a result, the search crawler can look quieter in your log than you expect.

The OAI SearchBot user agent, and its robots.txt variant

The published string looks like a desktop Chrome browser with compatible; OAI-SearchBot/1.4 at the end. OpenAI notes that the version number may change, so match on the token, not on the full line.

There is also a second form. When the crawler fetches your robots.txt file, OpenAI says it may add robots.txt; to the string. Both forms carry the same token, so one rule covers them.

How to block it, if you really want to

OpenAI says this crawler obeys robots.txt, so one rule is enough. Then wait about a day, because OpenAI says search results need roughly 24 hours to reflect a change.

robots.txt: leave ChatGPT search
User-agent: OAI-SearchBot
Disallow: /

We do not show a server rule here on purpose. A crawler that follows robots.txt needs no 403, and a hard block can also hide your robots.txt from it. Likewise, Pagereckon Pro's one-click blocking only covers bots that crawl hard and give nothing back, such as Bytespider. This one does give something back: links from ChatGPT answers.

Verify a real OAI SearchBot request

First, take the IP address from the log line. Next, compare it with the ranges in openai.com/searchbot.json. If it falls inside one, the request came from OpenAI. If not, someone copied the name.

This check matters because scrapers often borrow AI crawler names. A fake visitor can still eat your bandwidth, while a real one helps your search visibility.

Where Pagereckon shows it

Pagereckon reads your access log and files OAI-SearchBot under AI search, the middle group between training and live reads. You see which posts it indexes and how often it returns.

Then compare those pages with the ones that get ChatGPT referral visits. A page the search crawler reads often but that never earns a click may need a clearer answer near the top. Start with the AI crawler checker, then look at AI referral traffic for the visits.

OAI SearchBot questions

Should I block OAI SearchBot?

Only if you do not want to appear in ChatGPT search. OpenAI says blocked sites drop out of search answers, apart from navigational links.

Is OAI-SearchBot used for training?

OpenAI keeps training on GPTBot and lets you set each one on its own. If you allow both, OpenAI may reuse one crawl for both purposes.

How long does a robots.txt change take?

OpenAI says about 24 hours for search results to adjust.

Can I see OAI SearchBot visits for free?

Yes. The free Pagereckon plugin lists it per page under AI search. Download the free plugin to start.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.