OAI SearchBot: the crawler behind ChatGPT search
OAI SearchBot is the OpenAI crawler that finds and indexes pages so ChatGPT search can show and link them. If you want your posts cited in ChatGPT answers with a link, this is the agent you should let in.
It is easy to confuse with GPTBot. The two share a company and a log format, but they answer different questions: one is about search visibility, the other about model training.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
- Operator
- OpenAI
- Type
- AI search indexer
- robots.txt token
OAI-SearchBot- Obeys robots.txt
- Yes, by the operator's documentation. OpenAI says changes can take about 24 hours to show in search
- How to verify
- Match the IP against openai.com/searchbot.json
- Official docs
- developers.openai.com
Why OAI SearchBot matters more than GPTBot
OpenAI states that sites which opt out of this crawler will not appear in ChatGPT search answers. They can still show up as plain navigational links, but not as cited sources inside an answer.
In contrast, blocking GPTBot only affects training. OpenAI treats the two settings as independent. So a common setup is to disallow training and allow search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /When you allow both crawlers, OpenAI says it may use one crawl for both jobs. As a result, the search crawler can look quieter in your log than you expect.
The OAI SearchBot user agent, and its robots.txt variant
The published string looks like a desktop Chrome browser with compatible; OAI-SearchBot/1.4 at the end. OpenAI notes that the version number may change, so match on the token, not on the full line.
There is also a second form. When the crawler fetches your robots.txt file, OpenAI says it may add robots.txt; to the string. Both forms carry the same token, so one rule covers them.
How to block it, if you really want to
OpenAI says this crawler obeys robots.txt, so one rule is enough. Then wait about a day, because OpenAI says search results need roughly 24 hours to reflect a change.
User-agent: OAI-SearchBot
Disallow: /We do not show a server rule here on purpose. A crawler that follows robots.txt needs no 403, and a hard block can also hide your robots.txt from it. Likewise, Pagereckon Pro's one-click blocking only covers bots that crawl hard and give nothing back, such as Bytespider. This one does give something back: links from ChatGPT answers.
Verify a real OAI SearchBot request
First, take the IP address from the log line. Next, compare it with the ranges in openai.com/searchbot.json. If it falls inside one, the request came from OpenAI. If not, someone copied the name.
This check matters because scrapers often borrow AI crawler names. A fake visitor can still eat your bandwidth, while a real one helps your search visibility.
Where Pagereckon shows it
Pagereckon reads your access log and files OAI-SearchBot under AI search, the middle group between training and live reads. You see which posts it indexes and how often it returns.
Then compare those pages with the ones that get ChatGPT referral visits. A page the search crawler reads often but that never earns a click may need a clearer answer near the top. Start with the AI crawler checker, then look at AI referral traffic for the visits.
OAI SearchBot questions
Should I block OAI SearchBot?
Only if you do not want to appear in ChatGPT search. OpenAI says blocked sites drop out of search answers, apart from navigational links.
Is OAI-SearchBot used for training?
OpenAI keeps training on GPTBot and lets you set each one on its own. If you allow both, OpenAI may reuse one crawl for both purposes.
How long does a robots.txt change take?
OpenAI says about 24 hours for search results to adjust.
Can I see OAI SearchBot visits for free?
Yes. The free Pagereckon plugin lists it per page under AI search. Download the free plugin to start.