Skip to content
AI bots / ByteDance

Bytespider: what we know about ByteDance's crawler

Bytespider is a web crawler run by ByteDance, the company behind TikTok. Unlike OpenAI, Anthropic or Google, ByteDance has published no English documentation for it that we could find: no stated purpose, no full user agent string, no IP list and no robots.txt policy.

So this page is careful. We separate what you can see for yourself in a server log from what the press has reported, and we label each.

The Pagereckon team Updated
Bytespider at a glance, from Pagereckon
User-agent string
Bytespider
Operator
ByteDance
Type
Several purposes
robots.txt token
Bytespider
Obeys robots.txt
Not stated by the operator

What is documented, and what is only reported

First, the token itself is easy to confirm, because it shows up in access logs as part of the user agent. Beyond that, the official record is thin. The card above links to ByteDance's Toutiao Search webmaster platform, but we found no page there that describes this crawler.

However, press coverage fills some of the gap, with clear caveats:

  • Reported: a Fortune article from October 2024, citing research by the bot-management firm Kasada, said the crawler does not respect robots.txt.
  • Reported: the same article quoted Kasada's CEO saying it scraped at about 25 times the rate of GPTBot.
  • Not confirmed: what ByteDance uses the data for. Fortune described AI model work, but ByteDance did not respond to its request for comment.

Still, we could not confirm any of these points from ByteDance. Treat them as reports, and check your own log before you decide.

How to block Bytespider in WordPress

First, start with robots.txt anyway. It costs nothing, and it records your choice in the standard place.

robots.txt: states your choice
User-agent: Bytespider
Disallow: /

Then, because the operator gives no robots.txt policy and reports say it may not follow one, add a server rule. It answers 403 to any request whose user agent contains the token.

.htaccess (Apache, LiteSpeed)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]
</IfModule>
Nginx server block
if ($http_user_agent ~* "Bytespider") {
    return 403;
}

One click with Pagereckon Pro

This is one of the bots that Pagereckon Pro offers to block with one click. Pro writes a marked section at the top of your .htaccess on Apache or LiteSpeed, so the block also stops requests that a page cache would answer before PHP runs. On Nginx, it shows a ready snippet for your server block. One more click removes the block again.

What blocking Bytespider costs you

As far as anyone can show, very little. We know of no ByteDance product that documents sending visitors from this crawler. Since there is no published purpose, there is also no published benefit to weigh against the load it adds.

However, keep TikTok's link previews in mind. The Pagereckon classifier sees a separate agent, TikTokSpider, fetch link previews for pages shared inside the app. Blocking the token above does not touch that agent, because the user agent differs.

Spotting Bytespider without an IP list

Also, ByteDance publishes no IP list or reverse DNS rule that we could find. As a result, the user agent is the only label, and anyone can copy it.

In practice that matters less here than with Googlebot. Whether the request is real or a copycat, you probably want it blocked, and the server rule handles both.

Bytespider in Pagereckon

Pagereckon lists the crawler with the AI crawlers and marks it as a bot that crawls hard and gives nothing back. Next to it you see the pages it requested and how often. Once you block it with Pro, the bot shows as blocked, and one click lifts the block.

Want the bigger picture? The AI crawler checker shows every AI agent on your site, and the AI bots list explains each one. For a crawler that does publish its rules, compare with GPTBot.

Bytespider questions

Does Bytespider respect robots.txt?

ByteDance has published no policy we could find. A 2024 Fortune report, citing Kasada research, said it does not. Add a server rule if you want a firm block.

Who runs Bytespider?

ByteDance, the company that owns TikTok.

Will blocking Bytespider hurt my TikTok traffic?

Not that anyone has shown. In our classifier, TikTok link previews arrive under a different agent, TikTokSpider, which the rule above does not match.

Can Pagereckon block it for me?

Yes, with Pagereckon Pro. It is one of the bots with a Block button. See pricing for the plans.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.