Bytespider: what we know about ByteDance's crawler
Bytespider is a web crawler run by ByteDance, the company behind TikTok. Unlike OpenAI, Anthropic or Google, ByteDance has published no English documentation for it that we could find: no stated purpose, no full user agent string, no IP list and no robots.txt policy.
So this page is careful. We separate what you can see for yourself in a server log from what the press has reported, and we label each.
Bytespider
- Operator
- ByteDance
- Type
- Several purposes
- robots.txt token
Bytespider- Obeys robots.txt
- Not stated by the operator
- Official docs
- zhanzhang.toutiao.com
What is documented, and what is only reported
First, the token itself is easy to confirm, because it shows up in access logs as part of the user agent. Beyond that, the official record is thin. The card above links to ByteDance's Toutiao Search webmaster platform, but we found no page there that describes this crawler.
However, press coverage fills some of the gap, with clear caveats:
- Reported: a Fortune article from October 2024, citing research by the bot-management firm Kasada, said the crawler does not respect robots.txt.
- Reported: the same article quoted Kasada's CEO saying it scraped at about 25 times the rate of GPTBot.
- Not confirmed: what ByteDance uses the data for. Fortune described AI model work, but ByteDance did not respond to its request for comment.
Still, we could not confirm any of these points from ByteDance. Treat them as reports, and check your own log before you decide.
How to block Bytespider in WordPress
First, start with robots.txt anyway. It costs nothing, and it records your choice in the standard place.
User-agent: Bytespider
Disallow: /Then, because the operator gives no robots.txt policy and reports say it may not follow one, add a server rule. It answers 403 to any request whose user agent contains the token.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]
</IfModule>if ($http_user_agent ~* "Bytespider") {
return 403;
}One click with Pagereckon Pro
This is one of the bots that Pagereckon Pro offers to block with one click. Pro writes a marked section at the top of your .htaccess on Apache or LiteSpeed, so the block also stops requests that a page cache would answer before PHP runs. On Nginx, it shows a ready snippet for your server block. One more click removes the block again.
What blocking Bytespider costs you
As far as anyone can show, very little. We know of no ByteDance product that documents sending visitors from this crawler. Since there is no published purpose, there is also no published benefit to weigh against the load it adds.
However, keep TikTok's link previews in mind. The Pagereckon classifier sees a separate agent, TikTokSpider, fetch link previews for pages shared inside the app. Blocking the token above does not touch that agent, because the user agent differs.
Spotting Bytespider without an IP list
Also, ByteDance publishes no IP list or reverse DNS rule that we could find. As a result, the user agent is the only label, and anyone can copy it.
In practice that matters less here than with Googlebot. Whether the request is real or a copycat, you probably want it blocked, and the server rule handles both.
Bytespider in Pagereckon
Pagereckon lists the crawler with the AI crawlers and marks it as a bot that crawls hard and gives nothing back. Next to it you see the pages it requested and how often. Once you block it with Pro, the bot shows as blocked, and one click lifts the block.
Want the bigger picture? The AI crawler checker shows every AI agent on your site, and the AI bots list explains each one. For a crawler that does publish its rules, compare with GPTBot.
Bytespider questions
Does Bytespider respect robots.txt?
ByteDance has published no policy we could find. A 2024 Fortune report, citing Kasada research, said it does not. Add a server rule if you want a firm block.
Who runs Bytespider?
ByteDance, the company that owns TikTok.
Will blocking Bytespider hurt my TikTok traffic?
Not that anyone has shown. In our classifier, TikTok link previews arrive under a different agent, TikTokSpider, which the rule above does not match.
Can Pagereckon block it for me?
Yes, with Pagereckon Pro. It is one of the bots with a Block button. See pricing for the plans.