We've had this thing hammering our servers. The scraper uses randomized user-agents browser/OS combinations and comes from a number of distinct IP ranges in different datacenters around the world, but all the IPs track back to Bytedance.
I had to block ByteSpider at work because it can't even parse HTML correctly and just hammers the same page and accounts to sometimes 80% of the traffic hitting a customer's site and taking it down.
The big problem with AI scrapers is unlike Google and traditional search engines, they just scrape so aggressively. Even if it's all GETs, they hit years old content that's not cached and use up the majority of the CPU time on the web servers.
Scraping is okay, using up a whole 8 vCPU instance for days to feed AI models is not. They even actively use dozens of IPs to bypass the rate limits too, so theyre basically DDoS'ing whoever they scrape with no fucks given. I've been woken up by the pager way too often due to ByteSpider.
My next step is rewriting all the content with GPT-2 and serving it to bots so their models collapse.
Bullshit. This bot doesn't identify itself as a bot and doesn't rate limit itself to anything that would be an appropriate amount. We were seeing more traffic from this thing that all other crawlers combined.
It's illegal when a regular person steals something, but it's innovation and courage, when a huge corporation steals something. Interesting how that works
Honestly it’s fucking angering. So much regulation and geo-restrictions and licensing schemes… but it’s cool that there are data brokers, and shit like this. On top of it all Chrome screwing us with manifest v3 and killing ad blocking on chrome. It’s already in canary build.
Google are actually doing really awesome work with manifest v3. A pimp needs to smack their b1tches around every once in a while to remind them who's boss.
My personal website that primarily functions as a front end to my home server has been getting BEAT by these stupid web scrapers. Every couple of days the server is unusable because some web scraper demanded every single possible page and crashed the damn thing
They're too late, there's going to be way too much AI generated garbage in their data and so many social media platforms like Reddit and Twitter have already taken measures to curb scrapers.
Like those platforms aren't already full of AI garbage as well. Training new models will require a cut-off date before the genie was let out of the bottle.