Non-Cloudflare AI blocking?
Non-Cloudflare AI blocking?
Now that we know AI bots will ignore robots.txt and churn residential IP addresses to scrape websites, does anyone know of a method to block them that doesn't entail handing over your website to Cloudflare?
You're viewing a single thread.
If I'm reading your link right, they are using user agents. Granted there's a lot. Maybe you could whitelist user agents you approve of? Or one of the commenters had a list that you could block. Nginx would be able to handle that.
They just Fake User Agents If you Block them
Thank you for the reply, but at least one commenter claims they'll impersonate Chrome UAs.
You can read more Here
If you try to rate-limit them, they'll just switch to other IPs all the time. If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.
Except it's not denying service, so it's just a D.
In the hackernews comments for that geraspora link people discussed websites shutting down due to hosting costs, which may be attributed in part to the overly aggressive crawling. So maybe it's just a different form of DDOS than we're used to.