2026-06-10 Less banning, more garbage ===================================== Starting a week ago, the number of requests have gone up again, and with that, my automated ban-hammer has been banning whole autonomous systems. This in turn means that every network they control is banned. And that means that every single IP address range that they control is banned. At the firewall. It's harsh, and a lot of innocent people get banned. I tell myself that it's OK because residential and commercial networks are usually separate. You can host your artificial intelligence (AI) scraper bot on a commercial network but it shouldn't be possible to get thousands of IP addresses in residential networks. The sad truth is, however, that some internet service providers still serve those AI scraper bots because all the super rich companies that need to train their large language models (LLM) pay good money. And where are universities that do it for research? And what about the companies that sell development kits to app developers that help monetize "free" apps by renting out bandwidth? All the innocents using such free apps are participating in this blasting of the open web. So innocents are going to get caught up. One solution that @algernon@come-from.mad-scientist.club has been proposing is to ban less and serve more garbage. And perhaps he's right. So I made two changes today. The first change concerns the various watchers: # watch-recent-bans watch-active-autonomous-systems 19242 watch-attempted-edits 26 watch-expensive-end-points 30181 As you can see, watch-expensive-end-points triggered the most bans. But some of these endpoints I know to be no longer available, so I could also just serve garbage. In my Apache config: # Block all the old expensive endpoints for Oddmuse RewriteCond "%{QUERY_STRING}" "\baction=(rss|rc|contrib)\&|\bsearch=" RewriteRule "^" - [redirect=410,last] The error document for the error 410 is my garbage generator: # The 410 GONE response also contains garbage ErrorDocument 410 /nobot/ RewriteRule "^/nobot/.*" "unix:/run/garbage.sock|http://localhost/" [proxy,last] To test it, let's simulate this request from the log: communitywiki.org:443 102.96.140.237 - - [09/Jun/2026:01:00:14 +0200] "GET /wiki?action=rss&all=0&days=28&full=1&rcfilteronly=tag%3Ainteraction&showedit=1 HTTP/1.1" 200 5283 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Brave Chrome/84.0.4147.89 Safari/537.36" 8076 Using curl, we get garbage for this request: curl 'https://communitywiki.org/wiki?action=rss&all=0&days=28&full=1&rcfilteronly=tag%3Ainteraction&showedit=1' The garbage generator also serves all bots: curl --user-agent "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \ "https://campaignwiki.org/" Oh, and I discovered the following user-agent in my logs: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 (compatible; vision_data-collector; +https://github.com/rom1504/img2dataset) > Easily turn large sets of image urls to an image dataset. Can > download, resize and package 100M urls in 20h on one machine. -- > img2dataset Needless to say, that also goes onto the block list. Fuck you, developers of such tools. I can opt-out, of course. So 38 developers support this project and every single webmaster has to change the config of their web server in order to opt out. Or pay the price, of course, and spend compute and bandwidth to serve the images to these people. Because opt-in doesn't work. Who would want to opt in? Nobody. And of course it would be super expensive to ask every single one of us. So it's easier for them to just take the things they want. I remember that article naming this for what it is, taking what you want without consent. It's a four-letter word culture. > This field (or fields, whatever) is now describing the future in the > language of coercion. Because this appears to be something that the > leaders in this field are very comfortable with. Force. – How to use > NO as a complete sentence, by Mike Monteiro And I made another change, based on what I heard @algernon@come-from.mad-scientist.club does for iocaine: If the user-agent claims to be a modern browser, check if it sends the Sec-Fetch-Site header. If not, serve garbage. # Block user-agents that says Chrome or Firefox but don't set Sec-Fetch-Site (like iocaine does) RewriteCond "%{HTTP_USER_AGENT}" "Chrome|Firefox|Safari" RewriteCond "%{HTTP:Sec-Fetch-Site}" "^$" RewriteRule "^" - [redirect=410,last] Use curl, this gets garbage: curl --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0" \ "https://alexschroeder.ch/view/index" Whereas this gets the legit HTML: curl --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0" \ --header "Sec-Fetch-Site: same-site" "https://alexschroeder.ch/view/index" It's an arms race and I'm in it. But I'm hoping to outlast the AI fuckers. One day soon, the next AI winter will come. The bubble will burst. ⁂ And finally, at the end, some graphs. From June 3 to June 5, Apache connections increased by a factor of 10. Currently we're at 2.5; the average over the last few days is 10, the maximum was over 70. From June 3 to June 5, the number of banned IP address ranges rose from around two thousand to ten thousand. Just now there was another surge and now there are about sixteen thousand IP address ranges banned. Total hits right now is about 2 per second with a maximum of 22 per second. Garbage is served at a rate of 0.5 hits per second with a maximum of 8.5 hits per second. Add least the load average remains under control. With an occasional spike of up to 8. #Butlerian_Jihad 2026-06-10. A side trek about this user-agent from img2dataset. I just realised that it contains Firefox/72 – and that rings a bell. I blogged about it in 2024 because @alexskunz@mas.to had mentioned it – and he blogged about it, too. People in my private life keep wondering why the AI scrapers keep downloading all this stuff and this is the answer: they treat the web as their library, they treat your website as their storage medium. They don’t want to store a gazillion images! That would take far too much space and it would look like theft. So their bots download your images, train their AIs, and delete the copy they made. And next month, when it’s time to update their model, they do it again. Everybody who trains an image-processing AI does it, again and again. Downloading all your images. Every single time. The URLs of your images are in their datasets, not your images. Now you know how I feel about AI-generated images.