FINDING YOUR (SERVER) LIMITS

A little postscript to my last post, 
2025-06-07My_Session_with_the_Bots.txt, written before I forget the 
details completely. Before I ended with an Apache MaxRequestWorkers 
setting of 350, since 450 caused the server's 1GB of RAM to run out 
before queuing connections to the PHP script which was getting 
DDoS'ed for some reason. Of course having written that I soon 
discovered my server bogged down again, stuck on that Apache 
process limit. Gradually raising it, I found that the RAM 
requirements just to serve the 503 error response to the 
now-blocked requests containing PHPSESSID were less than I thought 
before. I was able to up the process limit to 950 and actual 
simultaneous Apache processes topped out around 800 while logs 
showed hundreds of requests from random Brazilian/SE-Asian IPs 
being denied per second.

This is obviously a limitation of Apache's configuration options, 
because before blocking the requests based on that pointless 
PHPSESSID query string the Apache processes used much more RAM to 
serve the dynamic PHP page rather than just a few headers with an 
error response (the bots didn't retrieve the error page). What I 
really need is a way to automatically scale the process limit based 
on available RAM and average RAM usage per Apache process, but 
strangely that doesn't seem to be available.

It seems to me that it should be possible to write a script to do 
this by editing the MaxRequestWorkers setting and reload Apache. 
But testing it would be time consuming and at this point my tally 
of jobs to do Vs time to do them is already hopeless. Plus any new 
job that doesn't immediately serve a profit motive ought to be 
excluded given my current situation, and so long as my website 
works now, improvements ought to first and foremost be to 
making/finding products to sell there that people want to buy.

I haven't got Apache's caching module enabled, something I'm 
considering now, but presumably the different PHPSESSID values 
would have defeated that anyway. Anyway, the attack slowly died 
away in request volume and then went back again to the old noise of 
one to three AmazonBot hits per second.

By the way isn't it odd that in order to provide a website that few 
real people actually want to look at, one now has to cater for 
demand equivalent to it being viewed my millions of people a day. 
How is the internet even still working when this sort of load is 
being placed on it? Yes other people like to block large IP ranges 
rather than trying to absorb the bots, or use one of the 
bot-blocker services that are suddenly blocking me everywhere now, 
but that's really just breaking the internet even more directly. As 
one of the comparatively few real humans online, I demand not to 
have every other website accuse me of being a bot unless I run 
Firefox with lots of random scripts allowed through NoScript. 
Scripts from third-party services who doubtless have 
side-businesses of collecting and selling data on this real living 
human that they caught out there in the sea of bots.

At least it is kind-of cool that the cheapest VPS I found could 
serve millions of dynamic webpage requests per day from real 
humans, if they weren't all accompanied by a larger swarm of hungry 
bots. The lightweight code and small size of content on my site 
makes it super fast even though Apache (chosen because I didn't 
expect to be dealing with such loads) isn't the best choice for 
performance. Imagine how low-spec a system just to serve human 
requests could be? A Rapberry Pi Zero would be huge overkill. But 
then it's that sort of cheap modern processing power that allows 
people to run these DDoS/scraper bots, so it's all a vicious cycle, 
and I only won this time because this attacker/scraper was so 
idiotic as to just hit one PHP script with a pointless query string.

But all this broad thinking is obviously where I'm going wrong in 
life. Stuff the whys and wheres, I need to make money somehow. Hmm, 
what if I made a huge bot farm running half-arsed code to scrape 
all the websites on the internet to death and feed it onto some AI 
model I can sell to people making their own half-arsed AI junk? 
Yeah, great idea, that'd put food on the table.

 - The Free Thinker