Subj : Re: Opinions on a door idea
To   : All
From : Jay Harris
Date : Wed Jul 03 2024 13:33:04

From: nospam.Jay.Harris@f664.n229.z1.fidonet.org (Jay Harris)

On 02 Jul 2024, Daniel said the following...

 Da> If I attempt accessing it with lynx, I get this generic error:
 Da> 
 Da> "Due to aggressive automated scraping of FederalRegister.gov and
 Da> eCFR.gov, programmatic access to these sites is limited to access to our
 Da> extensive developer APIs."
 Da> 
 Da> I don't get this message on a normal javascript enabled browser. I
 Da> surmise it views queries from non-javascript browsers as a curl-like
 Da> command.

I don't know if you can change the user agent in lynx, but try and see if it 
works with curl.

e.g:
agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101
Firefox/127.0"

curl -A "$agent" <url>


I've had to do this in the past when pulling information for weather, they
stopped allowing curl, but simply changing the user agent allowed it to work.


Jay

..... Modem: What landscapers do to dem lawns

--- NewsGate v1.0 gamma 2
 * Origin: News Gate @ Net396 -Huntsville, AL - USA (1:396/4)

.