2025-08-16 Let's Encrypt and the Butlerian Jihad ================================================ I'm having difficulties getting a certificate for search.transjovian.org. The Apache error log shows that the secondary validation is failing: [Sat Aug 16 14:58:04.325222 2025] [md:error] [pid 201594:tid 201596] ACME server authz: challenge 'invalid' for search.transjovian.org at https://acme-v02.api.letsencrypt.org/acme/authz/825400867/569770887957. Exact response was: {"identifier":{"type":"dns","value":"search.transjovian.org"},"status":"invalid","expires":"2025-08-23T12:58:02Z","challenges":[{"type":"http-01","url":"https://acme-v02.api.letsencrypt.org/acme/chall/825400867/569770887957/WMb5Bg","status":"invalid","validated":"2025-08-16T12:58:02Z","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: 178.209.50.237: Fetching http://search.transjovian.org/.well-known/acme-challenge/LiHh8qQ9y4NgW3kSxmWN1vseB4u7369Nqk8lIPHU3Rw: Connection refused","status":400},"token":"LiHh8qQ9y4NgW3kSxmWN1vseB4u7369Nqk8lIPHU3Rw","validationRecord":[{"url":"http://search.transjovian.org/.well-known/acme-challenge/LiHh8qQ9y4NgW3kSxmWN1vseB4u7369Nqk8lIPHU3Rw","hostname":"search.transjovian.org","port":"80","addressesResolved":["178.209.50.237","2a02:418:6a04:178:209:50:237:1"],"addressUsed":"2a02:418:6a04:178:209:50:237:1"}]}]} [Sat Aug 16 14:58:04.328364 2025] [md:error] [pid 201594:tid 201596] (22)Invalid argument: md[transjovian.org] while[Monitoring challenge status for transjovian.org] detail[domain authorization for search.transjovian.org failed, CA considers answer to challenge invalid.] [Sat Aug 16 14:58:04.334038 2025] [md:error] [pid 201594:tid 201596] (22)Invalid argument: AH10056: processing transjovian.org: Error waiting on domain names to be validated [Sat Aug 16 14:58:13.476015 2025] [md:error] [pid 201594:tid 201596] ACME server authz: challenge 'invalid' for search.transjovian.org at https://acme-v02.api.letsencrypt.org/acme/authz/825400867/569770940657. Exact response was: {"identifier":{"type":"dns","value":"search.transjovian.org"},"status":"invalid","expires":"2025-08-23T12:58:10Z","challenges":[{"type":"http-01","url":"https://acme-v02.api.letsencrypt.org/acme/chall/825400867/569770940657/LlZQgA","status":"invalid","validated":"2025-08-16T12:58:10Z","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: 178.209.50.237: Fetching http://search.transjovian.org/.well-known/acme-challenge/yQU4DjVWrCXtK1xdnZngKvrvd44RdI85A1-mcAEMxfE: Connection refused","status":400},"token":"yQU4DjVWrCXtK1xdnZngKvrvd44RdI85A1-mcAEMxfE","validationRecord":[{"url":"http://search.transjovian.org/.well-known/acme-challenge/yQU4DjVWrCXtK1xdnZngKvrvd44RdI85A1-mcAEMxfE","hostname":"search.transjovian.org","port":"80","addressesResolved":["178.209.50.237","2a02:418:6a04:178:209:50:237:1"],"addressUsed":"2a02:418:6a04:178:209:50:237:1"}]}]} [Sat Aug 16 14:58:13.480108 2025] [md:error] [pid 201594:tid 201596] (22)Invalid argument: md[transjovian.org] while[Monitoring challenge status for transjovian.org] detail[domain authorization for search.transjovian.org failed, CA considers answer to challenge invalid.] [Sat Aug 16 14:58:13.487338 2025] [md:error] [pid 201594:tid 201596] (22)Invalid argument: AH10056: processing transjovian.org: Error waiting on domain names to be validated [Sat Aug 16 14:58:22.582286 2025] [md:warn] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: acme problem urn:ietf:params:acme:error:rateLimited: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:32 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account [Sat Aug 16 14:58:22.585610 2025] [md:error] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: AH10056: processing transjovian.org: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:32 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account [Sat Aug 16 14:58:47.558946 2025] [md:warn] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: acme problem urn:ietf:params:acme:error:rateLimited: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:38 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account [Sat Aug 16 14:58:47.562322 2025] [md:error] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: AH10056: processing transjovian.org: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:38 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account [Sat Aug 16 14:59:15.500546 2025] [md:warn] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: acme problem urn:ietf:params:acme:error:rateLimited: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:30 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account [Sat Aug 16 14:59:15.504838 2025] [md:error] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: AH10056: processing transjovian.org: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:30 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account [Sat Aug 16 15:00:29.786596 2025] [md:warn] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: acme problem urn:ietf:params:acme:error:rateLimited: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:30 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account [Sat Aug 16 15:00:29.788931 2025] [md:error] [pid 201594:tid 201596] (70013)Missing parameter for the specified command line option: AH10056: processing transjovian.org: too many failed authorizations (5) for "search.transjovian.org" in the last 1h0m0s, retry after 2025-08-16 13:04:30 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account And indeed, the access log shows some successes: search.transjovian.org:80 2600:3000:2710:200::81 - - [16/Aug/2025:14:58:03 +0200] "GET /.well-known/acme-challenge/LiHh8qQ9y4NgW3kSxmWN1vseB4u7369Nqk8lIPHU3Rw HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 281 search.transjovian.org:80 2600:3000:2710:200::85 - - [16/Aug/2025:14:58:11 +0200] "GET /.well-known/acme-challenge/yQU4DjVWrCXtK1xdnZngKvrvd44RdI85A1-mcAEMxfE HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 339 On the forum, I learned that Let's Encrypt currently has one primary and 4 secondary centres; the primary and all but one of the secondaries must succeed. So I though, I must be blocking these at the firewall. Let's try to unban them all: # fail2ban-client unban --all 663 The graph shows the number of banned addresses and address ranges dropping from around 700 to zero. But it didn't help. 😭 And now here's the banger: When I run iptables --list I see a ton of entries in the f2b-butlerian-jihad chain! 😨 This is not good. At one point, the two must have gone out of sync. iptables --flush f2b-recidive iptables --flush f2b-butlerian-jihad iptables --flush f2b-butlerian-jihad-week ip6tables --flush f2b-recidive ip6tables --flush f2b-butlerian-jihad ip6tables --flush f2b-butlerian-jihad-week #Butlerian_Jihad #Administration #fail2ban #Let's_Encrypt