Subj : Cloudflare says a bad update broke its logging systems, made it l
To   : All
From : TechnologyDaily
Date : Wed Nov 27 2024 16:30:05

Cloudflare says a bad update broke its logging systems, made it lose data

Date:
Wed, 27 Nov 2024 16:28:00 +0000

Description:
55% of data logs have been lost after a dodgy Cloudflare update broke its 
systems.

FULL STORY
======================================================================Cloudfla
re confirms update made it lose customer log data The incident lasted 3.5 
hours in total, leading to a 55% loss of logs Despite a five-minute fix, the 
bug caused knock-on issues 

Cloudflare has confirmed a bad software update caused it to lose log data for 
its customers recently. The incident, which lasted around 3.5 hours, resulted 
in more than half (55%) of logs being lost. 

Embarrassed that the error occurred, the California company apologized to 
customers in a blog post , promising a similar issue should not happen again. 

Cloudflare also noted that failures within systems at scale are inevitable, 
but subsystems should be built to protect themselves in the event of wider 
issues. Cloudflare admits to losing data logs 

The problem originated with Cloudflares Logpush service, which bundles and 
sends logs from its global network to customers for compliance, debugging and 
analytics. A routine update to support a new data set ended up misconfiguring 
the service, causing the issue. 

The company says a configuration bug effectively told one of its internal 
servers, Logfwdr, that none of its customers had configured logs to be sent, 
leading to the loss. Although engineers identified and fixed the bug within 
five minutes, the issue triggered a deeper bug. 

A built-in fail-safe, which sends logs to all customers rather than just 
those with active Logpush jobs, ended up overwhelming the system. The 
buffering system, Buftee, had to manage 40 times its usual capacity, 
rendering the system unresponsive. 

We accept that mistakes and misconfigurations are inevitable. All our systems 
at Cloudflare need to respond to these predictably and gracefully," the 
company wrote. 

Looking ahead, Cloudflare has committed to conducting regular overload tests 
to simulate this error, providing confidence that its systems can handle 
future bugs of a similar nature. You might also like Check out the best cloud 
hosting providers Weve listed the best cloud storage providers Cloudflare 
thinks it has a way to make millions of web pages faster, for free



======================================================================
Link to news story:
https://www.techradar.com/pro/cloudflare-says-a-bad-update-broke-its-logging-s
ystems-made-it-lose-data


--- Mystic BBS v1.12 A47 (Linux/64)
 * Origin: tqwNet Technology News (1337:1/100)

.