Faulty CrowdStrike Update Grounded Airlines And Knocked TV Stations Offline

AND THE FIX

Faulty CrowdStrike Update Grounded Airlines And Knocked TV Stations Offline

July 19, 2024

A faulty update from CrowdStrike has caused widespread chaos, rendering hundreds of thousands of Windows computers and servers unusable across the globe, affecting major sectors like transport, broadcasting, finance, and retail. The glitch triggered an endless blue-screen-of-death loop, forcing IT teams to manually intervene, often needing to work through the weekend to resolve the issue. Airlines paused flights, and TV stations went offline, highlighting the massive impact of this incident. CrowdStrike has identified and isolated the problem, but recovery remains a daunting task for many organizations.

How A Faulty CrowdStrike Update Grounded Airlines And Knocked TV Stations Offline 1

United, American, and Delta Airlines have all halted flights globally, and Sky News has been taken off the air in the United Kingdom.

What took place?

Its endpoint security agent, Crowdstrike, is now proven to be the source of what first appeared to be a Microsoft issue.

The broken update causes Windows hosts to enter a blue screen of death (BSOD) cycle, which can be avoided, per Crowdstrike’s advice, by:

1. Booting Windows into Safe Mode or the Windows Recovery Environment

2. Navigating to the C:\Windows\System32\drivers\CrowdStrike directory

3. Locating the file matching “C-00000291*.sys” and deleting it, then

4. Booting the host normally.

Regretfully, restoring a large fleet of Windows PC workstations at a company will take some time, especially on a Friday, and will often require direct intervention using a local administrator account. It appears that the IT and support teams will need to work over the weekend.

Security experts are attempting to analyze the “bad” version in the same way that Crowdstrike is doing in order to determine what went wrong.

Even though the outage’s root cause—a straightforward code error, most likely—was not intentional sabotage brought on by an undetected supply-chain intrusion, it nonetheless impacted availability, which is one of the three CIA triad components, and so constituted an information security breach.

UPDATE: 05:45 a.m. ET July 19, 2024

“The .sys files causing the issue are channel update files, they cause the top-level CS driver to crash as they’re invalidly formatted. It’s unclear how/why Crowdstrike delivered the files and I’d pause all Crowdstrikes updates temporarily until they can explain,” security researcher Kevin Beaumont noted.

“This is going to turn out to be the biggest ‘cyber’ incident ever in terms of impact, just a spoiler, as recovery is so difficult.”

UPDATE: 06:25 a.m. ET July 19, 2024

George Kurtz, President and CEO of CrowdStrike, reports that the organization is actively assisting clients affected by this problem, which was brought on by a single Windows host content update.

“Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed,” he explained.

“We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels.”

Even though the repair has been made available, many computers that are caught in the BSOD loop won’t be able to apply it without first requiring technical assistance. The process could take days or even weeks in firms with a large number of Windows computers (real or virtual) and a limited IT support staff.

UPDATE: 07:20 a.m. ET July 19, 2024

The IT outage caused by Crowdstrike has nothing to do with the majority of Microsoft 365 apps and services that have been sporadically unavailable in the US for the past 12 hours.

“The underlying cause [a configuration change in a portion of our Azure backend workloads] has been fixed, however, residual impact is continuing to affect some Microsoft 365 apps and services. We’re conducting additional mitigations to provide relief,” Microsoft says.

Additionally, the business has acknowledged that Windows 365 Cloud PCs were impacted by the flawed Crowdstrike upgrade and that “users may restore their Windows 365 Cloud PC to a known good state prior to the release of the update (July 19, 2024)”.

Last year, GreatGameInternational reported that a post about the Blue Screen of Death on a Ford car had gone viral on social media, which was found to be due to the failure of an OTA update.

GreatGameIndia

Me: I freeze the registry of my computer and don't update it with a software program called Deep Freeze. Another program which does the same thing is CryptDrive.