CrowdStrike, Microsoft Outage Causes Global IT Meltdown

Cybersecurity firm CrowdStrike’s update early Friday wreaked havoc on Microsoft Windows hosts globally, canceling flights, impacting hospitals, banks, news organizations, railways, and other critical services as companies scramble to find a fix.

CrowdStrike CEO George Kurtz on his LinkedIn account said the outage was not the result of a cyberattack and blamed a defective update to its Falcon antivirus software. The Austin-based CrowdStrike has become a major player in IT with 24,000 customers globally. The company boasts usage by about half of Fortune 500 firms.

CrowdStrike CEO George Kurtz speaking during the RSA Conference in San Francisco. Photo by Shane Snider

“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts,” he wrote.  “Mac and Linux hosts are not impacted. This is not the result of a security incident or cyberattack. The issue has been identified, isolated, and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”

In a response on LinkedIn, Jose Calderon, IT director for the city of El Segundo, Calif., wrote, “A fail this historic deserves to have the fix be posted on your homepage and ALL your socials ASAP. Teams all around the world are running fire drills right now to get a handle on things and you want us to open a ticket?!!!”

In a statement to InformationWeek, a Microsoft spokesperson wrote, "Earlier today, a CrowdStrike update was responsible for bringing down a number of IT systems globally. We are actively supporting customers to assist in their recovery.”

Microsoft noted that the company does not believe the latest outage is related to a July 18 outage that impacted some Azure customers.

Former Microsoft CIO and author Jim DuBois tells InformationWeek, "Anytime a vendor does an update, they have a chance to screw things up, if you don't test well. When multiple suppliers are involved, it is more complex."

The travel industry was one of the biggest victims of the day, with Airports in the US, Australia, Japan, India, Europe, and more causing outages and delays. Hospitals were also badly hit. Israel said 15 hospitals had to switch to manual processes and ambulances were told to take cases to other hospitals, according to BBC.

CrowdStrike’s support forum posted an alert early Friday, saying the problem was “related to Falcon Sensor,” which is its cloud-based security service. The support forum describes workaround steps, including:

The update is causing Windows to crash, and reports of blue screen errors have been common. Several commenters on LinkedIn have been using #BLUEFRIDAY to describe the event.

Manual Fix Could Take ‘Days’

Omer Grossman, CIO at Israeli firm CyberArk, tells InformationWeek in an email that the issue could take “days” to fix.

“The current event appears -- even in July -- that it will be one of the most significant cyber issues of 2024,” he says. “The damage to business processes at the global level is dramatic … There are two main issues on the agenda: The first is how customers get back online and regain continuity of business processes. It turns out that because the endpoints have crashed -- the ‘Blue Screen of Death’ -- they cannot be updated remotely and this problem must be solved manually, endpoint by endpoint. This is expected to be a process that will take days.”

He adds, “The second is around what caused the malfunction … CrowdStrike’s analysis and updates in the coming days will be of the utmost interest.”

How CIOs Should Respond

Grossman says CIOs can take several steps to respond to the major incident.

“Immediate steps for IT and security teams would be to manually bypass the BOD (Blue Screen Of Death) at the endpoints," he writes. "The correct sources of information here are CrowdStrike’s formal guidelines. If they have experienced the same issues on their servers, IT can centrally manage the situation using their suite of infrastructure management tools. Finally, they should turn off automatic updates on critical systems/infrastructure and so on.”

In the future, organizations will likely be more careful as they make updates. "Perhaps the biggest lesson is to adopt a program of phased deployments of updates, which is best practice in many industries and organizations in any case. Always start with a test group. Make sure to categorize what are critical systems and what aren't."

DuBois says CIOs will need to take a hard look at their update practices and vendor relationships. "CIOs probably don't have a lot of leverage int he short term unless they are about to make a decision. They will need to make sure they believe the story about lessons learned and actions to prevent reoccurrence, and push back if it isn't good enough... and start looking at alternatives when it comes time to renew."

This story is developing, and InformationWeek will update.