CrowdStrike Releases Root Cause Analysis to Dissect Global IT Outage

A finicky sensor is to blame for the July 19 IT outage that canceled thousands of flights and disrupted critical services globally, according to a root cause analysis released Tuesday as part of CrowdStrike’s ongoing effort to explain the incident.

CrowdStrike detailed what it calls the “Channel 291 Incident,” which stemmed from a new sensor capability introduced to the company’s popular AI-powered Falcon sensors in March. On July 19, the company sent out a Rapid Response Content update to certain Windows hosts. The sensor expected the update to have 20 input fields, but it had 21 input fields. The mismatch caused systems to crash worldwide.

“We are using the lessons learned from this incident to better serve our customers,” CrowdStrike Founder and CEO George Kurtz said in a statement. “To this end, we have already taken decisive steps to help prevent this situation from repeating, and to help ensure that we -- and you -- become even more resilient.”

The outage is said to have cost Fortune 500 companies $5.4 billion. Lawsuits are already underway, including class-action lawsuits from a group of investors and from a group of travelers who were stranded due to the outage. Delta Air Lines has also threatened to sue CrowdStrike, saying the outage cost the airline $500 million. CrowdStrike attorneys called Delta’s threat “meritless.”

Related:CrowdStrike Aftermath: Lessons Learned for Future Recovery

“We are deeply sorry for the impact this had on you,” Kurtz said in the statement. “Nothing is more important than regaining your trust and confidence.”

CrowdStrike says it has taken several steps and outlined future actions, including updating its content configuration system test procedures, adding additional deployment layers and acceptance checks, providing additional controls for customers over the rapid response content updates, and more.

The company says it will engage two independent third-party software security vendors to conduct further review of the Falcon sensor code.

“Looking ahead, CrowdStrike is focused on using the lessons learned from this incident to better serve our customers,” a company spokesperson said in an email.