CrowdStrike and Microsoft: all the latest news on the global IT outage

A senior CrowdStrike executive will testify before the House Homeland Security Committee next month about the IT outage that grounded planes and workplaces to a halt globally on July 19th.

Adam Meyers, CrowdStrike’s senior vice president of counter adversary operations, has agreed to appear before the panel on September 24th at 2PM ET, the committee announced. Committee leaders had previously called on CEO George Kurtz to testify, but he’s not currently listed as a witness.

Microsoft is hosting an important summit on Windows security at its Redmond, Washington, headquarters next month. The Windows Endpoint Security Ecosystem Summit on September 10th will bring together Microsoft engineers and vendors like CrowdStrike to discuss improvements to Windows security and third-party best practices to try and prevent another CrowdStrike incident.

“Microsoft, CrowdStrike and key partners who deliver endpoint security technologies will come together for discussions about improving resiliency and protecting mutual customers’ critical infrastructure,” says Aidan Marcuss, corporate vice president of Microsoft Windows and devices. “Our objective is to discuss concrete steps we will all take to improve security and resiliency for our joint customers.”

CrowdStrike president Michael Sentonas personally accepted a “Most Epic Fail” award over the weekend at the annual Las Vegas Def Con hacking conference for the company’s software update that caused a global IT outage last month. While accepting the award, he said it was “super important to own it when you do things horribly wrong, which we did in this case.”

The Pwnie Awards, given out in an annual ceremony at Def Con, celebrate “the achievements (and failures) of security researchers and the security community.” This year’s categories included awards for mobile, desktop, and crypto bugs, one for “Lamest Vendor Response,” and an “Epic Achievement” award for researchers who discovered critical vulnerabilities.

Microsoft has responded to Delta Air Lines’ criticism of Windows and CrowdStrike after the giant IT outage last month. Delta CEO Ed Bastian wants compensation from both CrowdStrike and Microsoft for the estimated $500 million Delta lost due to the outage. Now, Microsoft says Delta refused its free help on multiple occasions and even ignored an email from CEO Satya Nadella to Bastian.

“Microsoft empathizes with Delta and its customers regarding the impact of the CrowdStrike incident. But your letter and Delta’s public comments are incomplete, false, misleading, and damaging to Microsoft and its reputation,” says Mark Cheffo, co-chair of Dechert’s global litigation practice, in a letter on behalf of Microsoft to Delta’s lawyers.

CrowdStrike refutes Delta Air Lines’ allegations that the cybersecurity firm is to blame for a dayslong flight disruption following last month’s catastrophic system outage, saying that the airline rejected repeated offers to help restore impacted systems. 

In an interview with CNBC last week, Delta CEO Ed Bastian said the outage cost the company $500 million after more than 6,000 flights were grounded, and that the airline has “no choice” but to seek legal compensation from CrowdStrike and Microsoft. 

The CrowdStrike global IT outage that downed some 8.5 million Windows PCs, halting flights and affecting hospitals, banks and more, is now apparently also a scapegoat for the terrible conditions at the set of a reality game show competition put on by mega-popular YouTuber MrBeast, aka Jimmy Donaldson, with $5 million on the line.

The New York Times is reporting that after 2,000 contestants arrived at Allegiant Stadium this July, they were barely fed and didn’t receive their prescription medication or clean underwear on time — despite providing it to the organizers themselves. (The 1,000 contestants who make it through can return for the Beast Games Amazon show, but this segment is for Donaldson’s YouTube channel.)

In an interview with CNBC, Delta Air Lines CEO Ed Bastian said the July 19th outage caused by a CrowdStrike update cost his company half a billion dollars in five days. Delta canceled more than 5,000 flights that weekend and had blue error screens still visible at airports days after the initial crash. Among the costs Bastian said Delta incurred were more than 40,000 servers that “we had to physically touch and reset” as well as compensation payments to travelers left in the lurch.

Asked about a continuing relationship with Microsoft after the crash, Bastian said he regards it as “probably the most fragile platform” and asked the question, “When was the last time you heard of a big outage at Apple?” He placed some blame on the valuations of big tech companies, which lately have been lifted by generative AI hype, saying, “...they’re building the future, and they have to make sure they fortify the current.”

Microsoft is still helping CrowdStrike clean up the mess that kicked off a week ago when 8.5 million PCs went offline due to a buggy CrowdStrike update. Now, the software giant is calling for changes to Windows and has dropped some subtle hints that it’s prioritizing making Windows more resilient and is willing to prevent security vendors like CrowdStrike from accessing the Windows kernel.

While CrowdStrike has blamed a bug in its testing software for its botched update, its software runs at the kernel level — the core part of an operating system that has unrestricted access to system memory and hardware. This means that if something goes wrong with CrowdStrike’s app, it can take down Windows machines with a Blue Screen of Death.

As CrowdStrike scrambled to respond to an influx of crash reports early Friday morning, engineers inside Microsoft also noticed that something was majorly wrong. Millions of Windows machines were being sent into a perpetual Blue Screen of Death, taking down important servers and PCs around the world.

Microsoft quickly declared the incident a “severity zero,” or what’s known internally as sev0, according to sources familiar with the situation. This is the highest, most urgent level of an incident impacting Microsoft products or services. A sev0 incident is rare at Microsoft, and it means people get woken up in the middle of the night and on-call engineers immediately start figuring out what happened and how to respond.

CrowdStrike has published a post incident review of the buggy update it published that took down 8.5 million Windows machines last week. The detailed post blames a bug in test software for not properly validating the content update that was pushed out to millions of machines on Friday. CrowdStrike is promising to more thoroughly test its content updates, improve its error handling, and implement a staggered deployment to avoid a repeat of this disaster.

CrowdStrike’s Falcon software is used by businesses around the world to help manage against malware and security breaches on millions of Windows machines. On Friday, CrowdStrike issued a content configuration update for its software that was supposed to “gather telemetry on possible novel threat techniques.” These updates are delivered regularly, but this particular configuration update caused Windows to crash.

On Friday morning, shortly after midnight in New York, disaster started to unfold around the world. In Australia, shoppers were met with Blue Screen of Death (BSOD) messages at self-checkout aisles. In the UK, Sky News had to suspend its broadcast after servers and PCs started crashing. In Hong Kong and India, airport check-in desks began to fail. By the time morning rolled around in New York, millions of Windows computers had crashed, and a global tech disaster was underway.

In the early hours of the outage, there was confusion over what was going on. How were so many Windows machines suddenly showing a blue crash screen? “Something super weird happening right now,” Australian cybersecurity expert Troy Hunt wrote in a post on X. On Reddit, IT admins raised the alarm in a thread titled “BSOD error in latest CrowdStrike update” that has since racked up more than 20,000 replies.

It all started with a configuration file. A seemingly routine update, the kind that happens hundreds of times a year to millions of computers around the world. But last week, that update crashed 8.5 million computers and wreaked havoc on banks, airlines, schools, and more.

On this episode of The Vergecast, The Verge’s Tom Warren joins the show to talk about the story and legacy of the CrowdStrike crash. We talk about what exactly happened, how CrowdStrike and Microsoft scrambled to fix it, and whether we’re due for a reckoning over how much we rely on complex and fragile technology. (Don’t forget to subscribe to Notepad!)

CrowdStrike CEO George Kurtz was called to testify before the House Homeland Security Committee over the major outage affecting Windows PCs spurred by a faulty update that brought flights, hospital procedures, and broadcasters to a halt on Friday, The Washington Post reported.

“Recognizing that Americans will undoubtedly feel the lasting, real-world consequences of this incident, they deserve to know in detail how this incident happened and the mitigation steps CrowdStrike is taking,” Homeland Security Chair Mark Green (R-TN) and Cybersecurity and Infrastructure Protection Subcommittee Chair Andrew Garbarino (R-NY) wrote in a letter later shared with The Verge. They asked that CrowdStrike schedule a hearing with the subcommittee by end of day Wednesday.

The CrowdStrike outage that hit millions of Windows machines on Friday has left IT workers scrambling to get their organizations’ computer infrastructure back up and running. Images and stories shared online are illustrating just how tedious and overwhelming this task is.

Microsoft and CrowdStrike don’t have a way to push a fix for the issue to crashed computers. And in many cases, systems administrators can’t repair the machines remotely, either. That leaves them working in person to do things like use a Microsoft-created tool on a USB drive, delete a specific file while in Safe Mode, or restart affected computers over and over again, hoping an update comes through.

CrowdStrike has published a new “Remediation and Guidance Hub” that collects details related to its faulty update that crashed 8.5 million Windows computers across the globe on Friday.

The page includes technical information on what caused the outage, what systems are affected, and CEO George Kurtz’s statement. It contains links to Bitlocker key recovery processes and to various third-party vendor pages about dealing with the outage, as well.

Microsoft has released a recovery tool that’s designed to help IT admins repair Windows machines that were impacted by CrowdStrike’s faulty update that crashed 8.5 million Windows devices on Friday. The tool creates a bootable USB drive that IT admins can use to help quickly recover impacted machines.

While CrowdStrike has issued an update to fix its software that led to millions of Blue Screen of Death errors, not all machines are able to automatically receive that fix. Some IT admins have reported rebooting PCs multiple times will get the necessary update, but for others, the only route is having to manually boot into Safe Mode and deleting the problematic CrowdStrike update file.

CrowdStrike’s faulty update caused a worldwide tech disaster that affected 8.5 million Windows devices on Friday, according to Microsoft. Microsoft says that’s “less than one percent of all Windows machines,” but it was enough to create problems for retailers, banks, airlines, and many other industries, as well as everyone who relies on them.

Separately, the technical breakdown from CrowdStrike released Friday explains more about what happened and why so many systems were affected all at once.