What Happened

CrowdStrike’s Falcon Sensor update led to a BSOD loop on SQL Nodes, impacting many MSSQL servers worldwide with the error: DRIVER_OVERRAN_STACK_BUFFER. Additionally, the CrowdStrike agent caused BSODs on MS Server 2022 Terminal servers, displaying the error “KERNEL_SECURITY_CHECK_FAILURE,” which is often linked to a stack-based buffer overrun. This CrowdStrike kernel level failure underscores the challenges and risks associated with traditional endpoint security solutions that rely heavily on kernel-level interventions.

Technically What Happened

The BSOD issues with CrowdStrike update, and similar outage incidents with other vendors like Sophos, Cisco, Symantec, Microsoft Defender, Check Point, Trellix, and VMware Carbon Black, all highlight a common theme: insufficiently tested kernel driver updates. Kernel drivers, operating at the core of the operating system, manage critical functions and have privileged access to system resources. When these drivers are updated to combat new threats, any flaws or bugs introduced can lead to severe system instability and crashes.

Why is the Kernel Important? Kernel drivers are essential for managing and facilitating communication between the operating system and hardware. They have high privileges and direct access to system memory and hardware resources, making them critical for the overall functioning and performance of the system. However, this high level of access also makes any issues or bugs within kernel-level drivers potentially catastrophic, leading to severe system instability, crashes, and security vulnerabilities.

Should You Use Security Software That Requires Kernel Updates? Using security software that frequently requires kernel updates can be risky due to the potential for system instability and operational disruptions. The recent CrowdStrike kernel level failure underscores these risks. While traditional cybersecurity solutions need these updates to combat emerging threats, they can lead to significant issues if not thoroughly tested.

Enhanced File Monitoring and Protection: Key Features of File Mini-Filter Drivers

Continuous File System Monitoring: File mini-filter drivers offer real-time surveillance of file system activities, including file creation, modification, deletion, and access. This continuous monitoring allows endpoint protection products to identify suspicious or malicious behaviors, such as unauthorized data access or ransomware attempts to encrypt files.

Enforcement of Access Control Policies: These drivers can implement security policies by managing file access based on predefined rules. They can block unauthorized applications from accessing sensitive data or prevent the execution of potentially harmful files, thereby reducing the risk of malware infections.

Integrity Verification: File mini-filter drivers enable endpoint protection solutions to perform integrity checks on critical system files and important user data. This process helps detect and prevent file-based attacks that involve tampering or corruption of essential files.

Content Scanning for Threat Detection: By intercepting file read and write operations, mini-filter drivers allow security software to scan file content for known malware signatures or suspicious patterns. This scanning occurs seamlessly in the background, without interrupting the user’s activities.

On-the-Fly Encryption and Decryption: Some endpoint protection solutions utilize mini-filter drivers to execute real-time encryption and decryption of files. This feature ensures that data remains protected at rest and is accessible only by authorized users and applications.

Comprehensive Logging and Auditing: Mini-filter drivers can generate detailed logs of all file activities by intercepting file system operations. These logs are crucial for forensic analysis, enabling security teams to understand the nature and scope of a security incident.

Operating at the kernel level, file mini-filter drivers provide robust and comprehensive security features that are difficult to bypass. However, their integration and deployment must be meticulously managed to avoid system instability and ensure compatibility with other kernel-level components.

Industries Affected and Specific Examples of Who of those 8.5M Microsoft Windows Devices Affected by CrowdStrike Outage

Industries Affected

The recent CrowdStrike IT outage has sent shockwaves across a wide array of industries, revealing the broad economic and societal impacts of such disruptions. Affecting 8.5 million Windows devices, the incident has highlighted the critical vulnerabilities within enterprises that run many essential services. Financial institutions, healthcare providers, airlines, IT service companies, retail chains, manufacturing plants, and educational institutions have all faced severe operational challenges due to this outage. This comprehensive report delves into the specific impacts across these sectors, serving as the definitive source for understanding the full scope of the incident’s repercussions.

Financial Institutions

• Banks and Financial Service Providers: Banks and financial service providers were significantly affected by the CrowdStrike outage. Operational disruptions were rampant, and the UK Cyber Security Centre reported a spike in phishing attempts during this period. This highlights the vulnerability of financial institutions to IT outages and the need for robust cybersecurity measures.
  • Bank of America: Reports of system crashes affecting online banking services.
  • Wells Fargo: Encountered issues with ATMs and online banking due to BSODs.
  • JP Morgan Chase: Experienced disruptions in trading operations.
  • ASB Bank: Their New Zealand, FastNet Classic and Mobile app were impacted.
  • Kiwibank in New Zealand
  • Capitec: South African bank
  • Frost Bank
  • TD Bank
  • Chase Bank
  • Charles Schwab
  • NAB in Australia
  • Commonwealth in Australia
  • Bendigo in Australia
• Stock Exchanges
  • London Stock Exchange: Website experienced issues, as well “RNS news service is currently experiencing a third-party global technical issue, preventing news from being published on www.londonstockexchange.com,” the company said in a statement.

Healthcare Providers & Phrama

• Hospitals and Healthcare Facilities: Hospitals and healthcare facilities relying on Windows-based systems faced major challenges due to the outage. The inability to access critical patient data and administrative functions potentially compromised patient care and operational efficiency. This disruption underscores the importance of reliable IT infrastructure in the healthcare sector.
  • Mayo Clinic: Reported system downtimes impacting patient records and appointment scheduling.
  • Cleveland Clinic: Encountered disruptions in diagnostic equipment and electronic health records (EHR) systems.
  • Mount Sinai Health System: Faced issues with their patient management systems.
  • Scheper Hospital in Emmen, Netherlands: The hospital’s emergency department closed because of the outage.
  • Massachusetts General Hospital: “All previously scheduled non-urgent surgeries, procedures, and medical visits” were canceled Friday. Its emergency rooms remained open.
  • Britain’s National Health Service: Outage was causing issues with its patient record systems at general practitioner offices.
• Pharma
  • Biogen: About 5-10% of individual systems, but no business continuity concerns.
  • Amgen: Confirmed they were assessing impacted systems.
  • GSK British Drug Maker
  • AstraZeneca British Drug Maker

Airlines and Travel Industry

• Airlines Using Windows-Based Systems: The airline industry was hit hard by the CrowdStrike outage. Delta Air Lines experienced over 600 flight cancellations and numerous delays. Other airlines also faced similar disruptions, causing significant inconvenience to passengers and operational challenges for the airlines. This incident highlights the criticality of IT systems in maintaining smooth airline operations. More than 13,000 flights experienced delays and disruptions in flight schedules.
  • Delta Airlines: Reported system crashes leading to delays in check-ins and flight schedules.
  • American Airlines: Encountered issues with ticketing and boarding systems.
  • United Airlines: Told Business Insider that it was holding all aircraft at their departure airports while it works to restore systems. “Flights already airborne are continuing to their destinations,” they added.
  • Frontier Airlines: Flight operations are currently being impacted by a major Microsoft technical outage
  • Spirit Airlines: Said, it was unable to rebook affected customers because of the outage.
  • Allegiant Airlines:
  • RyanAir: Advised passengers to arrive early as the outage caused “disruption across the network.”
  • KLM:  Said it had to “largely suspend operations” as the outage made “flight handling impossible.”
  • Others impacted: British Airways, Wizz Air, Turkish Airlines, Eurowings, Lufthansa, and Qantas were also among those who said they were affected.
• Airports
  • UK Airports Heathrow, Gatwick, and Luton have reported issues, with some warning of delays and disruption.
  • All US Airports

IT and Telecom Service Companies

• Companies Providing IT Services and Support: Companies providing IT services faced significant challenges during the CrowdStrike outage. The widespread issues with Windows devices hampered their ability to deliver services to clients effectively. This incident underscores the reliance of IT service companies on stable and secure software infrastructure.
  • Amazon Web Services (AWS): AWS collaborated closely with Microsoft to share information and accelerate fixes, suggesting AWS infrastructure was also affected indirectly by the outage (Press Center).
  • Google Cloud Platform (GCP): Similar to AWS, Google Cloud worked with Microsoft to address the impacts seen across the industry, indicating a level of disruption within its services (NZ Herald).
  • Accenture: Reported problems with internal systems affecting client support operations.
  • IBM: Encountered system crashes impacting cloud services and client support.
  • Deloitte: Experienced disruptions in IT consultancy and support services.
  • Microsoft: According to Microsoft’s service health status page, numerous Azure services reported issues during the period of the CrowdStrike IT outage. Key services affected included Azure AI Search, Azure Kubernetes Service (AKS), and Azure SQL Database among others. Microsoft said it has been actively working to restore full functionality and ensure stability across all affected services
• Telecom
  • Telstra in Australia

Retail

• Large Retail Chains: Retail chains experienced disruptions in customer services due to the CrowdStrike outage. Point-of-sale systems and online shopping platforms were affected, leading to potential revenue losses and customer dissatisfaction. This impact highlights the need for resilient IT systems in the retail sector.
  • Walmart: Reported issues with checkout systems and inventory management.
  • Target: Encountered problems with online orders and in-store POS systems.
  • Best Buy: Faced system crashes affecting sales and customer service operations.
  • McDonalds Japan: Suspended about 1/3 of its stores with issues their cash registers.
  • Wollworths Grocery Store: That some stores had “been impacted as a result of the global IT issue.” All but six stores were open for business, but some had fewer functioning checkouts.
  • Home Depot
  • Macy’s
  • Starbucks

Manufacturing

• Manufacturing Plants and Facilities: Manufacturing plants faced delayed production schedules due to the outage. The reliance on Windows-based systems for operational processes meant that any disruption could significantly impact production timelines and efficiency. This incident underscores the importance of robust IT infrastructure in the manufacturing industry.
  • General Motors: Reported production delays due to system crashes in their manufacturing process.
  • Ford: Encountered issues with their supply chain management systems.
  • Boeing: Faced disruptions in their production and inventory systems.

Education

• Educational Institutions: Educational institutions also felt the impact of the CrowdStrike outage. Administrative and educational activities were disrupted, affecting both staff and students. This highlights the critical role of reliable IT systems in the smooth functioning of educational institutions.
  • Harvard University: Reported issues with their online learning platforms and administrative systems.
  • Stanford University: Encountered system crashes affecting their student management systems.
  • University of California, Berkeley: Faced disruptions in online classes and administrative operations.

Public Safety

• 911 Alaska State Troopers
• NYCT Subway
• Washington DC Metro
• State Drivers Services:
  • Georgia Department of Driver Services
  • Tennessee Department of Safety
  • North Carolina’s Department of Motor Vehicles

Hotels

• Ocean Park Marriott in Hong Kong, staff were using pen and paper to check guests in and said the outage was affecting their systems globally.
• Marriott International later told CNN in a statement that “certain hotel systems” have been affected
• Seattle Children’s Hospital: Closed its outpatient clinics Friday.
• Memorial Sloan Kettering Cancer Center: Said it was pausing the start of procedures with anesthesia.
• Brigham and Women’s Hospital
• Duke Health

Media

• SkyNews: Unable to broadcast live TV.
• Canadian Broadcasting Corp (CBC) News
• Scripps News

Logistics

• FedEx
• UPS

Summary of Businesses and Organizations Impacted from CrowdStrike Kernel Update Outage

The CrowdStrike kernel level failure caused a massive IT outage, which has affected 8.5 million Windows devices, underscores the significant risks associated with relying on cybersecurity solutions that require frequent updates. Financial institutions experienced operational disruptions, with banks facing increased phishing attempts. Healthcare providers, including hospitals and healthcare facilities, struggled to maintain patient care due to compromised systems. Airlines, notably Delta Air Lines, saw thousands of flights grounded and delays, impacting both operations and customer service. IT service companies faced widespread issues, affecting their ability to deliver services to clients.

Retail chains experienced disrupted customer services, while manufacturing plants faced delayed production schedules. Educational institutions also encountered challenges, with the outage affecting administrative and educational activities. The societal impacts reflect the use of CrowdStrike by enterprises that run many critical services, emphasizing the need for robust and stable cybersecurity solutions. This outage also saw major tech giants like Amazon Web Services and Google Cloud Platform working closely with Microsoft to accelerate a fix, highlighting the interconnected nature of modern IT infrastructure.

Source Links

1. Bank of America and Wells Fargo Issues
2. Mayo Clinic and Cleveland Clinic Reports
3. Delta and American Airlines Disruptions
4. Accenture and IBM Problems
5. Walmart and Target Retail Impact
6. General Motors and Ford Manufacturing Issues
7. Harvard and Stanford University Disruptions

This detailed analysis and running list provide a comprehensive overview of the industries and specific companies affected by the CrowdStrike IT outage, establishing an authoritative resource for future reference. If we are missing anything please send us a note.

Criticality Scale

Given the potential for widespread operational disruptions, system instability, and significant downtime, the criticality score for such kernel-level vulnerabilities is high. Businesses relying on these traditional solutions face considerable risks, emphasizing the need for more reliable and less intrusive security approaches. As frequent kernel updates often require system reboots, which are disruptive in business environments as we have seen with this most recent CrowdStrike kernel level failure example.

Criticality Score: 9/10

• Impact on Operations: Severe, causing widespread disruptions across multiple sectors.
• Duration of Downtime: Prolonged, with ongoing efforts to fully resolve the issues.
• Reputational Damage: High, affecting customer trust and operational credibility.
• Financial Loss: Significant, with substantial costs incurred due to downtime and system failures.
• Regulatory Implications: Potential for increased scrutiny and regulatory actions in the affected industries.

Why It Matters

Frequent kernel driver updates, necessary for traditional cybersecurity solutions to stay ahead of new threats, pose significant risks. These include system crashes, downtime, and productivity loss, which can severely impact business operations. The CrowdStrike kernel level failure is a real-world example of these risks materializing, affecting numerous businesses and highlighting the urgent need for a more stable and secure approach.

Company Response

CrowdStrike acknowledged the impact to Microsoft outage and has been working none stop to address the problem that started in on Friday, 18 Jul 24. They are not allow in having similar issues, other vendors like Sophos, Cisco, Symantec, Microsoft, Check Point, Trellix, and VMware have also faced similar challenges and have issued updates and workarounds to mitigate the impact. However, these responses often involve reactive measures rather than addressing the root cause of the problem.

What Can You Do Now

To avoid the pitfalls of frequent kernel updates and the associated risks: