On July 19, consumers worldwide experienced firsthand what it means when a critical software-defined product fails. CrowdStrike, a giant security vendor, experienced a major incident that caused a global IT outage that affected Windows computers worldwide running on Microsoft’s Azure cloud platform. From airports and train stations to car manufacturing facilities, logistic centers, banks, hospitals, and media broadcasters, everything was disrupted or brought to a complete halt.
According to Microsoft, around 8.5 million devices were affected, representing less than 1% of the total number of Windows devices worldwide. The incident was likely caused due to a faulty signature of a software package, part of a routine configuration update, that wasn’t tested properly before being deployed in production.
Since the outage, CrowdStrike has opened a Remediation and Guidance Hub with numerous messages, FAQs, videos, and a detailed workaround that could help affected systems usually function until a permanent solution is found. One option is for users to “reboot the [affected computer] to allow it to download the reverted channel file,” referring to the fixed file.
**
Why didn’t the Falcon Sensor Update get Tested Properly?
CrowdStrike released a sensor configuration update to Windows systems, which caused the Outage. These configuration updates are an ongoing part of the protection mechanisms of the Falcon platform and occur several times a day, which is part of the architecture of the Falcon Sensor. Any company developing software products must integrate automated security testing tools into their CI/CD pipeline and have a dedicated environment for sandboxing, to continuously monitor and evaluate external code.
Integrating security tests into the CI/CD process for external vendor code is essential for mitigating security risks, ensuring compliance, maintaining code quality, preventing supply chain attacks, and enhancing visibility. By rigorously testing third-party code, organizations can protect themselves from potential vulnerabilities and threats, ensuring a secure and reliable software development lifecycle.
**
Leveraging DevSecOps Tools for Supply Chain Security Management
While CrowdStrike works to analyze the root cause of the issue, it’s possible that DevSecOps practices such as “shift left”, intelligent testing, and validation could have helped mitigate the recent CrowdStrike BSOD (Blue Screen of Death) issue, along with supply chain security management practices, which weren’t fully incorporated here. Many software companies use 3rd-party code developed by an external vendor and integrated into the product. That code can contain various elements, such as Open-Source Software (OSS), SDKs, Plugins, APIs and Cloud Services, COTS Software, Libraries, Firmware, and more.
Managing the security of third-party software code involves ensuring that these components are free from vulnerabilities, compliant with security policies, and continuously monitored and updated to mitigate risks. This is a critical aspect of supply chain security management, as vulnerabilities in third-party code can expose the entire system to potential threats.
**
How to Reduce the Risk of a CrowdStrike-like Incident on Our EV Charging Infrastructure
By deploying DevSecOps processes and tools, such as EVSec Platform, companies can manage product security activities internally and with suppliers, resulting in more efficient development and operations throughout the product lifecycle. By leveraging these tools and processes, we can reduce the risk of a CrowdStrike-like incident on our charging infrastructure:
Continuous Integration and Continuous Deployment (CI/CD)
Implementing CI/CD pipelines ensures that code changes are automatically tested and integrated, even on ‘low-level’ software packages, reducing the likelihood of bugs and compatibility issues that can lead to critical failures.
Sharing and Collaboration with Centralized Catalogs
The centralized catalog approach allows multiple stakeholders to maintain a unified level of product security assets. It allows internal and external teams to contribute to the security work, dynamically updating the Cyber Model in real time to calculate the new risk.
Automated Testing
Automated tests can run frequently and consistently in a simulated environment (‘shift left’), catching regressions and defects that might break in production, even when adding encryption keys and signatures to an existing software package.
Uncovering Edge Cases
Intelligent fuzzing can expose edge cases and unusual conditions before deploying them to production. By simulating various unexpected inputs, strings, and scenarios, deep and intelligent fuzzing can help identify potential issues that standard testing might overlook.
**
Schedule a demo to learn more about EVSec Product Security DevSecOps Platform and how it dynamically handles supply chain security management, out-of-the-box.
