Crowdstrike BSOD

[vbwins]

Beware. An update to this antivirus package has taken out most windows devices that use it globally. Systems get stuck in a reboot recovery loop

[Arnoutdv]

It seems to be caused by csagent.sys
Workaround Steps:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.

https://www.reddit.com/r/crowdstrike...strike_update/

Big news and impact in Australia:
https://www.abc.net.au/news/2024-07-...alia/104119960

[yereverluvinuncleber]

I remember Clamwin noting a part of Windows as being malware, which it immediately quarantined, meaning that Windows would no longer function. As we all know Norton is virtually Malware. I just uninstalled Avast as it was interfering with Windows or VB6 almost continuously. A badly operating or poorly configured A/v tool is equal to a good bit of malware any day for fecking up our systems totally.

I worked on fault tolerant mini systems for decades and when we ended virtualising them we could not keep the PC people away, trying to install enterprise management systems and anti-virus tools. Could not get them to understand that their method of 'PC' management was the risk. Always an argument trying to get them to uninstall a/v software and unplug the ethernet and physically block/or lock the USBs. A machine you cannot access is safe. No need for av tools.

P.S Currently, I cannot access my bank account with Natwest nor pay for an appointment with my healthcare provider this afternoon as both their systems are down. I may have to pay in cash and now hoping that the hole-in-the-wall is going to be operating!

[vbwins]

The fix to this involves hoping that you can keep a system up long enough to download a "fixed" version of Falcon or failing that boot in safe mode, delete a file and reboot.

Try doing that with a thousand failed VM's in AWS or Azure that are stuck in a reboot repair cycle.

How Kernel mode code this bad made it into the world I do not understand. Perhaps Crowdstrike dont test their AV internally on their own platforms.

[2kaud]

No doubt social media will be awash with various conspiracy theories....

But was it incompetence or deliberate? Either way, it was very effective in crippling much of essential IT world-wide.

[couttsj]

Got an alert from CISA this morning about a problem with a CloudStrike update. Not knowing what CloudStrike does required a little research. CloudStrike itself had a pretty good explanation:
--------------------------------
Falcon is the CrowdStrike platform purpose-built to stop breaches via a unified set of cloud-delivered technologies that prevent all types of attacks — including malware and much more. Today’s sophisticated attackers are going “beyond malware” to breach organizations, increasingly relying on exploits, zero days, and hard-to-detect methods such as credential theft and tools that are already part of the victim’s environment or operating system, such as PowerShell. CrowdStrike Falcon® responds to those challenges with a powerful yet lightweight solution that unifies next-generation antivirus (NGAV), endpoint detection and response (EDR), cyber threat intelligence,managed threat hunting capabilities and security hygiene — all contained in a tiny, single, lightweight sensor that is cloud-managed and delivered.
--------------------------------
This particular update only affected Windows based hosts (servers) in large organizations that utilize Falcon. Consequently it was not as widespread as the media would like to suggest, as Microsoft has a relatively small portion of the server market. None the less, certain areas of the global infrastructure may have been hit harder than others.

J.A. Coutts

[techgnome]

Only non essential things... like airports, banks, communication companies, hospitals... It looked like it was pretty widespread through Europe and the UK... And I know a few people that were affected... they're having a bad date with their remote systems... They can downplay it all they want, but it's still a pretty big miss and deal on their end.

I don't care if it was Kevin's first day... should be his last... and who approved his PR? hmmmmm?


-tg

[jdc2000]

This is yet another example of what happens when the only testing done on software updates is "It works on my box!".

[OptionBase1]

"It works on my box!".

That's what she said.

[NTShpikho]

I am proficient in several OS. BSOD doesn't effect me.

[Steve R Jones]

The NEWS people talked about how alarms started going off around the world.....One country after another.... They just knew we'd be next...

On a similar note- News reported that if just a few Electrical Power Station were to blow up - the whole country could be with out power for at least 18 months.

[2kaud]

Initially I thought someone with a warped sense of humour had got the MS BSOD screensaver installed as part of an update as a joke as I couldn't believe that a company like Crowdstrike could actually issue such faulty code as to cause a BSOD for real......

https://learn.microsoft.com/en-us/sy...ads/bluescreen

[2kaud]

Apparently Linux machines were also effected back in April:
https://www.computing.co.uk/news/433...-outages-april

[2kaud]

Having now looked more closely into the cause, IMO MS must bear some responsibility. The .sys files used by Crowdstrike are effectively 'pcode' that is used by the Crowstrike kernel driver. The driver has passed MS tests and is signed but the .sys files are not. Hence you can have signed kernal code that 'runs' unsigned code. The .sys file that caused the problem consisted all of 0's (incompetence or deliberate??). The driver uses a part of the .sys file as a base access address which causes illegal memory access in the kernel (as the read address is 0) and hence the BSOD.

Why was a driver passed and signed by MS that didn't do such a basic check of an address read from a .sys file not being valid? If the driver had had code that did this basic check then the BSOD would never have happened even if the .sys file was invalid. Black mark MS!

[PlausiblyDamp]

I could be wrong on this but driver signing only verifies the source of the driver, not the quality. A driver signed by Crowdstrike's certificate proves that Crowdstrike made the driver, but nothing else.

If the driver was part of something like WHQL testing then MS would have been involved in issuing a WHQL certificate for a particular driver version, however this would involve a turnaround due to how you run the tests, submit the logs, etc. Often time is of the essence with Day 0 vulnerabilities and tools like Crowdstrike are pushing updates at short notice to combat new exploits etc.

As far as I can tell this is entirely down to the failure of Crowdstrike to properly test a release, I doubt MS were involved in testing this update at all.

[2kaud]

I understand that the driver was tested by MS and a WHQL certificate issued. What Crowdstrike issue regularly are not updated drivers but the .sys files used by the main driver. These .sys files are not a driver, are only used by the main Crowdstrike driver and are not tested/signed by MS.

See https://www.youtube.com/watch?v=wAzEJxOo1ts
starting about 13:00

(the first part is mainly about what is a driver, kernel etc)

[2kaud]

MS are getting in their excuses early. They're now blaming EU red tape for allowing the update to cause the IT meltdown....

[jdc2000]

Here is an update on the issue with some of CrowdStrike's analysis:

https://www.zdnet.com/article/what-c...as-the-answer/

Apparently, their automatic "content validator" passed the update even though it had issues. Of course, they did not actually test the update by installing it on an actual computer running Windows and then reboot it to see what happens, and they probably will not add that test into their future validation procedures either.