The CrowdStrike Meltdown – The Rest of the Story
CrowdStrike is a major vendor of user workstation security software used by many Fortune 500 companies and government agencies. On Friday, they caused a major meltdown of Windows workstations across the globe.
CrowdStrike insists that this was simply a software update gone wrong and maybe that is true. But there are plenty of conspiracy theorists who think this was a cyberattack. The tin foil hat crowd thinks it was intentional on the part of the deep state.
Truth is that we may never know but in today’s world it is really hard to keep secrets, so if it was anything other than an error, we will eventually find out.
CrowdStrike had a similar problem in April affecting Linux machines, so this is not their first dumpster fire rodeo.
Microsoft says that the failure affected 8.5 million computers – THAT THEY KNOW OF – and because these computers are “dead” you cannot remotely fix them. It requires someone to touch each and every one of them. That will take a while.
Then there is the domino effect.
Airlines were among the biggest organizations affected. With multiple computers at every gate, every workstation, every airport, every ticket office, touching them manually is a gigantic task. Companies like Kaseya and Microsoft are offering to help folks who have computers that have become doorstops, but even so, that is a major task.
Then we have the follow on effects. Remember Southwest Airlines meltdown around Christmas 2022. That was blamed out an outdated, creaky crew scheduling system. Well, they fixed that (and they are not a CrowdStrike user), but this time the problem belonged to Delta. They cancelled over 3,000 flights over the weekend and more than 900 today. Airline passengers spent 10 to 20 hours trying to reach an airline representative. Since airlines are running at capacity. there really are very few options for the hundreds of thousands of passengers affected. Not to mention which, if you were trying on Friday to get to a Saturday wedding, offering you a flight on Wednesday is not much help. Especially, if it is your wedding :).
And airlines aren’t alone. Metropolitan Transit Authorities in many cities were also down.
Around the world, stock exchanges shut down and even TV networks went off the air. Even some border crossings shut down.
Even today over 1,000 flights were cancelled and 9,000 flights delayed. Delta, who seems to be the most significant meltdown, is offering Skymiles and vouchers. Federal regulations require them to offer cash – which they would really like not to have to do. Passengers are talking about suing Delta which they could probably do in small claims court. Delta says that they don’t plan to compensate passengers for lost income.
Of course hackers don’t want to miss out on an opportunity. Enter scams for fixes. One loads a remote control app if you install the fix. The scams involve fake emails, fake support calls and fraudulent offers of fixes.
How did this happen? In part you can thank our friends on the other side of the pond. The EU forced Microsoft to open certain APIs to make things fair for competitors. Those APIs, which grant access to the Windows kernel or soft underbelly of the operating system, if misused could and did cause systems to melt down.
Apple wasn’t affected by all of this because somehow they have escaped the scrutiny of authorities that required Microsoft to expose parts of the system that should never have been exposed. That decision, which dates back to 2009 is likely irreversible now but the EU definitely shares in the blame. Part of the problem is the need for speed. These APIs dig into the operating system at a really vulnerable level. The alternative is to move these APIs out of the kernel and lose 20 percent or more performance – which would not make anyone happy.
Finally, Microsoft has released a “recovery tool” for IT administrators. Still not a simple task for those super overworked IT folks, but at least it helps a little bit.
It may be weeks before everyone is back to normal.
One thing that this points out is that companies need to think about whether they need to rethink their business continuity strategy – what they will do if the very unexpected happens. Clearly, a lot of people did not plan for something like this.
If you were affected and need some assistance planning a smoother path for the next time (and there will be a next time), please contact us. If you were not affected but are worried that next time you might be, still contact us.
Credits:
by 