Major Software & Hardware Vendors Cause Self-Inflicted Downtime

Let’s Encrypt is the free HTTPS encryption service that is used by millions of web sites. Since it started out as a good idea of two Mozilla employees in 2012, it has issued about 2 billion free TLS certificates.

The history behind this organization is long and convoluted. The industry has a high bar for entery for a new player and in 2012, they had to get someone that the industry trusted to, kind of, co-sign their HTTPS certificates.

They knew that co-sign process was a short term solution and about 4 years ago they convinced the “Internet authorities” that they were the real deal and replaced that co-signed certificate with a new one.

Browsers and other software vendors have been incorporating this new software since 2017.

Let’s Encrypt, itself, has been warning people for about a year that the old certificate was going to expire today and software vendors needed to upgrade.

We expected that old, unsupported software like Windows XP and old hardware like Android phones running Android 7, would have a problem today.

That turned out to be true.

What we did not expect is that mainstream websites like Shopify, mainstream tech vendors like Palo Alto and Cisco and mainstream service vendors like Monday.com, Google Cloud monitoring and Quickbooks would be caught, napping or completely asleep at the switch.

Unfortunately, we were wrong.

These vendors and many others went dark about about 8 AM Mountain Time this morning.

Some of them fixed the issue. Shopify, for example, recovered at about 3:30 PM.

Others, like Fortinet, seem to continue to be asleep at the switch and have told their customers to turn off the security feature that warns you when you have a security issue. That is not a great solution, but for some Fortinet customers, that is their only option.

Many more likely have not been detected yet – like IoT devices that just stopped working but that no one has either noticed or figure out why.

And, importantly, if these software or hardware products are no longer supported, you are probably out of luck and will have to replace it.

In some cases, you have the ability to tell the system to ignore the error and move forward, but most of the time, that is not an option.

I am writing this because, I think, this is day one of an extended discovery process. Likely there are things that are down and people don’t know they are down or don’t know why they are down. This will take a while to discover and to fix. In some cases, the fix will be expensive and extended.

I wrote about this a few months ago. This should not have happened as the industry knew exactly what day it was going to be a problem 9 years ago. Still we, as an industry, create self inflicted wounds.

For more details, check out this article at ZDNet.

by