Amazon Broke the Internet Today – Really

Sometimes there is too much news so planned items get moved down the list. Today is one of these days.

As I am sure most of you know, starting around 12:11 AM Monday Pacific Time, Amazon Web Services had a meltdown.

To paraphrase a quote, when Amazon sneezes, everyone gets sick. That is what happened today.

This includes Amazon retail, banks, government agencies, Fortune 500 companies. JUST Some name brands affected were:

• Adobe
• Apple
• Delta
• HBO
• Hulu
• Instacart
• Lyft
• McDonalds
• Reddit
• Signal
• Slack
• Wordle
• Zoom

And tens of thousands more.

As the day wore on, more and more cascading side effects appeared. You can follow the chronology of what happened at the Amazon health log. This will eventually roll off but if you need a copy of it, please contact me.

Ultimately, this cascading failure was because of a dependency on Amazon’s DNS service, which runs in US East 1, which Amazon itself uses, even if you do not. It appears to be a single point of failure, and has been in the past, but I am sure Amazon will work very diligently to attempt to mitigate that now.

But here is the problem – just an example.

Let’s say we have a client, call them Company A.

Company A uses them for certain services.

They use service provider B to fulfill those services.

Service provider B uses Amazon to run that service.

That is considered a fifth party effect.

Today there were millions of fifth party and more effects.

DoD measures affects down to, at least, 10th party effects.

So today, Amazon sneezed and everyone got sick.

This is probably tolerable if the service you are providing does not stop your company from working, does not cause you to be in breach of contract and does not have anyone die.

It is not so good if the converse is true.

Now here is the big problem.

I tried to get to a website this afternoon. It was not responding. Do they use Amazon? Does one of their providers use Amazon? How would I know.

How do you, as the average company, understand all of this?

And if you are delivering a mission critical service and you go offline due to a service provider three levels removed from you having a bad hair day, are you still liable?

Regarding the liability, that is a question for your lawyer. But the answer is that likely, unless you specifically have a get out of jail free card in your customer contract, the answer is probably yes.

You also should figure out whether your cyber insurance will cover your lost income. We can help figure that out.

And you should be doing vendor risk assessments. You want to ask them what their dependencies are and who is liable if they have a bad hair day because of one of their suppliers that you have no control of. We can help you with this too.

It is HIGHLY unlikely that this is the last time some cloud provider five levels removed will have a bad hair day. That means you may need to tune up your business continuity strategy, which we can assist with. Unless being down for most of the day is okay with your company and with your customers.

I am sure that there are companies who permanently lost customers today. They could not get to you so they moved on to the next provider who, coincidentally, does not use Amazon. They were able to do their work and maybe, they won’t come back to you.

Be prepared. Be prepared.

Credit: Tech Crunch

by