This is Your Cloud Provider – What is Your Plan?
This is an actual event, not a drill.
SDIS 67 / Laurent Schoenferber
SDIX 67 / Laurent Schoenferber
The date is March 10,2021. The time is 12:40 AM. The location is an OVH data center on the west bank of the Rhine, where France borders Germany.
The data center, a small one by today’s standards, it housed 30,000 servers and it burned to the ground. It hosted 3.6 million websites for 464,000 domains.
This is one of four data center buildings on the site. One of the buildings housed the fiber connections that connected the site to the Internet. While that was not destroyed it was seriously damaged.
Firefighters arrived 19 minutes after the alarm came in. The fire was so intense due to the racks and racks of servers that the French fire brigade called in a firefighting ship that could fight the fire from the Rhine.
At 1:13 AM firefighters got the electric utility to cut power to the entire site. Given the batteries and generators, that didn’t help. The site was still electrified, making firefighting pretty exciting.
Six hours later the fire was declared under control. It burned for days after that.
Still, OVH personnel were not allowed onto the site. Six days after the fire OVH handed over electrical equipment (which is where the fire is believed to have started) to the police.
The other three buildings, while either undamaged or slightly damaged had no way to get to the Internet.
Customers thought OVH was handling things for them but the CEO held a press conference and advised customers to implement their disaster recovery plan. Hundreds threatened to sue. Two customers tried.
So why am I bringing up this situation?
Because our customers are becoming more reliant on the cloud day by day. This “disaster” is a real possibility. While neither Amazon nor Microsoft have had a fire of this scale, just in the last five years all have had data center fires. Each one represents a risk to consider. Your business continuity plan needs to consider a scenario like this.
Just this past week we saw a fiber cut – what is affectionately called “backhoe fade”, when a vendor dug through a fiber bundle. It got fixed but it took many hours to figure out how to route around the failed fiber.
If you don’t already have a shared responsibility matrix document with each of your business critical vendors, now would be a good time to do that. The customers of OVH thought on that early March morning that their vendor would take care of them. Not so. Even Microsoft and Amazon say that backing up your data is your responsibility.
If you need help sorting through this, please contact us. Credit: Data Center Dynamics
by 