What Can the Nashville Bombing Teach Us?

The terrible events in Nashville will take a long time to recover from. The bomb, likely intentionally, severely damaged an AT&T “wire center” near downtown, shutting off AT&T service for millions of customers. This included some 911 call centers, cell phone users, land line customers, Internet service and even cable TV.

Those 911 call centers that went down did not have the appropriate level of fault tolerance. For your business to be down for 3-5 days might be acceptable. Or it might not. For a 911 call center, that is just short of malfeasance.

While there seems to be minimal human injury, there is millions of dollars of damage and it will take months or more to repair.

In the meantime, many businesses went “off the air” when this happened. This included businesses in Tennessee, Kentucky, Alabama and other states.

Given the size of AT&T, it has the ability to call in resources from all over the country to help it recover.

What AT&T’s actions represent is an amazingly fine tuned disaster recovery plan. This includes driving in 25 cell sites on wheels that can be parked on the streets of Nashville to provide basic cell service.

Next up was bringing in 24 disaster recovery trailers – basically an AT&T facility on wheels from different parts of the country. Even for AT&T, that took 48 hours.

Then they had to pump out 3 feet of water from the basement and bring in portable generators to replace the generators that were damaged in the blast. In order to get the power into the building from the generators in the street, they had to drill holes through the walls. Given the building didn’t have walls, they had to figure out how heat it. And distribute power inside.

All this while the building is an active crime scene and where a fire reignited hours after the blast. For more information about AT&T’s efforts, check out this web page.

But most of you are not running an international telecommunications company, so how does this apply?

Given the scale of the damage and the resources AT&T was able to bring to bear, it still will be months or more before things are back to normal.

If your facility or your data center was damaged in some sort of natural or man-induced disaster, are you ready to mobilize the resources that you need to get your company back in operation in whatever time window is acceptable to you and your customers? Do you have the resources (AT&T had to bring in structural engineers just to figure out if the building was stable enough to let their technicians in there to begin assessing the damage)? Do you know what resources you would call on? Are the contracts in place? If you have insurance, do you know what your carrier will demand in order to make sure that you have coverage?

Your disaster recovery and business continuity program needs to address all of this and you need to have a team that is identified AND TRAINED. They need to go through mock exercises again and again.

AT&T was able to bring a large percentage of their customers back online within 72 hours. What they did was not pretty, but it works. Likely they will have to redo most of what they did in the first few days, but at least they are operating. Could you say the same?

by