The $10 Million Alternative to Paying Ransomware

Earlier this year, the Erie County Medical Center in Buffalo, New York was hit with a ransomware attack.  ECMC is a level 1 trauma center, teaching hospital and regional center for a variety of medical services – including, unfortunately, ransomware.

At 2 A.M. on Palm Sunday computer screens across the medical center flashed “What happened to your files?” and thus began a saga which is still playing out.

In the end 6,000 PCs were affected and many were infected.  The hackers wanted 1.7 Bitcoin for a key to decrypt each PC or 24 Bitcoins to decrypt all of the computers.  At the time, that represents about $25,000 to $30,000.

By 3:30 AM they had shut down all computer systems as a precaution.

The next decision was whether to pay the ransom or not.  By 5:30 AM they had called in cyber security experts from the consulting firm of Grey Castle from nearby Troy, NY.  Their incident response plan was working.  As Grey Castle’s experts explained to the management team what happened, they were in shock.  Kind of like their patients sometimes.  And, like those patients, they were making life or death decisions about the hospital’s IT systems.

After considering their options, they decided not to pay the ransom for a variety of reasons – they had backups, they could use a regional health information network called HealtheLink to get records from up to the time of the attack and they didn’t really know if they could trust the outcome if they did pay the ransom.  Would the data be intact and could they even trust the hackers to deliver the keys?

The hospital borrowed laptops and placed them in the emergency room and ICU and created an ad hoc network to get access to HealtheLink.

In the mean time, the disaster plan came into effect.  The hospital went back to paper patient charts.  Many hospital staffers had never worked with paper charts in their lives so the road was a bit bumpy.

All in all the hospital’s disaster recovery plan worked.  From the initial attack on April 9 they marched forward.  By April 19 – 10 days later, they had wiped computers and started delivering rebuilt computers to some critical departments such as emergency and critical care.

By early May doctors could begin to upload progress notes.

By mid May doctors could enter electronic prescriptions again.

In addition to working with Grey Castle, the hospital engaged experts from Microsoft, Cisco, Symantec and Meditech (their electronic health records vendor).   They brought in IT staff from Catholic Health Services and other hospitals and staff worked on their days off.   This was truly a all hands on deck effort.

Amazingly, the emergency room never went on diversion, critical because they are a level 1 trauma center.  Diversion is a process where ambulances are sent to more distant and sometimes less qualified hospitals because the primary hospital cannot not accept new patients for some reason.

Six weeks after the attack they were close to back to normal.

There are lessons here; some of which the hospital had in place and others that they learned as a result.

ECMC says expenses tied to the event were nearly $10 million.

Half of that money was for new hardware, software and assistance.  The other half was for overtime pay and other expenses and reduced revenue.

In addition, the hospital predicts expenses going forward of $250,000 to $400,000 a month for employee education, system upgrades and hardening of systems.

So what are the lessons?

• Having a tested incident response plan allowed them to respond to the situation quickly and be able to not have to turn away customers (ambulances).
• A tested disaster recovery/business continuity plan allowed the hospital to operate minus all the hardware and software they were used to working with.
• The ability to get help from (competing) hospital systems in town gave them some much needed extra resources.  Whether by formal agreement (usually called mutual aid and commonly used by emergency services) or informally, having a plan to marshal outside resources can be very helpful.
• Practicing for emergencies is critically important and that is not a one time event.  Just like anything complex, it needs to be rehearsed over and over until it is automatic.
• A big part of their success can be attributed to their cyber insurance.  Just last year they made a decision to increase the insurance policy amount from $2 million to $10 million and while insurance never covers all costs, if it covers 75%, that allows the hospital to do what they need to do.  Insurance will never pay for those things that you should have done but didn’t, but it will pay for a lot of things – IF YOU HAVE THE RIGHT POLICY!

On the other side of the equation, however are lessons learned out of the incident.

• How come they did not detect the event more quickly?
• How come the ransomware was able to attack so many computers (HINT:  the network was not partitioned effectively)?
• The fact that they are having to spend that $250-$400k a month NOW is because they did not take cyber security seriously enough before.  Would you rather spend $25k a month now or $400k a month later?  Kicking the cyber security can down the road was an expensive lesson for ECMC.

The good news is that any business can learn from events like this.  Are you prepared- really prepared?

Information for this post came from the Buffalo News and another Buffalo News article.

by