Cloudflare Exposes Customer Secrets
Cloudflare, the company that helps web sites perform when under stress, including when under denial of service attacks, was the victim of a self induced cyber breach. For those who are not familiar with Cloudflare, it acts as a front end to a customer company’s web servers. With Cloudflare in front of a company’s servers, the servers can stand up to incredible loads and massive denial of service attacks.
What is more amazing is how they handled it.
First a little bit of the story.
The bug likely exposed data between September of last year and this month.
Cloudflare modifies web pages that pass through its servers as part of the process that it uses. To do that, they created some software that parses web pages and makes the needed changes.
Tavis Ormandy, a security researcher that works for Google, discovered a bug that caused the Cloudflare servers to send unintended data out with the modified web page. Among the data that was exposed included authentication tokens, cookies, encryption keys and text of the whole packet. To make matters worse, the data that was exposed might be from any of it’s customers, not just the web site that the user was visiting.
In addition to that, some of the data was cached by Google. While they didn’t say, it is likely that Google web page crawlers were probably among the “users” that visited Cloudflare cached web sites.
Now the good part.
Once Tavis figured out what was going on, it was a Friday night and he knew that he needed to act fast. An email to the help desk wasn’t going to cut it.
So it put out an emergency plea on his Twitter page. Given who Tavis is, a LOT of people follow his Twitter feed. The plea said that he needed to talk to someone on Cloudflare’s security team NOW!.
Again, given who Tavis is, Twitter did it’s Twitter thing and Cloudflare security reached out to Tavis quickly. He explained the problem to them and within 47 minutes they had deployed a fix that mitigated the problem, but did not completely fix it.
Because of Cloudflare’s size, they were able to quickly create a cross functional team in San Francisco and another in London to work on the problem. Working 12 hour shifts, they handed off the work internationally 24 hours a day until they were convinced they had all of the leaked data under control.
Within 7 hours they had a complete fix in place but it took several days to work with Google to delete all of the cached data off Google’s servers. Working 24×7 with Google they now feel that all of the leaked data has been purged, so they were able to notify customers of the situation.
I already received one email from a web site hosted behind Cloudflare telling me that I should change my password. They said that we should expect many more notices given that Cloudflare protects millions of web sites.
Obviously, this was a pretty subtle bug but what was amazing was that within 47 minutes they were able to deploy the initial mitigating changes and within 7 hours they were able to deploy a complete set of fixes. Right now, by comparison, the same team that Tavis works for, Google’s Project Zero, just disclosed a Microsoft bug because Microsoft was not able to even release a fix, never mind get it deployed, in 90 days. 7 hours vs 90 days+ is the power of the cloud. One platform; total control over the environment. That is an amazing benefit of cloud based services.
While there is nothing for you to do regarding this breach, watch out for notices that tell you to change your password. Unless you want to suffer the same fate that the DNC did last during the election cycle last year, DO NOT click on any link in those emails – Go to the appropriate website yourself, log in and navigate to the password change page to change your password yourself.
Pretty amazing story.
Information for this post came from Ars Technica and the Cloudflare Blog.
by 