Cloudflare Bug That Leaked Sensitive User Data From Various Websites and Apps Now Fixed

Friday February 24, 2017 8:43 AM PST by Mitchel Broussard

Content delivery network Cloudflare has confirmed the existence of a bug that caused search engines to cache sensitive user data from a variety of well-known apps and websites. Google researcher Tavis Ormandy discovered and reported the bug to Cloudflare, and the company has since fixed the bug and published a detailed blog post about exactly what happened.

According to Cloudflare, the period of greatest impact for the "parser bug" ran from February 13 to February 18, although the extent of the leak stretches back months. The heart of the issue was a security problem with Cloudflare edge servers, which were returning corrupted web pages by some HTTP requests running on Cloudflare's large network.


In what the company referred to as "some unusual circumstances," occasionally private information was returned as well, including "HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data."

It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.

The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence.

As shared in a tweet by Ormandy this week, that data also included private dating site messages from OKCupid, full messages from a "well-known chat service," passwords from password managing apps like 1Password, and more (via Fortune). In response, some companies -- like 1Password -- have published blog posts confirming that "no 1Password data is put at any risk through the bug reported about CloudFlare."

To expedite a solution, Cloudflare responded to Ormandy's discovery and turned off three minor features of the network -- email obfuscation, Server-side Excludes, and Automatic HTTPS Rewrites -- discovered to be using the same HTML parser chain "that was causing the leakage."

In its blog post, the company said that it has "not discovered any evidence of malicious exploits" in relation to the time that the parser bug was active. It also noted that, while serious, the scale of the bug was still relatively low: around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulted in memory leakage. "That’s about 0.00003% of requests," the company noted.

Cloudflare worked with the affected search engines, including Google, Yahoo, and Bing, to erase any remnants of the sensitive data from their caches. The company's chief technology officer, John Graham-Cumming, concluded the blog saying, "We are very grateful to our colleagues at Google for contacting us about the problem and working closely with us through its resolution. All of which occurred without any reports that outside parties had identified the issue or exploited it."

Earlier this week, it was reported that Apple cut ties with server supplier Super Micro Computer in order to avoid a potential future scenario where user data might be put at risk, similar to Cloudflare's leak. Early in 2016, Apple was said to have discovered a potential security vulnerability in one of Super Micro Computer's data center servers and effectively ended its business relationship with the network company shortly thereafter.

For a technical dive into Cloudflare's parser bug and its origins, check out the company's blog post.

Top Rated Comments

(View all)
Avatar
41 months ago
If you use 1Password you were never at risk anyway ('https://blog.agilebits.com/2017/02/23/three-layers-of-encryption-keeps-you-safe-when-ssltls-fails/').
Score: 5 Votes (Like | Disagree)
Avatar
41 months ago

If you use 1Password you were never at risk anyway ('https://blog.agilebits.com/2017/02/23/three-layers-of-encryption-keeps-you-safe-when-ssltls-fails/').

Note that this means your master password is safe if you happen to use a 1Password account. This doesn't mean that there is any less risk for other data (including logins and other sensitive data).
Score: 1 Votes (Like | Disagree)
Avatar
41 months ago

I get how 1Password can say 'not affected' but I don't get how some of the others can. I have domains and DNS at Namecheap, and read their page about their investigation. I don't understand how they can say not affected, though. Unless I'm misunderstanding what happened, how would they even know (or be able to investigate)? No one logged into their site during the affected time periods?

Anyway, yea, using a password manager is a very good idea, as you can have a good, strong UNIQUE password for every site (i.e.: if one gets compromised, it's only that site). But, changing them can still be a pain for things like Dropbox, email, etc. where the change impacts all your systems and devices.

Sorry for the delay, a client popped in with Scotch - here in the office we couldn't say no to either one… ;)

I'd been poring over my own resources about getting to the bottom of this as well, and cruising the web for a more-narrowed or focused explanation and found one on Wired with some quotes from Cloudflare's CEO that broke it down for me, the whole post is a good read and the CEO's comments begin about halfway down starting with the "What Happens Now" header:
https://www.wired.com/2017/02/crazy-cloudflare-bug-jeopardized-millions-sites/

Keep in mind that the CEO cites a number of affected customers and not a number of affected web sites or portals… Cheers!
Score: 1 Votes (Like | Disagree)

Top Stories

Apple Acquires Weather App Dark Sky

Tuesday March 31, 2020 10:22 am PDT by Juli Clover
Apple has acquired weather app Dark Sky, Dark Sky's developers announced today. Dark Sky is one of the most popular weather apps on the App Store, known for its accuracy and storm warnings. Our goal has always been to provide the world with the best weather information possible, to help as many people as we can stay dry and safe, and to do so in a way that respects your privacy. There is no ...

Apple's 2020 MacBook Air vs. 2020 iPad Pro

Wednesday April 1, 2020 2:45 pm PDT by Juli Clover
Apple in March updated both the MacBook Air and the iPad Pro, and with the iPad Pro increasingly positioned as a computer replacement, we thought we'd compare both new machines to see how they measure up and which one might be a better buy depending on user needs. Subscribe to the MacRumors YouTube channel for more videos. We're comparing the base model 12.9-inch iPad Pro and the base model...

Zoom Accused of Misleading Users With 'End-to-End Encryption' Claims Amid Other Security Issues [Updated]

Wednesday April 1, 2020 2:47 am PDT by Tim Hardwick
Zoom is facing fresh scrutiny today following a report that the videoconferencing app's encryption claims are misleading. Zoom states on its website and in its security white paper that the app supports end-to-end encryption, a term that refers to a way of protecting user content so that the company has no access to it whatsoever. However, an investigation by The Intercept reveals that...

Case for Upcoming Low-Cost iPhone Shows Up at Best Buy With Alleged April 5 Stock Date

Monday March 30, 2020 4:25 pm PDT by Juli Clover
Apple has a new low-cost iPhone in the works, which is supposed to be launching sometime in the first half of 2020. Given the ongoing situation in the United States and other countries, it's been unclear if the device is going to launch within the planned timeline, but there are signs that it could be coming soon. We started seeing cases for the new low-cost iPhone back in early February,...

Apple Adding Some 2013 and 2014 MacBook Air and MacBook Pro Models to Vintage Products List at End of April

Wednesday April 1, 2020 2:24 pm PDT by Joe Rossignol
In an internal memo obtained by MacRumors, Apple has indicated that the following 2013 and 2014 models of the MacBook Air and MacBook Pro will be added to its vintage and obsolete products list on April 30:MacBook Air (11-inch, Mid 2013) MacBook Air (13-inch, Mid 2013) MacBook Air (11-inch, Early 2014) MacBook Air (13-inch, Early 2014) MacBook Pro (13-inch, Mid 2014)Apple defines vintage...

AirTags Referenced in New Apple Support Video

Thursday April 2, 2020 12:12 pm PDT by Joe Rossignol
Apple has accidentally referenced its widely rumored AirTags item tracking tags in a video that it uploaded to its Apple Support channel on YouTube today. The video was first spotted by the blog Appleosophy and has quickly been removed. The video was titled "How to erase your iPhone." AirTags were mentioned in Settings > Apple ID > Find My > Find My iPhone under Enable Offline Finding, with...

Testing Brydge's New Pro+ Keyboard With Trackpad for iPad Pro

Monday March 30, 2020 2:04 pm PDT by Juli Clover
Well ahead of when Apple introduced trackpad support in iOS 13.4, Brydge announced an iPad Pro keyboard with a built-in multi-touch trackpad. We have one of Brydge's new Pro+ keyboards on hand, and thought we'd check it out to see how it works with Apple's new 2020 iPad Pro models. Subscribe to the MacRumors YouTube channel for more videos. The Brydge Pro+ keyboard is similar in design to...

Intel Unveils 10th-Gen Processors Suitable for Next 16-Inch MacBook Pro With Wi-Fi 6 and Turbo Boost Speeds Above 5GHz

Thursday April 2, 2020 7:53 am PDT by Joe Rossignol
Intel today announced the launch of its latest 10th-generation Core processors for high-end notebooks, potentially including the next 16-inch MacBook Pro. The batch of 45W chips, part of the Comet Lake family, are built on Intel's 14nm++ architecture. The new H-series chips have the same base clock speeds as the 9th-generation chips in the current 16-inch MacBook Pro, but Turbo Boost speeds...

Apple's Work on New Upcoming Products Progressing Normally as Employees Adjust to Telecommuting

Monday March 30, 2020 11:58 am PDT by Juli Clover
Apple's development of upcoming products is progressing as usual despite the fact that Apple employees around the world are working from home, according to a new report today out from Bloomberg. Apple is still working on new versions of the HomePod, Apple TV, MacBook Pro, budget iPads, Apple Watch, iPhone, and iMac, all of which could be released "as early as later this year" and have been...

2020 iPad Pro May Not Have a U1 Ultra Wideband Chip After All

Wednesday April 1, 2020 8:49 pm PDT by Joe Rossignol
While it was previously reported that all 2020 iPad Pro models feature the same Apple-designed U1 chip as the iPhone 11 lineup, enabling Ultra Wideband support, we have compiled evidence to suggest that this may not be the case. As a reminder, Apple's tech specs for the iPhone 11 and iPhone 11 Pro list an Ultra Wideband chip for spatial awareness, but the chip is not mentioned in Apple's...