Testing for "reverse" Heartbleed

While patching our systems for the recent Heartbleed vulnerability, we found that some sites (including huge web properties), which had patched their servers were still vulnerable to a variant of the attack that we're calling "reverse heartbleed." This is a post about how the vulnerability works, what we found with our public tester tool, and how you might be affected.

What's Heartbleed and "reverse" Heartbleed?

The Heartbleed vulnerability in OpenSSL allows a malicious TLS implementation to extract random chunks of memory from an unpatched peer. If you're not up to speed on Heartbleed, check out the excellent documentation on that site and check your servers ASAP to see if you might be vulnerable.

Most of the attention around the Heartbleed attack has focused on the simplest and most obvious scenario: a malicious client attacking an HTTPS server to steal cookies, private keys, and other secrets. But this isn't the only attack possible: a malicious server can also send bad heartbeat packets to a client that uses OpenSSL and extract data from that client. The TLS heartbeats used in this attack are symmetric: they can be initiated by either the "client" or the "server" in a TLS connection, and both endpoints use the same vulnerable parsing code.

What kinds of clients are vulnerable?

Anything that speaks TLS using OpenSSL is potentially vulnerable, but there are two main classes of client apps that are worth mentioning:

1) Traditional clients are things like web browsers, apps that use HTTP APIs (everything from Dropbox to Microsoft Office), and of course many mobile apps on both iOS and Android. It might be easy to direct one of these clients to connect to a malicious server (as in the case of a web browser) or it might require a man-in-the-middle (MITM) attack to redirect a client to an evil endpoint.

2) Open agents are clients that can be driven by an attacker but don't reside on an attacker's machine. If you can direct some remote application to fetch a URL on your behalf, then you could theoretically attack that application. The web is full of applications that accept URLs and do something with them; any of these have the potential to be vulnerable:

  • Social networks that do smart things with URLs; e.g. Facebook, which fetches any URL that you type in to a status update in order to generate a preview of that URL.
  • File sharing apps like image thumbnailers, image hosters, Gravatar, and anything else that can "upload" an image or other user-supplied data via a URL.
  • Web spiders like the Googlebot that can stumble on a URL and index it – they can be directed to a malicious server just by linking to it.
  • API consumers that allow integrations across websites. For example, Redbooth integrates with Dropbox to allow users to upload files to projects. If I can convince the Redbooth servers via MITM to send their Dropbox requests to my server, I can potentially exploit them.
  • Identity federation protocols, such as OpenID and WebFinger, allow low-trust users to direct high-trust servers to arbitrary URLs that the user can control. The StackOverflow login page prompts the user for a URL that can be used to log in with OpenID – therefore, the code that StackOverflow uses to fetch that URL must not be vulnerable.
  • Webhooks, which allow a user to register interest in a certain event happening and get a callback. I can tell Github that I'd like to be notified at a URL I control whenever someone pushes to a repository, and Github's agent will connect to that URL over TLS if specified.

The surface of exposed clients is potentially very broad – any code in an application that makes outbound HTTP requests must be checked against reverse Heartbleed attacks. The traditional Heartbleed attack goes after servers that are at the "edges" of a company's security perimeter and therefore may be more hardened, but these open agents are usually deep within an organization's infrastructure.

We found exactly this problem in Meldium's architecture. We run hosts that can make HTTP requests to user-defined URLs. These hosts also have extremely sensitive data in their process memory. While we were not vulnerable to reverse heartbleed, this exercise has pointed out that there needs to be a wide "gap" between anything that touches the outside web and our secure hosts, regardless of whether the network connections are inbound or outbound.

What did we find?

After we developed this tool but before we made it public, we ran some ad-hoc tests against a number of major web properties. These ad-hoc tests found three sites that had patched against Heartbleed on their perimeter hosts, but had not patched their agents and thus could be exploited:

  • An unnamed top 5 social network (we're waiting for confirmation of their fix) that fetched our URL to generate a preview. The memory we extracted from their agent included results from internal API calls and snippets of python source code.
  • Reddit, which can use a URL to suggest a name for a new post, used a vulnerable agent that they've now patched. The memory we were able to extract from this agent was less sensitive, but we didn't get as many samples because they patched so quickly (nice work!).
  • We registered a webhook to our malicious URL at rubygems.org to notify us whenever a gem was published. Within a few minutes, we captured chunks of S3 API calls that the Rubygems servers were making. After the disclosure, they quickly updated OpenSSL and are now protected (really nice work, especially from an all-volunteer staff!).

Is this harder to exploit than Heartbleed?

At a high-level, the exploit code is very similar to the client-initiated attack, but there are a couple of factors that make "reverse Heartbleed" a little harder to pull off:

  • If a client is using certificate pinning, then the window for exploitation is much smaller, because as soon as the client realizes the server certificate doesn't match, they'll cut off the TLS connection.
  • Some clients that support TLS heartbeats will not advertise that support when negotiating a handshake. In our exploit code, we removed the OpenSSL checks that determined whether or not heartbeat requests were allowed based on the negotiated and just sent them anyway.
  • Server-initiated heartbeat requests can only be sent once the TLS connection has been established, so they need to be properly encrypted and MACed. We ended up building our exploit on top of a modified version of OpenSSL instead of writing one from scratch to keep the code simpler.
  • Most clients are trying to complete an HTTP request and then close the channel, so our server uses a few tricks to keep the connection open as long as possible and maximize the number of heartbeat requests we can make. We slowly bleed out the HTTP response headers, we stream the HTTP response body, and we send malformed heartbeat requests interleved throughout. We've found that vulnerable clients can actually be made to send hundreds of 16kb chunks of memory back, making it much easier to explore the client's memory space.

What can I do?

The mitigation steps are the same as for the regular Heartbleed attack: don't use vulnerable versions of OpenSSL. The important takeaway is that it's not enough to patch your perimeter hosts - you need to purge bad OpenSSL versions from your entire infrastructure. And you should keep a healthy distance between agent code that fetches user-provided URLs and sensitive parts of your systems.

Try the reverse heartbleed tester now on your sites, and please practice responsible disclosure if you find vulnerabilities. We'll continue to work to improve the tester to cover more cases, and we hope to release the source code soon.