The ‘HollowByte’ vulnerability enables unauthorized attackers to create a denial of service (DoS) on OpenSSL servers using a mere 11-byte malicious payload.
The OpenSSL team addressed this vulnerability discreetly, implementing a patch across older releases without assigning a specific identifier.
Given that OpenSSL is critical for secure Internet communications, organizations must prioritize upgrading to the patched version of this library.
Understanding the HollowByte Vulnerability
In a recent advisory, Okta’s Red Team elaborated on the mechanisms of the HollowByte DoS vulnerability and its potential impacts in practice.
The researchers noted that during a TLS handshake, each message is prefixed with a 4-byte header that indicates message size. Vulnerable versions of OpenSSL mistakenly allocate memory based on this header before verifying the actual size of the payload.
Every TLS handshake message starts with a 4-byte header, followed by a 3-byte length field that denotes the size of the handshake data.
Without proper payload validation, the server trusts the incoming packet and allocates memory accordingly. As a result, “the worker thread blocks, waiting indefinitely for data that never arrives,” describes Okta.
An unauthenticated attacker can exploit HollowByte by initiating a TLS connection and transmitting 11 bytes of malicious data, misleading the server into thinking a larger message will follow.
By repeating this process across multiple connections, the attacker forces the server to allocate significant amounts of memory while only sending minimal data.
According to Okta researchers, although OpenSSL releases memory when connections terminate, the GNU C Library (glibc) manages memory differently—retaining small-to-medium allocations instead of immediately returning them to the operating system.
“The attacker obstructs the allocator from reusing freed chunks by launching multiple connections with varied request sizes,” Okta explains.
This leads to extensive heap fragmentation, causing the server’s resident set size (RSS) to grow continuously. Even after the attacker disconnects, the server remains bloated.
The only way to fully recover allocated memory is to restart the process.
Impact and Solutions
The widely-used OpenSSL library underpins various software projects, including NGINX, Apache web servers, and numerous programming languages (Node.js, Python, Ruby, PHP, etc.), as well as databases like MySQL and PostgreSQL. It is pre-installed on most Linux distributions for TLS encryption and certificate management.
Okta’s tests on NGINX indicated that while HollowByte can quickly deplete memory in low-capacity environments, more robust servers can lose up to 25% of memory even when attack bandwidth remains below critical thresholds.
Though DoS vulnerabilities are generally viewed as less severe than issues allowing data breaches or code execution, they can still result in operational disruptions and reputational harm.
The HollowByte DoS vulnerability was effectively patched in OpenSSL version 4.0.1 and also backported to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The fix ensures that header claims are ignored and buffers expand solely as data arrives.
Even while categorized as a “hardening fix” rather than a critical security flaw, Okta strongly advises all users to “upgrade OpenSSL packages in their distributions immediately.”
Security teams document that only 54% of successful attacks get reported, and merely 14% are flagged to users. The remaining incidents go unnoticed.
Picus’ whitepaper provides insights on how to test your SIEM and EDR capabilities in breach simulations to ensure threats remain detected.
Source: www.bleepingcomputer.com