What the latest OpenSSH security issue is actually about
CVE-2024-6387 is a scary security flaw in the world most used SSH and SFTP server: OpenSSH. It allows unauthenticated remote code execution as root and is affecting many millions of internet-facing servers.
OpenSSH did publish a patch. If that’s not done already, let’s install the new release:
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://security.debian.org/debian-security bookworm-security InRelease
Hit:3 http://deb.debian.org/debian bookworm-updates InRelease
Hit:4 https://linux.dell.com/repo/community/openmanage/930/bionic bionic InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
94 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: http://linux.dell.com/repo/community/openmanage/930/bionic/dists/bionic/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
libssl3 openssh-client openssh-sftp-server openssl
Suggested packages:
keychain libpam-ssh monkeysphere ssh-askpass molly-guard ufw
The following packages will be upgraded:
libssl3 openssh-client openssh-server openssh-sftp-server openssl
5 upgraded, 0 newly installed, 0 to remove and 89 not upgraded.
Need to get 4,952 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Do you want to continue? [Y/n]Y
Get:1 http://security.debian.org/debian-security bookworm-security/main amd64 openssh-sftp-server amd64 1:9.2p1-2+deb12u3 [65.8 kB]
Get:2 http://deb.debian.org/debian bookworm/main amd64 libssl3 amd64 3.0.13-1~deb12u1 [2,022 kB]
Get:3 http://security.debian.org/debian-security bookworm-security/main amd64 openssh-server amd64 1:9.2p1-2+deb12u3 [456 kB]
Get:4 http://security.debian.org/debian-security bookworm-security/main amd64 openssh-client amd64 1:9.2p1-2+deb12u3 [991 kB]
Get:5 http://deb.debian.org/debian bookworm/main amd64 openssl amd64 3.0.13-1~deb12u1 [1,418 kB]
Fetched 4,952 kB in 2s (2,032 kB/s)
Reading changelogs... Done
Preconfiguring packages ...
(Reading database ... 74084 files and directories currently installed.)
Preparing to unpack .../libssl3_3.0.13-1~deb12u1_amd64.deb ...
Unpacking libssl3:amd64 (3.0.13-1~deb12u1) over (3.0.11-1~deb12u2) ...
Preparing to unpack .../openssh-sftp-server_1%3a9.2p1-2+deb12u3_amd64.deb ...
Unpacking openssh-sftp-server (1:9.2p1-2+deb12u3) over (1:9.2p1-2+deb12u2) ...
Preparing to unpack .../openssh-server_1%3a9.2p1-2+deb12u3_amd64.deb ...
Unpacking openssh-server (1:9.2p1-2+deb12u3) over (1:9.2p1-2+deb12u2) ...
Preparing to unpack .../openssh-client_1%3a9.2p1-2+deb12u3_amd64.deb ...
Unpacking openssh-client (1:9.2p1-2+deb12u3) over (1:9.2p1-2+deb12u2) ...
Preparing to unpack .../openssl_3.0.13-1~deb12u1_amd64.deb ...
Unpacking openssl (3.0.13-1~deb12u1) over (3.0.11-1~deb12u2) ...
Setting up libssl3:amd64 (3.0.13-1~deb12u1) ...
Setting up openssl (3.0.13-1~deb12u1) ...
Setting up openssh-client (1:9.2p1-2+deb12u3) ...
Setting up openssh-sftp-server (1:9.2p1-2+deb12u3) ...
Setting up openssh-server (1:9.2p1-2+deb12u3) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
ssh.socket is a disabled or a static unit not running, not starting it.
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for libc-bin (2.36-9+deb12u4) ...
The security flaw is a signal handler race condition in sshd (aka the openssh daemon). In its default configuration, whenever a client does not authenticate within “LoginGraceTime” seconds, openssh server will call the SIGALARM handler in an asynchronous way but this handler does use some non async signal safe functions, oops.
To exploit the issue, the security team who reported the issue did manage to find a path that when interrupted at the right time by SIGALRM, leaves sshd in an inconsistent state, state that is then exploited inside the SIGALRM handler.
Practically speaking, they were able to do exactly this by interrupting a call to both a free() and malloc() with a SIGALRM within the sshd public-key parsing code, leaving the heap in an inconsistent state, and exploit this inconsistent state in another call to free() and malloc(), within the syslog calls of the SIGALRM handler.
The Qualys report goes onto great details about the exploitation of such issue which is nothing less than amazing:
------------------------------------------------------------------------
Theory
------------------------------------------------------------------------
But that's not like me, I'm breaking free
-- The Interrupters, "Haven't Seen the Last of Me"
The SIGALRM handler of this OpenSSH version calls packet_close(), which
calls buffer_free(), which calls xfree() and hence free(), which is not
async-signal-safe:
------------------------------------------------------------------------
302 grace_alarm_handler(int sig)
303 {
...
307 packet_close();
------------------------------------------------------------------------
329 packet_close(void)
330 {
...
341 buffer_free(&input);
342 buffer_free(&output);
343 buffer_free(&outgoing_packet);
344 buffer_free(&incoming_packet);
------------------------------------------------------------------------
35 buffer_free(Buffer *buffer)
36 {
37 memset(buffer->buf, 0, buffer->alloc);
38 xfree(buffer->buf);
39 }
------------------------------------------------------------------------
51 xfree(void *ptr)
52 {
53 if (ptr == NULL)
54 fatal("xfree: NULL pointer given as argument");
55 free(ptr);
56 }
------------------------------------------------------------------------
Consequently, we started to read the malloc code of this Debian's glibc
(2.2.5), to see if a first call to free() can be interrupted by SIGALRM
and exploited during a second call to free() inside the SIGALRM handler
(at lines 341-344, above). Because this glibc's malloc is not hardened
against the unlink() technique pioneered by Solar Designer in 2000, we
quickly spotted an interesting code path in chunk_free() (which is
called internally by free()):
------------------------------------------------------------------------
1028 struct malloc_chunk
1029 {
1030 INTERNAL_SIZE_T prev_size; /* Size of previous chunk (if free). */
1031 INTERNAL_SIZE_T size; /* Size in bytes, including overhead. */
1032 struct malloc_chunk* fd; /* double links -- used only if free. */
1033 struct malloc_chunk* bk;
1034 };
------------------------------------------------------------------------
2516 #define unlink(P, BK, FD) \
2517 { \
2518 BK = P->bk; \
2519 FD = P->fd; \
2520 FD->bk = BK; \
2521 BK->fd = FD; \
2522 } \
------------------------------------------------------------------------
3160 chunk_free(arena *ar_ptr, mchunkptr p)
....
3164 {
3165 INTERNAL_SIZE_T hd = p->size; /* its head field */
....
3177 sz = hd & ~PREV_INUSE;
3178 next = chunk_at_offset(p, sz);
3179 nextsz = chunksize(next);
....
3230 if (!(inuse_bit_at_offset(next, nextsz))) /* consolidate forward */
3231 {
....
3241 unlink(next, bck, fwd);
....
3244 }
3245 else
3246 set_head(next, nextsz); /* clear inuse bit */
....
3251 frontlink(ar_ptr, p, sz, idx, bck, fwd);
------------------------------------------------------------------------
To exploit this code path, we arrange for sshd's heap to have the
following layout (chunk_X, chunk_Y, and chunk_Z are malloc()ated chunks
of memory, and p, s, f, b are their prev_size, size, fd, and bk fields):
-----|---+---------------|---+---------------|---+---------------|-----
... |p|s|f|b| chunk_X |p|s|f|b| chunk_Y |p|s|f|b| chunk_Z | ...
-----|---+---------------|---+---------------|---+---------------|-----
|<------------->|
user data
- First, if a call to free(chunk_Y) is interrupted by SIGALRM *after*
line 3246 but *before* line 3251, then chunk_Y is already marked as
free (because chunk_Z's PREV_INUSE bit is cleared at line 3246) but it
is not yet linked into its doubly-linked list (at line 3251): in other
words, chunk_Y's fd and bk pointers still contain user data (attacker-
controlled data).
- Second, if (inside the SIGALRM handler) packet_close() calls
free(chunk_X), then the code block at lines 3230-3244 is entered
(because chunk_Y is marked as free) and chunk_Y is unlink()ed (at line
3241): a so-called aa4bmo primitive (almost arbitrary 4 bytes mirrored
overwrite), because chunk_Y's fd and bk pointers are still attacker-
controlled. For more information on the unlink() technique and the
aa4bmo primitive:
https://www.openwall.com/articles/JPEG-COM-Marker-Vulnerability#exploit
http://phrack.org/issues/61/6.html#article
- Last, with this aa4bmo primitive we overwrite the glibc's __free_hook
function pointer (this old Debian version does not have ASLR, nor NX)
with the address of our shellcode in the heap, thus achieving remote
code execution during the next call to free() in packet_close().
------------------------------------------------------------------------
Practice
------------------------------------------------------------------------
Now they're taking over and they got complete control
-- The Interrupters, "Liberty"
To mount this attack against sshd, we interrupt a call to free() inside
sshd's parsing code of a DSA public key (i.e., line 144 below is our
free(chunk_Y)) and exploit it during one of the free() calls in
packet_close() (i.e., one of the lines 341-344 above is our
free(chunk_X)):
------------------------------------------------------------------------
136 buffer_get_bignum2(Buffer *buffer, BIGNUM *value)
137 {
138 u_int len;
139 u_char *bin = buffer_get_string(buffer, &len);
...
143 BN_bin2bn(bin, len, value);
144 xfree(bin);
145 }
------------------------------------------------------------------------
Initially, however, we were never able to win this race condition (i.e.,
interrupt the free() call at line 144 at the right time). Eventually, we
realized that we could greatly improve our chances of winning this race:
the DSA public-key parsing code allows us to call free() four times (at
lines 704-707 below), and furthermore sshd allows us to attempt six user
authentications (AUTH_FAIL_MAX); if any one of these 24 free() calls is
interrupted at the right time, then we later achieve remote code
execution inside the SIGALRM handler.
------------------------------------------------------------------------
678 key_from_blob(u_char *blob, int blen)
679 {
...
693 switch (type) {
...
702 case KEY_DSA:
703 key = key_new(type);
704 buffer_get_bignum2(&b, key->dsa->p);
705 buffer_get_bignum2(&b, key->dsa->q);
706 buffer_get_bignum2(&b, key->dsa->g);
707 buffer_get_bignum2(&b, key->dsa->pub_key);
------------------------------------------------------------------------
With this improvement, we finally won the race condition after ~1 month:
we were happy (and did a root-shell dance), but we also felt that there
was still room for improvement.
wow