Hi,

Summary:

This message is about issues in grub-set-bootflag.c commonly installed as grub2-set-bootflag, which is Red Hat's addition (not part of upstream GRUB project) used at least in Fedora and RHEL and its downstreams. It is a SUID root program. I think its latest development source code is currently located in this branch:

https://github.com/rhboot/grub2/tree/fedora-40

On non-OSTree distros, this program's purpose appears to be purely cosmetic - hide the boot menu if the system had already successfully booted up with its current kernel and a user had successfully logged in.

Impact of the issues I identified (through my work at CIQ on Rocky Linux) is rather limited - denial of service and resource limit bypass.

I pre-notified Red Hat grub2 package maintainers about upcoming issues in this program in late December, and reported them in detail via Red Hat Bugzilla on January 3:

https://bugzilla.redhat.com/showbug.cgi?id=2256678

(This is currently a private "bug", hopefully it will be opened soon.)

I also reported this to linux-distros on January 24, and today February 6 is the coordinated public disclosure.

Attached are my currently proposed patches (two revisions, see below), tested by me on Rocky Linux 9.3, and (for the later revision) also by people at Red Hat.

Red Hat assigned this issue CVE-2024-1048 and rated it as CVSSv3.1 Base Score 3.3 and Moderate severity, which I agree with:

CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:L - 3.3

Technically, the RLIMITNPROC bypass could mean S:C A:H, resulting in a score of 6.5, however in practice for this to matter the resource limits would need to be set up, which by default and on most systems they are not anyway. That is, by default almost the same kind and extent of DoS is possible by a simple "fork bomb" from the user's account, so there's no additional vulnerability.

I'd like to thank Red Hat, and especially Marta Lewandowska for her help in coordinating this disclosure and testing the patches.

Overall, I think that at least on Enterprise Linux distros unprivileged setting of boot flags should be disabled by default. It is of questionable value and isn't worth the risk. That said, I understand that for now it may be easier for distros to patch than to re-think it.

Detail:

In 2019, Tavis Ormandy reported that the original implementation of grub2-set-bootflag could be abused to truncate the grubenv file. This is CVE-2019-14865 and was fixed back then:

https://bugzilla.redhat.com/showbug.cgi?id=CVE-2019-14865 https://access.redhat.com/errata/RHSA-2020:0335

Taking a fresh look at grub2-set-bootflag, I saw some other ways in which users could still abuse this little program:

1. After CVE-2019-14865 fix, grub2-set-bootflag no longer rewrites the grubenv file in-place, but writes into a temporary file and renames it over the original, checking for error returns from each call first. This prevents the original file truncation vulnerability, but it can leave the temporary file around if the program is killed before it can rename or remove the file. There are still many ways to get the program killed, such as through RLIMITFSIZE triggering SIGXFSZ (tested, reliable) or by careful timing (tricky) of signals sent by process group leader, pty, pre-scheduled timers, SIGXCPU (probably not an exhaustive list). Invoking the program multiple times fills up /boot (or if /boot is not separate, then it can fill up the root filesystem). Since the files are tiny, the filesystem is likely to run out of free inodes before it'd run out of blocks, but the effect is similar - can't create new files after this point (but still can add data to existing files, such as logs).

2. After CVE-2019-14865 fix, grub2-set-bootflag naively tries to protect itself from signals by becoming full root. (This does protect it from signals sent by the user directly to the PID, but e.g. "kill -9 -1" by the user still works.) A side effect of such "protection" is that it's possible to invoke more concurrent instances of grub2-set-bootflag than the user's RLIMITNPROC would normally permit (as specified e.g. in /etc/security/limits.conf, or say in Apache httpd's RLimitNPROC if grub2-set-bootflag would be abused by a website script), thereby exhausting system resources (e.g., bypassing RAM usage limit if RLIMITAS was also set).

3. umask is inherited. Again, due to how the CVE-2019-14865 fix creates a new file, and due to how mkstemp() works, this affects grubenv's new file permissions. Luckily, mkstemp() forces them to be no more relaxed than 0600, but the user ends up being able to set them e.g. to 0. Luckily, at least in my testing GRUB still works fine even when the file has such (lack of) permissions.

The attached -1 patch deals with my example abuses above as follows:

1. RLIMITFSIZE is pre-checked, so this specific way to get the process killed should no longer work. However, this isn't a complete fix because there are other ways to get the process killed after it has created the temporary file.

The patch also fixes bug 1975892 ("RFE: grub2-set-bootflag should not write the grubenv when the flag being written is already set") and similar for "menushowonce", which further reduces the abuse potential.

2. RLIMITNPROC bypass should be avoided by not becoming full root (aka dropping the partial "kill protection").

3. A safe umask is set.

The -1 patch is a partial fix (temporary files can still accumulate, but this is harder to trigger). It should be safe to use.

The attached -7 patch additionally switches to usage of per-user fixed temporary filenames along with a weird locking mechanism, which is explained in source code comments. This is a more complete fix (temporary files can't accumulate). Unfortunately, it introduces new risks (by working on a temporary file shared between the user's invocations), which are hopefully avoided by the patch's elaborate logic. I actually got it wrong at first, which suggests that this logic is hard to reason about, and more errors or omissions are possible. It also relies on the kernel's primitives' exact semantics to a greater extent (nothing out of the ordinary, though).

Both patches also fix potential 1- or 2-byte over-read of env[] if its content is malformed - this was not a security issue since the grubenv file is trusted input, and the fix is just for robustness.

Also attached is a program I wrote and used to test the unusual approach to locking implemented in the -7 patch here.

Remaining issues that I think cannot reasonably be fixed without a redesign (e.g., having per-flag files with nothing else in them) and without introducing new issues:

A. A user can still revert a concurrent user's attempt of setting the other flag - or of making other changes to grubenv by means other than this program.

B. One leftover temporary file per user is still possible.

Needs comments by people more familiar with GRUB and its configurations in use:

C. One hopefully non-issue (but I am not sure): can "menushowonce" possibly make the system stuck at next boot? Apparently, not with defaults, but maybe along with other GRUB settings in place? If so, it could be unsafe to expose setting this flag to users. A misfeature?

Security hardening not yet implemented (would require changes or at least decisions outside of this program's code):

D. If this program's functionality is really desirable anywhere at all, perhaps its availability should vary by distro - e.g., have it on (some builds of) Fedora, but not on Enterprise Linux distros - and then don't make this program SUID root where that is not needed.

E. The program could refuse to work (exit early) if invoked by an unexpected system pseudo-user. Apparently, it's expected to be invoked by all normal users, but we can nevertheless disallow uid < 1000, so the program couldn't be abused by a compromised system pseudo-user account in a multi-vulnerability multi-step attack.

F. grubenv could be made a symlink into a subdirectory writable by a group, then SGID to that group could be used, mostly to reduce impact of some other (yet unidentified) vulnerabilities/attacks on the program.

Regarding remaining issue/idea D above, even RHEL installs /usr/lib/systemd/user/grub-boot-success.service, which then fails to run upon user login when the program is not user-accessible. The impact from this failure, however, appears to be very limited - just some noise in the logs. The -7 patch includes a piece to reduce such noise if the program is installed e.g. mode 755.

Overall, my understanding is that the program (and other related parts using the boot success flag) is most useful on systems with OSTree, which means some builds of Fedora, right? Per Wikipedia it's "Fedora's atomic spins (Silverblue, Kinoite, and Sericea)". https://en.wikipedia.org/wiki/OSTree

Should we get rid of it on other distros? Or on the contrary, should we make real, non-cosmetic use of the boot flag? If not setting the flag would trigger automatic fallback to the previous kernel, that could be a valuable enough feature to justify some risks, but on the other hand such fallback would also be unexpected by many and it'd be a security concern on its own. A server could successfully boot into the new kernel and be in use without any Unix user logins to it occurring until next reboot. It shouldn't then revert to the old kernel just because no one had logged in. So the feature would need to be opt-in by the sysadmin or/and the criteria for fallback would need to be different.

Alexander

Hi,

Great findings by Qualys, as usual!

Below are some comments on my attempt at reproducing the issue against Rocky Linux 9.5's systemd-coredump (systemd-252-46.el95.3.x8664):

On Thu, May 29, 2025 at 05:17:08PM +0000, Qualys Security Advisory wrote: Local information disclosure in systemd-coredump (CVE-2025-4598) ========================================================================

------------------------------------------------------------------------ Background ------------------------------------------------------------------------

While working on Ubuntu's apport, we remembered that various other distributions (Red Hat Enterprise Linux 9 and Fedora for example) use systemd-coredump as a core-dump handler in /proc/sys/kernel/corepattern (instead of apport). We began to wonder: how does systemd-coredump solve the kill-and-replace race condition that we exploited against apport?

Similarly to apport, systemd-coredump writes all core files into a hard-coded directory, /var/lib/systemd/coredump/. Before December 2022, systemd-coredump allowed users to read all of their core files (through file ACLs), including the core files of SUID or SGID programs, which of course allowed local attackers to read the contents of /etc/shadow by simply crashing su for example; this vulnerability was CVE-2022-4415, discovered and published by Matthias Gerstner:

https://www.openwall.com/lists/oss-security/2022/12/21/3 FWIW, when run on Fedora 34, my reproducer trying to trigger the new bug instead triggers the above older bug - file ACLs are in fact set to enable the non-root user to read a coredump from a SUID program. This old vulnerability was patched by introducing a new function, grantuseraccess(), which decides whether a user should be allowed to read a core file or not, by analyzing the /proc/pid/auxv of the crashed process: if its ATUID and ATEUID match, and if its ATGID and ATEGID match, and if its ATSECURE flag is 0, then read access is allowed; otherwise (if the crashed process is SUID or SGID), read access is denied (only root can read the core file).

------------------------------------------------------------------------ Analysis ------------------------------------------------------------------------

Unfortunately, we soon realized that systemd-coredump does not provide any protection at all against the kill-and-replace race condition that we exploited in apport. In other words, an attacker can simply crash a SUID process such as unixchkpwd, SIGKILL and replace it with a non-SUID process (before its /proc/pid/auxv is analyzed by systemd-coredump), and therefore gain read access to the core file of the crashed SUID process, and hence to the contents of /etc/shadow.

On the one hand, exploiting systemd-coredump is easier than exploiting apport, because we do not need to replace the crashed SUID process with a namespaced process: we can replace it with any non-SUID process, whose ATUID and ATEUID match, whose ATGID and ATEGID match, and whose ATSECURE flag is 0.

On the other hand, winning the kill-and-replace race condition against systemd-coredump is harder: unlike apport, systemd-coredump is written in C, and its initialization takes little time. To widen the window of the race condition, we pass an argv[0] of 128K '\177' characters to the SUID process: this slows down the analysis of its /proc/pid/cmdline (by systemd-coredump, before the analysis of its /proc/pid/auxv) and gives us enough time to replace the crashed SUID process with a non-SUID process.

------------------------------------------------------------------------ Proof of concept ------------------------------------------------------------------------

$ grep PRETTYNAME= /etc/os-release PRETTYNAME="Fedora Linux 41 (Server Edition)"

$ id uid=1001(evey) gid=1001(evey) groups=1001(evey) context=unconfinedu:unconfinedr:unconfinedt:s0-s0:c0.c1023

$ while true; do pid="$(printf 'whatever\0' | ./CVE-2025-4598 /usr/sbin/unixchkpwd "$USER" nullok)"; pidwait -f /usr/lib/systemd/systemd-coredump; if coredumpctl -1 dump "$pid" 2>/dev/null | strings -a | grep '\$[0-9A-Za-z]\+\$[0-9A-Za-z./]'; then break; fi; done

... pid 364536 tid 364521 tid 364540 died in main: 177 theadmin:$y$j9T$APKdqQO.brzhEbC2JFd.5zb7$Rz2q.0umBr8AmkwlozWr8/yphm/ckEHIOMo9vcj.Wj/::0:99999:7::: evey:$y$j9T$QUW3HEErO9CYuGrRhiQjt.$.befySFW/nA48280u/Hk1XrcA2yDZ6Z1s7iRf91nJuA:20188:0:99999:7::: I've attached my attempt at partially reconstructing the Qualys' exploit (which I haven't seen) that the above script uses, as well as the script with some edits.

I think I implemented most of what Qualys described (of the parts relevant to systemd-coredump rather than only to apport), except that I simply use fork() rather than clone() (slower PID reuse) and I didn't implement usage of inotify (harder to win the race leading to password hashes in dump). I've been testing this after:

sysctl kernel.pidmax=2000 control unixchkpwd public # Undo SIG/Security hardening

With the PID range reduced from the default of 4M down to 2K, PID reuse is quick even with simple fork(). I am getting frequent unixchkpwd coredumps (without password hashes in them, which is as expected without inotify), but none of them are getting ACLs set for read by the user (unexpected - I thought I'd win this easier race once in a while), e.g.:

Target pid 1588, current pid 1589 - missed target, retrying Target pid 1590, current pid 1591 - missed target, retrying Replaced pid 1592 getfacl: Removing leading '/' from absolute path names file: var/lib/systemd/coredump/core.unixchkpwd.1000.17099079ebb84acbbb2dc4d8dd38e858.1592.1748566368000000.zst owner: root group: root user::rw- group::r-- other::---

I'd appreciate any hints here.

Meanwhile, Red Hat confirms RHEL 9 and 10 are affected, and curiously lists not only systemd, but also NetworkManager and rpm-ostree among affected packages - I wonder why?

Alexander

Hi, Hi,

Great findings by Qualys, as usual!

Below are some comments on my attempt at reproducing the issue against Rocky Linux 9.5's systemd-coredump (systemd-252-46.el95.3.x8664):

On Thu, May 29, 2025 at 05:17:08PM +0000, Qualys Security Advisory wrote: Local information disclosure in systemd-coredump (CVE-2025-4598) ========================================================================

------------------------------------------------------------------------ Background ------------------------------------------------------------------------

While working on Ubuntu's apport, we remembered that various other distributions (Red Hat Enterprise Linux 9 and Fedora for example) use systemd-coredump as a core-dump handler in /proc/sys/kernel/corepattern (instead of apport). We began to wonder: how does systemd-coredump solve the kill-and-replace race condition that we exploited against apport?

Similarly to apport, systemd-coredump writes all core files into a hard-coded directory, /var/lib/systemd/coredump/. Before December 2022, systemd-coredump allowed users to read all of their core files (through file ACLs), including the core files of SUID or SGID programs, which of course allowed local attackers to read the contents of /etc/shadow by simply crashing su for example; this vulnerability was CVE-2022-4415, discovered and published by Matthias Gerstner:

https://www.openwall.com/lists/oss-security/2022/12/21/3 FWIW, when run on Fedora 34, my reproducer trying to trigger the new bug instead triggers the above older bug - file ACLs are in fact set to enable the non-root user to read a coredump from a SUID program. This old vulnerability was patched by introducing a new function, grantuseraccess(), which decides whether a user should be allowed to read a core file or not, by analyzing the /proc/pid/auxv of the crashed process: if its ATUID and ATEUID match, and if its ATGID and ATEGID match, and if its ATSECURE flag is 0, then read access is allowed; otherwise (if the crashed process is SUID or SGID), read access is denied (only root can read the core file).

------------------------------------------------------------------------ Analysis ------------------------------------------------------------------------

Unfortunately, we soon realized that systemd-coredump does not provide any protection at all against the kill-and-replace race condition that we exploited in apport. In other words, an attacker can simply crash a SUID process such as unixchkpwd, SIGKILL and replace it with a non-SUID process (before its /proc/pid/auxv is analyzed by systemd-coredump), and therefore gain read access to the core file of the crashed SUID process, and hence to the contents of /etc/shadow.

On the one hand, exploiting systemd-coredump is easier than exploiting apport, because we do not need to replace the crashed SUID process with a namespaced process: we can replace it with any non-SUID process, whose ATUID and ATEUID match, whose ATGID and ATEGID match, and whose ATSECURE flag is 0.

On the other hand, winning the kill-and-replace race condition against systemd-coredump is harder: unlike apport, systemd-coredump is written in C, and its initialization takes little time. To widen the window of the race condition, we pass an argv[0] of 128K '\177' characters to the SUID process: this slows down the analysis of its /proc/pid/cmdline (by systemd-coredump, before the analysis of its /proc/pid/auxv) and gives us enough time to replace the crashed SUID process with a non-SUID process.

------------------------------------------------------------------------ Proof of concept ------------------------------------------------------------------------

$ grep PRETTYNAME= /etc/os-release PRETTYNAME="Fedora Linux 41 (Server Edition)"

$ id uid=1001(evey) gid=1001(evey) groups=1001(evey) context=unconfinedu:unconfinedr:unconfinedt:s0-s0:c0.c1023

$ while true; do pid="$(printf 'whatever\0' | ./CVE-2025-4598 /usr/sbin/unixchkpwd "$USER" nullok)"; pidwait -f /usr/lib/systemd/systemd-coredump; if coredumpctl -1 dump "$pid" 2>/dev/null | strings -a | grep '\$[0-9A-Za-z]\+\$[0-9A-Za-z./]'; then break; fi; done

... pid 364536 tid 364521 tid 364540 died in main: 177 theadmin:$y$j9T$APKdqQO.brzhEbC2JFd.5zb7$Rz2q.0umBr8AmkwlozWr8/yphm/ckEHIOMo9vcj.Wj/::0:99999:7::: evey:$y$j9T$QUW3HEErO9CYuGrRhiQjt.$.befySFW/nA48280u/Hk1XrcA2yDZ6Z1s7iRf91nJuA:20188:0:99999:7::: I've attached my attempt at partially reconstructing the Qualys' exploit (which I haven't seen) that the above script uses, as well as the script with some edits.

I think I implemented most of what Qualys described (of the parts relevant to systemd-coredump rather than only to apport), except that I simply use fork() rather than clone() (slower PID reuse) and I didn't implement usage of inotify (harder to win the race leading to password hashes in dump). I've been testing this after:

sysctl kernel.pidmax=2000 control unixchkpwd public # Undo SIG/Security hardening

With the PID range reduced from the default of 4M down to 2K, PID reuse is quick even with simple fork(). I am getting frequent unixchkpwd coredumps (without password hashes in them, which is as expected without inotify), but none of them are getting ACLs set for read by the user (unexpected - I thought I'd win this easier race once in a while), e.g.:

Target pid 1588, current pid 1589 - missed target, retrying Target pid 1590, current pid 1591 - missed target, retrying Replaced pid 1592 getfacl: Removing leading '/' from absolute path names file: var/lib/systemd/coredump/core.unixchkpwd.1000.17099079ebb84acbbb2dc4d8dd38e858.1592.1748566368000000.zst owner: root group: root user::rw- group::r-- other::---

I'd appreciate any hints here.

Meanwhile, Red Hat confirms RHEL 9 and 10 are affected, and curiously lists not only systemd, but also NetworkManager and rpm-ostree among affected packages - I wonder why?

Alexander I hope it helps! David

On Wed, Jun 04, 2025 at 09:52:43AM +0200, David Fernandez Gonzalez wrote: I think I implemented most of what Qualys described (of the parts relevant to systemd-coredump rather than only to apport), except that I simply use fork() rather than clone() (slower PID reuse) and I didn't implement usage of inotify (harder to win the race leading to password hashes in dump). I've been testing this after:

sysctl kernel.pidmax=2000 control unixchkpwd public # Undo SIG/Security hardening

With the PID range reduced from the default of 4M down to 2K, PID reuse is quick even with simple fork(). I am getting frequent unixchkpwd coredumps (without password hashes in them, which is as expected without inotify), but none of them are getting ACLs set for read by the user (unexpected - I thought I'd win this easier race once in a while), e.g.: The POC looks good to me overall but the issue is that the replacement is not really happening while the dump is being generated.

Since the signal from the SUID process is not handled when it exits, it will remain defunct for too long. Either SIGIGN or waitpid for the signal right after SIGKILL. Then you need to spawn the extra processes to replace the PID "fast enough". fork is too slow for this I think, you may need to use clone as Qualys mentioned, for me it always works with clone. After that, it should work! Good point, but this wasn't the issue. A wait() was reached before the replacement PID would have been reached by the loop anyway.

Rather, as Qualys pointed out to me off-list, the biggest issue was that I had the replacement process exit immediately. I had copy-pasted this from the first into the second loop and didn't re-think it through.

Simply fixing this (and tuning a few other things while at it) made the attack work on Rocky Linux 9.5, but I do still have to lower kernel.pidmax as above (or even lower) to have it succeed quickly.

I've attached the revised files. As written, the script will stop when it sees a constant string that's part of unixchkpwd in the core dump. I've also tried editing and running it until it finds password hashes, which it actually did quite a few times as well (that's even without any inotify magic suggested by Qualys). Always the user's, but often also other users' and root's.

Somehow in my testing the core dumps after winning the race only appear when the target PID is low, in the ~300 to ~425 range. I tried this on two different systems (one bare metal and one VM) and observed this same behavior. I don't know why. Also changes to kernel.pidmax and to CPU affinity didn't affect this lucky range in my experiments.

Anyway, this is good enough now as a non-weaponized PoC to confirm that the vulnerability is indeed present on RHEL 9.5 rebuilds and to confirm that its mitigations or fixes make a difference.

Alexander

On Fri, Oct 13, 2023 at 11:19:18PM -0400, Neal Gompa wrote: On Fri, Oct 13, 2023 at 8:07???PM Martin Hecht <martin.hecht () hlrs de> wrote: I'd like to give an example against this. With the recent glibc issue (CVE-2023-4911) we were closely following the upcoming fixed packages. While we were installing the Rocky packages in the late evening of Thu Oct 5, I had the impression that the Redhat packages became available later on Friday. It might be attributed to some hours of delay between arriving on the repo servers vs. being announced via advisory. But, anyhow, accusing Rocky being late in providing packages at least is not valid in general imho. At least important ones, like this one, seem to arrive rather quickly. Without mentioning the distros, I have seen quite some announcements even around a week later. The fix for Rocky 8 and Rocky 9 are purely imports from RHEL:

R8: https://git.rockylinux.org/staging/rpms/glibc/-/commit/6433675bfaab392b362993d8ff8d576335e6bcd4 R9: https://git.rockylinux.org/staging/rpms/glibc/-/commit/610a8a6829e1e604ff018daccf6bf63620edd19d Right, but we also had an effective mitigation for R9 pushed publicly in Security SIG on Oct 3, same day as the vulnerability was made public:

https://sig-security.rocky.page/packages/glibc/

This was possible due to explicit permission I requested/obtained from Qualys in a thread on linux-distros. The rules do mention that things like that can be done "with the reporter's explicit approval". (This option also came up in the recent discussion around illumos distros.) I did see that Louis Abel attempted to do something for Rocky 8, but it was not shipped. Yes, I noticed this one too and asked him about it, and yes it was just a test that was not shipped.

I would be pushing a Security SIG package addressing the issue for R8 as well if we didn't get upstream's update for a day more (after Oct 5). I have also not seen much in terms of upstream engagement indicating the bidirectional relationship expected for members of linux-distros@. I agree this is something to improve, and I intend to be contributing to that with my CIQ or Rocky Linux hat on, as well as encourage others with Rocky Linux to contribute to upstreams more.

I did contribute to the linux-distros discussion on this glibc issue, to an extent greater than I would have without intent to push a mitigation or fix into Rocky Linux Security SIG. I think the point here is "not only being a rebuild of another distro". So, their engagement with SIG should already be a valid add-on to be honored. Anyhow, the fact that CIQ offers LTS branches and professional support, as well as their promise to provide backports of upstream fixes independent of RHEL clearly distinguishes them from a "pure distro rebuild".

https://ciq.com/products/rocky-linux/benefits/enterprise-level-support/ Quoting from the above web page:

"CIQ offers an additional paid service for customers: Rocky Linux Long Term Support ("LTS"). This is designed for organizations who want to remain on a previous minor release of Rocky Linux (like 8.6), which is no longer supported by the public project. CIQ staff will continue backporting security and bug fixes into the supported minor releases for a period of time, after the public project has retired that minor release.

Subscribed customers who value stability can remain on their desired release longer, while still enjoying minor security and bugfix updates to their packages. As of this writing, CIQ is supporting non-zero, even-numbered point releases (8.6, 8.8, 9.2, 9.4, etc.) with its LTS offerings. LTS support lasts for 18 months after the release is retired from the public project. For example, Rocky 8.6 was retired in November 2022. CIQ's LTS-8.6 support will last 18 months from that, or May 2024." The point I'm making is that SIGs do not count because they cannot obey embargo regulations. No open project or community project can do that without having some mechanism for private controls, which is antithetical to the community process. They fundamentally are ineligible to join because they cannot keep anything secret. SIGs are ineligible to join on their own. A distro's security team that would only push to SIGs on the coordinated release date can.

Alexander