See how ceph compares to other vendors in security performance
Hello all, A flaw was found in Ceph's RGW module. An attacker can use improper input validation to create a Denial Of Service within RGW. By using the argument x-amz-copy-source to put an object and specifying an empty string as its content , it will lead to the RGW daemon crashing, resulting in a DoS attack.
Our public advisory may be found at the following URL: https://github.com/ceph/ceph/security/advisories/GHSA-mgrm-g92q-f8h8
We have assigned it a CVE of CVE-2024-47866 with a CVSS Score of 7.5 CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H It is fixed in an upstream PR[1] and will be backported promptly to other releases. In downstream Ceph, it will be patched in Ceph 8.1z4 [2] set to be released this week.
[1]https://github.com/ceph/ceph/pull/65159 [2] https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=release-notes-81
Credits to: Ying Wang for discovery.
Sage McTaggart IBM Product Security
amct () redhat com
sagemct () ibm com
Pronouns:They/Them/Theirs
Ceph is a distributed object, block, and file storage platform. In versions up to and including 19.2.3, using the argument x-amz-copy-source to put an object and specifying an empty string as its content leads to the RGW daemon crashing, resulting in a DoS attack. As of time of publication, no known patched versions exist.
In the Linux kernel, the following vulnerability has been resolved:
ceph: avoid kernel BUG for encrypted inode with unaligned file size
The generic/397 test hits a BUGON for the case of encrypted inode with unaligned file size (for example, 33K or 1K):
[ 877.737811] run fstests generic/397 at 2025-01-03 12:34:40 [ 877.875761] libceph: mon0 (2)127.0.0.1:40674 session established [ 877.876130] libceph: client4614 fsid 19b90bca-f1ae-47a6-93dd-0b03ee637949 [ 877.991965] libceph: mon0 (2)127.0.0.1:40674 session established [ 877.992334] libceph: client4617 fsid 19b90bca-f1ae-47a6-93dd-0b03ee637949 [ 878.017234] libceph: mon0 (2)127.0.0.1:40674 session established [ 878.017594] libceph: client4620 fsid 19b90bca-f1ae-47a6-93dd-0b03ee637949 [ 878.031394] xfsio (pid 18988) is setting deprecated v1 encryption policy; recommend upgrading to v2. [ 878.054528] libceph: mon0 (2)127.0.0.1:40674 session established [ 878.054892] libceph: client4623 fsid 19b90bca-f1ae-47a6-93dd-0b03ee637949 [ 878.070287] libceph: mon0 (2)127.0.0.1:40674 session established [ 878.070704] libceph: client4626 fsid 19b90bca-f1ae-47a6-93dd-0b03ee637949 [ 878.264586] libceph: mon0 (2)127.0.0.1:40674 session established [ 878.265258] libceph: client4629 fsid 19b90bca-f1ae-47a6-93dd-0b03ee637949 [ 878.374578] -----------[ cut here ]------------ [ 878.374586] kernel BUG at net/ceph/messenger.c:1070! [ 878.375150] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 878.378145] CPU: 2 UID: 0 PID: 4759 Comm: kworker/2:9 Not tainted 6.13.0-rc5+ #1 [ 878.378969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 878.380167] Workqueue: ceph-msgr cephconworkfn [ 878.381639] RIP: 0010:cephmsgdatacursorinit+0x42/0x50 [ 878.382152] Code: 89 17 48 8b 46 70 55 48 89 47 08 c7 47 18 00 00 00 00 48 89 e5 e8 de cc ff ff 5d 31 c0 31 d2 31 f6 31 ff c3 cc cc cc cc 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 [ 878.383928] RSP: 0018:ffffb4ffc7cbbd28 EFLAGS: 00010287 [ 878.384447] RAX: ffffffff82bb9ac0 RBX: ffff981390c2f1f8 RCX: 0000000000000000 [ 878.385129] RDX: 0000000000009000 RSI: ffff981288232b58 RDI: ffff981390c2f378 [ 878.385839] RBP: ffffb4ffc7cbbe18 R08: 0000000000000000 R09: 0000000000000000 [ 878.386539] R10: 0000000000000000 R11: 0000000000000000 R12: ffff981390c2f030 [ 878.387203] R13: ffff981288232b58 R14: 0000000000000029 R15: 0000000000000001 [ 878.387877] FS: 0000000000000000(0000) GS:ffff9814b7900000(0000) knlGS:0000000000000000 [ 878.388663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 878.389212] CR2: 00005e106a0554e0 CR3: 0000000112bf0001 CR4: 0000000000772ef0 [ 878.389921] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 878.390620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 878.391307] PKRU: 55555554 [ 878.391567] Call Trace: [ 878.391807] <TASK> [ 878.392021] ? showregs+0x71/0x90 [ 878.392391] ? die+0x38/0xa0 [ 878.392667] ? dotrap+0xdb/0x100 [ 878.392981] ? doerrortrap+0x75/0xb0 [ 878.393372] ? cephmsgdatacursorinit+0x42/0x50 [ 878.393842] ? excinvalidop+0x53/0x80 [ 878.394232] ? cephmsgdatacursorinit+0x42/0x50 [ 878.394694] ? asmexcinvalidop+0x1b/0x20 [ 878.395099] ? cephmsgdatacursorinit+0x42/0x50 [ 878.395583] ? cephconv2tryread+0xd16/0x2220 [ 878.396027] ? rawspinunlock+0xe/0x40 [ 878.396428] ? rawspinrqunlock+0x10/0x40 [ 878.396842] ? finishtaskswitch.isra.0+0x97/0x310 [ 878.397338] ? schedule+0x44b/0x16b0 [ 878.397738] cephconworkfn+0x326/0x750 [ 878.398121] processonework+0x188/0x3d0 [ 878.398522] ? pfxworkerthread+0x10/0x10 [ 878.398929] workerthread+0x2b5/0x3c0 [ 878.399310] ? pfxworkerthread+0x10/0x10 [ 878.399727] kthread+0xe1/0x120 [ 878.400031] ? pfxkthread+0x10/0x10 [ 878.400431] retfromfork+0x43/0x70 [ 878.400771] ? pfxkthread+0x10/0x10 [ 878.401127] retfromforkasm+0x1a/0x30 [ 878.401543] </TASK> [ 878.401760] Modules l ---truncated---
On 6/26/25 15:09, Sage [They / Them] McTaggart wrote: Hello all, A flaw was found in CephFS. An unprivileged user can escalate to root privileges in a ceph-fuse mounted CephFS by chmod 777 a directory owned by root to gain access.
[...]
It is patched via 17.2.8 <https://github.com/ceph/ceph/pull/60314> , 18.2.5, and 19.2.3 in upstream Ceph. From that patch:
bool allowed = false; [...] allowed = true; } [...] if (perms.uid() != 0 && perms.uid() != in->uid && !allowed) goto out;
-- Jacob
Hello all, A flaw was found in CephFS. An unprivileged user can escalate to root privileges in a ceph-fuse mounted CephFS by chmod 777 a directory owned by root to gain access.
The result of this is that a user could read, write and execute to any directory owned by root as long as they chmod 777 it. This impacts confidentiality, integrity, and availability.
Our public advisory may be found at the following URL https://github.com/ceph/ceph/security/advisories/GHSA-89hm-qq33-2fjm
We have assigned it a CVE of CVE-2025-52555 with a CVSS Score of 6.5 CVSS:3.1/AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:L/A:N
It is patched via 17.2.8 <https://github.com/ceph/ceph/pull/60314> , 18.2.5, and 19.2.3 in upstream Ceph. Credits to: Ben Stöver for discovery, and Mark Nelson, Dan van der Ster, and Joshua Blanch for reporting.
Sage McTaggart IBM Product Security
amct () redhat com
sagemct () ibm com
A non-privileged user can change the permissions from a directory owned by the root user an can gain access to the targeted directory.
In the Linux kernel, the following vulnerability has been resolved:
ceph: fix memory leak in cephmdsauthmatch()
We now free the temporary target path substring allocation on every possible branch, instead of omitting the default branch. In some cases, a memory leak occured, which could rapidly crash the system (depending on how many file accesses were attempted).
This was detected in production because it caused a continuous memory growth, eventually triggering kernel OOM and completely hard-locking the kernel.
Relevant kmemleak stacktrace:
unreferenced object 0xffff888131e69900 (size 128): comm "git", pid 66104, jiffies 4295435999 hex dump (first 32 bytes): 76 6f 6c 75 6d 65 73 2f 63 6f 6e 74 61 69 6e 65 volumes/containe 72 73 2f 67 69 74 65 61 2f 67 69 74 65 61 2f 67 rs/gitea/gitea/g backtrace (crc 2f3bb450): [<ffffffffaa68fb49>] kmallocnoprof+0x359/0x510 [<ffffffffc32bf1df>] cephmdscheckaccess+0x5bf/0x14e0 [ceph] [<ffffffffc3235722>] cephopen+0x312/0xd80 [ceph] [<ffffffffaa7dd786>] dodentryopen+0x456/0x1120 [<ffffffffaa7e3729>] vfsopen+0x79/0x360 [<ffffffffaa832875>] pathopenat+0x1de5/0x4390 [<ffffffffaa834fcc>] dofilpopen+0x19c/0x3c0 [<ffffffffaa7e44a1>] dosysopenat2+0x141/0x180 [<ffffffffaa7e4945>] x64sysopen+0xe5/0x1a0 [<ffffffffac2cc2f7>] dosyscall64+0xb7/0x210 [<ffffffffac400130>] entrySYSCALL64afterhwframe+0x77/0x7f
It can be triggered by mouting a subdirectory of a CephFS filesystem, and then trying to access files on this subdirectory with an auth token using a path-scoped capability:
$ ceph auth get client.services [client.services] key = REDACTED caps mds = "allow rw fsname=cephfs path=/volumes/" caps mon = "allow r fsname=cephfs" caps osd = "allow rw tag cephfs data=cephfs"
$ cat /proc/self/mounts services@[REDACTED].cephfs=/volumes/containers /ceph/containers ceph rw,noatime,name=services,secret=<hidden>,msmode=prefer-crc,mounttimeout=300,acl,monaddr=[REDACTED]:3300,recoversession=clean 0 0
$ seq 1 1000000 | xargs -P32 --replace={} touch /ceph/containers/file-{} && \ seq 1 1000000 | xargs -P32 --replace={} cat /ceph/containers/file-{}
[ idryomov: combine if statements, rename rc to pathmatched and make it a bool, formatting ]
In the Linux kernel, the following vulnerability has been resolved:
ceph: fix possible deadlock when holding Fwb to get inlinedata
1, mount with wsync. 2, create a file with ORDWR, and the request was sent to mds.0:
cephatomicopen()--> cephmdscdorequest(openc) finishopen(file, dentry, cephopen)--> cephopen()--> cephinitfile()--> cephinitfileinfo()--> cephuninlinedata()--> { ... if (inlineversion == 1 || / initial version, no data / inlineversion == CEPHINLINENONE) goto outunlock; ... }
The inlineversion will be 1, which is the initial version for the new create file. And here the ci->iinlineversion will keep with 1, it's buggy.
3, buffer write to the file immediately:
cephwriteiter()--> cephgetcaps(file, need=Fw, want=Fb, ...); genericperformwrite()--> aops->writebegin()--> cephwritebegin()--> netfswritebegin()--> netfsbeginread()--> netfsrreqsubmitslice()--> netfsreadfromserver()--> rreq->netfsops->issueread()--> cephnetfsissueread()--> { ... if (ci->iinlineversion != CEPHINLINENONE && cephnetfsissueopinline(subreq)) return; ... } cephputcaprefs(ci, Fwb);
The cephnetfsissueopinline() will send a getattr(Fsr) request to mds.1.
4, then the mds.1 will request the rd lock for CInode::filelock from the auth mds.0, the mds.0 will do the CInode::filelock state transation from excl --> sync, but it need to revoke the Fxwb caps back from the clients.
While the kernel client has aleady held the Fwb caps and waiting for the getattr(Fsr).
It's deadlock!
URL: https://tracker.ceph.com/issues/55377
This vulnerability affects the RadosGW OIDC provider by allowing attackers to bypass authentication using JWTs with "none" as the algorithm (alg). The lack of signature enforcement creates a serious risk of unauthorized access and privilege escalation.
The vulnerability is probably in the RadosGW OIDC provider.
PoC
The HTTP request can be found below. But without the JWT:
POST / HTTP/2 Host: storage.xxx.se User-Agent: aws-sdk-go-v2/1.18.0 os/macos lang/go/1.21.1 X:nocoverageredesign md/GOOS/darwin md/GOARCH/arm64 api/sts/1.19.0 Content-Type: application/x-www-form-urlencoded Amz-Sdk-Invocation-Id: 30a74697-7d7e-4c02-b041-97d68156ee78 Amz-Sdk-Request: attempt=1; max=3 Content-Length: 1508 Accept-Encoding: gzip, deflate, br
Action=AssumeRoleWithWebIdentity&DurationSeconds=3600&RoleArn=arn%3Aaws%3Aiam%3A%3Aorgpentest002%3Arole%2Fu-pentest002STS&RoleSessionName=test&Version=2011-06-15&WebIdentityToken=ey..
Alexander Patrakov wrote: [...] What would be a good solution (as in: something that does not convert crashes into deadlocks) here? I understand that, after memory corruption, we are already in the UB territory, but is there anything better possible than what is implemented?
-- Jacob
Hi,
Given the recent discussions regarding the calls to non-async-signal-safe functions from signal handlers, let me add my two pesos to the discussion, even though I don't think there is anything security-related in this email.
A few months ago, I handled a customer report about a deadlock in Ceph RADOS Gateway. The Ceph version is 17.2.7. In that case, I attached gdb to the deadlocked process and applied the backtrace to all threads. One of the threads had this backtrace:
Thread 594 (Thread 0x7f41ac610700 (LWP 870073) "radosgw"): #0 0x00007f43ca050d0e in llllockwaitprivate () from target:/lib64/libc.so.6 #1 0x00007f43ca0568fa in malloc () from target:/lib64/libc.so.6 #2 0x00007f43ca0df884 in backtracesymbols () from target:/lib64/libc.so.6 #3 0x00007f43cb24485a in handleoneshotfatalsignal(int) () from target:/usr/lib64/libradosgw.so.2 #4 <signal handler called> #5 0x00007f43ca003d2b in raise () from target:/lib64/libc.so.6 #6 0x00007f43ca0053e5 in abort () from target:/lib64/libc.so.6 #7 0x00007f43ca049c87 in libcmessage () from target:/lib64/libc.so.6 #8 0x00007f43ca051d2a in mallocprinterr () from target:/lib64/libc.so.6 #9 0x00007f43ca0525f6 in unlinkchunk.isra () from target:/lib64/libc.so.6 #10 0x00007f43ca0555c0 in intmalloc () from target:/lib64/libc.so.6 #11 0x00007f43ca0567d8 in malloc () from target:/lib64/libc.so.6 #12 0x00007f43ca28e3fc in operator new(unsigned long) () from target:/usr/lib64/libstdc++.so.6 #13 0x00007f43cae3b42f in std::map<rgwobj, RGWObjState, std::less<rgwobj>, std::allocator<std::pair<rgwobj const, RGWObjState> > >::operator () from target:/usr/lib64/libradosgw.so.2 #14 0x00007f43caddb5a5 in RGWObjectCtx::setatomic(rgwobj const&) () from target:/usr/lib64/libradosgw.so.2 #15 0x00007f43caf189c0 in rgw::sal::RadosObject::setatomic(RGWObjectCtx) const () from target:/usr/lib64/libradosgw.so.2 #16 0x00007f43cad8319b in RGWDeleteMultiObj::handleindividualobject(rgwobjkey const&, optionalyield, boost::asio::basicdeadlinetimer<boost::posixtime::ptime, boost::asio::timetraits<boost::posixtime::ptime>, boost::asio::executor>) () from target:/usr/lib64/libradosgw.so.2 #17 0x00007f43cad83b99 in void boost::context::detail::contextentry<boost::context::detail::record<boost::context::continuation, boost::context::basicfixedsizestack<boost::context::stacktraits>, spawn::detail::spawnhelper<boost::asio::executorbinder<void ()(), boost::asio::strand<boost::asio::iocontext::basicexecutortype<std::allocator<void>, 0u> > >, RGWDeleteMultiObj::execute(optionalyield)::{lambda(spawn::basicyieldcontext<boost::asio::executorbinder<void ()(), boost::asio::strand<boost::asio::iocontext::basicexecutortype<std::allocator<void>, 0u> > > >)#3}, boost::context::basicfixedsizestack<boost::context::stacktraits> ::operator()()::{lambda(boost::context::continuation&&)#1}> (boost::context::detail::transfert) () from target:/usr/lib64/libradosgw.so.2 #18 0x00007f43cb293c1f in makefcontext () from target:/usr/lib64/libradosgw.so.2
Here is my attempt to make sense of it from the bottom up.
In frame #13, something was written into a C++ std::map; this has led to a memory allocation via malloc() in frame #11. However, the internal logic of the glibc implementation of malloc() detected some corruption, and there was, indeed this message in the log:
Jun 21 04:52:35 abc-osd08 radosgw[868945]: corrupted size vs. prevsize Jun 21 04:52:36 abc-osd08 radosgw[868945]: Caught signal (Aborted) Jun 21 04:52:36 abc-osd08 radosgw[868945]: in thread 7f41ac610700 threadname:radosgw
After printing this message, glibc tried to safely abort the process to prevent further damage/exploitation, see frame #6.
Ceph daemons, however, have a signal handler that catches SIGABRT and SIGSEGV and tries to format and log a backtrace. Their systemd units are also set to automatically restart the daemons. Without the handler, this is what would have happened. However, within the handler, radosgw calls backtracesymbols(), which, in turn, calls malloc() in frame #1, which is not async-signal-safe. So, here is a usage bug by definition, and it converted a restartable crash into a classical A-A deadlock because the outer malloc() call from frame #11 already holds a lock.
In Ceph public bug tracker, there is also a similar report with MDS instead of the RADOS gateway: https://tracker.ceph.com/issues/65039
The question here is whether this is only a Ceph bug or whether backtracesymbols() is an unusable piece of API a-la gets() because it is calling malloc() and is documented to do so.
The same manual page documents backtracesymbolsfd(), but, as-is, it is not good enough because Ceph also wants to demangle C++ symbols, to log the result, and to post the crash information into the designated directory.
I acknowledge that there is a todo comment in src/global/signalhandler.cc:
// TODO: don't use an ostringstream here. It could call malloc(), which we // don't want inside a signal handler. // Also fix the backtrace code not to allocate memory. ClibBackTrace bt(1); ostringstream oss; bt.print(oss); doutemergency(oss.str());
Apparently, the authors don't realize that the use of ostringstream is not the only cause of possible malloc() calls.
What would be a good solution (as in: something that does not convert crashes into deadlocks) here? I understand that, after memory corruption, we are already in the UB territory, but is there anything better possible than what is implemented?
-- Alexander Patrakov
Hello all, A flaw was found in Ceph RGW. An unprivileged user can write to any bucket(s) accessible by a given key if a POST's form-data contains a key called 'bucket' with a value matching the name of the bucket used to sign the request.
The result of this is that a user could actually upload to any bucket accessible by the specified access key as long as the bucket in the POST policy matches the bucket in said POST form part.
We have assigned it a CVE of CVE-2023-43040 and the patch is attached.
Credits to Lucas Henry of Digital Ocean.
Sage McTaggart IBM Product Security
amct () redhat com
sagemct () ibm com
Pronouns:They/Them/Theirs
Hello list,
this report is about a ceph user to root privilege escalation in the ceph-crash systemd service which is part of the ceph-base component of the Ceph distributed storage system project [1]. This report relates to Ceph version 16.2.9.
The Vulnerability =================
The ceph-crash.service [2] runs the ceph-crash Python script [3] as root. The script is operating in the directory /var/lib/ceph/crash which is controlled by the unprivileged ceph user (ceph:ceph mode 0750). The script periodically scans for new crash directories and forwards the content via ceph crash post. This constellation is subject to security issues that can allow the ceph user to either:
1) post arbitrary data as a "crash dump", even content from private files owned by root. The consequences of this are not fully clear to me, it could be an information leak if the security domain of "root" on the system is different to the security domain of wherever the ceph-crash data will be sent to / accessible afterwards. The ceph crash post command expects JSON input, however, thus the degree of freedom for this is reduced.
2) cause a denial-of-service by feeding large amounts of data into the ceph crash post process. This can cause high amounts of memory and CPU consumption. By placing a symlink or FIFO into the directory instead of an actual file, the script can be made to read from a device file like /dev/random or to block forever.
3) cause a local ceph to root user privilege escalation by tricking ceph-crash to move a ceph controlled file into a privileged file system location.
Item 3) is the most critical of these possibilities. The ceph-crash script basically does the following at a regular interval (by default every 10 minutes):
a) it iterates over all sub-directories of /var/lib/ceph/crash and for each sub-directory it does the following: b) it checks whether <crash>/meta is a regular file; if not then the dir is skipped. c) it checks whether <crash>/done is a regular file; if not then it sleeps for one second and checks again; if still not then the dir is skipped. d) it feeds the content of <crash>/meta to stdin of the command line timeout 30 ceph -n <auth> crash post -i - e) only if the crash post succeeded (exit code 0) will the script attempt to perform os.rename("/var/lib/ceph/crash/<crash>", "/var/lib/ceph/crash/posted/<crash>")
The sleep of one second in step c) makes it easier winning the involved race condition. A possible approach for a compromised ceph user account for exploiting this is the following:
- create a fake crash directory named 'mount', containing an empty 'meta' file:
ceph$ mkdir /var/lib/ceph/crash/mount ceph$ touch /var/lib/ceph/crash/mount/meta
- wait for c) to happen i.e. ceph-crash sleeps for a second to wait for the "done" file to appear. This can be done in an event triggered fashion by using the inotify API to detect the service opening the crash directory. While ceph-crash is sleeping create the "done" file and replace "meta" by a FIFO:
ceph$ touch /var/lib/ceph/crash/mount/done ceph$ rm /var/lib/ceph/crash/mount/meta ceph$ mkfifo /var/lib/ceph/crash/mount/meta
On success the "ceph-crash" script, upon returning from the one second sleep, will block on the FIFO until the attacker is writing data into it, giving the attacker enough time to stage the rest of the attack (30 seconds, because of the timeout frontend command used in step d).
- while ceph-crash is busy forwarding data to ceph crash post the ceph user can replace the "mount" directory by a regular file and prepare a symlink attack:
ceph$ mv /var/lib/ceph/crash/mount /var/lib/ceph/crash/oldmount ceph$ echo 'echo evil code' >/var/lib/ceph/crash/mount ceph$ chmod 755 /var/lib/ceph/crash/mount ceph$ mv /var/lib/ceph/crash/posted /var/lib/ceph/crash/posted.old ceph$ ln -s /usr/bin /var/lib/ceph/crash/posted # unblock the ceph-crash script ceph$ echo "$FAKEJSONDATA" >/var/lib/ceph/crash/oldmount/meta
If this succeeds in time then during step e) the ceph-crash script will rename the ceph controlled "mount" file to /usr/bin/mount, thereby replacing the system binary "mount" by the ceph controlled script. Any root process invoking this is then executing exploit code. Any other binary could be used for this, or also configuration files in /etc that could allow to crack the system.
Because /var/lib/ceph/crash is not world-writable and has no sticky bit, the Linux kernel's symlink protection is not coming to the rescue in this constellation. A precondition is, however, that the file system /var/lib/ceph is the same file system as the target directory for the rename(), because rename() does not work across file system boundaries. For many default Linux setups this is the case though.
Reproducer ==========
Attached to this e-mail is a proof of concept exploit script that demonstrates the vulnerability. Running the script with ceph:ceph credentials pretty reliably replaces /usr/bin/mount by a ceph controlled script. Since ceph-crash only executes its routine every 10 minutes it can take a bit of time to succeed if the race is not won, but it is well within reach to succeed in a real world scenario.
I did not test this in a real world Ceph setup. For testing purposes I let the invocation of "ceph crash post" always succeed. From reading the Python code executed by "ceph crash post" I believe that the JSON data I use in the exploit script should be accepted and lead to a zero exit code.
Possible Fix ============
To fix the issue the simplest route I see would be to execute the ceph-crash script also as ceph:ceph. If this is not possible for some reason though then a careful selection of system calls and/or temporary privilege drops will be necessary in the ceph-crash script to avoid any symlink attacks and other race conditions on file system level.
The systemd service, the ceph-crash script and also the directory permissions for /var/lib/ceph/crash are not specific to SUSE packaging but are already found in the upstream sources. Also Fedora Linux ships with the same setup, for example.
I reported this finding to the Ceph security mailing list a while ago. Red Hat assigned the CVE for the issue. I did not get a confirmation from their side yet whether the issue could be reproduced with a real world Ceph setup. I also didn't hear about upstream's plans and schedule for an actual bugfix.
Timeline ========
2022-09-22: I reported the vulnerability to security () ceph io suggesting an embargo period of 14 days. 2022-10-10: I provided some additional information to security () ceph io suggested two more weeks of embargo, because I wasn't available for some time and things didn't progress much. 2022-10-21: I inquired security () ceph io about the state of their analysis and bugfixing. I received the CVE for the issue. They suffers some delays in handling the issue but we agreed to publish the full report today anyway.
[1]: https://github.com/ceph/ceph [2]: https://github.com/ceph/ceph/blob/85454b3262ef887aacb8f54f23646d6f163df2ee/systemd/ceph-crash.service.in [3]: https://github.com/ceph/ceph/blob/85454b3262ef887aacb8f54f23646d6f163df2ee/src/ceph-crash.in
Best Regards
Matthias
-- Matthias Gerstner <matthias.gerstner () suse de> Security Engineer https://www.suse.com/security GPG Key ID: 0x14C405C971923553 SUSE Software Solutions Germany GmbH HRB 36809, AG Nürnberg Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
The key length for encrypted devices created using ceph-volume is incorrect. This is due to a bug in cephvolume/util/encryption.py, where upon writing a key using osddmcryptkeysize it does not pass the key size to the format and open operations following. The default key is them applied in cryptsetup. All versions since Luminous are assumed affected.
A flaw was found in Ceph-ansible v4.0.41 where it creates an /etc/ceph/iscsi-gateway.conf with insecure default permissions. This flaw allows any user on the system to read sensitive information within this file. The highest threat from this vulnerability is to confidentiality.
A flaw was found in the way the Ceph RGW Beast front-end handles unexpected disconnects. An authenticated attacker can abuse this flaw by making multiple disconnect attempts resulting in a permanent leak of a socket connection by radosgw. This flaw could lead to a denial of service condition by pile up of CLOSEWAIT sockets, eventually leading to the exhaustion of available resources, preventing legitimate users from connecting to the system.
A flaw was found in the ceph-ansible playbook where it contained hardcoded passwords that were being used as default passwords while deploying Ceph services. Any authenticated attacker can abuse this flaw to brute-force Ceph deployments, and gain administrator access to Ceph clusters via the Ceph dashboard to initiate read, write, and delete Ceph clusters and also modify Ceph cluster configurations. Versions before ceph-ansible 6.0.0alpha1 are affected.
A flaw was found in the Ceph RGW configuration with Beast as the front end handling client requests. An unauthenticated attacker could crash the Ceph RGW server by sending valid HTTP headers and terminating the connection, resulting in a remote denial of service for Ceph RGW clients.
Ceph does not properly sanitize encryption keys in debug logging for v4 auth. This results in the leaking of encryption key information in log files via plaintext.
Upstream Patch:
https://github.com/ceph/ceph/pull/25881/commits
Upstream Bug:
http://tracker.ceph.com/issues/37847
A flaw was found in rados gateway shipped as part of ceph. Unclosed file descriptors while denying TCP connections to SSL serving port pile up until exhaustion of resources leading to potencial remote denial of service.
A flaw was found in rados gateway shipped as part of ceph. Unclosed file descriptors while denying TCP connections to SSL serving port pile up until exhaustion of resources leading to potencial remote denial of service.
The ceph-iscsi-cli package starts the rbd-target-api service in debug mode by default, which enables the underlying python-werkzeug debug console. A remote unauthenticated user can use the debug console to execute abitrary commands.
Upstream Bug:
https://github.com/ceph/ceph-iscsi-cli/issues/120
In Ceph, a format string flaw was found in the way libradosstriper parses input from user. A user could crash an application or service using the libradosstriper library.
A flaw was found in the way ceph mon handles user requests. Any authenticated ceph user having read access to ceph can delete, and corrupt snapshot images.
This would require user to have read access and for that user must have key for authentication. It would only affect snapshots and images. So attacker with read access will only be able to corrupt data of snapshot images and rest of the ceph cluster should work as it is. Affect on integrity would be low and availability part can be controlled by mitigation using 'monallowpooldelete = false' in ceph.conf to disable deletion of pools
A flaw was found in the way signature calculation is handled by cephx protocol. The signature calculation is encrypting a 29 byte struct with 16-byte block AES cipher, and then using the first 8 bytes of the result as signature. This only covers first (16 by tes) cipher block, datacrc falls on second block.There are no known exploits against this, If attacker can alter the message payload any changes in datacrc will not be noticed or checked by signature check.
Red Hat Ceph Storage is a scalable, open, software-defined storage platform that combines the most stable version of the Ceph storage system with a Ceph management platform, deployment utilities, and support services.Security Fix(es): ceph: Unauthenticated malformed HTTP requests handled by rgwcivetweb.cc:RGW::initenv() can lead to denial of service (CVE-2018-7262) For more details about the security issue(s), including the impact, a CVSS score, acknowledgments, and other related information, refer to the CVE page(s) listed in the References section.
In ceph, HTTP request headers without a ":" character that are handled in rgwcivetweb.cc:RGW::initenv() can cause variables to be set to NULL, leading to a crash or other potentially unspecified behaviour.
Upstream Pull Request:
https://github.com/ceph/ceph/pull/20403
A flaw was discovered in ceph. Assertion in rgwiampolicy.cc can be reached by user input and fail through data passed in from a rest call causing it to crash.
Upstream patch:
https://github.com/ceph/ceph/commit/b3118cabb8060a8cc6a01c4e8264cb18e7b1745a
A resource-permission flaw was found in the tripleo-heat-templates package where ceph.client.openstack.keyring is created as world-readable. A local attacker with access to the key could read or modify data on Ceph cluster pools for OpenStack as though the attacker were the OpenStack service, thus potentially reading or modifying data in an OpenStack Block Storage volume. This has been patched in versions 7.0.6 and 8.0.0.