CVE-2026-17507: MLS membership checks compare a uint32 leaf_index as signed, admitting an out-of-range sender
In Bouncy Castle for Java before 1.86, the MLS implementation (org.bouncycastle.mls) holds RFC 9420's uint32 leafindex in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input, and it must still decode, since the MLS interop test vectors round-trip the full range. GroupKeySet.SecretTree.hasLeaf and Group.validateRemove compared the decoded value directly against the tree's leaf count, and a signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check. In the hasLeaf case the SenderData of an unprotected PrivateMessage could then drive LeafIndex.directPath through NodeIndex.parent() arithmetic that never reaches the tree root, growing the resulting node list without bound until the JVM exhausted its heap. A single small message from any current group member could therefore deny service to every other member of the group. Both comparisons now interpret the value as unsigned via Integer.toUnsignedLong, rejecting an out-of-range sender however it was encoded; well-formed leaf indices are unaffected.
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
Bouncy Castle for Javato a version that resolves this vulnerability.Fixed in 1.86
Event History
Frequently Asked Questions
Who can exploit this issue?
A current member of an affected MLS group can exploit it by sending a single small unprotected PrivateMessage with crafted SenderData. No prior privileges beyond group membership are described.
What is the practical impact of a successful exploit?
The crafted out-of-range sender value can cause unbounded growth of a node list during direct-path processing until the JVM exhausts heap memory. This can deny service to every other member of the MLS group.
Are valid MLS messages or full-range leaf-index encodings rejected by the fix?
No. The fix treats leaf indices as unsigned when checking bounds, rejecting only out-of-range senders regardless of encoding. Well-formed leaf indices remain unaffected, including legitimate wire encodings with the top bit set.
How is the issue fixed?
The affected membership checks in GroupKeySet.SecretTree.hasLeaf and Group.validateRemove were changed to interpret decoded leaf indices as unsigned using Integer.toUnsignedLong before comparing them with the tree leaf count.