GHSA-2vh9-42vm-xmv2: Critical severity pip/lmdeploy vulnerability
Summary
LMDeploy's PyTorch DistServe/PD-disaggregation control plane used recvpyobj() to deserialize messages received through a ZeroMQ PULL socket. PyZMQ implements recvpyobj() using Python pickle deserialization, which can execute arbitrary code while reconstructing an object.
The peer address used by the receiver was supplied through the POST /distserve/p2pconnect HTTP endpoint. An attacker who could reach an affected DistServe API server could cause the server to connect to an attacker-controlled ZeroMQ endpoint and deserialize a crafted pickle payload.
API-key authentication is not enabled unless the operator explicitly configures it. As a result, affected DistServe deployments without API keys allowed unauthenticated remote code execution with the privileges of the LMDeploy serving process.
This issue affects the PyTorch backend when PD-disaggregation/DistServe is enabled. Ordinary deployments that do not use the affected disaggregated-serving path do not expose this data flow.
Affected components
- HTTP entry point: lmdeploy/serve/openai/endpoints/distserve.py, POST /distserve/p2pconnect - Attacker-controlled peer address: DistServeConnectionRequest.remoteengineendpointinfo.zmqaddress - Vulnerable receiver: lmdeploy/pytorch/disagg/conn/engineconn.py, EngineP2PConnection.handlezmqrecv() - Unsafe operation: recvpyobj(), which performs pickle deserialization
Vulnerable data flow
1. A caller submits a DistServe P2P connection request containing a ZeroMQ address. 2. The LMDeploy engine connects its ZeroMQ PULL socket to that address. 3. handlezmqrecv() receives messages using recvpyobj(). 4. A malicious peer sends a crafted pickle object. 5. Python code executes during deserialization, before LMDeploy can perform any type or field validation.
A type check performed after recvpyobj() cannot mitigate this issue because pickle payload execution occurs during deserialization.
Impact
Successful exploitation allows arbitrary code execution as the LMDeploy serving process. This can expose model weights, prompts, credentials, attached storage, cluster-network services, and host or GPU resources. An attacker may also modify or terminate the serving process.
Affected versions
Affected versions:
- lmdeploy >= 0.9.2, < 0.16.0
The vulnerable P2P receiver was introduced in commit b0b705f7.
Remediation
The issue was fixed by replacing the pickle-based ZeroMQ protocol with JSON serialization:
- sendpyobj() was replaced with sendjson(). - recvpyobj() was replaced with recvjson(). - Received objects are validated using the DistServeCacheFreeRequest Pydantic schema before use. - Invalid or off-schema messages are rejected without terminating the receive loop.
Fix commit:
https://github.com/InternLM/lmdeploy/commit/f05b4ad8bf2e2d84101a1d63b3c44fadd99223b2
The fix was released in LMDeploy 0.16.0.
Workarounds
Users who cannot upgrade immediately should:
- Prevent untrusted clients from reaching /distserve/ endpoints. - Restrict the DistServe HTTP and ZeroMQ control planes to trusted cluster networks. - Configure API-key authentication. - Block arbitrary outbound ZeroMQ connections from serving nodes.
These measures reduce exposure but do not make pickle deserialization safe. Upgrading to LMDeploy 0.16.0 or later is recommended.
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
pip/lmdeployto a version that resolves this vulnerability.Fixed in 0.16.0 - Upgrade
Upgrade
lmdeployto a version that resolves this vulnerability.Fixed in 0.16.0 - Configuration
Ensure API-key authentication is explicitly enabled for the DistServe HTTP entry points to prevent unauthenticated remote code execution paths.
DistServe API-key authentication = enabled - Configuration
Ensure the receiver uses the JSON-based ZeroMQ protocol (replace `recv_pyobj()` with `recv_json()` and `send_pyobj()` with `send_json()`), which removes pickle deserialization during `POST /distserve/p2p_connect` handling.
DistServe P2P receiver ZeroMQ message serialization = JSON (recv_json/send_json) - Configuration
Validate the `remote_engine_endpoint_info.zmq_address` provided in `DistServeConnectionRequest` (the peer address used by the receiver) before the receiver connects; only allow expected/approved endpoints.
DistServeConnectionRequest remote_engine_endpoint_info.zmq_address = validated/allowed list - Compensating control
Block arbitrary outbound ZeroMQ connections from serving nodes so attackers cannot provide attacker-controlled ZeroMQ peer addresses for P2P connections.
- Compensating control
Prevent untrusted clients from reaching the `/distserve/*` endpoints (HTTP entry point) to reduce exposure to the vulnerable P2P connect control plane.
- Compensating control
Restrict the DistServe HTTP and ZeroMQ control planes to trusted networks/peers (only allow the expected callers and cluster networks).
Event History
Frequently Asked Questions
Which deployments are exposed to this issue?
The affected path is limited to LMDeploy deployments using the PyTorch backend with PD-disaggregation/DistServe enabled. Ordinary deployments that do not use this disaggregated-serving path do not expose the vulnerable data flow.
Does a default DistServe API deployment require authentication to exploit this?
No. API-key authentication is not enabled unless the operator explicitly configures it, so an affected deployment without API keys can be reached without authentication by an attacker who can access the DistServe API server.
What does an attacker need to exploit the issue?
The attacker needs network access to the affected DistServe API server. They can use the POST /distserve/p2p_connect endpoint to make the server connect to an attacker-controlled ZeroMQ endpoint and send a crafted pickle payload.
What can be done if patching is not immediately possible?
Explicitly configure API-key authentication and restrict access to the DistServe API server. If PD-disaggregation/DistServe is not required, avoid enabling the affected PyTorch disaggregated-serving path.
What level of access could successful exploitation provide?
Successful exploitation can result in remote code execution with the privileges of the LMDeploy serving process. The crafted payload is deserialized through Python pickle handling in PyZMQ.