CVE-2025-66455: LMDeploy has Remote Code Execution by Pickle Deserialization via handle_zmq_recv in lmdeploy/lmdeploy/pytorch/disagg/conn/engine_conn.py
Summary
LMDeploy's PyTorch DistServe/PD-disaggregation control plane used recvpyobj() to deserialize messages received through a ZeroMQ PULL socket. PyZMQ implements recvpyobj() using Python pickle deserialization, which can execute arbitrary code while reconstructing an object.
The peer address used by the receiver was supplied through the POST /distserve/p2pconnect HTTP endpoint. An attacker who could reach an affected DistServe API server could cause the server to connect to an attacker-controlled ZeroMQ endpoint and deserialize a crafted pickle payload.
API-key authentication is not enabled unless the operator explicitly configures it. As a result, affected DistServe deployments without API keys allowed unauthenticated remote code execution with the privileges of the LMDeploy serving process.
This issue affects the PyTorch backend when PD-disaggregation/DistServe is enabled. Ordinary deployments that do not use the affected disaggregated-serving path do not expose this data flow.
Affected components
- HTTP entry point: lmdeploy/serve/openai/endpoints/distserve.py, POST /distserve/p2pconnect - Attacker-controlled peer address: DistServeConnectionRequest.remoteengineendpointinfo.zmqaddress - Vulnerable receiver: lmdeploy/pytorch/disagg/conn/engineconn.py, EngineP2PConnection.handlezmqrecv() - Unsafe operation: recvpyobj(), which performs pickle deserialization
Vulnerable data flow
1. A caller submits a DistServe P2P connection request containing a ZeroMQ address. 2. The LMDeploy engine connects its ZeroMQ PULL socket to that address. 3. handlezmqrecv() receives messages using recvpyobj(). 4. A malicious peer sends a crafted pickle object. 5. Python code executes during deserialization, before LMDeploy can perform any type or field validation.
A type check performed after recvpyobj() cannot mitigate this issue because pickle payload execution occurs during deserialization.
Impact
Successful exploitation allows arbitrary code execution as the LMDeploy serving process. This can expose model weights, prompts, credentials, attached storage, cluster-network services, and host or GPU resources. An attacker may also modify or terminate the serving process.
Affected versions
Affected versions:
- lmdeploy >= 0.9.2, < 0.16.0
The vulnerable P2P receiver was introduced in commit b0b705f7.
Remediation
The issue was fixed by replacing the pickle-based ZeroMQ protocol with JSON serialization:
- sendpyobj() was replaced with sendjson(). - recvpyobj() was replaced with recvjson(). - Received objects are validated using the DistServeCacheFreeRequest Pydantic schema before use. - Invalid or off-schema messages are rejected without terminating the receive loop.
Fix commit:
https://github.com/InternLM/lmdeploy/commit/f05b4ad8bf2e2d84101a1d63b3c44fadd99223b2
The fix was released in LMDeploy 0.16.0.
Workarounds
Users who cannot upgrade immediately should:
- Prevent untrusted clients from reaching /distserve/ endpoints. - Restrict the DistServe HTTP and ZeroMQ control planes to trusted cluster networks. - Configure API-key authentication. - Block arbitrary outbound ZeroMQ connections from serving nodes.
These measures reduce exposure but do not make pickle deserialization safe. Upgrading to LMDeploy 0.16.0 or later is recommended.
Other sources
LMDeploy is a toolkit for compressing, deploying, and serving large language models. Starting in version 0.9.2 and prior to version 0.16.0, LMDeploy's PyTorch DistServe/PD-disaggregation control plane used recvpyobj() to deserialize messages received through a ZeroMQ PULL socket. PyZMQ implements recvpyobj() using Python pickle deserialization, which can execute arbitrary code while reconstructing an object. The peer address used by the receiver was supplied through the POST /distserve/p2pconnect HTTP endpoint. An attacker who could reach an affected DistServe API server could cause the server to connect to an attacker-controlled ZeroMQ endpoint and deserialize a crafted pickle payload. API-key authentication is not enabled unless the operator explicitly configures it. As a result, affected DistServe deployments without API keys allowed unauthenticated remote code execution with the privileges of the LMDeploy serving process. This issue affects the PyTorch backend when PD-disaggregation/DistServe is enabled. Ordinary deployments that do not use the affected disaggregated-serving path do not expose this data flow. The fix was released in LMDeploy 0.16.0. Users who cannot upgrade immediately should prevent untrusted clients from reaching /distserve/ endpoints, restrict the DistServe HTTP and ZeroMQ control planes to trusted cluster networks, configure API-key authentication, and block arbitrary outbound ZeroMQ connections from serving nodes. These measures reduce exposure but do not make pickle deserialization safe.
— MITRE
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
pip/lmdeployto a version that resolves this vulnerability.Fixed in 0.16.0 - Upgrade
Upgrade
LMDeployto a version that resolves this vulnerability.Fixed in 0.16.0 - Configuration
Configure API-key authentication for the DistServe API server (API-key authentication is not enabled unless the operator explicitly configures it).
DistServe HTTP API API-key authentication = enabled - Compensating control
Prevent untrusted clients from reaching the HTTP endpoints under `/distserve/*`.
- Compensating control
Restrict the DistServe HTTP and ZeroMQ control planes to trusted cluster networks.
- Compensating control
Block arbitrary outbound ZeroMQ connections from serving nodes to prevent the LMDeploy engine from connecting to attacker-controlled ZeroMQ endpoints.
Event History
Frequently Asked Questions
Which deployments are exposed?
Only LMDeploy deployments using the PyTorch backend with PD-disaggregation/DistServe enabled expose the affected message flow. Ordinary deployments that do not use disaggregated serving are not affected by this path.
What does an attacker need to exploit this?
The attacker needs network access to an affected DistServe API server. They can use the POST /distserve/p2p_connect endpoint to supply a ZeroMQ peer address and cause the server to deserialize a crafted message from an attacker-controlled endpoint.
Are default deployments protected by authentication?
No. API-key authentication is not enabled unless the operator explicitly configures it, so affected DistServe deployments without configured API keys may permit unauthenticated exploitation.
What can be done if patching cannot happen immediately?
Disable the PyTorch PD-disaggregation/DistServe path if it is not required. Otherwise, restrict network access to the DistServe API and configure API-key authentication to prevent unauthenticated access to the connection endpoint.
How can I identify whether my deployment is affected?
Check whether LMDeploy is running with the PyTorch backend and PD-disaggregation/DistServe enabled. Also determine whether the DistServe API exposes POST /distserve/p2p_connect and whether API-key authentication has been explicitly configured.