Summary

SageMaker Python SDK is an open source library for training and deploying machine learning models on Amazon SageMaker. An issue where the HMAC secret key is stored in environment variables and disclosed via the DescribeTrainingJob API has been identified.

Impact

- Function and Payload Tampering: Attackers with DescribeTrainingJob permissions may extract HMAC secret keys and forge serialized function payloads stored in S3. These tampered payloads would be processed and executed without triggering integrity validation errors, enabling unintended code substitution. - Arbitrary Code Execution in the Training Environment: An third party with both DescribeTrainingJob permissions and write access to the job's S3 output location can extract the HMAC key, craft inappropriate Python objects, and achieve remote code execution in the client's Python process when the victim retrieves remote function results. - Data and Credentials Handling: Arbitrary remote code execution may interact with sensitive data, model artifacts, environment variables, and potentially AWS metadata. - Cross-Tenant or Shared Environment Risks: In multi-tenant, shared S3 bucket, a disclosed HMAC key could act as a pivot point to perform inappropriate actions against other users' remote function workloads. This could leverage the IAM permissions, shared S3 buckets, or VPC resources to compromise adjacent services or data.

Impacted versions

- SageMaker Python SDK v3 < v3.2.0 - SageMaker Python SDK v2 < v2.256.0

Patches This issue has been addressed in SageMaker Python SDK version v3.2.0 and v2.256.0. Upgrading to the latest version immediately and ensuring any forked or derivative code is patched to incorporate the new fixes is recommended.

Workarounds Customers using self-signed certificates for internal model downloads should add their private Certificate Authority (CA) certificate to the container image rather than relying on the SDK’s previous insecure configuration. This opt-in approach maintains security while accommodating internal trusted domains.

Resources If there are any questions or comments about this advisory, contact AWS Security via the vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.

Summary Amazon SageMaker Python SDK is an open-source library for training and deploying machine learning models on Amazon SageMaker. An issue exists where, under certain circumstances, the ModelBuilder/Serve component stores an HMAC signing key in cleartext as a container environment variable, which is returned in plaintext by SageMaker describe APIs.

Impact When using ModelBuilder to build and deploy models with affected model servers (TorchServe, Multi-Model Server, TensorFlow Serving, SMD, or Triton), the SDK generates an HMAC secret key for model artifact integrity verification and stores it as the SAGEMAKERSERVESECRETKEY environment variable in the SageMaker model container configuration. This environment variable is returned in plaintext by the DescribeModel, DescribeEndpointConfig, and DescribeModelPackage APIs. A remote authenticated actor with permissions to call these describe APIs and S3 write access to the model artifact path could extract the key, forge valid integrity signatures for specially crafted model artifacts, and achieve code execution in inference containers with the SageMaker execution role's IAM permissions.

Impacted versions: >= v2.199.0 AND <= v2.257.1, >= v3.0.0 AND <= v3.7.1

Patches This issue has been addressed in Amazon SageMaker Python SDK v2.257.2 and v3.8.0. AWS recommend upgrading to the latest version and rebuilding any models previously created with ModelBuilder using the updated SDK. Models created with affected versions may still have the HMAC key stored in their container environment variables until they are rebuilt with the patched SDK. Ensure any forked or derivative code is patched to incorporate the new fixes.

Workarounds If upgrading is not immediately possible, users can manually remove the SAGEMAKERSERVESECRETKEY environment variable from existing SageMaker models by recreating the model without this variable in the container environment configuration.

References If there any questions or comments about this advisory, contact AWS Security via the vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.

Summary Amazon SageMaker Python SDK is an open-source library for training and deploying machine learning models on Amazon SageMaker. An issue exists where, under certain circumstances, the Triton inference handler deserializes model artifacts without performing integrity verification, allowing specially crafted pickle payloads to execute arbitrary code.

Impact When using ModelBuilder with the Triton inference server, the Triton handler did not perform integrity verification before deserializing model artifacts. A remote authenticated actor with S3 write access to the model artifact path could replace model files with a crafted payload that would execute automatically on the next container lifecycle event, achieving code execution with the SageMaker execution role's IAM permissions.

Impacted versions: >= v2.199.0 AND <= v2.257.1, >= v3.0.0 AND <= v3.7.1

Patches This issue has been addressed in Amazon SageMaker Python SDK v2.257.2 and v3.8.0. The Triton inference handler now performs integrity verification before deserializing model artifacts. AWS recommend upgrading to the latest version and rebuilding any Triton models previously created with ModelBuilder using the updated SDK. Ensure any forked or derivative code is patched to incorporate the new fixes.

Workarounds If upgrading is not immediately possible, users should restrict S3 write access to model artifact paths to only trusted principals and monitor for unintended modifications to files in model artifact S3 locations.

References If there any questions or comments about this advisory, contact AWS Security via vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.

Insertion of sensitive information into log files in the gitutils[.]py component in Amazon SageMaker Python SDK V3 before 3.10.0 and V2 before 2.257.3 might allow an authenticated remote user to obtain plaintext repository credentials from CloudWatch Logs when a git clone operation fails, because the runclonecommand() function does not redact credentials from subprocess error output.

To remediate this issue, users should upgrade to Amazon SageMaker Python SDK V3 3.10.0 or V2 2.257.3.