Exec-only access to sandboxes, the secure way

#What goes wrong

kubectl exec is a good front door for a sandbox. There is no SSH daemon to harden, no port to expose, and every session goes through the API server, which authenticates the user and can write an audit event. The RBAC behind it is where it goes wrong.

Exec is a subresource. A rule on pods does not cover pods/exec, and a rule on pods/exec without resourceNames allows exec into every pod in the namespace, including other users' sandboxes.

The verb changed under people's feet. kubectl exec used to send an HTTP POST (SPDY), which RBAC checks as create. Newer kubectl versions use a WebSocket when the server supports it, and the WebSocket handshake is an HTTP GET. On clusters without the fix, get on pods/exec is enough to open a shell, and plenty of "read-only" roles grant get on pods/*.

Other doors to the same room. pods/attach joins the container's main process. pods/portforward tunnels to any port in the pod. Ephemeral containers (kubectl debug) add a new container, possibly with a different image and profile. And nodes/proxy reaches the kubelet API directly, which can exec into any pod on the node without the API server's audit log or admission control.

Bindings outlive sandboxes. A RoleBinding created for a sandbox stays after the pod is gone. The next pod with the same name inherits the access.

#What the docs say

However, the create limitation applies only to top-level resources, not subresources. For example, you can use the resourceNames field with pods/exec.

Source: Kubernetes docs, Using RBAC Authorization

When the AuthorizePodWebsocketUpgradeCreatePermission feature gate is true, clients must be authorized to create Pod subresources even when triggering their creation using a WebSocket.

Source: Kubernetes docs, Feature Gates

This access bypasses audit logging and admission control, so care should be taken before granting any rights to this resource.

Source: Kubernetes docs, Role Based Access Control Good Practices (nodes/proxy)

The WebSocket fix is a feature gate, beta and on by default since 1.35. Clusters older than that, or with the gate turned off for old tooling, still accept get for exec. The RBAC page does not mention it; the feature gate list does.

#The secure configuration

1. One Role and one RoleBinding per sandbox, created and deleted with it. The user is an OIDC identity; the pod name is unique per session.

# k8s-exec-access.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sandbox-lab-4821
  namespace: labs
  labels:
    sandbox.example.com/session: "4821"   # the controller deletes by this label
rules:
  - apiGroups: [""]
    resources: ["pods"]
    resourceNames: ["lab-4821"]
    verbs: ["get"]                        # kubectl exec reads the pod first
  - apiGroups: [""]
    resources: ["pods/exec"]
    resourceNames: ["lab-4821"]
    verbs: ["create"]                     # create only; never get on pods/exec
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sandbox-lab-4821
  namespace: labs
  labels:
    sandbox.example.com/session: "4821"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: sandbox-lab-4821
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: oidc:[email protected]

No pods/attach, pods/portforward, pods/ephemeralcontainers, pods/log beyond what you choose, and no list: users cannot even see that other sandboxes exist.

2. A guard against debug containers. Even if a broader role slips into the namespace, only platform admins may add ephemeral containers there. Attach and port-forward are connection requests; keep them out of every Role, as above.

# k8s-exec-guard.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: sandbox-no-debug
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["UPDATE"]
        resources: ["pods/ephemeralcontainers"]   # what kubectl debug changes
  validations:
    - expression: "request.userInfo.groups.exists(g, g == 'oidc:platform-admins')"
      message: "only platform admins may add debug containers to sandbox pods"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: sandbox-no-debug
spec:
  policyName: sandbox-no-debug
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        sandbox.example.com/untrusted: "true"

3. Keep the WebSocket check on. On Kubernetes 1.35 and later, AuthorizePodWebsocketUpgradeCreatePermission is on by default; do not turn it off. On older clusters, find every role that grants get on pods/exec, pods/attach or pods/portforward (including pods/* and *) and remove it:

kubectl get roles,clusterroles -A -o json | jq -r '
  .items[] | . as $r | .rules[]?
  | select((.resources // []) | any(. == "pods/exec" or . == "pods/attach" or . == "pods/portforward" or . == "pods/*" or . == "*"))
  | select((.verbs // []) | any(. == "get" or . == "*"))
  | "\($r.kind)/\($r.metadata.namespace // "-")/\($r.metadata.name)"' | sort -u

4. Never grant nodes/proxy to people or to workloads in the sandbox cluster. Metrics collectors that need kubelet data should use nodes/metrics and nodes/stats, not nodes/proxy.

5. Record every exec. The audit policy in Kubernetes audit policy logs pods/exec at Metadata; the event's requestURI holds the command.

#Prove it

Run on a lab cluster (Kubernetes 1.34, gVisor release-20260921) with the Role, RoleBinding and admission policy above, and a learner pod lab-4821 running under gVisor. --as impersonates the learner's OIDC identity. Real output:

1. The learner can exec into their own pod and do nothing else:

U=oidc:[email protected]
kubectl -n labs auth can-i create pods/lab-4821 --subresource=exec --as=$U
kubectl -n labs auth can-i create pods/lab-9999 --subresource=exec --as=$U
kubectl -n labs auth can-i get pods/lab-4821 --subresource=exec --as=$U
kubectl -n labs auth can-i create pods/lab-4821 --subresource=portforward --as=$U
kubectl -n labs auth can-i list pods --as=$U

yes
no
no
no
no

2. The shell works, as root inside the sandbox:

kubectl -n labs exec -it lab-4821 --as=$U -- id

uid=0(root) gid=0(root) groups=0(root),10(wheel)

3. No debug containers:

kubectl -n labs debug -it lab-4821 --image=busybox:1.37 --as=$U

Defaulting debug container name to debugger-s8nfb.
Error from server (Forbidden): pods "lab-4821" is forbidden: User "oidc:[email protected]" cannot patch resource "pods/ephemeralcontainers" in API group "" in the namespace "labs"

RBAC stops it first. The admission policy is the second line: it answers only platform admins may add debug containers to sandbox pods if a broader role ever grants pods/ephemeralcontainers.

#Mistakes people make

#get on pods/exec

It looks like read access. Over WebSocket it opened a shell on clusters without the feature gate, and it gives nothing useful on clusters with it. Grant create only.

#Exec without resourceNames

create on pods/exec for the namespace lets every user into every sandbox. Subresources accept resourceNames; use them.

#Forgetting nodes/proxy

get on nodes/proxy reaches the kubelet API, which can exec into any pod on the node, with no API server audit event and no admission. It is often granted to monitoring tools; give them nodes/metrics instead.

#Bindings that outlive the sandbox

A RoleBinding left behind grants access to the next pod with the same name. Create Role, RoleBinding and pod together, label them with the session, and delete them together.

#Running SSH "for convenience"

An SSH daemon in the sandbox adds keys to manage, a port to expose and a second, unaudited path in. Exec already authenticates, authorizes and audits.

#Checklist

• Each sandbox has its own Role and RoleBinding, created and deleted with the pod.
• The Role allows get on the pod and create on pods/exec, both with resourceNames.
• No role in the cluster grants get on pods/exec, pods/attach or pods/portforward.
• AuthorizePodWebsocketUpgradeCreatePermission is enabled (default on 1.35+).
• No person or sandbox workload has any verb on nodes/proxy.
• Ephemeral containers are limited to admins in sandbox namespaces; no Role grants attach or port-forward.
• The audit policy records pods/exec at Metadata, and the log is shipped off the node.
• Sandbox pods run no SSH daemon.