Kubernetes audit policy, the secure way

#What goes wrong

The API server writes audit events for each request, one per stage (RequestReceived, ResponseStarted for long-running requests such as watches, ResponseComplete, Panic) unless a stage is omitted, at the level of the first rule that matches. The levels are None, Metadata (who, what, when, the URL, the response code), Request (plus the request body) and RequestResponse (plus the response body).

Three things go wrong with real policies:

Rule order. The first matching rule wins. A broad rule such as "log writes at Request" placed above the Secrets rule records every created or updated Secret with its data. The rule you wrote for Secrets never runs.

Credentials in bodies. Secrets are not the only objects that carry them. A TokenReview request body contains the bearer token being checked. A serviceaccounts/token response contains a fresh token. ConfigMaps often hold connection strings. At Request or RequestResponse, the audit log becomes the easiest place in the cluster to steal credentials from, and it is usually shipped to a log platform with far wider access than the cluster.

Volume. A catch-all RequestResponse rule logs every list and watch response. The API server holds more in memory, the disk fills, and the log rotates away the events you needed before anyone reads them.

The opposite mistake is common too: a Metadata-only policy with no rule for pods/exec, and no copy of the log off the node. On Talos the default policy is a single rule, level: Metadata, and the log stays on the control plane node's disk.

#What the docs say

The first matching rule sets the audit level of the event.

Source: Kubernetes docs, Auditing

Log configmap and secret changes in all other namespaces at the Metadata level.

Source: Kubernetes docs, Auditing (example policy)

The audit logging feature increases the memory consumption of the API server because some context required for auditing is stored for each request.

Source: Kubernetes docs, Auditing

Some fields are replaced instead of merged: KubeAuditPolicyConfig.configuration, KubeAuthenticationConfig.configuration, KubeCredentialProviderConfig.configuration and KubeStaticPodConfig.pod

Source: Sidero Labs docs, Configuration Patches

The Kubernetes example handles Secrets and ConfigMaps, but not token reviews or token requests, and it logs "all other resources in core and extensions" at Request level. The Talos default logs everything at Metadata, which is safe, but it has no rule that makes pods/exec or RBAC changes stand out. The Kubernetes page describes a webhook backend but does not tell you to keep a copy off the control plane node. The Talos Knowledge Base shows a promtail setup that reads /var/log/audit/kube, but as a logging recipe, not as a security control.

#The secure configuration

The policy, as a Talos patch. On other distributions, put the configuration block in a file, pass it with --audit-policy-file, and set --audit-log-path (Talos sets both for you).

# talos-audit-policy.yaml
# talosctl gen config ... --config-patch-control-plane @talos-audit-policy.yaml
# Talos replaces the whole policy with this one (no merge).
apiVersion: v1alpha1
kind: KubeAuditPolicyConfig
configuration:
  apiVersion: audit.k8s.io/v1
  kind: Policy
  omitStages:
    - RequestReceived            # no event before the request is handled; the response stages remain
  rules:
    # 1. Objects that carry credentials: never log bodies. First, because the first match wins.
    - level: Metadata
      resources:
        - group: ""
          resources: ["secrets", "configmaps", "serviceaccounts/token"]
        - group: "authentication.k8s.io"
          resources: ["tokenreviews"]
    # 2. Noise with no security value.
    - level: None
      nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version", "/openapi*"]
    - level: None
      resources:
        - group: ""
          resources: ["events"]
        - group: "events.k8s.io"
          resources: ["events"]
    # 3. Interactive access. Metadata keeps the request URI, which holds the exec command.
    - level: Metadata
      resources:
        - group: ""
          resources: ["pods/exec", "pods/attach", "pods/portforward", "nodes/proxy", "services/proxy"]
    # 4. Who changed permissions or admission: the full object before and after.
    - level: RequestResponse
      verbs: ["create", "update", "patch", "delete", "deletecollection"]
      resources:
        - group: "rbac.authorization.k8s.io"
          resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
        - group: "admissionregistration.k8s.io"
          resources:
            - validatingwebhookconfigurations
            - mutatingwebhookconfigurations
            - validatingadmissionpolicies
            - validatingadmissionpolicybindings
    # 5. Every other write: the request body.
    - level: Request
      verbs: ["create", "update", "patch", "delete", "deletecollection"]
    # 6. Everything else, including reads: metadata only.
    - level: Metadata

Ship the log off the node. On Talos the API server writes it under /var/log/audit/kube/kube-apiserver.log on each control plane node, rotated at 100 MB with 10 old files kept for up to 30 days. Run a log collector as a DaemonSet on control plane nodes that reads that directory with a read-only hostPath mount and sends events to storage the cluster's own credentials cannot delete. The files are written as the nobody user; the Talos Knowledge Base adds the DAC_READ_SEARCH capability to its promtail container so that it can read them. Alert on the events that matter (see Security alerting as code in Grafana): writes to clusterrolebindings, any pods/exec in kube-system, and create on serviceaccounts/token by a human user. Match exec on the subresource, not the verb: current kubectl opens exec over WebSockets, and the event's verb is get (Prove it, step 2).

#Prove it

The first check needs no cluster: secure-tests/kubernetes-audit-policy/check_policy.py validates the policy against the audit.k8s.io/v1 fields and walks the rules in order for sample requests, the way the API server does. Real output:

schema: valid audit.k8s.io/v1 Policy with 7 rules
create  core/secrets                                     -> rule 1: Metadata
update  core/configmaps                                  -> rule 1: Metadata
create  authentication.k8s.io/tokenreviews               -> rule 1: Metadata
create  core/serviceaccounts/token                       -> rule 1: Metadata
create  core/pods/exec                                   -> rule 4: Metadata
create  rbac.authorization.k8s.io/clusterrolebindings    -> rule 5: RequestResponse
patch   apps/deployments                                 -> rule 6: Request
list    core/pods                                        -> rule 7: Metadata
create  core/events                                      -> rule 3: None
get     /readyz                                          -> rule 2: None

Then the policy ran on a kind cluster: the configuration block saved as a file and passed to the API server with --audit-policy-file, and the log written to /var/log/kubernetes/audit/kube-apiserver.log on the control plane node. On Talos, read the log with talosctl -n <control plane> read /var/log/audit/kube/kube-apiserver.log instead of the docker exec ... cat used here.

1. A Secret's value never reaches the log:

$ kubectl -n audit-check create secret generic canary --from-literal=value=AuditCanary-7f3k
$ ... cat kube-apiserver.log | grep -c 'AuditCanary-7f3k'
0
$ ... | grep '"canary"' | grep '"verb":"create"' | tail -1 | jq -c '{level, verb, resource: .objectRef.resource, name: .objectRef.name, user: .user.username, hasRequestObject: has("requestObject")}'
{"level":"Metadata","verb":"create","resource":"secrets","name":"canary","user":"kubernetes-admin","hasRequestObject":false}

A later kubectl get secret canary was logged the same way: Metadata, no responseObject.

2. An exec records the command:

$ kubectl -n audit-check exec shell -- id
$ ... | grep '"subresource":"exec"' | tail -1 | jq -c '{level, verb, requestURI}'
{"level":"Metadata","verb":"get","requestURI":"/api/v1/namespaces/audit-check/pods/shell/exec?command=id&container=shell&stderr=true&stdout=true"}

Note the verb: get, not create.

3. An RBAC change records the object:

$ kubectl -n audit-check create rolebinding audit-test --clusterrole=view [email protected]
$ ... | grep '"audit-test"' | grep '"verb":"create"' | tail -1 | jq -c '{level, verb, resource: .objectRef.resource, subjects: .requestObject.subjects, roleRef: .responseObject.roleRef.name}'
{"level":"RequestResponse","verb":"create","resource":"rolebindings","subjects":[{"kind":"User","apiGroup":"rbac.authorization.k8s.io","name":"[email protected]"}],"roleRef":"view"}

Events were not logged at all (rule 3), and no RequestReceived stage appeared: the stages in the file were ResponseComplete and, for exec and watches, ResponseStarted. The log reached about 1 MB in its first minute on a small lab cluster, so size the rotation and the shipping for your own traffic. Clean up with kubectl delete namespace audit-check.

#Mistakes people make

#Writing the Secrets rule last

The first matching rule wins. A "writes at Request" rule above the Secrets rule logs Secret data on every create and update. Put credential-carrying resources first, and review the order on every change.

#Forgetting TokenReview

Webhook authenticators and some controllers send bearer tokens to the API server in TokenReview requests. At Request level, those tokens are written to disk and shipped to your log platform.

#RequestResponse for everything

It records every list response, including full lists of Secrets for anyone who runs kubectl get secrets -A -o yaml. It also raises API server memory and disk use enough to hurt availability.

#Patching one rule on Talos

KubeAuditPolicyConfig.configuration is replaced, not merged. A patch with one rule leaves a policy with one rule. Always patch the complete policy.

#Adding the document to a config from before Talos 1.14

Configs generated before Talos 1.14 hold the policy in .cluster.apiServer.auditPolicy, now deprecated. Talos rejects a KubeAuditPolicyConfig document while that field is set ("audit policy config is already set in v1alpha1 config"). Remove the old field in the same change, or keep patching the old field until you migrate.

#Keeping the only copy on the node

An attacker with node access or nodes/proxy can read or remove the log on the node. Ship it off the node as it is written.

#Checklist

• The first rule sets Metadata for secrets, configmaps, serviceaccounts/token and tokenreviews.
• No rule above it matches those resources.
• omitStages contains RequestReceived.
• Health checks and Events are at level None.
• pods/exec, pods/attach, pods/portforward and proxy subresources are logged at Metadata.
• RBAC and admission webhook or policy changes are logged at RequestResponse.
• The last rule is a Metadata catch-all.
• On Talos, the whole policy is in one KubeAuditPolicyConfig patch, and .cluster.apiServer.auditPolicy is not set.
• Audit logs are shipped off the control plane nodes to storage the cluster cannot delete.
• A canary Secret value was searched for in the log and not found.