Cilium L2 announcements in a cloud VPC, the secure way

#What goes wrong

Cilium's L2 announcements make one node answer ARP (IPv4) or NDP (IPv6) requests for a Service's IP. Clients on the same layer 2 segment send traffic for that IP to the node, and Cilium load-balances it. On an office network or a bare-metal VLAN, it is a simple load balancer.

A cloud VPC is not a layer 2 segment you own. The provider's network routes to the addresses it assigned to each interface, and it checks that a VM only sends from its own addresses. An IP that exists only in ARP replies usually gets no traffic. The common "fixes" are the security problem:

• Disabling source/destination checks (AWS) or enabling IP forwarding (Google Cloud) on the nodes. The VPC then accepts packets from those nodes with any source address, which is exactly what anti-spoofing prevents.
• Announcing externalIPs. Any user who can create a Service can put any address in spec.externalIPs. With announcements on for externalIPs, a node will answer ARP for that address: another host's IP, the gateway's IP. This is the pattern behind CVE-2020-8554.
• Announcing everywhere. A policy with loadBalancerIPs: true and no selectors announces every Service with a load balancer IP, from every node, on every interface. (Both loadBalancerIPs and externalIPs default to false, so a policy with neither announces nothing.)

#What the docs say

This selector is NOT a security feature, services will still be available via interfaces when not advertised (for example by hard-coding ARP/NDP entries).

Source: Cilium docs, L2 Announcements / L2 Aware LB (Beta)

If externalIPs is true all IPs in .spec.externalIPs field are announced. These IPs are managed by service authors.

Source: Cilium docs, L2 Announcements / L2 Aware LB (Beta)

Not all clients accept gratuitous ARP replies since they can be used for ARP spoofing.

Source: Cilium docs, L2 Announcements / L2 Aware LB (Beta)

By default, IP forwarding is disabled, and Google Cloud performs strict source address checking.

Source: Google Cloud docs, Use routes

Cilium's L2 page is written for on-premises networks and does not mention cloud VPCs. Elsewhere, its kube-proxy-free page notes that on clouds with source and destination checks (it names AWS) the checks must be disabled for DSR mode. Nothing on either page tells you that turning those checks off is a security trade, not a networking detail.

#The secure configuration

1. In a cloud VPC: use the provider's load balancer, or BGP where the provider supports it. Keep source/destination checks (AWS) on and IP forwarding (Google Cloud) off on Kubernetes nodes. If a design needs them off, treat the nodes as routers and put them under router-grade controls.

2. Where you do run L2 announcements (a segment you control):

# cilium-values.yaml
kubeProxyReplacement: true
k8sServiceHost: api.k8s.example.com
k8sServicePort: 6443
devices: eth1                  # the private segment only
l2announcements:
  enabled: true
bpf:
  disableExternalIPMitigation: false   # keep the CVE-2020-8554 mitigation (the default)

3. A reserved address range, for named namespaces only.

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: edge-pool
spec:
  blocks:
    - start: "10.20.255.100"
      stop: "10.20.255.149"
  serviceSelector:
    matchExpressions:
      - key: io.kubernetes.service.namespace
        operator: In
        values: ["edge"]

Reserve the range on the segment (DHCP exclusions, IPAM records) so no host ever gets one of these addresses.

4. Announce only labelled Services, from labelled nodes, on one interface, and never externalIPs.

apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: edge-announce
spec:
  serviceSelector:
    matchLabels:
      edge.example.com/announce: "true"
  nodeSelector:
    matchLabels:
      edge.example.com/l2-edge: "true"   # a few dedicated nodes, not every worker
  interfaces:
    - ^eth1$                              # anchored: exactly the private NIC
  loadBalancerIPs: true
  externalIPs: false

5. Control who picks addresses. Block spec.externalIPs for everyone, and let only the platform team request a specific load balancer IP:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: restrict-service-ips
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["services"]
  validations:
    - expression: >-
        !has(object.spec.externalIPs) || size(object.spec.externalIPs) == 0
      message: "spec.externalIPs is not allowed"
    - expression: >-
        request.userInfo.groups.exists(g, g == 'platform-admins') ||
        ((!has(object.spec.loadBalancerIP) || object.spec.loadBalancerIP == '') &&
         (!has(object.metadata.annotations) ||
          !('lbipam.cilium.io/ips' in object.metadata.annotations)))
      message: "only platform-admins may request a specific load balancer IP"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: restrict-service-ips
spec:
  policyName: restrict-service-ips
  validationActions: ["Deny"]

The API server's DenyServiceExternalIPs admission plugin also rejects new uses of externalIPs, if you manage its flags.

6. Protect the leases. Each announced Service has a lease cilium-l2announce-<namespace>-<service> in the Cilium namespace; its holder answers ARP. The same namespace holds the control plane's own leader-election leases, so check with kubectl auth can-i update leases -n kube-system --as=<user> that no user or CI identity outside Cilium and the control plane can write leases there.

#Prove it

Run on a kind cluster, whose docker network is a real L2 segment. In this lab the private NIC is eth0 on 172.18.0.0/16, so the pool is 172.18.255.100 to 172.18.255.149 and the interface regex is ^eth0$. Only lab-worker has the label edge.example.com/l2-edge=true. The checks ran from a host on the same segment.

1. Which node announces what:

$ kubectl -n edge get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)
announced       LoadBalancer   10.96.192.110   172.18.255.100   80:32172/TCP
not-announced   LoadBalancer   10.96.184.112   172.18.255.101   80:30145/TCP
$ kubectl -n kube-system get lease | grep cilium-l2announce
cilium-l2announce-edge-announced       lab-worker        15s
$ curl -s -m 4 -o /dev/null -w '%{http_code}\n' http://172.18.255.100/
200
$ curl -s -m 4 -o /dev/null -w '%{http_code}\n' http://172.18.255.101/
000
$ ip neigh show 172.18.255.100
172.18.255.100 dev br-7ca29625e034 lladdr 06:ac:ec:c7:84:72 REACHABLE

06:ac:ec:c7:84:72 is the MAC of lab-worker's eth0. The Service without the label got a pool address, but nobody answers ARP for it.

2. Only pool addresses are in use:

$ kubectl get ippools
NAME        DISABLED   CONFLICTING   IPS AVAILABLE   AGE
edge-pool   false      False         48              18s

3. externalIPs stay unannounced, and the admission policy rejects them. A Service in edge with the announce label and externalIPs: [172.18.0.50], an address outside the pool:

externalIPs: false (this page)   curl http://172.18.0.50/  ->  000, ARP INCOMPLETE, no lease
externalIPs: true                curl http://172.18.0.50/  ->  200, ARP answered by lab-worker
                                 cilium-l2announce-edge-sneaky   lab-worker

With externalIPs: true, anyone who can create a Service in a matching namespace can make a node answer for any address on the segment, including one that belongs to another host. With the admission policy applied:

$ kubectl -n edge create service clusterip probe --tcp=80:80 --dry-run=client -o yaml \
  | kubectl patch --local -f - -p '{"spec":{"externalIPs":["172.18.0.50"]}}' -o yaml \
  | kubectl apply --dry-run=server -f -
The services "probe" is invalid: : ValidatingAdmissionPolicy 'restrict-service-ips' with binding 'restrict-service-ips' denied request: spec.externalIPs is not allowed

And for requested addresses, as a user dev with edit in edge:

annotation lbipam.cilium.io/ips, group developers:       denied: only platform-admins may request a specific load balancer IP
spec.loadBalancerIP, group developers:                   denied: only platform-admins may request a specific load balancer IP
annotation lbipam.cilium.io/ips, group platform-admins:  service/pick created (server dry run)
plain LoadBalancer, group developers:                    service/plain created (server dry run)

4. The leases are not writable by users:

$ kubectl auth can-i update leases -n kube-system --as=dev
no
$ kubectl auth can-i update leases -n kube-system --as=system:serviceaccount:kube-system:cilium
yes

5. In a cloud VPC, anti-spoofing is still on (not run in this lab):

aws ec2 describe-instances --filters Name=tag:role,Values=k8s-node \
  --query 'Reservations[].Instances[].[InstanceId,SourceDestCheck]' --output text

What you should see: True for every node (on Google Cloud, check canIpForward is false).

#Mistakes people make

#Turning off source/destination checks to make ARP work

It makes the VPC accept traffic from the node with any source address, for every workload on that node. Use the cloud load balancer instead.

#Announcing externalIPs

externalIPs are chosen by whoever writes the Service. With announcements, that becomes "any user can make a node answer for any IP on the segment". Set externalIPs: false and reject the field at admission.

#Relying on the interfaces selector

Cilium says it plainly: it is not a security feature. A host that hard-codes an ARP entry still reaches the Service through another interface. Use it for tidiness; use network placement and policy for security.

#A pool that overlaps real hosts

If a pool range overlaps addresses assigned to hosts, an announced Service takes over their traffic. Reserve the range and check pools for conflicts.

#Announcing from every node

Every node that can hold a lease is a node that can answer for the IP. Limit it to a few edge nodes with a node selector.

#Checklist

• In cloud VPCs, Services are exposed with the provider's load balancer or BGP, not L2 announcements.
• Source/destination checks stay on and IP forwarding stays off on nodes.
• LB IPAM pools use a reserved range and select namespaces.
• The L2 announcement policy selects labelled Services, labelled nodes and one anchored interface.
• The policy sets externalIPs: false.
• Admission rejects spec.externalIPs and non-admin requests for specific IPs.
• The CVE-2020-8554 mitigation is not disabled.
• No identity other than Cilium and the control plane components can write leases in the Cilium namespace.