Cloud firewalls by tag, the secure way

#What goes wrong

A DigitalOcean cloud firewall applies to the Droplets you list by ID and to every Droplet that carries one of its tags. Tags are the only way to scale past ten Droplets per firewall, and they make autoscaling work: a new node with the right tag is protected the moment it starts.

That also makes the tag the security boundary. Three things go wrong:

A Droplet with no matching tag has no cloud firewall. DigitalOcean does not apply a default firewall. A node created by hand, by an old script, or with a misspelled tag is open to the internet on every port it listens on. The firewall's own status still says succeeded, because it did protect everything it was asked to protect.

Rules name IPs instead of tags. Rules such as "allow 10250 from 10.10.1.21" break when nodes are replaced, so people widen them to the whole VPC or 0.0.0.0/0. Tags as sources follow the nodes.

Allow-all outbound. The control panel suggests outbound rules that permit all traffic, and most firewalls keep them. A compromised node can then send data anywhere. Teams that start with no outbound rules hit a blocked download on day one and add the same allow-all rule.

Two more facts matter during changes. Tightening a firewall does not cut connections that are already open. And DigitalOcean shows no traffic logs for cloud firewalls, so you learn about a gap only by testing.

#What the docs say

Cloud firewalls block all traffic that isn't expressly permitted by a rule.

Source: DigitalOcean docs, Cloud Firewalls

You can have a maximum of 10 Droplets per firewall and 5 tags per firewall. If you have more than 10 Droplets that need the same firewall, tag the Droplets, then add that tag to the firewall.

Source: DigitalOcean docs, Firewalls Limits

Adding new rules to a firewall does not terminate existing connections.

Source: DigitalOcean docs, Firewalls Limits

The suggested outbound rules in the Control Panel permit all traffic to any destination on any port to make it easier to set up a new server because many fundamental services rely on outbound connections.

Source: DigitalOcean docs, Cloud Firewalls Quickstart

The docs say what a firewall blocks, but not what protects a Droplet that no firewall covers (nothing), and there is no built-in report of uncovered Droplets. The DHCP detail is on the limits page: Droplets from custom images, such as Talos, get their network settings over DHCP on port 67, so an outbound rule for it is required.

#The secure configuration

One firewall per role, attached by tag, with tags as sources. The examples use the tags talos-cp, talos-worker and bastion, and an admin address 198.51.100.7. Create the tags first; a firewall can only name existing tags.

1. All cluster nodes.

{
  "name": "talos-nodes",
  "tags": ["talos-cp", "talos-worker"],
  "inbound_rules": [
    {"protocol": "tcp", "ports": "50000", "sources": {"tags": ["bastion"]}},
    {"protocol": "tcp", "ports": "0", "sources": {"tags": ["talos-cp", "talos-worker"]}},
    {"protocol": "udp", "ports": "0", "sources": {"tags": ["talos-cp", "talos-worker"]}},
    {"protocol": "icmp", "ports": "0", "sources": {"tags": ["talos-cp", "talos-worker", "bastion"]}}
  ],
  "outbound_rules": [
    {"protocol": "tcp", "ports": "443", "destinations": {"addresses": ["0.0.0.0/0", "::/0"]}},
    {"protocol": "udp", "ports": "53", "destinations": {"addresses": ["1.1.1.1", "8.8.8.8"]}},
    {"protocol": "tcp", "ports": "53", "destinations": {"addresses": ["1.1.1.1", "8.8.8.8"]}},
    {"protocol": "udp", "ports": "123", "destinations": {"addresses": ["0.0.0.0/0", "::/0"]}},
    {"protocol": "tcp", "ports": "4460", "destinations": {"addresses": ["0.0.0.0/0", "::/0"]}},
    {"protocol": "udp", "ports": "67", "destinations": {"addresses": ["0.0.0.0/0"]}},
    {"protocol": "tcp", "ports": "0", "destinations": {"tags": ["talos-cp", "talos-worker"]}},
    {"protocol": "udp", "ports": "0", "destinations": {"tags": ["talos-cp", "talos-worker"]}},
    {"protocol": "icmp", "ports": "0", "destinations": {"tags": ["talos-cp", "talos-worker"]}}
  ]
}

• 50000 (Talos API) only from the bastion tag.
• Node-to-node traffic by tag, so replacements need no rule change. The host firewall on each node narrows this further (etcd only between control plane nodes), as in Talos Linux hardening.
• Outbound: HTTPS for images, DNS to two named resolvers, NTP and NTS (4460) for time, and DHCP (67) because Talos is a custom image. Nothing else leaves.

2. Control plane only: the Kubernetes API from the bastion.

{
  "name": "talos-control-plane",
  "tags": ["talos-cp"],
  "inbound_rules": [
    {"protocol": "tcp", "ports": "6443", "sources": {"tags": ["bastion"]}}
  ]
}

A Droplet can be in several firewalls; the rules add up. If a load balancer fronts 6443, add its ID under load_balancer_uids in the sources.

3. The bastion.

{
  "name": "bastion",
  "tags": ["bastion"],
  "inbound_rules": [
    {"protocol": "tcp", "ports": "22", "sources": {"addresses": ["198.51.100.7/32"]}}
  ],
  "outbound_rules": [
    {"protocol": "tcp", "ports": "443", "destinations": {"addresses": ["0.0.0.0/0", "::/0"]}},
    {"protocol": "udp", "ports": "53", "destinations": {"addresses": ["1.1.1.1", "8.8.8.8"]}},
    {"protocol": "tcp", "ports": "50000", "destinations": {"tags": ["talos-cp", "talos-worker"]}},
    {"protocol": "tcp", "ports": "6443", "destinations": {"tags": ["talos-cp"]}}
  ]
}

Create them with the API (each body in its own file):

for f in talos-nodes talos-control-plane bastion; do
  curl -sS -X POST -H "Authorization: Bearer $DO_TOKEN" -H "Content-Type: application/json" \
    -d @"fw-$f.json" https://api.digitalocean.com/v2/firewalls | jq -r '.firewall | "\(.name) \(.status)"'
done

4. Audit coverage. A read-only token (droplet:read, firewall:read) is enough. Save both lists, then run the script:

curl -sS -H "Authorization: Bearer $DO_READ_TOKEN" "https://api.digitalocean.com/v2/droplets?per_page=200" > droplets.json
curl -sS -H "Authorization: Bearer $DO_READ_TOKEN" "https://api.digitalocean.com/v2/firewalls?per_page=200" > firewalls.json
python3 firewall-coverage.py droplets.json firewalls.json

# firewall-coverage.py
"""Lists Droplets no cloud firewall covers, and firewall tags no Droplet carries.
Exit code 1 if either list is non-empty. Input: /v2/droplets and /v2/firewalls JSON."""
import json
import sys

droplets = json.load(open(sys.argv[1]))["droplets"]
firewalls = json.load(open(sys.argv[2]))["firewalls"]

covered = {}  # droplet id -> firewall names
for fw in firewalls:
    for did in fw.get("droplet_ids") or []:
        covered.setdefault(did, []).append(fw["name"])
    for d in droplets:
        if set(d.get("tags") or []) & set(fw.get("tags") or []):
            covered.setdefault(d["id"], []).append(fw["name"])

all_tags = {t for d in droplets for t in (d.get("tags") or [])}
problems = 0
for d in droplets:
    names = sorted(set(covered.get(d["id"], [])))
    status = ", ".join(names) if names else "NO FIREWALL"
    problems += not names
    print(f'{d["name"]:12} tags={",".join(d.get("tags") or []) or "-":24} {status}')
for fw in firewalls:
    for tag in fw.get("tags") or []:
        if tag not in all_tags:
            problems += 1
            print(f'firewall {fw["name"]}: tag "{tag}" is on no Droplet')
sys.exit(1 if problems else 0)

Run the audit on a schedule and alert on a non-zero exit.

#Prove it

Real output from secure-tests/cloud-firewalls-tag/. The three bodies above were validated against DigitalOcean's public OpenAPI spec:

fw-talos-nodes.json: valid POST /v2/firewalls request
fw-talos-control-plane.json: valid POST /v2/firewalls request
fw-bastion.json: valid POST /v2/firewalls request

The audit ran against fixture API responses that contain two classic mistakes: a worker with a misspelled tag and a Droplet created without tags.

python3 firewall-coverage.py droplets.json firewalls.json; echo "exit $?"

cp-1         tags=talos-cp                 talos-control-plane, talos-nodes
cp-2         tags=talos-cp                 talos-control-plane, talos-nodes
worker-1     tags=talos-worker             talos-nodes
worker-2     tags=talos-workers            NO FIREWALL
debug-box    tags=-                        NO FIREWALL
bastion-1    tags=bastion                  bastion
exit 1

On your account, also test from outside:

nc -vz -w 5 203.0.113.21 50000     # from a host that is not the bastion
nc -vz -w 5 203.0.113.21 10250

What you should see: both time out. From the bastion, 50000 connects.

#Mistakes people make

#Trusting the green status

status: succeeded means the firewall reached every Droplet it covers. It says nothing about Droplets it does not cover. Only a coverage audit does.

#Attaching by Droplet ID

IDs change when nodes are replaced, and a firewall holds at most ten. Attach by tag and let the tag follow the node.

#Allow-all outbound

An outbound rule for all ports to 0.0.0.0/0 turns a compromised node into a free exfiltration host. List the destinations nodes need.

#Forgetting DHCP on custom images

Talos and other custom images get their network settings by DHCP on DigitalOcean, and the firewall limits page says they need outbound UDP 67. Leave it out and networking breaks in ways that tempt people into an allow-all outbound rule.

#Tightening and assuming it applies now

New rules do not close open connections. After you remove access, restart the services on the affected nodes or reboot them, then test again.

#Loose tag permissions

Whoever can add or remove tags on Droplets can move them in and out of firewalls. Keep tag:create and Droplet update scopes out of CI tokens that do not need them.

#Checklist

• Every firewall is attached by tag, not by Droplet ID.
• Rules use tags as sources and destinations for traffic between your own Droplets.
• Ports 50000 and 6443 accept traffic only from the bastion tag (or a load balancer ID).
• Every firewall has explicit outbound rules; none allows all ports to 0.0.0.0/0.
• Custom-image Droplets allow outbound UDP 67.
• A scheduled audit reports Droplets with no firewall and firewall tags on no Droplet.
• Tokens that can change tags are limited to the provisioning pipeline.
• After tightening rules, open connections were closed by a restart and access was retested.