Talos on DigitalOcean without losing kernel hardening, the secure way

#What goes wrong

Two separate things go wrong on DigitalOcean, and both are quiet.

The kernel command line follows the installer image. DigitalOcean custom images must boot with BIOS, so Talos uses GRUB there, not systemd-boot. With GRUB, the kernel arguments live in the GRUB configuration, and the Talos Bootloader page says they are taken from the installer image. The first boot uses the command line baked into the digital-ocean disk image: talos.platform=digital-ocean, the KSPP flags, and whatever you added to your schematic. After that, installs and upgrades use whatever installer you name.

talosctl gen config, talosctl upgrade without --image, and the Talos upgrade guide all name factory.talos.dev/metal-installer/<vanilla schematic>. If the Bootloader page is right, a droplet upgraded with that image reboots with the metal command line of the vanilla schematic: talos.platform=metal and none of your extra arguments. The KSPP defaults (slab_nomerge, pti=on) are Talos defaults and appear in that line too. init_on_free=1, or any other flag you added, does not. The platform matters beyond the command line: the Talos DigitalOcean platform code is what reads the droplet's network settings from the metadata service, and the metal platform does not use it.

The machine config is served forever by the metadata service. The Talos DigitalOcean guide creates droplets with --user-data-file controlplane.yaml. That file holds the cluster CA keys, the etcd CA, the service account signing key, the secrets encryption key and the bootstrap token. DigitalOcean serves user data at http://169.254.169.254/metadata/v1/user-data to anything on the droplet that can reach that address, and user data cannot be changed after the droplet is created. Unless a network policy blocks it, pod traffic to that address leaves through the node like any other egress. One curl from a compromised pod returns the whole file.

#What the docs say

UEFI boot is not supported. Custom images must boot using BIOS.

Source: DigitalOcean docs, Custom Images limits

With GRUB, kernel arguments are stored in the GRUB configuration file. They are taken from the installer image, so build the image with an Image Factory schematic (customization.extraKernelArgs) and upgrade the machine to it to modify them.

Source: Sidero Labs docs, Bootloader

installer and initramfs images only support system extensions (kernel args and META are ignored)

Source: Sidero Labs docs, Image Factory

You cannot modify user data after a Droplet is created.

Source: DigitalOcean docs, How to Provide User Data During Droplet Creation

The two Sidero Labs pages disagree about whether an installer image carries kernel arguments: the Bootloader page says GRUB takes them from the installer image, the Image Factory page says installer images ignore them. The Bootloader page adds that .machine.install.grubUseUKICmdline decides which line GRUB uses: true means the command line embedded in the image, false means the legacy line that Talos builds from its defaults and your machine config. It defaults to false for installations upgraded to v1.12, is set to true in v1.14 when the UnattendedInstallConfig document is used, and the upgrade API introduced in v1.13 does not apply .machine.install.extraKernelArgs while it is false. Neither page says which answer holds for a droplet, so check the node after every upgrade instead of trusting either page. The Talos DigitalOcean guide never mentions upgrades or the digital-ocean-installer image, and it passes the full machine config as user data without a word about the metadata service.

#The secure configuration

1. One schematic for the disk image and the installer. Put your extra kernel arguments in a schematic and register it. The ID is a hash of the content, so the same file always gives the same ID.

# schematic.yaml
customization:
  extraKernelArgs:
    - init_on_free=1        # KSPP-recommended; zeroes freed memory. Not enabled in the Talos kernel by default.

curl -s -X POST --data-binary @schematic.yaml https://factory.talos.dev/schematics
# {"id":"2f2a22eb47a36e38b703cdb8110ed317c226fa9e5af63fad1ff349972cbaa67b", ...}
SCHEMATIC=2f2a22eb47a36e38b703cdb8110ed317c226fa9e5af63fad1ff349972cbaa67b

# Custom image for the droplet, imported straight from Image Factory:
doctl compute image create talos-v1-14-1 --region ams3 \
  --image-url "https://factory.talos.dev/image/${SCHEMATIC}/v1.14.1/digital-ocean-amd64.raw.gz"

2. Name the DigitalOcean installer of the same schematic in the machine config, so installs and upgrades use the same platform and schematic as the disk image the droplet booted from:

# talos-do-installer.yaml
# talosctl gen config ... --config-patch @talos-do-installer.yaml
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
  # <platform>-installer/<schematic>:<version>. Never metal-installer on a droplet,
  # and never a different schematic than the disk image.
  image: factory.talos.dev/digital-ocean-installer/2f2a22eb47a36e38b703cdb8110ed317c226fa9e5af63fad1ff349972cbaa67b:v1.14.1
provisioning:
  # This document replaces the generated one as a whole, so repeat the disk.
  diskSelector:
    match: disk.dev_path == "/dev/vda"   # droplet disks are virtio
  wipe: false

Upgrade with the same reference, changing only the version:

talosctl -n 10.10.1.11 upgrade \
  --image "factory.talos.dev/digital-ocean-installer/${SCHEMATIC}:v1.14.1"

3. Put the VPC address in the certificate. On DigitalOcean the Talos API certificate names the public address and 127.0.0.1, not the VPC address. Managing the node over the VPC then fails with x509: certificate is valid for 127.0.0.1, 203.0.113.21, not 10.10.1.11. Add the VPC address before you apply the config:

machine:
  certSANs:
    - 10.10.1.11      # the droplet's VPC address

4. Keep the machine config out of user data. Create the droplet with no user data, in a VPC, behind a cloud firewall that allows TCP 50000 only from a bastion in the same VPC (see Cloud firewalls by tag). DigitalOcean refuses a droplet from a custom image without an SSH key (The image for this droplet does not use root passwords, please use an SSH key); pass any key, Talos ignores it. Talos boots into maintenance mode. Read the certificate fingerprint from the droplet console in the control panel, then apply the config over the private network and pin the fingerprint. The fingerprint is the base64 SHA-256 of the certificate's public key; without the console, compute it from the bastion (this trusts the network path, so it is weaker than reading the console):

echo | openssl s_client -connect 10.10.1.11:50000 2>/dev/null \
  | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der \
  | openssl dgst -sha256 -binary | base64

# From the bastion, to the droplet's VPC address:
talosctl apply-config --insecure --nodes 10.10.1.11 \
  --cert-fingerprint '<fingerprint from the console>' \
  --file controlplane.yaml

The metadata service now has nothing secret to serve. If droplets were already created with user data, block 169.254.169.254 for all pods (see Blocking the cloud metadata endpoint from pods) and treat every secret in that file as exposed. Rotating the Talos and Kubernetes CAs with talosctl rotate-ca does not replace the etcd CA, the service account key or the secrets encryption key that are also in it.

#Prove it

Run with the schematic above on a throwaway droplet (s-2vcpu-4gb) created from the imported image, next to a bastion in the same VPC. The first check needs no droplet.

1. What your schematic puts on each platform's command line:

curl -sL "https://factory.talos.dev/image/${SCHEMATIC}/v1.14.1/cmdline-digital-ocean-amd64"; echo
curl -sL "https://factory.talos.dev/image/${SCHEMATIC}/v1.14.1/cmdline-metal-amd64"; echo

This step needs no droplet, so here is the real output for the schematic above:

talos.platform=digital-ocean console=ttyS0 console=tty0 console=tty1 net.ifnames=0 slab_nomerge pti=on consoleblank=0 printk.devkmsg=on selinux=1 module.sig_enforce=1 proc_mem.force_override=never init_on_free=1
talos.platform=metal console=tty0 slab_nomerge pti=on consoleblank=0 printk.devkmsg=on selinux=1 module.sig_enforce=1 proc_mem.force_override=never init_on_free=1

Both lines keep the KSPP flags and init_on_free=1, because both come from your schematic. The metal line drops the DigitalOcean platform, its serial console and net.ifnames=0. With the vanilla schematic that talosctl defaults to, init_on_free=1 disappears as well. These endpoints show the line Image Factory builds into boot assets. Whether an installer writes the same line into GRUB is exactly what the two Sidero Labs pages disagree on, so step 2 is the check that counts.

2. The droplet, as built above. A droplet from the imported image, with no user data, in the VPC, behind a cloud firewall that allows 50000 and 6443 from the bastion's VPC address only:

public internet -> 203.0.113.21:50000: timeout
public internet -> 203.0.113.21:6443:  timeout
bastion (VPC)   -> 10.10.1.11:50000:   open
bastion         -> 203.0.113.21:50000: timeout

(The droplet's public and VPC addresses are replaced with the example addresses used on this page.) In maintenance mode, before any config, the node already ran the schematic's command line:

BOOT_IMAGE=/A/vmlinuz talos.platform=digital-ocean console=ttyS0 console=tty0 console=tty1 net.ifnames=0 slab_nomerge pti=on consoleblank=0 printk.devkmsg=on selinux=1 module.sig_enforce=1 proc_mem.force_override=never init_on_free=1

Applying the config with a pinned fingerprint:

$ talosctl apply-config --insecure -n 10.10.1.11 --cert-fingerprint 'AAAA...AAA=' -f controlplane.yaml
... authentication handshake failed: leaf peer certificate doesn't match the provided fingerprints: [AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=]
$ talosctl apply-config --insecure -n 10.10.1.11 --cert-fingerprint '<public key hash>' -f controlplane.yaml
Applied configuration without a reboot

A hash of the whole certificate is refused the same way; Talos pins the public key.

3. What the node booted with, after every upgrade:

$ talosctl -n 10.10.1.11 upgrade --image "factory.talos.dev/digital-ocean-installer/${SCHEMATIC}:v1.14.1" --wait
$ talosctl -n 10.10.1.11 get cmdline -o jsonpath='{.spec.cmdline}'
talos.platform=digital-ocean console=ttyS0 console=tty0 console=tty1 net.ifnames=0 slab_nomerge pti=on consoleblank=0 printk.devkmsg=on selinux=1 module.sig_enforce=1 proc_mem.force_override=never init_on_free=1
$ talosctl -n 10.10.1.11 get extensions
schematic   2f2a22eb47a36e38b703cdb8110ed317c226fa9e5af63fad1ff349972cbaa67b

If the schematic ID is 376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba, the node was upgraded with the vanilla schematic and lost your arguments.

4. What the metadata service hands out (from an ordinary, non-root pod):

$ wget -qO- -T 3 http://169.254.169.254/metadata/v1/user-data
(empty)
$ wget -qO- -T 3 http://169.254.169.254/metadata/v1/id
<the droplet's numeric ID>

No user data, so no machine config to read, but the pod does reach the metadata service. On a droplet created with user data, that first request returns the full machine config; block the address for pods (see Blocking the cloud metadata endpoint from pods).

One more thing from the test: the node's DNS resolver from DigitalOcean's metadata (the VPC resolver) failed on the node with write: operation not permitted, and image pulls failed until DigitalOcean's public resolvers were set with a ResolverConfig document (Talos 1.14 refuses machine.network.nameservers there: .machine.network.nameservers is already set in v1alpha1 config). We did not find the cause; check talosctl logs dns-resolve-cache on a new droplet.

#Mistakes people make

#Managing over the VPC without the VPC address in certSANs

The node's certificate names its public address. Over the VPC, talosctl refuses the connection, and closing the public port leaves you locked out. Add the VPC address to machine.certSANs before the first apply.

#Letting talosctl pick the installer

talosctl upgrade without --image defaults to factory.talos.dev/metal-installer/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba at the talosctl version: the metal installer with the vanilla schematic. talosctl gen config writes the same image into the generated UnattendedInstallConfig. On a droplet that is the wrong platform and the wrong schematic.

#Mixing schematics

The disk image and the installer must come from the same schematic ID. A different ID means a different set of extensions and, by the Bootloader page's account, a different command line on GRUB.

#Treating user data like a one-time secret

User data is not consumed on first boot. It stays on the metadata service for the life of the droplet and cannot be edited. Anything you put there is readable from the node until the droplet is destroyed.

#Rotating the CAs and calling it done

talosctl rotate-ca replaces the Talos API and Kubernetes API CAs. The machine config in user data also holds the etcd CA, the service account signing key and the key that encrypts Secrets in etcd. Plan for those, or never put them in user data.

#Leaving the uploaded image public

The Talos guide asks you to upload the disk image to a Space as public. The image holds no secrets, but remove it or make it private after DigitalOcean imports it, so nobody builds on an image you no longer track.

#Checklist

• One schematic file holds all extra kernel arguments and is kept in git.
• Droplets boot from digital-ocean-amd64.raw.gz built from that schematic.
• UnattendedInstallConfig names factory.talos.dev/digital-ocean-installer/<same schematic>:<version>.
• Every talosctl upgrade passes --image with the digital-ocean installer and the same schematic.
• After each upgrade, talosctl get cmdline shows talos.platform=digital-ocean and your extra arguments.
• Droplets are created without user data; configs are applied over the VPC with --cert-fingerprint.
• TCP 50000 is reachable only from the bastion while nodes are in maintenance mode.
• Pods cannot reach 169.254.169.254.
• Clusters that were created with user data have a plan to replace every secret in it.