CloudNativePG TLS and pg_hba, the secure way

#What goes wrong

CloudNativePG turns on TLS for every cluster and issues certificates from its own CA. Replication and the connection pooler use client certificates. That part is secure by default.

Application connections are different. The operator writes pg_hba.conf in a fixed order: its own rules, then yours, then a default rule host all all all scram-sha-256. In PostgreSQL, host matches both TLS and plaintext connections. A client with sslmode=disable, or an old driver that does not ask for TLS, sends its login and all its data in the clear, and the server accepts it.

The second gap is verification. Clients that use sslmode=require encrypt the connection but do not check the certificate, so anyone who can intercept the traffic can pose as the database.

#What the docs say

Since the first matching rule is used for authentication, the pg_hba.conf file generated by the operator can be seen as composed of four sections

Source: CloudNativePG 1.30 docs, PostgreSQL Configuration, The pg_hba section

host records match SSL or non-SSL connection attempts as well as GSSAPI encrypted or non-GSSAPI encrypted connection attempts.

Source: PostgreSQL 18 docs, The pg_hba.conf File

By default, the operator sets both ssl_min_protocol_version and ssl_max_protocol_version to TLSv1.3.

Source: CloudNativePG 1.30 docs, Client TLS/SSL connections

Put the first two quotes together and the problem appears: the operator's last rule is a host rule, and a host rule does not require TLS. The docs show the default rule but do not point out that it accepts plaintext.

#The secure configuration

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: db
  namespace: team-a
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:18
  storage:
    size: 20Gi
  bootstrap:
    initdb:
      database: app
      owner: app
  # Server certificate from cert-manager instead of the operator's self-signed CA,
  # so clients can verify it against a CA you control and rotate.
  certificates:
    serverTLSSecret: db-server-tls
    serverCASecret: db-server-ca
  postgresql:
    parameters:
      # The operator already defaults both to TLSv1.3; pinning them makes a later change visible in review.
      ssl_min_protocol_version: "TLSv1.3"
      ssl_max_protocol_version: "TLSv1.3"
      password_encryption: "scram-sha-256"
      log_connections: "on"
    # User rules are placed after the operator's fixed rules and BEFORE its default
    # "host all all all scram-sha-256", which would also accept plaintext connections.
    pg_hba:
      # 1. No unencrypted TCP connections at all.
      - hostnossl all all all reject
      # 2. The app, over TLS, from the pod network only.
      - hostssl app app 10.244.0.0/16 scram-sha-256
      # 3. Everything else over TCP is refused; the default rule is never reached.
      - hostssl all all all reject

The server certificate, from cert-manager and your own CA:

# cert-manager Certificate for the server; the CA issuer is yours (for example an internal CA).
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: db-server
  namespace: team-a
spec:
  secretName: db-server-tls
  usages: [server auth]
  dnsNames:
    - db-rw
    - db-rw.team-a
    - db-rw.team-a.svc
    - db-rw.team-a.svc.cluster.local   # clients that use the full name verify too
    - db-r
    - db-ro
  secretTemplate:
    labels:
      cnpg.io/reload: ""               # without it, renewals are not loaded
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
    group: cert-manager.io

Two details decide whether this works. The CA must be allowed to sign these names: an intermediate with a DNS name constraint (such as the one in an offline internal CA, limited to internal.example.com) cannot, and the operator refuses to create the cluster. And the cnpg.io/reload label is what makes the instances load a renewed certificate; without it they keep serving the old one until a manual kubectl cnpg reload or a restart.

The operator also needs db-server-ca, a Secret with the CA certificate (ca.crt), so clients and the operator can verify the server. Clients then connect with full verification:

psql "host=db-rw.team-a.svc dbname=app user=app sslmode=verify-full sslrootcert=/etc/db-ca/ca.crt"

In application connection strings, use sslmode=verify-full (libpq and most drivers) and mount the CA from the Secret. Keep the pod network range in the hostssl rule as narrow as your cluster allows.

#Prove it

The pg_hba ordering was first reproduced in a plain PostgreSQL 18 container, with TLS on and the operator's default rule last: a plaintext connection was accepted, and it was rejected once hostnossl all all all reject came first. Then the manifests above ran on a lab cluster with the operator.

The issuer. With the Certificate from a name-constrained intermediate, the Cluster never started:

$ kubectl -n team-a get cluster db -o jsonpath='{.status.phase}'
Unable to create required cluster objects
$ kubectl -n cnpg-system logs deploy/cnpg-cloudnative-pg | grep error
... "error":"generating server TLS certificate: failed to verify certificate: x509: a root or intermediate certificate is not authorized to sign for this name: DNS name \"db-rw\" ...

With a CA that may sign these names:

$ kubectl -n team-a get cluster db
NAME   AGE    INSTANCES   READY   STATUS                     PRIMARY
db     102s   3           3       Cluster in healthy state   db-1

The rules, in the order PostgreSQL reads them:

$ kubectl exec -n team-a db-1 -c postgres -- grep -vE '^\s*(#|$)' /var/lib/postgresql/data/pgdata/pg_hba.conf
local all cnpg_metrics_exporter peer map=cnpg_metrics_exporter
local all all peer map=local
hostssl postgres streaming_replica all cert map=cnpg_streaming_replica
hostssl replication streaming_replica all cert map=cnpg_streaming_replica
hostssl all cnpg_pooler_pgbouncer all cert map=cnpg_pooler_pgbouncer
hostnossl all all all reject
hostssl app app 10.244.0.0/16 scram-sha-256
hostssl all all all reject
host all all all scram-sha-256

The operator's default rule is last, where nothing reaches it.

Connections, from a pod with psql and the CA mounted:

== sslmode=disable
psql: error: connection to server at "db-rw" (10.96.176.163), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.2.24", user "app", database "app", no encryption
== sslmode=verify-full, host db-rw.team-a.svc
t|TLSv1.3|TLS_AES_256_GCM_SHA384
== TLS 1.2 client
psql: error: connection to server at "db-rw" (10.96.176.163), port 5432 failed: SSL error: tlsv1 alert protocol version
== postgres user over TLS
psql: error: connection to server at "db-rw" (10.96.176.163), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.2.24", user "postgres", database "postgres", SSL encryption

(The verified connection ran select ssl, version, cipher from pg_stat_ssl where pid = pg_backend_pid().)

The full service name, and renewals. The first version of the Certificate had no db-rw.team-a.svc.cluster.local:

psql: error: connection to server at "db-rw.team-a.svc.cluster.local" (10.96.176.163), port 5432 failed: server certificate for "db-rw" (and 4 other names) does not match host name "db-rw.team-a.svc.cluster.local"

After adding the name, cert-manager renewed the Secret at once, but the instances kept serving the old certificate. Three minutes later:

$ kubectl -n team-a get secret db-server-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -ext subjectAltName
    DNS:db-rw, DNS:db-rw.team-a, DNS:db-rw.team-a.svc, DNS:db-rw.team-a.svc.cluster.local, DNS:db-r, DNS:db-ro
$ kubectl -n team-a exec db-1 -c postgres -- openssl x509 -in /controller/certificates/server.crt -noout -ext subjectAltName
    DNS:db-rw, DNS:db-rw.team-a, DNS:db-rw.team-a.svc, DNS:db-r, DNS:db-ro

With the cnpg.io/reload label on the Secret, the instance manager logged Requesting configuration reload, the new certificate was served, and verify-full against the full name returned t|TLSv1.3. No pod restarted. CloudNativePG's docs describe this:

If you want ConfigMaps and secrets to be reloaded by instances, you can add a label with the key cnpg.io/reload to it. Otherwise you must reload the instances using the kubectl cnpg reload subcommand.

Source: CloudNativePG docs, Certificates

secretTemplate.labels on the Certificate puts the label on the Secret, and cert-manager keeps it there across renewals.

#Mistakes people make

#Trusting "TLS is on" to mean "TLS is required"

TLS being available and TLS being mandatory are different settings. Only a hostnossl ... reject rule makes it mandatory.

#sslmode=require in every connection string

It encrypts, but it accepts any certificate. Use verify-full with the CA, so a stolen route or DNS entry cannot put a fake server in the middle.

#A catch-all allow rule for convenience

hostssl all all 0.0.0.0/0 scram-sha-256 lets every role log in from anywhere in the cluster. List databases, roles and networks per app, and close with hostssl all all all reject.

#Lowering the TLS version for one old client

ssl_min_protocol_version applies to every client. Upgrade the old driver, or put it behind a pooler that speaks TLS 1.3 to the database.

#Rules that assume fixed pod IPs

Pod IPs change on every restart. Use the pod network range, and rely on network policies to decide which pods can reach the database at all.

#Checklist

• spec.postgresql.pg_hba starts with hostnossl all all all reject.
• Each app has a hostssl rule for its own database, role and network with scram-sha-256.
• The user rules end with hostssl all all all reject.
• The server certificate comes from a CA you control, via serverTLSSecret and serverCASecret.
• Clients use sslmode=verify-full with the CA mounted.
• ssl_min_protocol_version is TLSv1.3.
• A network policy limits which pods can reach the database services.
• A test connection with sslmode=disable is rejected.