Internal tools behind a Kuma gateway, the secure way

#What goes wrong

Internal tools are some of the most valuable targets in a cluster. Grafana shows your data sources, Argo CD can deploy anything, and admin panels do what their name says. They are often published with the least care:

• A public load balancer by default. A Kuma MeshGatewayInstance creates a Kubernetes Service of type LoadBalancer unless you say otherwise. On most clouds that is an internet-facing address.
• Tool login as the only lock. Each tool has its own login, its own password policy and its own history of vulnerabilities in the login page.
• Side doors. The old Ingress, a NodePort, or a mesh permission that lets any service call the tool directly. The gateway is only a control if it is the only way in.
• One wildcard certificate. A *.example.com certificate on the gateway is a key that impersonates every host in the domain if the gateway leaks.

#What the docs say

ServiceType specifies the type of managed Service that will be created to expose the dataplane proxies to traffic from outside the cluster.

Source: Kuma MeshGatewayInstance CRD, spec.serviceType (default LoadBalancer)

TLS sessions are terminated on a Gateway by specifying the “HTTPS” protocol, and providing a server certificate configuration.

Source: Kuma docs, Configuring built-in listeners

In order to route HTTP traffic for a MeshGateway, you need to target the MeshGateway in spec.targetRef and set spec.to[].targetRef.kind: Mesh.

Source: Kuma docs, MeshHTTPRoute

Mutual TLS has to be enabled to make MeshTrafficPermission work.

Source: Kuma docs, MeshTrafficPermission

The gateway docs start from serviceType: LoadBalancer, which suits a public edge. They do not cover authentication at the gateway, so for internal tools you add it yourself, and you need mesh permissions to make sure the gateway is the only path.

#The secure configuration

1. The gateway pods, on a private Service.

apiVersion: kuma.io/v1alpha1
kind: MeshGatewayInstance
metadata:
  name: tools-gateway
  namespace: tools-gateway
spec:
  replicas: 2
  serviceType: ClusterIP            # reached from the VPN / tailnet router, not the internet

Reach the Service from your private access path: a VPN or tailnet subnet router that advertises the cluster Service range. The other option is an internal load balancer on your cloud: keep serviceType: LoadBalancer and set the cloud's internal annotation in serviceTemplate.metadata.annotations. Private DNS names *.tools.example.com point at that address.

2. One HTTPS listener and certificate per tool.

apiVersion: kuma.io/v1alpha1
kind: MeshGateway
mesh: default
metadata:
  name: tools-gateway
spec:
  selectors:
    - match:
        kuma.io/service: tools-gateway_tools-gateway_svc
  conf:
    listeners:
      - port: 8443
        protocol: HTTPS
        hostname: grafana.tools.example.com   # one listener per tool, no wildcard
        tls:
          mode: TERMINATE
          certificates:
            - secret: grafana-tools-example-com
        tags:
          name: grafana
      - port: 8443
        protocol: HTTPS
        hostname: argocd.tools.example.com
        tls:
          mode: TERMINATE
          certificates:
            - secret: argocd-tools-example-com
        tags:
          name: argocd

Each certificate is a Kuma Secret in kuma-system holding the private key and the certificate chain in one PEM value:

cat grafana.key grafana-fullchain.pem > grafana-bundle.pem
kubectl -n kuma-system create secret generic grafana-tools-example-com \
  --type=system.kuma.io/secret --from-file=value=grafana-bundle.pem
kubectl -n kuma-system label secret grafana-tools-example-com kuma.io/mesh=default

No plain HTTP listener: a plaintext request to port 8443 fails the TLS handshake and never reaches a route.

3. Route every host to an SSO proxy. Tools are not reachable without an identity-provider login first. The proxy (for example oauth2-proxy) forwards authenticated requests to the tool.

apiVersion: kuma.io/v1alpha1
kind: MeshHTTPRoute
metadata:
  name: tools-routes
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshGateway
    name: tools-gateway
  to:
    - targetRef:
        kind: Mesh
      hostnames:
        - grafana.tools.example.com
        - argocd.tools.example.com
      rules:
        - matches:
            - path:
                type: PathPrefix
                value: /
          default:
            backendRefs:
              - kind: MeshService       # name/namespace/port form needs MeshService
                name: sso-proxy         # enabled on the Mesh (meshServices.mode)
                namespace: tools
                port: 4180

MeshService is opt-in in Kuma 2.14. If the Mesh does not set meshServices.mode, reference the backend by its kuma.io/service value instead: kind: MeshService with name: sso-proxy_tools_svc_4180.

4. Make the gateway the only way in.

apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: sso-proxy-inbound
  namespace: tools
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: sso-proxy
  from:
    - targetRef:
        kind: MeshSubset
        tags:
          kuma.io/service: tools-gateway_tools-gateway_svc
      default:
        action: Allow
---
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: tools-inbound
  namespace: tools
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      tools.example.com/behind-sso: "true"
  from:
    - targetRef:
        kind: MeshSubset
        tags:
          kuma.io/service: sso-proxy_tools_svc_4180
      default:
        action: Allow

Kuma 2.14 accepts these and warns that from is deprecated and will be removed in 3.0, in favour of rules with MeshIdentity. Plan the move before upgrading to 3.0.

This needs strict mTLS and no allow-all in the mesh (see Default-deny MeshTrafficPermission). Then delete the tools' old Ingress objects, LoadBalancer and NodePort Services.

#Prove it

Run on a lab cluster with Kuma 2.14.3. The SSO proxy was oauth2-proxy v7.12.0; the tool was an nginx stand-in labelled tools.example.com/behind-sso: "true"; certificates came from a private CA. The mesh did not enable MeshService, so the route used the sso-proxy_tools_svc_4180 form. Checks ran from a pod outside the mesh (standing in for a VPN client) and from pods inside it.

1. The gateway has no public address:

$ kubectl -n tools-gateway get svc -o wide
tools-gateway   ClusterIP   10.96.43.78   <none>   8443/TCP   2m52s   app=tools-gateway
$ kubectl get svc -A -o json | jq -r '... LoadBalancer or NodePort ...' | grep -E '^tools'
(none)

2. TLS per host, with the right certificate:

-servername grafana.tools.example.com  ->  X509v3 Subject Alternative Name: critical  DNS:grafana.tools.example.com
-servername argocd.tools.example.com   ->  X509v3 Subject Alternative Name: critical  DNS:argocd.tools.example.com

A plain HTTP request to port 8443 got no response at all (curl exit 56).

3. No tool answers without login:

$ curl -s -o /tmp/b -w '%{http_code}\n' https://grafana.tools.example.com:8443/
403
$ grep -o '<title>[^<]*</title>' /tmp/b
<title>Sign In</title>

oauth2-proxy answered with its sign-in page. With --skip-provider-button it redirects straight to the provider instead:

302 https://github.com/login/oauth/authorize?approval_prompt=force&client_id=lab-client&redirect_uri=https%3A%2F%2Fgrafa...

Either way, never the tool.

4. No side door from inside the mesh. A pod in another meshed namespace calling the tool directly, and a pod with the SSO proxy's identity:

with the mesh's default allow-all:
  app/sidedoor (another app in the mesh)   -> 200
  pod with the sso-proxy identity          -> 200
allow-all removed:
  app/sidedoor (another app in the mesh)   -> 403
  pod with the sso-proxy identity          -> 200
  VPN client via the gateway               -> 403 <title>Sign In</title>

With an allow-all MeshTrafficPermission in place, the permissions on this page change nothing: every app in the mesh walks straight past the SSO proxy. Many meshes carry one from a quickstart or a migration. Check with kubectl get meshtrafficpermissions -A and remove it before relying on the gateway.

#Mistakes people make

#Keeping the default LoadBalancer

serviceType defaults to LoadBalancer. For internal tools, set ClusterIP and add a private path to it.

#Trusting each tool's own login

Tool logins differ in strength and in bugs. Put one identity-provider login, with MFA, in front of all of them, and keep the tools' own logins as a second layer.

#Leaving the old Ingress in place

The gateway is only a control if it is the only way in. Search for Ingress, LoadBalancer, NodePort and HTTPRoute objects that still point at the tools.

#A wildcard certificate on the gateway

A leaked wildcard key impersonates every host in the zone. Issue one certificate per tool hostname.

#Letting every mesh service call the tools

Without permissions, any compromised pod in the mesh reaches Grafana directly, skipping the gateway and the login. Allow only the SSO proxy.

#Checklist

• The MeshGatewayInstance uses serviceType: ClusterIP or an internal load balancer.
• Tool hostnames resolve only on private DNS.
• Each tool has its own HTTPS listener and certificate; no plain HTTP listener.
• Certificates are Kuma Secrets in kuma-system, not inline.
• Every route goes to the SSO proxy, not to a tool.
• A MeshTrafficPermission allows only the gateway to call the SSO proxy.
• A MeshTrafficPermission allows only the SSO proxy to call the tools.
• No Ingress, LoadBalancer or NodePort Service points at a tool.
• An unauthenticated request gets the SSO login, never a tool page.