Loki multi-tenancy, the secure way

#What goes wrong

Loki separates data by tenant. With auth_enabled: true, every write and every query names a tenant in the X-Scope-OrgID header, and Loki keeps each tenant's streams apart. That part works well.

What Loki does not do is check who is asking. The header is not a secret and not a credential. Any client that can reach Loki's HTTP port can send X-Scope-OrgID: finance and read the finance team's logs, or write fake log lines into their tenant.

So the security of a multi-tenant Loki is the security of whatever sits in front of it. If that is nothing, there is no isolation, only labels.

#What the docs say

Grafana Loki does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services.

Source: Grafana Loki docs, Manage authentication

When auth_enabled: true, a request that does not include the X-Scope-OrgID header fails with an HTTP 401 Unauthorized error and the message no org id.

Source: Grafana Loki docs, Manage tenant isolation

Specify multiple tenants in the query request HTTP header X-Scope-OrgID by separating the tenant IDs with the pipe character (|).

Source: Grafana Loki docs, Manage tenant isolation

The docs say "you must" run a proxy. They do not say what the proxy must do with a header the client already sent. If it passes the client's header through, the proxy authenticates the user and then lets them pick any tenant.

#The secure configuration

Loki, with tenants on and cross-tenant queries off:

# loki.yaml: single-binary Loki with tenants turned on.
auth_enabled: true            # every request must carry X-Scope-OrgID; data is stored per tenant
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  # Defaults for every tenant; tighten or loosen per tenant in the runtime file.
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 8
  max_query_series: 5000
  retention_period: 720h
querier:
  # Keep cross-tenant queries ("tenant-a|tenant-b") off. This is the default; set it explicitly.
  multi_tenant_queries_enabled: false
runtime_config:
  file: /etc/loki/overrides.yaml

# /etc/loki/overrides.yaml
# Per-tenant limits, reloaded at run time.
overrides:
  tenant-b:
    ingestion_rate_mb: 1
    retention_period: 168h

The gateway authenticates each caller, maps the user to one tenant, and overwrites the header:

# nginx.conf: the only way into Loki. It authenticates the caller and SETS the tenant header.
events {}
http {
  # Credentials file: one user per tenant (htpasswd format).
  map $remote_user $loki_tenant {
    default   "";
    team-a    "tenant-a";
    team-b    "tenant-b";
  }
  server {
    listen 8080;
    location /loki/api/ {
      auth_basic           "loki";
      auth_basic_user_file /etc/nginx/htpasswd;
      # A user with no mapping gets an empty value: nginx then drops the header and Loki answers 401.
      # Overwrite whatever the client sent. Never pass a client-supplied tenant through.
      proxy_set_header X-Scope-OrgID $loki_tenant;
      proxy_pass http://loki:3100;
    }
    location / { return 404; }
  }
}

Then make the gateway the only way in:

• Do not publish Loki's port. In Kubernetes, use a ClusterIP Service and a network policy that allows port 3100 only from the gateway pods (and from Loki's own components).
• Give log shippers (Alloy, Promtail, Fluent Bit) their own credentials at the gateway, one per tenant, instead of letting them set the header.
• For Grafana, create one data source per tenant with its own gateway credentials, and restrict data source access by team.

#Prove it

One line written into each tenant, straight to Loki:

push as tenant-a: HTTP 204
push as tenant-b: HTTP 204

Without a tenant header Loki refuses the request:

curl -s http://loki:3100/loki/api/v1/labels

no org id
 (HTTP 401)

Each tenant sees only its own line, and a cross-tenant query is refused because multi_tenant_queries_enabled is off:

curl -s -G -H "X-Scope-OrgID: tenant-a" --data-urlencode 'query={job="app"}' http://loki:3100/loki/api/v1/query_range
curl -s -G -H "X-Scope-OrgID: tenant-b" --data-urlencode 'query={job="app"}' http://loki:3100/loki/api/v1/query_range
curl -s -G -H "X-Scope-OrgID: tenant-a|tenant-b" --data-urlencode 'query={job="app"}' http://loki:3100/loki/api/v1/query_range

tenant-a: tenant-a: invoice 1001 paid
tenant-b: tenant-b: password reset for bob
multiple org IDs present (HTTP 500)

The trap. Nothing stops a client from naming another tenant:

someone claiming tenant-b: tenant-b: password reset for bob

Through the gateway: no credentials and a wrong password are refused, and a team-a user who sends X-Scope-OrgID: tenant-b still gets tenant-a data, because nginx replaced the header:

curl -s -o /dev/null -w "%{http_code}" http://gateway:8080/loki/api/v1/labels
curl -s -o /dev/null -w "%{http_code}" -u team-a:guess http://gateway:8080/loki/api/v1/labels
curl -s -G -u team-a:*** --data-urlencode 'query={job="app"}' http://gateway:8080/loki/api/v1/query_range
curl -s -G -u team-a:*** -H "X-Scope-OrgID: tenant-b" --data-urlencode 'query={job="app"}' http://gateway:8080/loki/api/v1/query_range

no credentials: HTTP 401
wrong password: HTTP 401
team-a: tenant-a: invoice 1001 paid
team-a sending X-Scope-OrgID: tenant-b: tenant-a: invoice 1001 paid

The query output above is reduced to the log line by the test script, secure-tests/loki-multi-tenancy/run.sh.

#Mistakes people make

#Treating the header as authentication

X-Scope-OrgID is a label on the request, chosen by the sender. Only the gateway's credential check is authentication.

#A proxy that passes the header through

proxy_pass forwards client headers by default. Always set the header from the authenticated identity, so a client value is overwritten.

#Loki reachable from the whole cluster

A gateway that anyone can walk around is a suggestion. Block direct access to port 3100 with a network policy or firewall.

#Turning on multi-tenant queries for convenience

multi_tenant_queries_enabled: true lets one request read several tenants. If you need it for a central security team, give that team its own gateway credential and log its use.

#One shared shipper credential

If every log shipper uses the same credential and sets its own tenant, one compromised node can write into every tenant. One credential per tenant.

#Checklist

• auth_enabled: true is set.
• multi_tenant_queries_enabled is false, or its use is limited and logged.
• Loki's HTTP and gRPC ports are not reachable except from the gateway and Loki's own components.
• The gateway authenticates every request.
• The gateway sets X-Scope-OrgID from the authenticated identity and overwrites any client value.
• Each tenant has its own shipper credential and its own Grafana data source.
• Per-tenant limits are set in the runtime overrides file.
• A test sends a forged X-Scope-OrgID through the gateway and checks it is ignored.