Hunting secrets on a host, the secure way

#What goes wrong

Secrets spread. A developer tests a connection by typing the password on the command line, and the shell saves it. A deploy writes a .env file with mode 644. A service gets its API key through an environment variable, which any process running as the same user, or root, can read from /proc. A key is committed to a repository, then deleted in the next commit, and stays in the history forever.

Attackers know all of these places. After getting a foothold, the first thing most of them do is search for exactly these files, because one database password is often worth more than the host.

The goal of a hunt is to find them first, rotate them, and move them to one place with tight permissions.

#What the docs say

This file contains the initial environment that was set when the currently executing program was started via execve(2).

Source: proc_pid_environ(5)

The dir (aliases include files, directory) command lets you scan directories and files.

Source: gitleaks README

Once the secret is revoked or rotated, it can no longer be used for access, and that may be sufficient to solve your problem.

Source: GitHub Docs, Removing sensitive data from a repository

The GitHub page gets the order right: rotate first. Scanners find patterns, not meaning; they miss passwords that look like normal words, and they flag things that are not secrets. The hunt needs both tools and a person reading the results.

#The secure configuration

Run the hunt as root, and write findings to a file only root can read. Redact values in any report that leaves the host.

umask 077
# 1. Files that are usually secret stores, with mode and owner.
find / -xdev \( -name ".env" -o -name "*.pem" -o -name "*_key" -o -name "id_*" \
  -o -name "config.json" -o -name ".git-credentials" -o -name ".pgpass" -o -name "*.kdbx" \) \
  -type f -not -path "/proc/*" -not -path "/usr/*" -printf "%m %u %p\n" 2>/dev/null | sort -k3

# 2. Private keys by content, wherever they are named.
grep -rIl --exclude-dir={proc,sys,usr,dev} -e "BEGIN OPENSSH PRIVATE KEY" -e "BEGIN RSA PRIVATE KEY" \
  -e "BEGIN EC PRIVATE KEY" -e "BEGIN PRIVATE KEY" / 2>/dev/null

# 3. Passwords and tokens typed on command lines.
grep -HnE -- "(-p[^ ]{4,}|--password[= ]|PASSWORD=|token=)" /root/.*history /home/*/.*history 2>/dev/null

# 4. Secrets in the environment of running processes.
for e in /proc/[0-9]*/environ; do
  tr "\0" "\n" < "$e" 2>/dev/null | grep -E "^[A-Z_]*(KEY|TOKEN|SECRET|PASSWORD)[A-Z_]*=" | sed "s|^|$(dirname "$e" | cut -d/ -f3): |"
done | sort -u

# 5. Pattern scan of the places people write to (gitleaks; verify the release checksum first).
for d in /srv /home /root /opt /etc; do gitleaks dir "$d" --redact --no-banner --report-path "/root/gitleaks-$(basename "$d").json"; done

# 6. Every git repository on the host, including history.
find / -xdev -type d -name .git -not -path "/proc/*" 2>/dev/null | while read -r g; do
  gitleaks git "$(dirname "$g")" --redact --no-banner --report-path "/root/gitleaks-git-$(echo "$g" | tr / _).json"
done

For each finding:

1. Decide if it is real and still valid.
2. Rotate it at the source (database, cloud console, token issuer).
3. Remove it from the host (and from git history if the repository is shared).
4. Store the new value in a secret store or a root-owned 0600 file read by the service.
5. Check the logs of the system the secret protected for use you do not recognize.

#Prove it

The test planted fake secrets in six places. The name search finds three files, with their modes. Two of them are readable by everyone (644):

600 root /home/deploy/.ssh_backup_key
644 root /root/.docker/config.json
644 root /srv/app/.env

Private keys by content, and a password typed on a command line:

/home/deploy/.ssh_backup_key
/home/deploy/.bash_history:1:mysql -u root -pHunter2-not-real -h db.example.net

A process started with a key in its environment. Anyone who is root, or the same user, can read it for as long as the process runs:

3445: API_KEY=sk_live_not_real_0123456789

gitleaks on /srv, /home and /root, reduced to rule and file:

for d in /srv /home /root; do gitleaks dir "$d" --redact --no-banner --no-color --report-path /tmp/gl.json; grep -E '"(RuleID|File)"' /tmp/gl.json | paste - -; done

 "RuleID": "github-pat",	 "File": "/srv/app/.env",
 "RuleID": "private-key",	 "File": "/home/deploy/.ssh_backup_key",
 "RuleID": "generic-api-key",	 "File": "/root/.docker/config.json",

gitleaks did not flag the database password in /srv/app/.env (postgres://app:Hunter2-not-real@...); the name search and a person reading the file did. And the git repository: the current settings.ini reads the key from an environment variable, but the first commit still holds the real one:

gitleaks git /opt/tool --redact --no-banner --no-color --report-path /tmp/gl-git.json
grep -E '"(RuleID|Commit|Message)"' /tmp/gl-git.json | paste - - -

WRN leaks found: 1
 "RuleID": "aws-access-token",	 "Commit": "fe1cf35",	 "Message": "add settings",

The test script is secure-tests/hunting-secrets-host/run.sh. All the secrets in it are fake.

#Mistakes people make

#Deleting the file and calling it fixed

The secret was readable for as long as the file existed, and it may be in backups, logs and git history. Rotate it. Deleting comes after.

#Only scanning the current tree

A commit that removes a key does not remove it from history. Scan with gitleaks git, and rotate anything it finds, even in old commits.

#Printing secrets in the report

A hunt that writes every secret into a world-readable report creates a new leak. Use --redact, umask 077, and keep raw findings on the host.

#Trusting the scanner alone

Pattern scanners miss plain passwords and custom token formats. Read the files the name search finds.

#Secrets in environment variables of long-running services

They show up in /proc/<pid>/environ, crash dumps and debug endpoints. Prefer a file with mode 0600 owned by the service user, or a secret store the service reads at start.

#Checklist

• The name search, content search, history search and /proc search have been run as root.
• gitleaks has scanned /srv, /home, /root, /opt and /etc, with --redact.
• Every git repository on the host has been scanned including history.
• Every real finding has been rotated at its source.
• Secret files are owned by the service user and have mode 0600 or tighter.
• Shell histories with secrets have been cleaned, and the people involved know why.
• No service receives long-lived secrets through command-line arguments.
• The hunt is repeated after incidents and on a schedule.