Secrets and PKI

Never put a secret on the command line, the secure way

You typed the password as an argument because the tool allowed it, and it worked. It also worked for every other user on the box who ran `ps` in that second.

The short answer

By default, any user on a Linux host can read every process's arguments through `ps` and `/proc/<pid>/cmdline`, and your shell saves what you type. Pass secrets through a 0600 file, a file descriptor or stdin, read them with `read -rs`, keep them out of history, and mount `/proc` with `hidepid=invisible`.

Updated Houssam Hammoudi, CTOTested with Alpine 3.22, OpenSSL 3.5, curl 8, bash 5.2, procps-ng

On this page
  1. What goes wrong
  2. What the docs say
  3. The secure configuration
  4. Prove it
  5. Mistakes people make
  6. Checklist

What goes wrong

Process arguments are public on Linux. Any user can list them with ps or read /proc/<pid>/cmdline, with no special rights. A password passed as -p secret, pass:secret or user:secret is readable by every account on the host while the program runs.

The same line goes into your shell history file, where it stays for months and gets copied with your home directory.

Environment variables are better, but not private in the way people think. Every child process inherits them, crash reports and debug endpoints dump them, and docker inspect shows a container's.

Monitoring agents, audit logs and CI logs often record full command lines. The secret then leaves the host too.

What the docs say

Since the password is visible to utilities (like 'ps' under Unix) this form should only be used where security is not important.

Source: OpenSSL docs, openssl-passphrase-options

This is not enough to protect credentials from possibly getting seen by other users on the same system as they still are visible for a moment before being cleared.

Source: curl man page, -u, --user

If the list of values includes ‘ignorespace’, lines which begin with a space character are not saved in the history list.

Source: GNU Bash manual, Bash Variables: HISTCONTROL

As an additional bonus, as /proc//cmdline is unaccessible for other users, poorly written programs passing sensitive information via program arguments are now protected against local eavesdroppers.

Source: Linux kernel docs, The /proc filesystem: mount options

Each project documents its own piece. The kernel docs mark hidepid=off as the default, so every argument on the host is public unless someone changed the mount. The OpenSSL and curl pages warn about process listings, but not about the shell history, which keeps the line after the process is gone.

The secure configuration

Read the secret from a file only its owner can read:

bash
umask 077
mkdir -p ~/.config
${EDITOR:-vi} ~/.config/backup.pass            # type it in the editor, not the shell

openssl enc -aes-256-cbc -pbkdf2 -pass file:"$HOME/.config/backup.pass" -in data.tar -out data.tar.enc
curl -K ~/.config/api.curlrc https://api.example.com/   # the file holds: user = "admin:..."

Or read it once without echo and hand it over on a file descriptor or stdin:

bash
#!/bin/bash
# use-pass.sh: nothing secret in argv, nothing in history.
read -rs -p "passphrase: " P; echo
# printf is a bash builtin, so its arguments never appear in ps.
openssl enc -aes-256-cbc -pbkdf2 -pass fd:3 -in data.tar -out data.tar.enc 3< <(printf '%s\n' "$P")
unset P

Tools with a stdin option, used the same way:

bash
printf '%s' "$TOKEN" | docker login --username ci --password-stdin registry.example.com
bao login -method=userpass username=alice        # prompts; never password=... on the line

For services, use systemd credentials instead of Environment=:

ini
# /etc/systemd/system/app.service.d/secret.conf
[Service]
LoadCredential=db-password:/etc/app/db-password    # file mode 0600, owner root
# The service reads $CREDENTIALS_DIRECTORY/db-password

Keep typed lines out of history, and hide other users' processes:

bash
# ~/.bashrc
HISTCONTROL=ignoreboth        # ignorespace + ignoredups: a leading space skips history
conf
# /etc/fstab: other users cannot see your processes or their arguments
proc  /proc  proc  defaults,hidepid=invisible  0  0

Daemons that must inspect other users' processes, such as a process monitoring agent, need an exception. The kernel docs describe gid= for this: put the daemon's user in a group and add gid=<group> to the mount options.

Prove it

From secure-tests/never-put-secret-command-line/. Alice runs a command with the passphrase as an argument. Bob is a separate unprivileged user:

bash
sleep 5 | openssl enc -aes-256-cbc -pbkdf2 -pass pass:example-hunter2 -out backup.enc &   # alice
ps -eo user,args | grep [o]penssl                                                         # bob
tr "\0" " " < /proc/$(pgrep openssl)/cmdline                                              # bob
text
alice    openssl enc -aes-256-cbc -pbkdf2 -pass pass:example-hunter2 -out backup.enc
openssl enc -aes-256-cbc -pbkdf2 -pass pass:example-hunter2 -out backup.enc

Some tools overwrite their arguments after they start. curl does, so one second later bob sees stars. Before that overwrite, the argument is as public as openssl's, and your shell history still has it.

text
alice    curl -s -u ********************* --max-time 8 https://198.51.100.1/api

The environment is readable by its owner only, and by any process that owner runs:

bash
cat /proc/$(pgrep sleep)/environ                            # bob
tr "\0" "\n" < /proc/$(pgrep sleep)/environ | grep TOKEN    # alice
text
cat: can't open '/proc/<pid>/environ': Permission denied
API_TOKEN=example-token-42

History, with and without ignorespace. The second line was typed with a leading space:

text
$ cat ~/.bash_history   # HISTCONTROL unset
export DB_PASSWORD=example-typed-secret
 export API_KEY=example-leading-space
exit
$ cat ~/.bash_history   # HISTCONTROL=ignorespace
export DB_PASSWORD=example-typed-secret
exit

The file and file-descriptor versions. Bob sees a path or a number, and cannot read the file:

bash
ps -eo user,args | grep [o]penssl; cat /home/alice/.config/backup.pass   # bob
ps -eo user,args | grep -E "[o]penssl|[u]se-pass"                          # bob, during use-pass.sh
text
alice    openssl enc -aes-256-cbc -pbkdf2 -pass file:/home/alice/.config/backup.pass -out backup.enc
cat: can't open '/home/alice/.config/backup.pass': Permission denied
alice    /bin/bash /tmp/use-pass.sh
alice    openssl enc -aes-256-cbc -pbkdf2 -pass fd:3 -out /home/alice/backup.enc

With /proc mounted hidepid=invisible, the bad command is hidden from bob, but not from root:

bash
grep " /proc " /proc/mounts
ps -eo user,args | grep [o]penssl; echo "exit $?"   # bob
ps -eo user,args | grep [o]penssl                   # root
text
proc /proc proc rw,nosuid,nodev,noexec,relatime,hidepid=invisible 0 0
exit 1
alice    openssl enc -aes-256-cbc -pbkdf2 -pass pass:example-hunter2 -out backup.enc

Mistakes people make

Trusting a tool that hides its arguments

curl and a few others overwrite argv once they start. The value is public until then, and it is still in history, audit logs and CI logs. Use the file or stdin option.

export SECRET=... typed at the prompt

The line goes into history like any other. Use read -rs SECRET and export SECRET on its own, or a leading space with HISTCONTROL=ignorespace.

echo "$SECRET" | tool in a POSIX sh script

In bash and busybox sh, echo and printf are builtins and do not create a process. If a script calls /bin/echo or /usr/bin/printf explicitly, the secret becomes an argument again.

Secrets in Environment= or a Dockerfile ENV

They show up in systemctl show, docker inspect and image history. Use LoadCredential=, a mounted secret file, or your orchestrator's secret store.

Treating hidepid as the fix

It hides processes from other users. Root, your monitoring agent and the history file still see everything. It is a second line, not the first.

Cleaning history and forgetting the copies

Deleting a line from ~/.bash_history does not remove it from backups, other open shells (which write their own history on exit), or central logging. If a secret was typed, rotate it.

Checklist

  • Search scripts and runbooks for -p, --password=, pass:, -u user: and token= arguments.
  • Replace each with a file, stdin or file-descriptor option.
  • Create secret files with umask 077 and check they are mode 0600.
  • Use read -rs to take a secret interactively.
  • Set HISTCONTROL=ignoreboth in shell profiles.
  • Mount /proc with hidepid=invisible and a gid= for daemons that need it.
  • Use systemd LoadCredential= instead of Environment= for service secrets.
  • Rotate any secret that has been typed as an argument or found in history.

If `ps` can print it, so can everyone else. Keep secrets in files and pipes, where Unix permissions can actually do their job.

FND

Learn it on a live range

Linux 3: securing Linux, in Foundation: a real host in your browser, and every objective checked on the machine.

Start free

The Secure Way

More on secrets and pki

OpenBao, External Secrets, internal certificate authorities and keeping secrets off the command line.

All secrets and pki guides