Log review with journalctl, the secure way

#What goes wrong

Three problems stop people from answering "what happened on this host?".

The journal may not survive a reboot. With Storage=auto, journald keeps logs in /run (memory) unless /var/log/journal exists. After a crash or a reboot, the evidence is gone.

People trust the text of a log line. A local program can write a message that claims to come from sshd or sudo. The MESSAGE and SYSLOG_IDENTIFIER fields are whatever the sender said.

An attacker with root can edit log files. Without a way to detect changes, you cannot tell a clean journal from an edited one.

#What the docs say

Fields prefixed with an underscore are trusted fields, i.e. fields that are implicitly added by the journal and cannot be altered by client code.

Source: systemd.journal-fields(7), Trusted Journal Fields

Note that the journal service does not validate the values of any structured journal fields whose name is not prefixed with an underscore

Source: systemd.journal-fields(7)

Forward Secure Sealing (FSS) for all persistent journal files is enabled. FSS is based on Seekable Sequential Key Generators by G. A. Marson and B. Poettering (doi:10.1007/978-3-642-40203-6_7) and may be used to protect journal files from unnoticed alteration.

Source: journald.conf(5), Seal=

The docs say sealing protects "from unnoticed alteration". They do not stress the operational part: the verification key printed by --setup-keys must leave the host. If it stays next to the journal, an attacker with root has everything needed.

#The secure configuration

# /etc/systemd/journald.conf.d/10-security.conf
[Journal]
# Keep logs on disk across reboots.
Storage=persistent
# Seal persistent journal files (needs a key from journalctl --setup-keys).
Seal=yes
# Bound disk use and age, so logs never fill the disk.
SystemMaxUse=2G
MaxRetentionSec=90day

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald
# Create the sealing key. The command prints a VERIFICATION key:
# store it off the host (password manager, offline), then clear the terminal.
journalctl --setup-keys --interval=15min

Daily review commands:

journalctl -b -p warning                         # this boot, warning and worse
journalctl -u ssh --since "24 hours ago"         # one unit ("sshd" on RHEL family)
journalctl SYSLOG_FACILITY=4 SYSLOG_FACILITY=10  # auth and authpriv, any program
journalctl -t sshd --grep "Failed password" -o cat | grep -oE "from [0-9a-f.:]+" | sort | uniq -c | sort -rn
journalctl _COMM=sudo --since today              # entries really written by the sudo binary
journalctl -o verbose -n 1 -t sudo               # every field of one entry
journalctl --verify --verify-key=<key from your vault>

Only the adm and systemd-journal groups can read everyone's logs. Keep those groups small. Send a copy to a central log store too (systemd-journal-upload, rsyslog, or an agent), because a root attacker can still delete the whole journal, and sealing only tells you afterwards.

#Prove it

Auth events (facility 4 auth, facility 10 authpriv) with ISO timestamps:

journalctl --no-pager -o short-iso SYSLOG_FACILITY=4 SYSLOG_FACILITY=10

2026-09-24T22:10:29+0000 web-1 sshd[431]: Failed password for invalid user admin from 203.0.113.9 port 51237 ssh2
2026-09-24T22:10:29+0000 web-1 sshd[432]: Failed password for invalid user oracle from 203.0.113.9 port 51244 ssh2
2026-09-24T22:10:29+0000 web-1 sshd[433]: Failed password for invalid user test from 203.0.113.9 port 51251 ssh2
2026-09-24T22:10:29+0000 web-1 sshd[434]: Failed password for invalid user ubuntu from 203.0.113.9 port 51258 ssh2
2026-09-24T22:10:29+0000 web-1 sshd[435]: Failed password for invalid user admin from 203.0.113.9 port 51265 ssh2
2026-09-24T22:10:29+0000 web-1 sshd[436]: Accepted publickey for alice from 198.51.100.20 port 50022 ssh2: ED25519 ...
2026-09-24T22:10:29+0000 web-1 sudo[437]:    alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/cat /etc/shadow

Failed logins counted by source, and everything at warning or worse:

journalctl --no-pager -o cat -t sshd --grep "Failed password" | grep -oE "from [0-9.]+" | sort | uniq -c
journalctl --no-pager -o short -p warning

      5 from 203.0.113.9
Sep 24 22:10:29 web-1 app[439]: disk almost full

The "sudo" line looks real. Its trusted fields say otherwise: _COMM=logger. The test wrote every line above with logger, and journald recorded that. A real sudo entry has _COMM=sudo:

journalctl --no-pager -o verbose -t sudo | grep -E "^ +(PRIORITY|SYSLOG_FACILITY|SYSLOG_IDENTIFIER|_PID|_UID|_COMM|MESSAGE)="

    _UID=0
    _COMM=logger
    PRIORITY=5
    SYSLOG_FACILITY=10
    SYSLOG_IDENTIFIER=sudo
    MESSAGE=   alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/cat /etc/shadow
    _PID=437

A user outside adm and systemd-journal sees nothing from the system:

su bob -c "journalctl --no-pager -t sshd | tail -2"

Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.

Sealing, with a 10-second interval so the test does not wait 15 minutes. The verification passes, then one word in the file is changed and it fails:

journalctl --verify --verify-key="$(cat /root/fss-verification-key.txt)"
# edit "alice" to "mallo" inside the Accepted publickey entry, then:
journalctl --verify --verify-key="$(cat /root/fss-verification-key.txt)"

PASS: /var/log/journal/<machine-id>/system.journal
=> Validated from Thu 2026-09-24 22:10:27 UTC to Thu 2026-09-24 22:10:30 UTC, final 25.006862s entries not sealed.
3920f8: Invalid object contents: Bad message
File corruption detected at /var/log/journal/<machine-id>/system.journal:3920f8 (of 8388608 bytes, 44%).
FAIL: /var/log/journal/<machine-id>/system.journal (Bad message)

This edit was caught by the per-object hash, which a careful attacker could recompute. The seal is the part they cannot recompute without the verification key. Note final 25.006862s entries not sealed: entries newer than the last seal are not yet protected. The test (verification key kept in the container only for the test) is secure-tests/log-review-journalctl/run.sh.

#Mistakes people make

#Volatile logs on servers

Check with ls /var/log/journal. If the directory is missing and Storage=auto, every reboot erases the evidence.

#Believing the identifier

-t sshd matches what the sender claimed. For evidence, filter on trusted fields: _COMM=sshd, _EXE=/usr/sbin/sshd, _SYSTEMD_UNIT=ssh.service.

#Keeping the verification key on the host

A root attacker who has it can edit and re-seal. Store it off the host, and run --verify from a trusted copy of the logs.

#Only local logs

Sealing detects edits. It does not bring back deleted files. Ship logs to a central store that the host cannot delete from.

#Everyone in adm

Logs contain user names, IP addresses, and sometimes tokens that programs logged by mistake. Treat read access to the journal like read access to a database.

#Checklist

• /var/log/journal exists and Storage=persistent is set.
• SystemMaxUse and MaxRetentionSec match your retention policy.
• journalctl --setup-keys was run and the verification key is stored off the host.
• journalctl --verify with the key passes, and runs on a schedule.
• Only admins are in the adm and systemd-journal groups.
• Logs are forwarded to a central store.
• Reviews filter on trusted _ fields for anything used as evidence.
• The host clock is synchronized, so timestamps line up with other hosts.