Linux for Defenders 12: Who Did What as Root? (auditd, ausearch and aureport)

The Linux kernel can record security events itself: files opened, programs run, accounts changed. auditd is the service that collects those records and writes them to disk. Install it and ask the kernel for its status:

$ sudo apt install auditd
$ systemctl is-active auditd
active
$ sudo auditctl -s | head -3
enabled 1
failure 1
pid 2204
$ sudo auditctl -l
No rules

enabled 1 means the kernel is auditing and pid is the auditd process collecting it (your number will differ). failure 1 means that if auditing ever breaks, the kernel complains in the system log rather than halting the machine. No rules is the important line: out of the box, auditd records logins and a few system events, but nothing you would need in an investigation. The rules are your job.

Why this matters. "Do you have auditing on?" is a standard audit question, and "the service is running" is not the answer. With no rules, auditd is a camera with the lens cap on.

A watch tells the kernel to record any access of a chosen kind to a file. Start with the two files that define who can log in:

$ sudo auditctl -w /etc/passwd -p wa -k identity
$ sudo auditctl -w /etc/shadow -p rwa -k shadow
$ sudo auditctl -l
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p rwa -k shadow

-w is the file to watch. -p is which kinds of access to record: r read, w write, x execute, a attribute change (owner, permissions). -k is a label you choose, so you can search for it later. /etc/passwd is world-readable by design, so watching its reads would only make noise; /etc/shadow holds the password hashes, so even a read is worth recording.

Why this matters. Labels (-k) are what make audit logs usable. A real server writes thousands of records an hour; you will search by label, not by scrolling.

Log in as alex (the admin from Lab 01) and add a user, the way a helpdesk would:

$ sudo useradd -m bob
$ sudo ausearch -k identity --format text
At 13:24:48 10/23/26 alex, acting as root, successfully opened-file /etc/passwd using /usr/sbin/useradd
At 13:24:48 10/23/26 alex, acting as root, successfully renamed /etc/passwd+ to /home/alex/(null) using /usr/sbin/useradd

--format text turns each raw record into one readable sentence. Read the first line carefully: alex, acting as root. The program ran as root (it had to, to change /etc/passwd), but the kernel still knows the person who logged in was alex. That original login identity is called the audit UID (auid). It is set at login and survives sudo, su and sudo -i. Your output will also start with a line for the add_rule itself (trimmed here): changes to audit rules are audited too. The odd (null) path on the rename line is a display quirk of the text format on renames; sudo ausearch -k identity -i shows the full raw record with the real file names.

Why this matters. On a shared admin box, every change in the system log says "root". The audit UID is how an investigator answers "which person was root at 13:24?"

Still as alex, try to read the password hashes without sudo, then search for failures only:

$ cat /etc/shadow
cat: /etc/shadow: Permission denied
$ sudo ausearch -k shadow --success no --format text
At 13:24:48 10/23/26 alex unsuccessfully opened-file /etc/shadow using /usr/bin/cat

Blocked attempts are recorded too, and --success no filters to just those. Drop the filter and you will also see lines like alex successfully opened-file /etc/shadow using /usr/bin/sudo. That is normal: sudo reads the shadow file to check passwords. Knowing what normal looks like is half of reading an audit log.

Why this matters. A failed read of /etc/shadow by a normal user is rarely an accident. It is the kind of small signal that turns into a timeline when something bigger happens later.

Watches cover files. For commands, use a syscall rule. This one records every program started with root's power (euid=0) by someone who logged in as a real user:

$ sudo auditctl -a always,exit -F arch=b64 -S execve -F euid=0 -F 'auid>=1000' -F 'auid!=unset' -k root-cmds
$ sudo id
uid=0(root) gid=0(root) groups=0(root)
$ sudo ls /root
$ sudo ausearch -k root-cmds --format text
At 13:25:17 10/23/26 alex successfully executed /usr/bin/sudo
At 13:25:17 10/23/26 alex, acting as root, successfully executed /usr/bin/id
At 13:25:17 10/23/26 alex successfully executed /usr/bin/sudo
At 13:25:17 10/23/26 alex, acting as root, successfully executed /usr/bin/ls
$ sudo ausearch -k root-cmds -i | grep EXECVE | tail -1
type=EXECVE msg=audit(10/23/26 13:25:17.259:155) : argc=2 a0=ls a1=/root

Piece by piece: -a always,exit adds a rule checked when a syscall finishes, -S execve is the "start a program" syscall, arch=b64 is 64-bit programs, euid=0 means it ran with root's power, auid>=1000 limits it to people (system accounts sit below 1000), and auid!=unset is needed as well because "unset" is stored as 4294967295, which would otherwise pass >=1000: it skips services that never logged in. The quotes matter: without them the shell reads > as "redirect to a file". The EXECVE record keeps the full command line, arguments included. sudo itself appears because it is a set-UID program and starts with root's power.

32-bit programs. On a 64-bit machine, a 32-bit program uses a different syscall table and slips past an arch=b64 rule. Hardened baselines add the same rule a second time with -F arch=b32.

Why this matters. This rule is closely related to the privileged-command rules in Linux compliance baselines (CIS and DISA STIG use variations of it). It also records the arguments, so you see what was done, not just that sudo ran.

Rules added with auditctl vanish at reboot. Permanent rules live in files under /etc/audit/rules.d/, and augenrules merges them into /etc/audit/audit.rules. Create the lab's file:

$ sudo nano /etc/audit/rules.d/50-lab.rules

Paste this in and save:

## StashGrid Linux for Defenders 12
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p rwa -k shadow
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=unset -k root-cmds

Check, load, and confirm:

$ sudo augenrules --check
/usr/sbin/augenrules: Rules have changed and should be updated
$ sudo augenrules --load
$ sudo auditctl -l
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p rwa -k shadow
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d -p wa -k sudoers
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -F key=root-cmds

--load also prints the kernel status a few times; that is normal. Two details in the list: auditctl shows unset as -1, and the trailing slash on the folder is dropped. Both mean the same thing. Files in rules.d load in name order, which is why the lab's file is 50- and the lock in step 8 is 99-.

Why this matters. An auditor checks the files, not your shell history. If it is not in rules.d, it is not your policy; it is something you did once.

Here is the kind of change audit rules exist for. As alex, give bob full root with no password, the way an attacker (or a rushed admin) might, without touching the main sudoers file:

$ sudo bash -c 'echo "bob ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/99-helpdesk'
$ sudo ausearch -k sudoers --format text | tail -1
At 13:25:39 10/23/26 alex, acting as root, successfully opened-file /etc/sudoers.d/99-helpdesk using bash
$ sudo ausearch -k root-cmds -i | grep EXECVE | tail -1
type=EXECVE msg=audit(10/23/26 13:25:39.559:189) : argc=3 a0=bash a1=-c a2=echo "bob ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/99-helpdesk

Two rules, two angles: the sudoers watch says which file changed and who did it; the root-cmds rule shows the exact line they wrote. Now get the overview the way you would on a real incident, then undo the change (your counts will differ):

$ sudo aureport -k --summary

Key Summary Report
===========================
total  key
===========================
16  shadow
12  root-cmds
3  identity
3  sudoers
$ sudo rm /etc/sudoers.d/99-helpdesk

Add --start today (or --start recent for the last ten minutes) to any ausearch to narrow it to a time window. sudo aureport -x --summary -i lists which programs ran most.

Why this matters. A file in sudoers.d grants admin rights without anyone editing the file people usually check. It is exactly the kind of persistence Lab 01's sudo -l audit would only catch later; auditd catches it the moment it is written, with a name attached.

An audit log is only useful if it is still there and still true. Three checks. First, what happens when the log fills up:

$ sudo grep -E '^(max_log_file|num_logs|max_log_file_action|space_left_action|disk_full_action) ' /etc/audit/auditd.conf
max_log_file = 8
num_logs = 5
max_log_file_action = ROTATE
space_left_action = SYSLOG
disk_full_action = SUSPEND

Ubuntu keeps five files of 8 MB each, rotating the oldest out, and suspends auditing (keeps the machine running) if the disk fills. Busy servers raise max_log_file; strict environments set disk_full_action = HALT. Second, who can read it:

$ sudo ls -l /etc/audit/auditd.conf /var/log/audit/audit.log
-rw-r----- 1 root root   881 Jul 13 20:10 /etc/audit/auditd.conf
-rw-r----- 1 root adm 72950 Oct 23 13:26 /var/log/audit/audit.log
$ sudo grep ^log_group /etc/audit/auditd.conf
log_group = adm
$ getent group adm
adm:x:4:ubuntu,syslog

The config file is readable by root only (root root), hence sudo. On Ubuntu, members of the adm group can read the audit log, which now contains full command lines. Put that group on the same admin list as sudo from Lab 01. Third, lock the rules so nobody can quietly delete them. Do this last, because after it you cannot change any rule until you reboot:

$ echo "-e 2" | sudo tee /etc/audit/rules.d/99-finalize.rules
-e 2
$ sudo augenrules --load
$ sudo auditctl -s | head -1
enabled 2
$ sudo auditctl -D
Error deleting rule (Operation not permitted)
There was an error while processing parameters

enabled 2 is immutable mode: even root cannot add or remove rules until the next boot, and because both files stay in rules.d, the machine comes back locked with the same rules after every reboot. To change rules later, delete 99-finalize.rules, edit, and reboot. To undo the lab: sudo rm /etc/audit/rules.d/50-lab.rules /etc/audit/rules.d/99-finalize.rules, sudo userdel -r bob, then reboot (or simply revert the snapshot).

Why this matters. An attacker with root who knows about auditd will try auditctl -D first. In immutable mode that fails, and changing the rules needs a reboot, which is noisy. It does not make the log untouchable: root can still stop auditd or edit files in /var/log/audit. That is why real setups also ship the log off the machine as it is written (see the Splunk lab).