Lab 09 — Read your own logs: a first-response triage with journalctl
0 / 6 steps
The goal
Investigate your own Linux machine the way an analyst would: find out what logs you have, who used admin rights, whether any accounts changed, what is running and listening, and turn it all into a short timeline. No extra tools — just what ships with Linux.
Your machine only. Everything here runs on the Kali VM from Lab 00 and only reads its own logs. The two small test actions (a wrong password, a throwaway account) are made on purpose on a box you own.
Why this lab. A SIEM like Wazuh or Splunk is only a faster way of asking questions of logs. Asking them by hand, on one machine, is how you learn which questions matter — and it is exactly what you do when the SIEM is down or the machine was never connected to it.
Before you start. Open a terminal on your Kali VM. Several commands use sudo because security logs are readable only by admins — that is by design.
Steps
01Know what you are reading
Modern Linux keeps its logs in the systemd journal, and journalctl is how you read it. Start by seeing how much history there is — every boot the journal remembers:
$ journalctl --list-boots
Each line is one boot, newest at the bottom. -b means “this boot”, -b -1 the one before. If you only see one line, the journal is being kept in memory and wiped at every reboot — worth knowing before you ever rely on it.
Why start here. The first question in any investigation is “what evidence do I actually have?” Logs you assumed existed but didn’t survive a reboot are one of the most common dead ends in real incident response.
02Find who used sudo — and who failed
Every sudo command is logged: who ran it, from where, and what they ran. Failures are logged too. Show today’s:
$ sudo journalctl _COMM=sudo --since today
Lines with COMMAND= are successful runs. Lines with authentication failure or incorrect password attempts are the failures. If the list is empty, make one on purpose: run sudo -k && sudo true, type a wrong password, and look again.
Why this matters. Privileged commands are the first thing an investigator reads, because they show what someone did with admin rights. A burst of failures followed by a success is a pattern every analyst learns to spot.
03Spot account changes
Creating, deleting or changing accounts leaves its own entries, tagged with the program that did it. Search the last week for all of them at once:
$ sudo journalctl -t useradd -t userdel -t usermod -t passwd --since "7 days ago"
Nothing back is a good sign on a lab VM. To see what a hit looks like, create and remove a throwaway account — sudo useradd -M sg-logtest && sudo userdel sg-logtest — and run the search again.
Why this matters. A new account is one of the classic ways someone keeps access to a machine after getting in. “Were any accounts created or changed?” is on every incident checklist for exactly that reason.
04See what is running and listening
Logs tell you what happened; the system tells you what is happening now. List the services that are running:
$ systemctl list-units --type=service --state=running
Then see which programs are listening for network connections, and on which ports:
$ sudo ss -tulpn
Read the Local Address:Port column: 127.0.0.1 means only this machine can connect; 0.0.0.0 or * means anything that can reach it can. The last column names the process.
Why this matters. An unexpected service or an unexpected listening port is often the first concrete clue that a machine has been changed. You can only spot “unexpected” if you know what normal looks like — so look now, while it is clean.
05Cut the noise with priorities
The journal is loud. Every entry carries a priority, from emergencies down to debug chatter. Show only errors and worse from this boot:
$ sudo journalctl -p err -b
Swap err for warning to widen the net. Add -u and a service name — -u ssh, say — to look at one service only.
Why this matters. Triage is mostly deciding what not to read. Filtering by severity, service and time is the same skill a SIEM query does for you at scale — learn it by hand first and the SIEM makes much more sense.
06Build a timeline
Put the last hour into a file with exact, sortable timestamps, then pull out just the security-relevant lines:
$ sudo journalctl --since "1 hour ago" -o short-iso > timeline.txt
$ grep -Ei "sudo|useradd|userdel|session opened|authentication failure" timeline.txt
You now have a short, time-ordered story of what happened on this machine in the last hour. Read it top to bottom and check it matches what you actually did.
Why this matters. Every incident report is built around a timeline. Being able to say “at 14:02 a sudo failure, at 14:03 a success, at 14:05 a new account” — with evidence for each line — is what turns a hunch into a finding.
You have checked what evidence exists, read privileged activity, looked for account changes, mapped running services and open ports, filtered by severity, and built a timeline. That is a first-response triage of a Linux host, done with nothing but built-in tools.
If it didn’t work
- “No journal files were found” or only this boot shows. The journal is in memory only. Make it persistent with sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald; history starts from then.
- You see “Hint: You are currently not seeing messages from other users and the system”. You left off sudo. Security entries are admin-only.
- The account search returns nothing even after the test. Check the spelling of the -t names, and widen the time with --since "1 hour ago". Some systems log account changes under a different tag — sudo journalctl --since "1 hour ago" | grep -i user will find them either way.
- ss shows no process names. Run it with sudo; without it, Linux hides other users’ processes.
- The timeline file is empty. Nothing matched in the last hour. Do one of the test actions again, or change 1 hour ago to today.
Your turn — no walkthrough
Answer three questions about your own VM using only the commands from this lab and what you learned in Lab 03 (grep, sort, uniq): when did this machine last boot; how many sudo commands ran today, and which was run most; and which service logged the most errors this boot.
Keep the evidence. For each answer, save the exact command that produced it next to the result. An answer without the command behind it is an opinion; with it, anyone can check your work — which is the whole point of evidence.
Clean up
Delete the timeline with rm timeline.txt (or keep it in your lab notes). Make sure the test account is gone: id sg-logtest should say “no such user”.
Before you leave — prove it
Without scrolling up: one, what does it mean if --list-boots shows only one boot? Two, in ss output, what is the difference between a service on 127.0.0.1 and one on 0.0.0.0? Three, why do investigators build a timeline instead of just listing what they found?
If the second answer came easily, you already think like someone hardening a server.
What you learned
- journalctl --list-boots — checking what evidence exists before you rely on it
- _COMM= and -t — filtering the journal to one program
- systemctl and ss -tulpn — what is running now, and what is listening
- -p err, -u, --since — cutting noise by severity, service and time
- -o short-iso — exporting a sortable timeline
Put it on your resume. Performed first-response triage on a Linux host with journalctl (admin use, account changes, listening services) and wrote a timeline. Change it to match what you actually did.
Tried it? Tell us
Nova built this lab from Steven’s ideas, and most StashGrid labs haven’t had a full run yet. If a step broke, behaved differently for you, or you know a better way, tell us. Ideas for the next lab are welcome too — no promises on timing.
NextLab 10 — Test your detections in Wazuh→
Go deeper
- journalctl cheat sheet — every command from this lab and more, on one page
- Lab 08 — Watch your own machine with a Wazuh agent — the same questions, automated
- Set Up Splunk Free With Docker and the SPL cheat sheet — timelines and searches at SIEM scale
- Linux privilege escalation cheat sheet — the attacker’s view of the same machine
Original walkthrough. Every command is standard systemd, iproute2 and coreutils usage on a Debian-based system. Distributions differ in the details — if something here does not match what your terminal does, tell me and I will fix it.