Linux for Defenders 02: What’s Running That You Didn’t Start? systemd Services

Ask systemd for every service in the running state. This is the live picture — what is executing this second.

$ systemctl list-units --type=service --state=running

On the machine this lab was built on, the running list was short and every name was explainable:

cron.service          Regular background program processing daemon
dbus.service          D-Bus System Message Bus
rsyslog.service       System Logging Service
ssh.service           OpenBSD Secure Shell server
systemd-journald.service   Journal Service
systemd-logind.service     User Login Management

You should be able to say, in one sentence, why each one is there. The description column helps. A name you cannot explain is the thread you pull.

Why this matters. You are building the “known good” picture of your own machine. You will never recognise the odd one out until the ordinary ones are familiar — which is exactly why an attacker names their service something that blends in.

A service that accepts connections has an open port. ss -tulpn lists every listening TCP and UDP socket and — the important part — the program behind each one.

$ sudo ss -tulpn | grep LISTEN
tcp LISTEN 0 100       0.0.0.0:25   0.0.0.0:* users:(("master",pid=389))
tcp LISTEN 0 4096      0.0.0.0:22   0.0.0.0:* users:(("sshd",pid=819))
tcp LISTEN 0 4096 127.0.0.53%lo:53  0.0.0.0:* users:(("systemd-resolve",pid=77))

That is real output. Read each line as port → program: :22 is sshd, :25 is the mail master, :53 on 127.0.0.53 is the local DNS stub. The 0.0.0.0 addresses listen to the whole network; the 127.x ones listen only to the machine itself.

The attacker’s view. A backdoor usually listens on a port so it can be reached. 0.0.0.0 on a high, unexplained port is the shape of a planted listener.

The defender’s move. Every listening port should map to a service you can name. An open port with a program you do not recognise is the finding — the pid in that same line is how you chase it down.

“Running now” and “starts at boot” are different questions. A service can be enabled to start every reboot even while it is stopped right now. Ask for the enabled list.

$ systemctl list-unit-files --type=service --state=enabled
cron.service          enabled
fail2ban.service      enabled
rsyslog.service       enabled
ssh.service           enabled
...

Why this matters. Persistence is about surviving a reboot, so the enabled list is where it has to show up. A service can be stopped this moment and still be set to launch itself the next time the machine starts. Check both lists — running and enabled — because the dangerous one may only be in the second.

Now play the other side. Create a service with an official-sounding name that just sleeps forever and is set to restart and to start at boot — the skeleton of real Linux persistence (MITRE ATT&CK technique T1543.002).

$ sudo tee /etc/systemd/system/sysupdate.service >/dev/null <<'EOF'
[Unit]
Description=System Update Helper
[Service]
ExecStart=/bin/bash -c "while true; do sleep 3600; done"
Restart=always
[Install]
WantedBy=multi-user.target
EOF
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now sysupdate

It is now running and set to come back every reboot. In the running list from step 1 it hides in plain sight:

$ systemctl list-units --type=service --state=running | grep sysupdate
sysupdate.service   loaded active running System Update Helper

The attacker’s view. “System Update Helper” reads as something the OS ships. Restart=always brings it back if killed; WantedBy=multi-user.target brings it back on reboot. Real malware runs a payload where this one sleeps.

Two ways to catch it. First, read what a service actually runs — not its friendly name, but its command and its file on disk.

$ systemctl show sysupdate -p ExecStart -p FragmentPath
ExecStart={ path=/bin/bash ; argv[]=/bin/bash -c while true; do sleep 3600; done ... }
FragmentPath=/etc/systemd/system/sysupdate.service

The friendly name said “update helper”; the ExecStart says it runs a bash loop, and systemctl cat sysupdate prints the whole file. Name and behaviour do not match — that mismatch is the tell. Second way: the planted file is newer than everything the OS shipped, so sort the service directory by time.

$ ls -lt --time-style=+%Y-%m-%d\ %H:%M /etc/systemd/system/*.service | head -3
-rw-rw-rw- 1 root root 159 2026-10-20 18:56 /etc/systemd/system/sysupdate.service
lrwxrwxrwx 1 root root  35 2026-10-20 18:55 sshd.service -> .../ssh.service

Newest at the top, and it is the one you planted. On a machine installed months ago, a single service file dated yesterday is worth a very close look.

Why this matters. “What does it run” beats “what is it called” every time. A planted service lies in its name and its description; it cannot lie in its ExecStart or its file timestamp.

Removing a service is three steps in order: stop it, disable it so it will not come back at boot, then delete its file and reload systemd. Skip the disable and it returns on the next reboot.

$ sudo systemctl stop sysupdate
$ sudo systemctl disable sysupdate
$ sudo rm /etc/systemd/system/sysupdate.service
$ sudo systemctl daemon-reload
$ systemctl status sysupdate
Unit sysupdate.service could not be found.

That last line — “could not be found” — is your proof it is gone, running and at boot. That before (step 4) and after is the evidence for a ticket.

Why this matters. Order is the whole lesson. Delete the file without disabling first and systemd keeps a dangling enable symlink; stop it without deleting the file and it is back next boot. Stop, disable, delete, reload — every time.