Linux for Defenders 06: Packages You Can Trust — Updates, Signing and File Integrity

Every package arrives from a source listed in one folder. Read it.

$ ls /etc/apt/sources.list.d/
ubuntu.sources
$ grep -vE '^\s*(#|$)' /etc/apt/sources.list.d/ubuntu.sources
Types: deb
URIs: http://archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Types: deb
URIs: http://security.ubuntu.com/ubuntu/
Suites: noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Two sources: the main archive and the security archive, which carries only security fixes (noble-security). The line that matters most is Signed-By. apt checks every package list against that key, so a mirror cannot slip you a modified package without the signature failing. Ubuntu 24.04 uses this newer one-block-per-source format; older systems keep the same information as one-line entries in /etc/apt/sources.list.

Why this matters. Every file in sources.list.d/ is someone you trust to run code as root on your machine. A repository you did not add — or one with a key you do not recognise — is a finding. On a clean Ubuntu, this folder holds exactly one file.

Refresh the package lists, then ask what has a newer version and where that version comes from.

$ sudo apt update
$ apt list --upgradable
Listing...
tzdata/noble-updates,noble-security 2026e-0ubuntu0.24.04.1 all [upgradable from: 2024a-2ubuntu1]

One package is behind. Read the line left to right: the package, the sources offering the new version (noble-security among them, so it counts as a security update), the new version, and the version you have. Our test VM was fully patched, so to have something to show we rolled the time-zone package back to its original release with sudo apt install --allow-downgrades tzdata=2024a-2ubuntu1. Do the same if your VM is up to date too.

The attacker’s view. A package that is behind on a security fix is a known, published weakness — anyone can read what it fixes. The defender’s move: “which packages have a security update waiting” is the first question in any patch review, and apt list --upgradable | grep security answers it.

Ubuntu ships unattended-upgrades for this. Check it is switched on, ask what it would do, then let it run.

$ cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
$ systemctl list-timers 'apt-daily*' --no-pager
NEXT                        LEFT UNIT                    ACTIVATES
Wed 2026-10-21 06:32:51 UTC  11h apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2026-10-21 16:36:05 UTC  21h apt-daily.timer         apt-daily.service
$ sudo unattended-upgrade --dry-run -v 2>&1 | grep -E 'Allowed origins|will be upgraded'
Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, ...
Packages that will be upgraded: tzdata
$ sudo unattended-upgrade -v
All upgrades installed
$ apt list --upgradable
Listing...

The two "1" lines mean “refresh the lists daily, install daily”, and the two timers from the cron and timers lab do the work. Allowed origins is the safety rule: by default only the security archive is installed automatically, so a big feature update never lands unannounced. After the real run, nothing is waiting. What it did is logged in /var/log/unattended-upgrades/.

Why this matters. The gap between a fix being published and being installed is the window attackers use. Automatic security updates close it without anyone having to remember. If 20auto-upgrades is missing or says "0", turn it on with sudo dpkg-reconfigure -plow unattended-upgrades.

When you meet an unfamiliar program, the first question is whether a package put it there.

$ dpkg -S /usr/sbin/sshd
openssh-server: /usr/sbin/sshd
$ dpkg -S /usr/bin/w
procps: /usr/bin/w
$ dpkg -l openssh-server | tail -1
ii  openssh-server 1:9.6p1-3ubuntu13.20 amd64        secure shell (SSH) server, for secure access from remote machines
$ dpkg -S /usr/local/bin/nothing
dpkg-query: no path found matching pattern /usr/local/bin/nothing

The SSH server and the w command (who is logged in) both belong to packages, and dpkg -l gives the exact installed version — the number you compare against a security advisory. The last line is the interesting answer: no package owns that path.

Why this matters. A program in a system folder that no package owns did not arrive through apt. Sometimes that is an admin’s own script; sometimes it is something dropped by an attacker. Either way, it is worth a look — and /usr/local/, /opt/ and /tmp/ are where unowned programs usually live.

Play the attacker. Back up /usr/bin/w, then change it by adding a few harmless bytes to the end.

$ sudo cp /usr/bin/w /root/w.bak
$ echo '#tampered' | sudo tee -a /usr/bin/w >/dev/null
$ w >/dev/null && echo "w still runs"
w still runs

The program still works exactly as before, and its size went up by ten bytes that nobody would spot by eye. To every check you are about to run, “ten extra bytes” and “a different program” look the same: the file no longer matches what the package installed.

Why this matters. A changed system program is one of the clearest signs a machine has been tampered with, and you cannot see it by running the program. You need something that remembers what the file should look like.

dpkg kept a fingerprint of every file it installed. Check one package, then the whole system.

$ sudo dpkg --verify procps | grep -v '^missing'
??5??????   /usr/bin/w
$ sudo debsums -c procps
/usr/bin/w
$ time (sudo dpkg --verify | grep -v '^missing')
??5??????   /usr/bin/w

real	0m1.800s
$ md5sum /usr/bin/w; grep ' usr/bin/w$' /var/lib/dpkg/info/procps.md5sums
ce458fc071e5d8c3b1fb5d682f795d7c  /usr/bin/w
d7bc645e50af0617aad80ae84058d9d4  usr/bin/w

Caught, three ways. In ??5??????, the 5 means the file’s checksum no longer matches; the question marks are checks dpkg does not run. debsums -c lists changed files plainly. The whole-system check found the one changed file among every package in under two seconds. The last two lines show why: the file’s fingerprint now differs from the one dpkg recorded at install.

Read this before you rely on it. The grep -v '^missing' hides lines about deleted documentation, which minimal and cloud images strip on purpose; on a full desktop install you may not need it. And the fingerprints live on the same machine, so an attacker with root could edit them too. That is why the next lab builds a separate baseline with AIDE.

Do not copy your backup over it. Reinstall the package from the signed archive, then verify again.

$ sudo apt install --reinstall procps
Unpacking procps (2:4.0.4-4ubuntu3.3) over (2:4.0.4-4ubuntu3.3) ...
Setting up procps (2:4.0.4-4ubuntu3.3) ...
$ sudo dpkg --verify procps | grep -v '^missing'
$ md5sum /usr/bin/w
d7bc645e50af0617aad80ae84058d9d4  /usr/bin/w

No output from the verify is the result you want, and the fingerprint matches the recorded one again. You can delete /root/w.bak now.

Why reinstall. The reinstall pulls a fresh, signature-checked copy, so you know it is the real one. A backup you made on a machine you now suspect is only as trustworthy as the machine. On a real incident, a tampered system binary means you treat the whole machine as compromised and rebuild — but finding it is how you know.