Linux for Defenders 13: Disks, fstab and a Backup You Have Actually Restored

You need a spare disk to format and mount without risking the system disk. The cleanest way is to add a 1 GB virtual disk to the VM in VMware or VirtualBox (it shows up as /dev/sdb). If you can't, make a file that behaves like a disk. The outputs below use the file method:

$ sudo truncate -s 1G /srv/lab-disk.img
$ sudo losetup --find --show /srv/lab-disk.img
/dev/loop0
$ lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS /dev/loop0
NAME  SIZE TYPE FSTYPE MOUNTPOINTS
loop0   1G loop

losetup attaches the file as a block device and prints its name. On Ubuntu, snaps already use several loop devices, so yours may well be /dev/loop7 or similar: use the name it printed everywhere this lab says /dev/loop0. If you added a real virtual disk instead, use its name (/dev/sdb, /dev/vdb or /dev/nvme0n2 depending on the hypervisor), and check the name with lsblk first: formatting the wrong disk is the one mistake in this lab you cannot undo.

Why this matters. Every storage task starts with "which disk is which?" lsblk answers it with sizes and mount points, so you never guess from a device name.

Create a filesystem with a label, then ask for its identity:

$ sudo mkfs.ext4 -q -L labdata /dev/loop0
$ sudo blkid /dev/loop0
/dev/loop0: LABEL="labdata" UUID="e0565f4f-dff4-4c54-b825-7e061318cb61" BLOCK_SIZE="4096" TYPE="ext4"

The UUID is the filesystem's permanent ID; yours will be different. Device names like /dev/sdb can change between boots when disks are added or reordered. The UUID does not, which is why /etc/fstab should always use it.

Why this matters. "The server came up with the data disk mounted in the wrong place" is almost always an fstab that used /dev/sdX names. UUIDs prevent it.

Back up /etc/fstab, then add one line for the new disk, using your own UUID from step 2:

$ sudo cp /etc/fstab /etc/fstab.bak
$ sudo mkdir -p /srv/data
$ sudo nano /etc/fstab

Add this as the last line:

UUID=e0565f4f-dff4-4c54-b825-7e061318cb61  /srv/data  ext4  defaults,nofail,nodev,nosuid,noexec  0  2

The columns are: what to mount, where, filesystem type, options, a legacy dump flag (0), and the check order at boot (2 = after the root disk). The options are the security part: nodev (no device files), nosuid (set-UID programs lose their power here), noexec (programs on this disk cannot be started directly). nofail tells systemd not to stop the boot if this disk is missing, which matters for the file method because the loop disk is not set up again after a reboot. Now check the file before anything depends on it:

$ sudo systemctl daemon-reload
$ sudo findmnt --verify
Success, no errors or warnings detected
$ sudo mount -a
$ findmnt /srv/data
TARGET    SOURCE     FSTYPE OPTIONS
/srv/data /dev/loop0 ext4   rw,nosuid,nodev,noexec,relatime

Why this matters. A data disk holds files users upload or copy in. There is no reason anything on it should run as a program or act as a device, so these three options remove whole classes of abuse for free. CIS benchmarks require them on /tmp, /var/tmp and /dev/shm for the same reason.

Here is the mistake that sends a server to the emergency console after a reboot (for a line without nofail; with nofail the boot waits about 90 seconds, then carries on with the disk silently not mounted, and anything written to /srv/data lands on the root disk instead). Make a copy of fstab with one wrong character in the UUID and check it:

$ sed 's/e0565f4f/e0565f4e/' /etc/fstab > /tmp/fstab.typo
$ sudo findmnt --verify --tab-file /tmp/fstab.typo

0 parse errors, 1 error, 0 warnings
/srv/data
   [E] unreachable on boot required source: UUID=e0565f4e-dff4-4c54-b825-7e061318cb61

Use your own first eight UUID characters in the sed line and change the last one. Run findmnt --verify with sudo: without it, it cannot read the disks and adds "Permission denied" warnings. findmnt --verify caught it: no disk with that UUID exists. Make it a habit to run it after every fstab edit, and before every reboot. If you ever forget systemctl daemon-reload, it also warns that systemd is still using the old version of the file.

Why this matters. A bad fstab line is one of the most common self-inflicted outages on Linux, and on a cloud server with no console it can mean rebuilding the machine. One command prevents it.

Put a tiny script on the disk and try to run it two ways:

$ printf '#!/bin/sh\necho hello from data\n' | sudo tee /srv/data/hello.sh > /dev/null
$ sudo chmod +x /srv/data/hello.sh
$ sudo /srv/data/hello.sh
sudo: unable to execute /srv/data/hello.sh: Permission denied
$ sh /srv/data/hello.sh
hello from data

Running it directly fails, even as root, because the mount says noexec. Handing it to an interpreter (sh, bash, python3) works, because then the program being started is /usr/bin/sh, which lives on a normal disk.

Why this matters. noexec stops drive-by execution (a downloaded binary, a malicious upload run by path), not a determined attacker. Write it in a hardening report as "reduces", never as "prevents".

Put some realistic files on the disk, record a checksum of every file, then archive the whole folder (tar records owners and permissions; they come back when root extracts it):

$ sudo mkdir -p /srv/data/reports /srv/data/config
$ for m in jan feb mar; do echo "quarterly numbers for $m" | sudo tee /srv/data/reports/$m.csv > /dev/null; done
$ echo "db_host=127.0.0.1" | sudo tee /srv/data/config/app.conf > /dev/null
$ sudo chmod 600 /srv/data/config/app.conf
$ sudo chown alex:alex /srv/data/reports/mar.csv
$ sudo ln -s reports/mar.csv /srv/data/latest.csv
$ cd /srv/data && sudo find . -type f -exec sha256sum {} + | sort -k2 | sudo tee /root/data-manifest.sha256 > /dev/null
$ sudo mkdir -p /backup
$ sudo tar --exclude=lost+found -cpzf /backup/data-2026-10-23.tar.gz -C /srv data
$ ls -l /backup/
total 4
-rw-r--r-- 1 root root 385 Oct 23 13:33 data-2026-10-23.tar.gz

Look at those permissions: -rw-r--r-- means every user on the machine can read the archive, including config/app.conf, which was 600 (owner only) on the disk. A backup quietly undoes the file permissions you set. Fix it (the file first: once /backup is 700, your own shell can no longer list it, so a * in the path would fail), and record a checksum of the archive itself:

$ sudo chmod 600 /backup/data-2026-10-23.tar.gz
$ sudo chmod 700 /backup
$ sudo sh -c 'sha256sum /backup/data-2026-10-23.tar.gz > /backup/data-2026-10-23.tar.gz.sha256'

tar option order. Put --exclude before the folder names. Written after them (... -C /srv data --exclude=lost+found), GNU tar 1.35 refuses: "The following options were used after non-option arguments", and exits with an error.

Why this matters. Backups are a favourite target exactly because they collect everything in one readable file. An attacker who cannot read app.conf may well be able to read last night's archive.

Simulate the bad day. This scrambles the three reports into .locked files and deletes the originals, the way ransomware leaves a folder:

$ cd /srv/data/reports && for f in *.csv; do sudo sh -c "base64 $f > $f.locked && rm $f"; done
$ ls /srv/data/reports
feb.csv.locked
jan.csv.locked
mar.csv.locked
$ cat /srv/data/latest.csv
cat: /srv/data/latest.csv: No such file or directory

Now restore, in the right order: check the archive is intact, restore it somewhere safe first, prove every file is right, and only then put it back:

$ sudo sha256sum -c /backup/data-2026-10-23.tar.gz.sha256
/backup/data-2026-10-23.tar.gz: OK
$ sudo mkdir -p /root/restore-test && sudo tar -xpzf /backup/data-2026-10-23.tar.gz -C /root/restore-test
$ sudo sh -c 'cd /root/restore-test/data && sha256sum -c /root/data-manifest.sha256'
./config/app.conf: OK
./hello.sh: OK
./reports/feb.csv: OK
./reports/jan.csv: OK
./reports/mar.csv: OK
$ sudo rsync -a --dry-run --itemize-changes /root/restore-test/data/reports/ /srv/data/reports/
>f+++++++++ feb.csv
>f+++++++++ jan.csv
>f+++++++++ mar.csv
$ sudo rsync -a /root/restore-test/data/reports/ /srv/data/reports/
$ sudo rm /srv/data/reports/*.locked
$ cat /srv/data/latest.csv
quarterly numbers for mar
$ ls -l /srv/data/reports
total 12
-rw-r--r-- 1 root root 26 Oct 23 13:33 feb.csv
-rw-r--r-- 1 root root 26 Oct 23 13:33 jan.csv
-rw-r--r-- 1 alex alex 26 Oct 23 13:33 mar.csv

Every checksum matches, mar.csv is owned by alex again (when root extracts, tar restores owners by default; -p on extract restores the exact permissions), and the latest.csv shortcut works because its target is back. The --dry-run --itemize-changes line showed exactly what rsync was about to write before it wrote anything (>f+++++++++ = new file).

Why this matters. A backup that has never been restored is a hope, not a backup. /root is readable only by root, which is why the check runs inside sudo sh -c. Restoring to a test folder first means a bad archive can never overwrite what little you still have.

Many people "back up" with rsync --delete to a second disk. Watch what happens to a mistake:

$ sudo rsync -a --delete /srv/data/ /backup/mirror/
$ sudo rm /srv/data/reports/jan.csv
$ sudo rsync -a --delete --dry-run --itemize-changes /srv/data/ /backup/mirror/
*deleting   reports/jan.csv
.d..t...... reports/
$ sudo rsync -a --delete /srv/data/ /backup/mirror/
$ sudo ls /backup/mirror/reports
feb.csv
mar.csv

The mirror faithfully copied the mistake: jan.csv is gone from both. The dated archive still has it, and you can pull back a single file:

$ sudo tar -tzf /backup/data-2026-10-23.tar.gz data/reports/jan.csv
data/reports/jan.csv
$ sudo tar -xpzf /backup/data-2026-10-23.tar.gz -C /srv data/reports/jan.csv
$ ls /srv/data/reports
feb.csv
jan.csv
mar.csv

Dated archives need a retention rule, or the backup disk fills. Preview which ones are old before deleting anything (the lab box had copies dated 10, 14 and 20 October). -mtime +7 means more than seven whole days since the file was last modified, so in practice 8 days or older; it reads the file's own timestamp, not the date in its name, and copying an archive without cp -p resets that timestamp:

$ sudo find /backup -maxdepth 1 -name 'data-*.tar.gz' -mtime +7 -print
/backup/data-2026-10-14.tar.gz
/backup/data-2026-10-10.tar.gz

When the list is right, the same command with -delete instead of -print removes them. To finish, undo the lab. Leave the folder first (cd ~), or umount says "target is busy". Then sudo umount /srv/data, sudo cp /etc/fstab.bak /etc/fstab, sudo systemctl daemon-reload, and for the file method sudo losetup -d /dev/loop0 (your loop name) and sudo rm /srv/lab-disk.img. Delete the lab copies too: sudo rm -r /backup /root/restore-test /root/data-manifest.sha256 /etc/fstab.bak.

Why this matters. The 3-2-1 rule (three copies, two kinds of storage, one off the machine) exists because mirrors copy deletions and ransomware instantly. Versioned, dated copies are what let you go back to before the mistake.