Linux for Defenders 11: Run Containers Without Handing Out Root (Rootless Podman and Quadlet)

Before installing anything, check the classic problem. With Docker, the daemon runs as root, and Docker's own documentation warns that the docker group grants root-level privileges. Anyone in that group can, in practice, do anything root can. So the first audit line is:

$ getent group docker
$ echo $?

$ getent group docker
$ echo $?
2

No output and exit code 2 means there is no docker group on this machine. On a box that has one, every name after the last colon is effectively an administrator, exactly like the sudo group in Lab 01, and should be on the same list.

Why this matters. "Just add yourself to the docker group so you don't need sudo" is one of the most copied tips on the internet, and it quietly creates admins nobody tracks. Rootless containers remove the problem instead of managing it.

Podman runs containers with no background daemon, and as a normal user. It needs a block of spare user IDs for each user, which useradd reserves in /etc/subuid and /etc/subgid:

$ sudo apt install podman
$ podman --version
podman version 4.9.3
$ grep alex /etc/subuid /etc/subgid
/etc/subuid:alex:165536:65536
/etc/subgid:alex:165536:65536

That line says: user alex may use 65,536 user IDs starting at 165536 inside their containers. Now, as alex (log in as alex, or ssh alex@localhost, so you get a full session), ask Podman how it is running:

$ podman info --format 'rootless={{.Host.Security.Rootless}}'
rootless=true

Why this matters. If a user has no /etc/subuid line, rootless containers fail in confusing ways. Accounts created before Podman was installed, or by scripts that bypass useradd, are the usual culprits.

Still as alex, start a tiny container and ask it who it is:

$ podman run --rm docker.io/library/busybox:latest id
uid=0(root) gid=0(root) groups=10(wheel)

Inside, the process is root. That is normal: plenty of software expects to be root in its container. The question a defender asks is what that "root" really is on the machine. Leave one running so you can look at it from outside:

$ podman run -d --name web2 docker.io/library/busybox:latest sleep 600

Why this matters. "The container runs as root" sounds alarming in an audit finding. Whether it actually is depends entirely on the next step.

From a second terminal, look at the same process as the host sees it, and ask Podman to show both identities side by side:

$ ps -o user,pid,cmd -C sleep
USER         PID CMD
alex        5334 sleep 600
$ podman top web2 user huser pid hpid
USER        HUSER       PID         HPID
root        1001        1           5334
$ cat /proc/5334/uid_map
         0       1001          1
         1     165536      65536

To the machine, it is just alex (UID 1001). podman top shows the mapping: root inside is host UID 1001 outside, process 1 inside is process 5334 outside. The uid_map file is the kernel's record of it: container UID 0 maps to 1001, and container UIDs 1 and up map to alex's reserved block starting at 165536. Use your own PID from ps in the cat line.

Why this matters. This mapping is the whole security model. If something breaks out of this container, it lands as an ordinary user with no admin rights, not as root.

Mount the host's /etc into the container, read-only, and try to read the password hashes. First as alex (rootless), then as root (rootful) for comparison:

$ podman run --rm -v /etc:/host-etc:ro docker.io/library/busybox:latest cat /host-etc/shadow
cat: can't open '/host-etc/shadow': Permission denied
$ sudo podman run --rm -v /etc:/host-etc:ro docker.io/library/busybox:latest sh -c 'cut -d: -f1 /host-etc/shadow | head -1'
Trying to pull docker.io/library/busybox:latest...
root

Same image, same command, opposite result. Rootless "root" gets Permission denied, because to the kernel it is alex. Rootful root reads the file (only the first user name is printed here, never the hash). Notice too that the sudo run had to pull the image again: root and each user keep completely separate image stores.

Why this matters. Mounting host folders is the most common way a container gets more than it should. With rootless, a careless mount is a nuisance instead of a breach.

This is the defender's catch. With web2 still running as alex, look from root:

$ sudo podman ps -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

Empty. Root's podman ps only shows root's own containers. Each user's containers live in their own home folder. So an inventory has to go user by user:

$ loginctl list-users --no-legend
1001 alex yes lingering
$ sudo ls -d /home/*/.local/share/containers
/home/alex/.local/share/containers
$ for u in $(loginctl list-users --no-legend | awk '{print $2}'); do echo "-- $u"; sudo -iu "$u" podman ps --format '{{.Names}} {{.Image}}'; done
-- alex
web2 docker.io/library/busybox:latest

lingering means alex's services keep running after they log out (set with loginctl enable-linger, step 8), which is how a rootless container survives a reboot. Stop web2 now: podman stop web2.

Why this matters. If your asset list only comes from sudo podman ps or docker ps, every user's containers are invisible to it. Add the per-user loop, and the containers folders, to the Lab 08 health check.

Even rootless, a container gets a default set of Linux capabilities. Compare the default with none at all, then test two lock-downs:

$ podman run --rm docker.io/library/busybox:latest grep CapEff /proc/self/status
CapEff:	00000000800405fb
$ podman run --rm --cap-drop=all docker.io/library/busybox:latest grep CapEff /proc/self/status
CapEff:	0000000000000000
$ podman run --rm --cap-drop=all docker.io/library/busybox:latest chown nobody /etc/hostname
chown: /etc/hostname: Operation not permitted
$ podman run --rm --read-only docker.io/library/busybox:latest touch /evil
touch: /evil: Read-only file system

CapEff is a bitmap of the capabilities the process can use: the default has several (changing file owners, binding ports and so on); --cap-drop=all makes it zero. --read-only stops anything writing to the container's own files, so a dropped web shell or a changed binary has nowhere to go.

Why this matters. Least privilege, applied to software. Most simple services need none of the default capabilities, and a read-only container can still write to the folders you deliberately give it.

Containers you want running all the time belong in systemd, not a terminal. Podman's Quadlet turns a short file into a proper service. As alex:

$ mkdir -p ~/.config/containers/systemd
$ cat > ~/.config/containers/systemd/web.container <<'EOF'
[Unit]
Description=Lab web page (rootless)

[Container]
Image=docker.io/library/busybox:latest
Exec=sh -c "echo hello from a rootless container > /tmp/index.html && exec busybox httpd -f -p 8080 -h /tmp"
PublishPort=127.0.0.1:8080:8080
ReadOnly=true
DropCapability=ALL
NoNewPrivileges=true

[Install]
WantedBy=default.target
EOF
$ systemctl --user daemon-reload
$ systemctl --user start web
$ systemctl --user is-active web
active

Every lock-down from step 7 is one line, plus NoNewPrivileges (nothing inside can gain privileges through set-UID programs) and a port published on 127.0.0.1 only. Check it from the host and from inside:

$ curl -s 127.0.0.1:8080
hello from a rootless container
$ ss -tlnp 'sport = :8080'
LISTEN 0      4096       127.0.0.1:8080      0.0.0.0:*    users:(("rootlessport",pid=5092,fd=5))
$ podman exec systemd-web sh -c 'touch /evil; grep CapEff /proc/1/status; id'
touch: /evil: Read-only file system
CapEff:	0000000000000000
uid=0(root) gid=0(root)
$ loginctl enable-linger alex

If the service fails with a cgroup "permission denied" error. The lab box runs inside a container with the older cgroup v1, so it needed one extra line under [Container]: PodmanArgs=--cgroups=disabled. A normal Ubuntu 24.04 or RHEL 9 VM uses cgroup v2 and does not need it. A rootless user also normally cannot publish ports below 1024 (sysctl net.ipv4.ip_unprivileged_port_start prints 1024), which is why the page is on 8080.

Why this matters. This is what "container hardening" looks like on a real server: a non-root owner, no capabilities, a read-only image, loopback-only exposure, and systemd restarting it. Every line is checkable, which is exactly what an auditor will do.