Linux for Defenders 09: What Is This Machine Connected To? Network, DNS and Listening Ports

Start with the two views every defender reads first: the interfaces (is the cable in?) and the addresses on them.

$ ip -br link
$ ip -br addr

Straight off the machine this lab was built on:

$ ip -br link
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0@if6         UP             ea:27:27:67:c3:48 <BROADCAST,MULTICAST,UP,LOWER_UP>
$ ip -br addr
lo               UNKNOWN        127.0.0.1/8
eth0@if6         UP             172.17.0.2/16

Your names and numbers will differ: a VMware or VirtualBox VM usually shows ens33 or enp0s3 and a 192.168.x.x address. The columns mean the same thing everywhere: the interface, whether it is up, and its address with the network size after the slash (the /16 here is the subnet mask you met in the subnetting lab).

Why this matters. Every other check in this lab hangs off this list. An interface you do not recognise (a second card, a VPN tun0, a bridge someone left behind) is the first sign that traffic can leave by a door you did not know about.

The routing table decides where every packet goes. The line that matters most is default: anything not on a local network leaves that way.

$ ip route
$ ip route get 1.1.1.1

$ ip route
default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2
$ ip route get 1.1.1.1
1.1.1.1 via 172.17.0.1 dev eth0 src 172.17.0.2 uid 0
    cache

ip route get is the most useful network command almost nobody knows: give it any address and it tells you exactly which gateway, which interface and which source address this machine would use to reach it. No guessing from the table.

Why this matters. A second default route, or a route for one network through an unexpected interface, is how traffic quietly goes somewhere else: a VPN that leaks, or a tunnel an attacker added. One default line, through the gateway you expect, is the healthy answer.

Every connection by name starts with a DNS question. On Ubuntu the question goes to a small local resolver first, and it decides which real DNS servers to ask.

$ ls -l /etc/resolv.conf
$ grep -v '^#' /etc/resolv.conf
$ resolvectl status | head -5

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Oct 21 15:30 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
$ grep -v '^#' /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search .
$ resolvectl status | head -5
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
       DNS Servers: 8.8.8.8 8.8.4.4

127.0.0.53 is the local stub (systemd-resolved); the real upstream servers are on the DNS Servers line. resolv.conf mode: stub is the normal Ubuntu setup. If /etc/resolv.conf is a plain file instead of that link, something has taken over name resolution, which is worth knowing even when it is harmless (a VPN client, a container runtime).

Why this matters. Whoever answers your DNS decides where your machine connects. A changed DNS server line is one of the quietest ways to send every login page to the wrong place.

Before any DNS server is asked, the machine checks /etc/hosts. Add one fake line and see what each tool believes. 203.0.113.66 is a documentation address that belongs to nobody.

$ sudo cp /etc/hosts /root/hosts.bak
$ echo '203.0.113.66 example.com' | sudo tee -a /etc/hosts
$ getent hosts example.com
$ dig +short example.com A
$ dig +short @8.8.8.8 example.com A

$ getent hosts example.com
203.0.113.66    example.com
$ dig +short example.com A
203.0.113.66
$ dig +short @8.8.8.8 example.com A
104.20.23.154
172.66.147.243

Every program on the box now goes to the fake address, ping and browsers included. The surprise is the middle one: even dig is fooled, because on Ubuntu it asks the local stub, and systemd-resolved reads /etc/hosts too. Only a question sent straight to an outside server (@8.8.8.8) gets the real answer. That comparison is the check. Put the file back:

$ sudo cp /root/hosts.bak /etc/hosts
$ grep -vE '^\s*(#|$)' /etc/hosts

127.0.0.1	localhost
172.17.0.2	labvm

Why this matters. Malware has edited the hosts file for decades to block security updates or send a bank's name to a copy of its site. It needs no network access and leaves no DNS logs. Two or three lines (localhost, the machine's own name) is normal; anything else needs a reason.

Lab 04 showed what a firewall lets in. This is the other half: which programs are waiting for connections, and on which address.

$ sudo ss -tulpn

Netid State  Local Address:Port  Process
udp   UNCONN    127.0.0.53%lo:53  users:(("systemd-resolve",pid=1662,fd=14))
tcp   LISTEN    127.0.0.53%lo:53  users:(("systemd-resolve",pid=1662,fd=15))
tcp   LISTEN          0.0.0.0:22  users:(("sshd",pid=163,fd=3),("systemd",pid=1,fd=73))
tcp   LISTEN          0.0.0.0:25  users:(("master",pid=513,fd=13))

(Columns trimmed to fit.) Read the address column: 127.0.0.53 and 127.0.0.1 mean only this machine can connect; 0.0.0.0 means every network it is attached to. SSH on all addresses is expected. But port 25 is Postfix, a mail server (its main process is called master), and on this box it was installed earlier and forgotten, open to the whole network. It only needs to send this machine's own mail, so tie it to loopback:

$ postconf inet_interfaces
inet_interfaces = all
$ sudo postconf -e 'inet_interfaces = loopback-only'
$ sudo systemctl restart postfix
$ sudo ss -tlnp 'sport = :25'
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      100        127.0.0.1:25        0.0.0.0:*    users:(("master",pid=2584,fd=13))

Why this matters. A firewall rule can be removed by mistake; a service that only listens on loopback cannot be reached from the network at all. Fixing the bind address is the stronger fix, and the firewall becomes the second layer.

On Ubuntu, permanent network settings live in YAML files in /etc/netplan/. Add a small test interface: a bridge, because it needs no spare network card and nothing else uses it. 192.0.2.10 is another documentation address.

$ sudo tee /etc/netplan/60-lab.yaml <<'EOF'
network:
  version: 2
  renderer: networkd
  bridges:
    br-lab:
      addresses:
        - 192.0.2.10/24
EOF
$ sudo netplan generate

** (generate:1739): WARNING **: Permissions for /etc/netplan/60-lab.yaml are too open. Netplan configuration should NOT be accessible by others.

Netplan files can hold Wi-Fi passwords and VPN keys, so netplan wants them readable by root only. Fix the permissions, then apply:

$ sudo chmod 600 /etc/netplan/60-lab.yaml
$ sudo netplan apply
$ ip -br addr show br-lab
br-lab           DOWN           192.0.2.10/24

The interface is there with its address (DOWN only because nothing is plugged into the bridge). On a VM you run over SSH, never use netplan apply for a real interface first. Use netplan try, which is the next step.

Why this matters. Every change to a server's network is one typo from locking you out. Files under /etc/netplan/ are also the place to look when an address changes after a reboot and nobody admits to it.

netplan try applies a change and undoes it after a countdown unless you press Enter. Make a deliberate mistake and do not press anything:

$ sudo cp /etc/netplan/60-lab.yaml /root/60-lab.good
$ sudo sed -i 's#192.0.2.10/24#198.51.100.7/24#' /etc/netplan/60-lab.yaml
$ sudo netplan try --timeout 15

Do you want to keep these settings?
Press ENTER before the timeout to accept the new configuration
Changes will revert in 15 seconds
...
Changes will revert in  1 seconds
Reverting.

While the countdown ran, br-lab really had 198.51.100.7. Afterwards it is back on 192.0.2.10. Now look at the file:

$ ip -br addr show br-lab
br-lab           DOWN           192.0.2.10/24
$ grep -A1 addresses /etc/netplan/60-lab.yaml
      addresses:
        - 198.51.100.7/24

On this machine (netplan 1.1.2) the live network was rolled back, but the edited file stayed. The next netplan apply, or the next reboot, would bring the bad address straight back. Always check the file after a revert, and put it back yourself:

$ sudo cp /root/60-lab.good /etc/netplan/60-lab.yaml
$ sudo chmod 600 /etc/netplan/60-lab.yaml
$ sudo netplan apply

Why this matters. A safety net that only half-catches you is worse than none if you trust it fully. "It reverted" and "it is fixed" are two different checks.

Put everything from steps 1 to 5 into one snapshot you can compare later. Leave out process IDs (they change every restart), so ss -tuln instead of -tulpn:

$ snap() { ip -br addr; ip route; ss -tuln | sort; resolvectl dns; grep -vE '^\s*(#|$)' /etc/hosts; }
$ snap | sudo tee /root/net-baseline.txt > /dev/null

Now change two things the way an intruder might: add the fake line from step 4 to /etc/hosts again, and add a second address (- 192.0.2.99/24) under addresses in the netplan file, then sudo netplan apply. Take a second snapshot and compare:

$ snap | diff /root/net-baseline.txt -
3c3
< br-lab           DOWN           192.0.2.10/24
---
> br-lab           DOWN           192.0.2.10/24 192.0.2.99/24
18a19
> 203.0.113.66 example.com

Both changes, and nothing else. Undo them (restore the hosts file and the good netplan file, then netplan apply) and run the same line again: no output means you are back on the baseline.

Why this matters. This is the network half of the Lab 08 health check. Add the snap line to that script, and a new listener, a new address or a hosts-file edit shows up in the morning report instead of in an incident.