Syslog and SNMP lab: build a monitoring server and see what it sees

dev (10.40.0.2) plays a network switch called switch01. nms (10.40.0.1) is the network monitoring server that collects from it.

# ip netns add nms; ip netns add dev
# ip link add mon0 netns nms type veth peer name eth0 netns dev
# ip -n nms addr add 10.40.0.1/24 dev mon0
# ip -n dev addr add 10.40.0.2/24 dev eth0
# ip -n nms link set lo up; ip -n nms link set mon0 up
# ip -n dev link set lo up; ip -n dev link set eth0 up
# mkdir -p /tmp/sglab

Write a tiny rsyslog config that listens on UDP port 514 (the standard syslog port) and writes everything it receives to one file, with the sender’s address and the message’s facility and severity on each line. Then start rsyslog with that config inside nms.

# cat > /tmp/sglab/collector.conf <<'EOF'
module(load="imudp")
input(type="imudp" address="10.40.0.1" port="514")
template(name="sgfmt" type="string" string="%timegenerated% %fromhost-ip% %syslogfacility-text%.%syslogseverity-text% %syslogtag% %msg%\n")
*.* action(type="omfile" file="/tmp/sglab/remote.log" template="sgfmt")
EOF
# ip netns exec nms rsyslogd -f /tmp/sglab/collector.conf -i /tmp/sglab/rsyslog.pid
# ip netns exec nms ss -ulnp

State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
UNCONN 0      0          10.40.0.1:514       0.0.0.0:*    users:(("rsyslogd",pid=123546,fd=3))

Why -f and -i? -f points this copy of rsyslog at the lab config instead of /etc/rsyslog.conf, and -i gives it its own PID file, so it runs alongside your machine’s normal rsyslog without touching it. UNCONN is normal for UDP: there are no connections, just a socket waiting for messages.

Start a capture on the collector (give it a second), then have switch01 send two alerts with logger, the same way a real switch sends syslog over UDP.

# ip netns exec nms tcpdump -n -l -A -i mon0 -c 1 'udp port 514' &
# ip netns exec dev logger -n 10.40.0.1 -P 514 -d -p local0.warning -t switch01 "port 3 link down"
# ip netns exec dev logger -n 10.40.0.1 -P 514 -d -p local0.err -t switch01 "fan 2 failed"
# cat /tmp/sglab/remote.log

IP 10.40.0.2.53113 > 10.40.0.1.514: SYSLOG local0.warning, length: 143
...<132>1 2026-10-27T12:30:19.264846-04:00 ... switch01 - - [timeQuality ...] port 3 link down
Oct 27 12:30:45 10.40.0.2 local0.warning switch01 port 3 link down
Oct 27 12:30:46 10.40.0.2 local0.err switch01 fan 2 failed

Decode the <132>. Every syslog message starts with a priority number: facility × 8 + severity. local0 is facility 16 and warning is severity 4, so 16 × 8 + 4 = 132. Severities run from 0 (emergency) to 7 (debug): 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info, 7 debug. Lower is worse, which trips people up in exams and in alert rules. The collector added the sender’s IP (10.40.0.2) on each line. (Our capture’s second “...” hides our test machine’s hostname; yours will show your own.)

Now send a message from the collector itself, pretending to be switch01 reporting a critical login.

# ip netns exec nms logger -n 10.40.0.1 -P 514 -d -p auth.crit -t switch01 "admin login from 10.40.0.2"
# tail -1 /tmp/sglab/remote.log

Oct 27 12:30:47 10.40.0.1 auth.crit switch01 admin login from 10.40.0.2

The attacker’s view. The collector accepted it without question. The name “switch01” is just text anyone can type; only the source address (10.40.0.1, not the switch’s .2) gives it away, and on UDP even that can be forged. Attackers use this to bury real alerts in noise or plant misleading ones. The defender’s move: accept syslog only from known device addresses, put devices on a management network, and where devices support it use syslog over TCP with TLS (RFC 5425), which encrypts and authenticates the sender.

The agent answers questions about the device on UDP port 161. This config allows read-only access with the community string sglabRO, and only from the monitoring server.

# cat > /tmp/sglab/snmpd.conf <<'EOF'
agentAddress udp:10.40.0.2:161
rocommunity sglabRO 10.40.0.1
sysLocation Lab rack 1
sysContact you@example.com
sysName switch01
EOF
# ip netns exec dev snmpd -C -c /tmp/sglab/snmpd.conf -p /tmp/sglab/snmpd.pid -Lf /tmp/sglab/snmpd.log
# ip netns exec dev ss -ulnp

State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
UNCONN 0      0          10.40.0.2:161       0.0.0.0:*    users:(("snmpd",pid=123592,fd=7))

Reading the command. -C -c means “use only this config file,” -p writes a PID file for clean-up, and -Lf sends the agent’s own log to a file. Restricting the community to one source address (the 10.40.0.1 on the rocommunity line) is the first hardening step on any real device.

SNMP questions are numbers called OIDs, branches of one big tree. Ask for the device’s name, location and uptime, then walk its list of interfaces.

# ip netns exec nms snmpget -v2c -c sglabRO 10.40.0.2 1.3.6.1.2.1.1.5.0 1.3.6.1.2.1.1.6.0 1.3.6.1.2.1.1.3.0
# ip netns exec nms snmpwalk -v2c -c sglabRO 10.40.0.2 1.3.6.1.2.1.2.2.1.2

iso.3.6.1.2.1.1.5.0 = STRING: "switch01"
iso.3.6.1.2.1.1.6.0 = STRING: "Lab rack 1"
iso.3.6.1.2.1.1.3.0 = Timeticks: (202) 0:00:02.02
iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo"
iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0"

Reading it. 1.3.6.1.2.1 is the standard MIB-2 branch every device supports. .1.5.0 is sysName, .1.6.0 sysLocation, .1.3.0 sysUpTime in hundredths of a second (202 = 2.02 seconds, because we had just started the agent). snmpget asks for exact OIDs; snmpwalk asks for everything under a branch, here the interface descriptions, where .2.2 means interface number 2 is eth0. A MIB file is the dictionary that turns these numbers into names; Ubuntu doesn’t ship the full set, which is why you see numbers.

Read eth0’s inbound byte counter (ifInOctets, interface 2), send some traffic, and read it again.

# ip netns exec nms snmpget -v2c -c sglabRO 10.40.0.2 1.3.6.1.2.1.2.2.1.10.2
# ip netns exec nms ping -c 20 -i 0.2 -q 10.40.0.2
# ip netns exec nms snmpget -v2c -c sglabRO 10.40.0.2 1.3.6.1.2.1.2.2.1.10.2

iso.3.6.1.2.1.2.2.1.10.2 = Counter32: 558
...
20 packets transmitted, 20 received, 0% packet loss, time 3852ms
iso.3.6.1.2.1.2.2.1.10.2 = Counter32: 2955

This is how bandwidth graphs work. The device never reports “Mbps”. It keeps a running byte count, and the monitoring tool polls it every few minutes and divides the difference by the time between polls. Here, 2955 − 558 = 2397 bytes: the 20 pings (98 bytes each on the wire, 1960 in all) plus the SNMP questions themselves. A Counter32 wraps back to zero after about 4.3 billion bytes, which on a busy gigabit link happens in under a minute; that is why fast links are polled with the 64-bit counters (ifHCInOctets).

Try the factory-default community public, then capture a normal request.

# ip netns exec nms snmpget -v2c -c public -t 1 -r 0 10.40.0.2 1.3.6.1.2.1.1.5.0; echo "exit $?"
# ip netns exec nms tcpdump -n -l -A -i mon0 -c 1 'udp port 161' 2>/dev/null &
# ip netns exec nms snmpget -v2c -c sglabRO 10.40.0.2 1.3.6.1.2.1.1.5.0 >/dev/null

Timeout: No Response from 10.40.0.2.
exit 1
IP 10.40.0.1.34376 > 10.40.0.2.161:  C="sglabRO" GetRequest(28)  .1.3.6.1.2.1.1.5.0

Two lessons. A wrong community gets no reply at all, so to a monitoring tool it looks exactly like a dead device or a firewall: check the community string before you open a network ticket. And the right community, sglabRO, crosses the wire in plain text for anyone capturing to read. Devices still shipping with public/private are a classic finding in audits and pentests. The defender’s move: use SNMPv3 with authPriv (a username, authentication and encryption), allow polling only from the monitoring server, and never enable write access you don’t need.

Stop the two lab daemons using their PID files, then delete both namespaces and the lab folder.

# pkill -F /tmp/sglab/rsyslog.pid; pkill -F /tmp/sglab/snmpd.pid
# ip netns del nms; ip netns del dev
# rm -rf /tmp/sglab
# exit