ARP Lab: Watch Your Network Find Its Neighbours (and Catch a Duplicate IP)

A namespace called sw holds a Linux bridge (a software switch). Three more namespaces, h1, h2 and h3, plug into it with easy-to-read MAC addresses ending in :01, :02 and :03.

# ip netns add sw; ip -n sw link add br0 type bridge; ip -n sw link set br0 up
# for i in 1 2 3; do ip netns add h$i; ip link add eth0 netns h$i address 02:00:00:00:00:0$i type veth peer name p$i netns sw; ip -n sw link set p$i master br0 up; ip -n h$i addr add 10.30.0.$i/24 dev eth0; ip -n h$i link set lo up; ip -n h$i link set eth0 up; done
# ip -n h1 neigh show

The last command prints nothing: h1 has never talked to anyone, so its neighbour cache (the ARP table) is empty.

About those MACs. Real network cards get a MAC address from the factory. A 02 at the start marks an address as “locally administered” (set by software), which is what virtual machines and containers use. We picked simple ones so you can read the captures at a glance.

h1 wants to ping h2 at 10.30.0.2, but to send anything on a switch it needs h2’s MAC address. Capture on h2, then ping. The sleep 1 gives tcpdump a second to start listening.

# ip netns exec h2 tcpdump -n -e -l -i eth0 -c 2 arp & sleep 1
# ip netns exec h1 ping -c 1 10.30.0.2

64 bytes from 10.30.0.2: icmp_seq=1 ttl=64 time=0.057 ms
02:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.30.0.2 tell 10.30.0.1, length 28
02:00:00:00:00:02 > 02:00:00:00:00:01, ethertype ARP (0x0806), length 42: Reply 10.30.0.2 is-at 02:00:00:00:00:02, length 28

Reading it. h1 sent the request to ff:ff:ff:ff:ff:ff, the broadcast address, so every machine on the switch heard “who has 10.30.0.2? tell 10.30.0.1.” Only h2 answered, and it answered h1 directly (unicast): “10.30.0.2 is at 02:00:00:00:00:02.” Then the ping went through. ARP only works inside one network (one broadcast domain); to reach another network, your computer ARPs for the gateway’s MAC instead.

# ip -n h1 neigh show

10.30.0.2 dev eth0 lladdr 02:00:00:00:00:02 REACHABLE

Wait about a minute without sending anything, then look again:

# ip -n h1 neigh show

10.30.0.2 dev eth0 lladdr 02:00:00:00:00:02 STALE

Reading it. REACHABLE means “confirmed recently.” After roughly 30 seconds without confirmation it becomes STALE: still used, but checked again next time h1 sends to it. That re-check is how a cache heals itself when a device is swapped. On Windows the same table is arp -a; there, entries are listed as dynamic or static. When someone replaces a printer or a router and “nothing can reach it for a few minutes,” an old cache entry is a prime suspect, and ip neigh flush (or arp -d on Windows) clears it.

Nothing on the switch owns 10.30.0.9. Capture on h1 and try to ping it.

# ip netns exec h1 tcpdump -n -l -i eth0 -c 3 arp & sleep 1
# ip netns exec h1 ping -c 2 -W 1 10.30.0.9
# sleep 3; ip -n h1 neigh show 10.30.0.9

ARP, Request who-has 10.30.0.9 tell 10.30.0.1, length 28
ARP, Request who-has 10.30.0.9 tell 10.30.0.1, length 28
2 packets transmitted, 0 received, 100% packet loss, time 1021ms
10.30.0.9 dev eth0 FAILED

Reading it. The questions go out again and again with no answer (you may see two or three), and the cache marks the address FAILED. Not a single ping packet ever left h1, because it never learned where to send it. On Windows this is the “Destination host unreachable” reply that comes from your own PC’s address: the clue that the problem is on your local network (wrong IP, device off, wrong VLAN), not out on the internet.

arping sends ARP questions on purpose and shows who answers. It works even when a firewall blocks ping, because a device can’t refuse ARP and still use the network.

# ip netns exec h1 arping -c 2 -I eth0 10.30.0.3

ARPING 10.30.0.3 from 10.30.0.1 eth0
Unicast reply from 10.30.0.3 [02:00:00:00:00:03]  0.537ms
Unicast reply from 10.30.0.3 [02:00:00:00:00:03]  0.540ms
Sent 2 probes (1 broadcast(s))
Received 2 response(s)

Reading it. One MAC answered every time: exactly one machine owns 10.30.0.3. That is the healthy result. The next step shows the unhealthy one.

Give h3 the same address h2 already has, a classic mistake when someone sets a static IP by hand. Then arping it from h1.

# ip -n h3 addr add 10.30.0.2/24 dev eth0
# ip netns exec h1 arping -c 2 -I eth0 10.30.0.2

ARPING 10.30.0.2 from 10.30.0.1 eth0
Unicast reply from 10.30.0.2 [02:00:00:00:00:03]  0.534ms
Unicast reply from 10.30.0.2 [02:00:00:00:00:02]  0.554ms
Unicast reply from 10.30.0.2 [02:00:00:00:00:02]  0.539ms
Sent 2 probes (1 broadcast(s))
Received 3 response(s)

Two different MACs answer for one IP. Now check it the way a computer does before it uses an address (duplicate address detection), from h2:

# ip netns exec h2 arping -D -c 2 -I eth0 10.30.0.2; echo "exit $?"

ARPING 10.30.0.2 from 0.0.0.0 eth0
Unicast reply from 10.30.0.2 [02:00:00:00:00:03]  0.536ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)
exit 1

Reading it. -D asks “is anyone else using this address?” from 0.0.0.0. Someone answered, so arping exits with 1: conflict. Windows does the same check and shows the “IP address conflict” pop-up. The symptom users report is “my connection drops in and out,” because other machines keep switching between the two MACs. Fix: find the MAC in your switch or DHCP server, and use DHCP reservations instead of hand-typed addresses. Remove the duplicate before the next step:

# ip -n h3 addr del 10.30.0.2/24 dev eth0

Clusters and firewalls in failover pairs move an IP from a failed box to a healthy one. Make h1 learn h2’s MAC, move 10.30.0.2 to h3, and have h3 announce it with a gratuitous ARP.

# ip -n h1 neigh flush all; ip netns exec h1 ping -c 1 10.30.0.2 >/dev/null; ip -n h1 neigh show 10.30.0.2

10.30.0.2 dev eth0 lladdr 02:00:00:00:00:02 REACHABLE

# ip -n h2 addr del 10.30.0.2/24 dev eth0; ip -n h3 addr add 10.30.0.2/24 dev eth0
# ip netns exec h1 tcpdump -n -e -l -i eth0 -c 1 arp & sleep 1
# ip netns exec h3 arping -U -c 1 -I eth0 10.30.0.2
# ip -n h1 neigh show 10.30.0.2

02:00:00:00:00:03 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.30.0.2 (ff:ff:ff:ff:ff:ff) tell 10.30.0.2, length 28
10.30.0.2 dev eth0 lladdr 02:00:00:00:00:03 STALE

Reading it. h3 broadcast an ARP message nobody asked for (“10.30.0.2? that’s me”), and h1 simply believed it: its cache now points 10.30.0.2 at :03, with no question asked. That is the whole failover trick, and also the weakness. ARP has no proof of identity, so anyone on the same network can make the same claim for the gateway’s address and put themselves in the middle of other people’s traffic (ARP poisoning, an on-path attack).

# for n in h1 h2 h3 sw; do ip netns del $n; done
# exit

Deleting the namespaces removes the switch, the three hosts and all their addresses.