I recently came across an interesting issue with DHCP and Client Identifiers.
Whilst working on the Junos as a DHCP server post, I found that only one client would get an address via DHCP, and the other client wouldn’t.
The MAC addresses were different, so it appeared that it should be working.
After some searching, it appears another symptom may appear as the different clients being given the same IP address. It just depends on your exact setup of virtualization and clients.
A Little Background Info
To help understand what is going on, you need to know a little bit about the lab setup.
This was all built on top of Eve-NG.
The lab uses a vQFX to provide switching. A vSRX and vMX are connected to the QFX with access ports in VLAN 20. Two Ubuntu Linux virtual machines also connect to the vQFX with access ports in VLAN 20.
The MAC addresses for the two Linux machines are:
Linux-1 – aa:aa:aa:00:00:01
Linux-2 – aa:aa:aa:00:00:02
To get the packet captures, I used analyzer ports on the vQFX to another Linux VM.
The Investigation
Since I already had some port mirroring in place, I took a capture and started looking through it with Wireshark.
Packets 11 through 14 show the expected behavior of the normal DHCP process. Note that the destination IP address is 192.168.30.21, which is the IP address being handed to Linux-1.
Where it starts to get a bit strange, is packets 18 through 32. You can’t tell from the image above, but these Discover messages are coming from Linux-2, and yet the DHCP server is responding and sending Offer messages destined to 192.168.30.21, the address given to Linux-1.
Looking at packet 21, the first offer for Linux-2, things seem to get even stranger.
The destination Ethernet address is pointing to Linux-1, and the Client MAC Address is set to the MAC of Linux-2.
It’s as if the DHCP server thinks both Linux-1 and Linux-2 are the same client. Time to look at the Discover messages in more detail.
Linux-1:
Linux-2:
And here we start to see something worth investigating further. You can see the source addresses of Linux-1 and Linux-2 differ, but both have the same Option 61 Client Identifier. That seems to be the cause of our issues.
Digging into RFC 2132 – DHCP Options and BOOTP Vendor Extensions we find a section on Client Identifiers. The following text seems relevant:
“DHCP servers use this value to index their database of address bindings. This value is expected to be unique for all clients in an administrative domain”
and
“For correct identification of clients, each client’s client-identifier MUST be unique among the client-identifiers used on the subnet to which the client is attached”
The Solution
The problem is our duplicate client identifiers. In my specific case, an Ubuntu server using netplan to manage the IP Addressing, it appears there are at least two possible solutions.
Option 1 – Telling Netplan What Client Identifier To Use
The first option is to tell netplan to use the mac address as the client identifier. This can be done by editing the configuration file found under /etc/netplan.
The interface configuration can be updated to the following:
network:
ethernets:
ens4:
dhcp4: true
dhcp-identifier: mac
version: 2Whilst this does work, and with the above configuration Linux-2 now gets an IP Address, option 2 is probably a better choice.
Option 2 – Reset the Ubuntu Machine ID
To reset the Machine ID, use the following
sudo rm /etc/machine-id
sudo systemd-machine-id-setup
On my VM it looked like this:
eve@linux-2:~$ sudo rm /etc/machine-id
eve@linux-2:~$ sudo systemd-machine-id-setup
Initializing machine ID from VM UUID.
eve@linux-2:~$After changing the Machine ID, I recommend a reboot. Netplan seemed to still use the old Client Identifier until the machine was restarted. After that DHCP worked fine without using the fix in Option 1.
Conclusions And Learnings
The most obvious learning, or re-learning as the case may be, is to remember that when you copy a virtual machine, you should change or update the identifiers.
There is the question of why didn’t I hit this issue when creating the lab for the DHCP post. The answer is that I only used one local and one remote DHCP client for that lab, so this issue didn’t occur.
I think that is a trap that we all fall into from time to time. Once we prove it works for one client, we assume it will work for all clients. Whether that is DHCP in this case or another form of connectivity. The lesson here is to make sure you test more than one client at a time.
The other lesson that stands out is this: it is not always the thing we are configuring that is broken. In this case, I assumed something was wrong with the way I had configured the DHCP server. I spent more time than I’m willing to admit looking and troubleshooting in the wrong place. The reality was it was an issue with the client configuration.