How do I debug DHCP?

Question

UPDATE: The DHCP in the camera does not work as expected and requires a static IP. However tcpdump is your first line of defense, see checked answer below.

I have the following setup:

• NVidia Xavier running Ubuntu 16.04
• NetworkManager managing wlan0, connected to my home router
• /etc/network/interfaces defining eth0, connected to a PoE switch
• isc-dhcp-server running on eth0
• Internet CCTV camera on ethernet switch

I'm logged into the Xavier through WiFi so that's all good.

However, when I plug anything into the switch, (including a test raspberry pi), nmap doesn't show anything connect nor does /var/lib/dhcp/dhcpd.leases.

No errors in journalctl.

Before I start dumping all of my config files, what is a standard process of logfile viewing and querying I can do to inspect the system and see where things are breaking?

I can see ifconfig shows eth0 configured as I expect. So the interface is good. How do I probe what's happening when something on the switch asks for DHCP config?

Solution 1

First, run tcpdump, or wireshark if you prefer, to monitor things at the packet level. In this example the internal NIC serviced by the DHCP server is enp2s0 at 192.168.111.1:

doug@DOUG-64:~$ sudo tcpdump -n -tttt -i enp2s0 port 67 and port 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
2020-05-18 14:42:30.656940 IP 192.168.111.122.68 > 192.168.111.1.67: BOOTP/DHCP, Request from 04:d4:c4:93:f4:55, length 265
2020-05-18 14:43:07.473233 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 04:d4:c4:93:f4:55, length 287
2020-05-18 14:43:07.473493 IP 192.168.111.1.67 > 192.168.111.122.68: BOOTP/DHCP, Reply, length 300
2020-05-18 14:43:07.473766 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 04:d4:c4:93:f4:55, length 299
2020-05-18 14:43:07.475214 IP 192.168.111.1.67 > 192.168.111.122.68: BOOTP/DHCP, Reply, length 300

O.K., so I re-booted the computer at 192.168.111.122 so as to create some DHCP traffic. Things at the packet level look O.K., and if things were not O.K. at least we would know if request packets were arriving.

Now, lets check the logs for this transaction:

doug@DOUG-64:~$ grep dhcpd /var/log/syslog | tail -10
May 18 11:34:12 DOUG-64 dhcpd[1141]: DHCPACK on 192.168.111.122 to 04:d4:c4:93:f4:55 via enp2s0
May 18 14:25:29 DOUG-64 dhcpd[1141]: DHCPREQUEST for 192.168.111.110 from a4:ee:57:e6:d0:0b via enp2s0
May 18 14:25:29 DOUG-64 dhcpd[1141]: DHCPACK on 192.168.111.110 to a4:ee:57:e6:d0:0b via enp2s0
May 18 14:28:07 DOUG-64 dhcpd[1141]: DHCPREQUEST for 192.168.111.112 from f4:6d:04:65:2d:8e via enp2s0
May 18 14:28:07 DOUG-64 dhcpd[1141]: DHCPACK on 192.168.111.112 to f4:6d:04:65:2d:8e via enp2s0
May 18 14:42:30 DOUG-64 dhcpd[1141]: DHCPRELEASE of 192.168.111.122 from 04:d4:c4:93:f4:55 via enp2s0 (not found)
May 18 14:43:07 DOUG-64 dhcpd[1141]: DHCPDISCOVER from 04:d4:c4:93:f4:55 via enp2s0
May 18 14:43:07 DOUG-64 dhcpd[1141]: DHCPOFFER on 192.168.111.122 to 04:d4:c4:93:f4:55 via enp2s0
May 18 14:43:07 DOUG-64 dhcpd[1141]: DHCPREQUEST for 192.168.111.122 (192.168.111.1) from 04:d4:c4:93:f4:55 via enp2s0
May 18 14:43:07 DOUG-64 dhcpd[1141]: DHCPACK on 192.168.111.122 to 04:d4:c4:93:f4:55 via enp2s0

So the transaction all looked O.K. and we know the DHCP server got, and dealt with, the traffic. This example lease was MAC based, and doesn't appear in the /var/lib/dhcp/dhcpd.leases, but that would be the next place to check for a pool based lease. Example:

lease 192.168.111.46 {
  starts 1 2020/05/18 01:53:46;
  ends 2 2020/05/19 01:53:46;
  cltt 1 2020/05/18 02:21:41;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet e4:e4:ab:08:6a:6b;
  uid "\001\344\344\253\010jk";
}

Solution 2

Make sure you have INTERFACES="eth0" defined in /etc/default/isc-dhcp-server. Once you define that directive be sure to restart the DHCP daemon.

Comments

[goo]

wireshark will let you see DHCP packets, as will tcpdump. Also check ip route.

[PeterT]

@waltinator Ah, tcpdump. I shall learn this tool and report back. Thank you. Why ip route?

[PeterT]

Yes, it is set to eth0 (as is the parameter for IPv4 interfaces). How can I tell if the isc-dhcp-server is getting the requests from the networking driver(s)/layer(s)?

[PeterT]

this is insightful. I'm only seeing the IP 0.0.0.0.68 > 255.255.255.255.67 requests from the target device and no messages in the syslog.

[PeterT]

OK. Cool. I realized the DHCP daemon had an issue (I had messed with the dhcpd.conf and removed a semicolon and it wasn't starting). So now I can see a test raspberry pi connect to the switch, ask for an IP address and get one from the Xavier. However when I plug the camera into the switch I don't see a request come across tcpdump. but if I plug the camera into my other main router, I can connect to :80 and use it. So why wouldn't tcpdump show me the DHCP request from the camera??

[sc608]

You can check to see DHCP activity on your server by using the following command journalctl -u isc-dhcp-server. Also journalctl -u isc-dhcp-server -r will provide the results most recent first.

[PeterT]

So it turns out the DHCP in the camera doesn't work correctly. I plugged it into another machine, set the static address to a range in my DHCP and now it works. In summary: tcpdump & netstat are your friends. Thanks for the help!