dhcp server gives "no free leases" error although there should be available IPs

Question

We have a small HPC cluster of about 40 machines running Rocks Linux Distro 6.2 (based on Centos 6.6). I have been trying to add 6 more machines to the system in the last few days. The first 3 were successfull but I can't pxe boot the other 3 machines. /var/log/messages shows "no free leases" error as below:

[root@topaz ~]# tail -n3 /var/log/messages 
Jun 29 17:56:00 topaz dhcpd: DHCPDISCOVER from d0:bf:9c:03:3b:40 via eth0: network 10.1.0.0/16: no free leases
Jun 29 17:56:03 topaz dhcpd: DHCPDISCOVER from d0:bf:9c:03:3b:40 via eth0: network 10.1.0.0/16: no free leases
Jun 29 17:56:07 topaz dhcpd: DHCPDISCOVER from d0:bf:9c:03:3b:40 via eth0: network 10.1.0.0/16: no free leases

All machines only have a local IP except for the frontend. There should be enough IPs availabe to assign to these new machines. There doesn't seem to be any unexpected machines connected to the network:

[root@topaz ~]# arp -a | awk '{ print $2 }' | sort   
(10.1.255.215)
(10.1.255.216)
(10.1.255.217)
(10.1.255.218)
(10.1.255.219)
(10.1.255.220)
(10.1.255.221)
(10.1.255.222)
(10.1.255.223)
(10.1.255.224)
(10.1.255.225)
(10.1.255.226)
(10.1.255.227)
(10.1.255.228)
(10.1.255.229)
(10.1.255.230)
(10.1.255.231)
(10.1.255.232)
(10.1.255.233)
(10.1.255.234)
(10.1.255.235)
(10.1.255.236)
(10.1.255.237)
(10.1.255.238)
(10.1.255.239)
(10.1.255.240)
(10.1.255.241)
(10.1.255.242)
(10.1.255.243)
(10.1.255.244)
(10.1.255.245)
(10.1.255.246)
(10.1.255.247)
(10.1.255.248)
(10.1.255.249)
(10.1.255.250)
(10.1.255.251)
(10.1.255.252)
(10.1.255.253)
(10.1.255.254)
(193.140.197.1)
(193.140.197.110)
(193.140.197.160)
(193.140.197.30)
(193.140.197.36)

Our leases file do not show any entries:

[root@topaz ~]# cat /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.1.1-P1

server-duid "\000\001\000\001\037\006r\351x\347\321\350\031\006";

And our configuration file looks like this:

[root@topaz ~]# cat /etc/dhcp/dhcpd.conf
ddns-update-style none;
subnet 10.1.0.0 netmask 255.255.0.0 {
        default-lease-time 1200;
        max-lease-time 1200;
        option routers 10.1.1.1;
        option subnet-mask 255.255.0.0;
        option domain-name "local";
        option domain-name-servers 10.1.1.1;
        option broadcast-address 10.1.255.255;
        option interface-mtu 1500;
        group "local" {
                host topaz-eth0 {
                        hardware ethernet 78:E7:D1:E8:19:06;
                        option host-name "topaz";
                        fixed-address 10.1.1.1;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host topaz-eth2 {
                        hardware ethernet 78:E7:D1:E8:19:0A;
                        option host-name "topaz";
                        fixed-address 10.1.1.1;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host topaz-eth3 {
                        hardware ethernet 78:E7:D1:E8:19:0C;
                        option host-name "topaz";
                        fixed-address 10.1.1.1;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-2-0-eth0 {
                        hardware ethernet 00:23:7d:9f:6f:ca;
                        option host-name "compute-2-0";
                        fixed-address 10.1.255.254;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-2-0-eth1 {
                        hardware ethernet 00:23:7d:a0:20:e4;
                        option host-name "compute-2-0";
                        fixed-address 10.1.255.254;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-0-0-eth0 {
                        hardware ethernet 1c:c1:de:76:6f:40;
                        option host-name "compute-0-0";
                        fixed-address 10.1.255.253;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-0-0-eth1 {
                        hardware ethernet 1c:c1:de:76:6f:44;
                        option host-name "compute-0-0";
                        fixed-address 10.1.255.253;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }

                (goes on like this)
}

I don't understand why am I getting this error. Any help is appreciated.

Solution 1

It looks like you miss either range definition or a specific host entry for those boxes, matching their MACs. Unless your failing boxes are on the list that "goes on like this", they won't have any address assigned.

Solution 2

Your ISC dhcpd config file (posted June 2016) has no range statement. No leases are free because the server can't find an IP address, in a defined range, to hand out. You need a line like

range 10.1.42.0 10.1.49.255;

inside your subnet{} block. Any range inside that subnet, that nothing else is already using, will work. If you have multiple subnet blocks, each one needs a range definition inside it.

Solution 3

I just had a weird case of this error, I'll mention it here in case it helps someone.

My dhcp3 is binding on two adapters. I couldn't get DHCP to work on one of them. It turns out I had an old MAC address reservation in place for the client computer. Since this reservation was for the other subnet, DHCP could only find a pre-determined IP address to give that was outside the range for this adapter, and so gave this no free leases error.

Comments

[none]

I was going to try the range setting as you said but somehow I can't reproduce the error anymore. I don't remember doing anything special though. I changed the connected ports on the switch and restarted the dhcp server a few times both of which I have tried numerous times before. Now 2 more machines are working but the last one still does not work. It boots from network but I can't ping to it afterwards. It's likely a different issue. Still, I think this is a helpful answer and you deserve the rep for giving me good luck. Thanks..

[Grigory Sergeev]

Glad to be of assistance, in any way :) What about this last host MAC? Is it on the list?

[none]

Well, I'm using insert-ethers tool from the rocks distro and it does assign an IP and add the MAC address to the conf file. I have checked the file after the boot and everything seems alright. It wasn't doing that before and I had tried to manually add the MAC addresses to the conf file at some point. Then the machine was booting up but couldn't connect to the network afterwards just like now. That's why I thought dhcpd was the problem and asked this question. I guess there were two problems back then and now I have only one. I suspect it could be an hardware error.

[DanSut]

Maybe not the answer to this question but I just hit exactly your situation with dhcp 4.1.1 and your answer probably saved me quite a bit of time head scratching :)