What is the /etc/hosts size limit?

Question

How can I determine or set the size limit of /etc/hosts? How many lines can it have?

Solution 1

Problematical effects include slow hostname resolution (unless the OS somehow converts the linear list into a faster-to-search structure?) and the potential for surprising interaction with shell tab completion well before any meaningful file size is reached.

For example! If one places 500,000 host entries in /etc/hosts

# perl -E 'for (1..500000) { say "127.0.0.10 $_.science" }' >> /etc/hosts

for science, the default hostname tab completion in ZSH takes about ~25 seconds on my system to return a completion prompt (granted, this is on a laptop from 2009 with a 5400 RPM disk, but still).

Solution 2

I don't think it has a size limit in terms of number of lines.

Before DNS (which was brought into use in 1985), it was this file that served as the only means for doing host name lookups, so I presume that this means that the file should be able to have thousands or at least hundreds of entries to be able to support the most well-connected pre-1985 Internet nodes.

Here's an example from 1985 (the format has changed somewhat): http://jim.rees.org/apollo-archive/hosts.txt This file has 1680 lines out of which 1325 are host lines. The remaining 355 lines are blank, comments, networks or gateways¹.

The only real limit that I could find was that on some systems, the individual lines are limited to less than BUFSIZ characters (1024 on my OpenBSD machine).

If you have more than a couple of handfuls of entries in /etc/hosts, you should consider setting up a local name server instead, but that's my personal opinion.

---

¹ _{Thanks to Jeff Schaller for digging this up.}

Solution 3

How can I determine the size limit of /etc/hosts?

It is a regular file, so the limit would correspond to the underlying filesystem's limits (which would itself be capped by the number of disks behind it), minus the space used by any other files in the same (probably root (/)) filesystem:

• ext2/3: 2 TiB
• ext4: 16 TiB (with the default 4KiB block size)
• xfs: 500 TiB

How can I set the size limit of /etc/hosts?

Since it's a manually-edited file, only manually:

sed -i '100,$d' /etc/hosts

(to remove lines 100 and beyond).

Solution 4

Size limits only apply when allocating static buffers. gethostbyname(3), which parses the entries in /etc/hosts, does not allocate static buffers - and never has. The original 1983 release of BSD 4.3 algorithm shows an open-file, while parse line, close-file pattern:

sethostent(0);
while (p = gethostent()) {
    if (strcmp(p->h_name, name) == 0)
        break;
    for (cp = p->h_aliases; *cp != 0; cp++)
        if (strcmp(*cp, name) == 0)
            goto found;
}
found:
endhostent();

Modern implementations retain this heritage in all essentials.

Anyway, internally, the *hostent family of functions store a file pointer to the current line in the file. sethostent opens the file and sets the file pointer position. gethostent gets data and advances the pointer. endhostent closes the file pointer. The GNU C Library offers a thorough reference on these functions.

As you might guess from the implementation, entries occuring earlier in the file resolve faster. If your hosts file is huge, this comes into play.

So, no matter how big the file is, the OS will consume it. Eventually, though, you'll hit filesystem limits (per Jeff Schaller's answer). You also have maximum line size limits (per Kusalananda's answer). But, in the end, you can make it as big as you want. But please, don't.

Solution 5

... I've racked my brain and for the life of me I can't think of a single situation or circumstance where you'd approach any kind of size limit issue in /etc/hosts -- you'd run into practical problems like the severe performance hit to the getaddrinfo() family of system calls that all have to consult the file before deciding whether to send out a DNS query, to say nothing of the problems maintaining a flat text file that size.

I suspect that what we have here is a failure to communicate at a higher level. What problem are you trying to solve with a gigantic /etc/hosts file? I'm almost certain that there's a better solution than this.

View more solutions

Comments

[Shadur]

... I think at the point where this becomes an actual worry rather than an academic curiosity you have far passed the point where it would be wiser to set up a private DNS server in terms of performance and maintainability.

[Basile Starynkevitch]

I'm surprised you are asking. Why do you expect some precise and hard meaningful size limit on configuration files?

[Geremia]

@BasileStarynkevitch I asked because a ~2 MB hosts file wasn't working on my router, but the issue was that I didn't sighup dnsmasq to re-read the hosts file.

[bsd]

@Geremia my hosts file is 500M, 15,000+ entries, no problems. It is always best to determine the root cause of your problem and solve it first

[cybernard]

Are you using your host file to block IP addresses? If so you should be using iptables with ipset. Using iptables alone will cause a major performance hit, however with ipset a list of nearly 500,000 ips has negligible impact.

[Michael]

@bsd you are averaging 33kbytes for each entry in your hosts file?

[bsd]

Meant 500k, not m

[Jeff Schaller]

I've voted to close this question as per the OP, they hadn't sighup'd the dnsmasq daemon to re-read the existing hosts file. It's a classic XY problem

[Geremia]

@JeffSchaller Who's "they"? I'm a single person. (cf. this article: "“They” is Destroying the English Language")

[Jeff Schaller]

My apologies, Geremia, I wasn't actually addressing you, but generally stating why I feel the question should be closed. When your answer is "HUP dnsmasq", the question isn't "how large can /etc/hosts be?

[Basile Starynkevitch]

True in principle, but the file system limits (e.g. terabytes) are practically largely irrelevant.

[PlasmaHH]

Using the glibc and linux, this is not so straightforward unfortunately. If you call gethostbyname and the system is setup accordingly (default on many systems) then it will instead of reading /etc/host call the nscd. I have no idea if the nscd will cache only hits in the file or try to cache it as a whole. In the later case, you would have a ram limitation for the file size (assuming that the nscd config allows that much entries)

[Bert]

Some people use a hosts file to blacklist ads/malware/tracking/etc. There are curated lists on the internet, the one I use is 41k lines and 1.1MB in size.

[reinierpost]

It would probably be better for performance to use dnsmasq for that - see e.g. dnsgate (which I haven't tried).

[Barmar]

Before DNS, I don't think it was common to convert the full Internet host table to /etc/hosts format. Most Unix systems weren't even on the Internet, and even if a machine was, it didn't need a complete host table, just the handful of the machines it needed to talk to. I'd be surprised if there were many machines with more than 100 entries.

[Mark Plotnick]

The 4.3BSD version is here. It supported a dbm-hashed version of /etc/hosts. IIRC, dbm imposed some size limits that could cause attempts to create a hashed db to fail.