Number of file descriptors: different between /proc/sys/fs/file-nr and /proc/$pid/fd?

Question

I would like to check how many file descriptors are actually used:

cat /proc/sys/fs/file-nr 
12750   0   753795

The first column (12750) indicates the number of file descriptors allocated since boot.

I would like to know why the number from the following command is different (assuming this one liner is returning the correct value:

for pid in $(lsof | awk '{ print $2 }' | uniq); do find /proc/$pid/fd/ -type l 2>&1 | grep -v "No"; done | wc -l

11069

Solution 1

lsof only lists the Process ID. To get info about threads, you should use ps -eLf. According to the man proc:

   /proc/[pid]/task (since Linux 2.6.0-test6)
          This is a directory that contains one subdirectory for each
          thread in the process.  The name of each subdirectory is the
          numerical thread ID ([tid]) of the thread (see gettid(2)).
          Within each of these subdirectories, there is a set of files
          with the same names and contents as under the /proc/[pid]
          directories.  For attributes that are shared by all threads,
          the contents for each of the files under the task/[tid]
          subdirectories will be the same as in the corresponding file
          in the parent /proc/[pid] directory (e.g., in a multithreaded
          process, all of the task/[tid]/cwd files will have the same
          value as the /proc/[pid]/cwd file in the parent directory,
          since all of the threads in a process share a working
          directory).  For attributes that are distinct for each thread,
          the corresponding files under task/[tid] may have different
          values (e.g., various fields in each of the task/[tid]/status
          files may be different for each thread).

          In a multithreaded process, the contents of the
          /proc/[pid]/task directory are not available if the main
          thread has already terminated (typically by calling
          pthread_exit(3)).

I would calculate the number of open file descriptors by running:

ps -eL | awk 'NR > 1 { print $1, $2 }' | \
while read x; do \
    find /proc/${x% *}/task/${x#* }/fd/ -type l; \
done | wc -l

The result is 17270.

Let's see how many file descriptors allocated since boot:

cat /proc/sys/fs/file-nr 
11616   0   398855

Why there is an excess of number of file descriptors in /proc/[pid]/task/[tid]/fd over the number of allocated file handles in /proc/sys/fs/file-nr? I suppose that they are created by forked child processes:

man fork:

The child inherits copies of the parent's set of open file descriptors.

man pthreads:

POSIX.1 also requires that threads share a range of other attributes (i.e., these attributes are process-wide rather than per-thread): - process ID

• parent process ID

• process group ID and session ID

• controlling terminal

• user and group IDs

• open file descriptors

Solution 2

http://www.netadmintools.com/part295.html Some of the open files which are not using file descriptors: library files, the program itself (executable text), and so on as listed above. These files are accounted for elsewhere in the kernel data structures (cat /proc/PID/maps to see the libraries, for instance), but they are not using file descriptors and therefore do not exhaust the kernel's file descriptor maximum.

Comments

[Timothy Sipples]

I don't think the higher number in /proc/sys/fs/file-nr is due to forked processes. See the first answer here: unix.stackexchange.com/questions/176967/… and also this quote here: (..but the kernel does not free these file handles when they are released by the application. The kernel recycles these file handles instead.) access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Lin‌​ux/…

[fholzer]

I think quanta/user367890 were referring to 17270 being greater than 11616. And the explanation give seems sound to me.