How can a file descriptor be reclaimed and reused later?

Question

In context of C and C++, is the close() system call the only way for kernel to reclaim the fd and reuse it later when calling open() in the same process lifetime? What about multi-threaded programs?

Solution

Read more carefully close(2) and Advanced Linux Programming

Your formulation is wrong: close is a system call (listed in syscalls(2) for Linux) by which the application tells the kernel to release a resource (not the other way round). You could use strace(1) to understand the system calls executed by some command or process. See also pthreads(7), credentials(7), fork(2), execve(2), clone(2)

And yes, file descriptors (and address space in virtual memory, see mmap(2)) are common to all the threads of a given process. However, you might (rarely) create "threads" directly with the low-level clone(2) syscall (that is in practice only used by implementors of thread libraries like pthreads), and in the unlikely case you are not using CLONE_FILES things are different. But calling directly clone is a black magic art.

For a given process of pid 1234, on Linux, you can query the set of file descriptors (in proc(5)) thru /proc/1234/fd/ and the memory mapping thru /proc/1234/maps (etc etc... there are many useful pseudo files and links in /proc/1234). From inside the process, you could use /proc/self/fd/ - e.g. as argument to opendir(3)

Of course, file descriptors are much more a POSIX or Linux thing than a standard C99 or C++11 one. For instance, fileno(3) & open is defined in POSIX, not in C99.

So if you see a given number -e.g. 49- been returned by open several times it is because some other part of your program (perhaps inside some library, in another thread) have called close with 49. The kernel is never "magically" closing file descriptors (except at process termination) without been asked. You might use strace or use your gdb debugger with a breakpoint (probably a conditional one) on close

Comments

[lordofire]

Thanks Basile, I think the issue I encountered is that I have seen thousands of fd reusing without seeing explicitly close() calling when fixing our own IPC related bugs, which makes me doubt if I missed any possible means which could lead to this situation.

[Basile Starynkevitch]

Sorry, I don't understand your comment above (and even your question was strangely formulated): how have you seen "fd reusing" - do you mean "file descriptors been reused"? And like you, I am not a native English speaker, but you might try to find better words in your questions & comments

[lordofire]

I think my question is as simple as the following: in a complex multi-threading code I'm trying to debug now, I have seen the same fd (say 49) coming from open(curr_dev, O_RDWR) a lot of times ( which means fd 49 has been reclaimed successfully by kernel before everytime I called open() ), but I did not see explicit close(fd) in the code flow. So just wondering if there is any other possible way for system to reclaim that fd 49 instead of using close(fd). Hope this time it makes it clear. Sorry for any misunderstood I have introduced in my question and previous comment.

[Basile Starynkevitch]

No, fd has not been reclaimed by the kernel, but some other part of your program (your code, or some library that you are using) has explicitly done a close(y) with y been 49

[lordofire]

cool, then I think I would narrow my concentration onto finding which function or lib I have called have done the close(fd) for me. Thanks for your sharing.

[Stéphane Chazelas]

@lordofire see also ltrace -S to help identify shared library functions that may call close()

[G-Man Says 'Reinstate Monica']

@lordofire: If you're trying to debug a program to find out where a file descriptor (e.g., 49) gets closed (i.e., you have the source code, and you can edit it and recompile), you might want to look at the fstat(2) system call.  fstat(49, buf) returns 0 (and fills in buf) if file descriptor 49 is open, and it returns −1 if it is closed.  So, if you suspect function foo() of calling close(49), then you should call fstat(49, buf) before and after calling foo().

[Basile Starynkevitch]

@StéphaneChazelas: IIRC, pthreads should share file descriptors. Indeed, some other (non-POSIX) thread libraries might have separate fd-s.