Moving a file while it's in use -- How does it work?
Solution 1
The directory entry is just a pointer to an inode. The inode contains the meta-information about the file (other than the name), and pointers to the file's data (if any). When you start to copy a file you get a handle to the inode.
The operating system maintains a count of references to the inode. As long as there are references to the inode, the inode and the file's data are kept. Once all references to the the inode are removed, the inode is and the space required by the file is released.
As you have the file open for copying it will be kept until your process closes the file. This should occur when the file transfer finishes, and will happen if the copy process fails. If the file transfer fails part way through and you have deleted all hard links to the file, you will be unable to successfully restart the transfer.
EDIT: As other have noted, file moves on the same device are done without moving the data. Instead a new directory entry is created in the destination directory, and the original directory entry is removed.
It is possible to have multiple directory entries for the same file. These are called hard links. They are created by making a new directory entry for the file without removing the original entry. The file system's inode has a reference count to record the number of directory entries pointing to the file.
EDIT2: If the process crashes or is killed, the file will be cleanly removed as the in memory access count will be reduced to zero. This is the action that occurs when the the program ends normally.
In the case of a power fail or other unorderly system shutdown, the disk will need an fsck
(file system check) before it can be fully mounted. Depending on the state of the on-disk inode and directory structures, the space will be recovered, the file will remain in the directory, or a new entry will be made in the lost+found
directory. The results will depend on which changes have been flushed to disk or written to the file systems journal.
Solution 2
As explained by Matt Jenkins, the OS (the filesystem) keeps track of files that are kept open by applications. As long as a process keeps a file open, its data stays on disk (even though it has been deleted and is no longer visible or accessible to other programs.
Note that a consequence of this is that the space occupied by a file can only be reclaimed once the last process using it has closed it. That's a FAQ for Linux/Unix file system operations: "'df' command says partition is full, while 'du' reports free space" (see e.g. http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html ). If you need to free up space, it's not enough to delete big files (e.g. logfiles), you must also make sure no process keeps them open (typically a problem with logfiles).
Solution 3
It's quite simple actually. The file maintains a list of references - processes that are accessing the file. When you delete the file it just removes the listing from the directory but not the file itself. The programs that still have the file open can still access it. The file is only actually deleted when all the programs that are accessing it close it.
Also, with moving the file - if it's within the same filesystem - the file doesn't actually move as such, it just changes the pointer to the directory the file is in.
Mario Zigliotto
Updated on September 17, 2022Comments
-
Mario Zigliotto over 1 year
I've noticed that on non-Windows OS.... ie Linux/Mac I can do things like:
- Send a zip to a friend over aim
- Delete the file while it's in transfer
And the transfer does not fail.
Or, I can do operations like..
- start a movie
- erase the file
- the movie still plays to completion (read from disk, not just buffered in memory)
Although the files are being "deleted", as i mentioned, they are actually being moved to a different location on the file system... ie a Trash directory or something. So it seems to me like the OS uses a pointer @ the file that is updated when it moves rather than accessing the files directly.
Can anyone shed some light on how this AWESOME capability is actually implemented? I'm not even sure what to google to learn more about it.
-
barlop about 13 yearsIn theory I suppose there are 2 ways it could happen. Both are reality. The file's in memory. Or the file system has references to the files and only the reference is deleted. Both are reality. The references fact is how undelete programs can undelete things. The idea that the file is actually deleted when all the programs acessing it close, sounds like nonsense. You could look up about linked lists(a programming data structure, to understand the concept in more depth). Or you could look up about particular file systems.
-
HikeMike about 13 years"The file maintains a list of references"?
-
Moab about 13 yearsShould be the "OS or Explorer" not file.
-
JRobert about 13 yearsThat would actually be the file-system.
-
sleske about 13 yearsTrue, but that's not what's happening here. The point is that even after the last hardlink is removed, the file still stays until all pocesses have closed their handles to them.
-
FJ de Brienne about 13 yearsThe OS uses the filesystem to store the references in the file (nlinks in the file header structure) - there - we're all happy ;)
-
mpez0 about 13 years"References" are both processes with an open handle to the file and hard links (including the first file name) to the file. The data blocks are not marked free and available for reuse until the reference count goes to zero.
-
HikeMike about 13 yearsNot true in case of OS X/HFS+: You can move files, e.g. to the Trash, but not across partitions or delete them (i.e. emptying the Trash).
-
Norman Gray about 13 yearsExactly. I couldn't put it better myself. Incidentally, a common trick, if you need some scratch file storage within a program, is to create a file in
/tmp
, and immediatelyunlink(2)
it. At that point, there's no file in a directory (so nothing to clean up at exit or crash), but your process still has access to the file, and no other process can accidentally or deliberately mess with it. That illustrates the property of interest. -
Jan-Philip Gehrcke over 11 yearsWhile you get it right roughly, my recommendation is not to try explaining things without being entirely sure that you get the facts straight. "The file maintains a list of references" as well as " The file is only actually deleted when all the programs that are accessing it close it" is how you believe things work. But you believe wrong.
-
Jason C almost 10 yearsDoes this mean that if an open file is removed while a program is using it and a power failure occurs, stranded data will remain on the drive occupying space? What about if the process using the file crashes or is killed?
-
BillThor almost 10 years@JasonC I have updated the response to answer your questions.