What happens when you 'mount over' an existing folder with contents?

Question

Right now /tmp has some temporary files in it. When I mount my hard drive (/dev/sdc1) on top of /tmp, I can see the files on the hard drive. What happens to the actual content of /tmp when my hard drive is mounted? Is it possible to perform r/w operations on the actual content of /tmp while the hard drive is mounted?

python@lanix / $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       286G   43G  229G  16% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.8G  4.0K  3.8G   1% /dev
tmpfs           766M  1.4M  765M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.8G   38M  3.8G   1% /run/shm
none            100M   24K  100M   1% /run/user
/dev/sdb1       7.5G  2.7G  4.9G  35% /mnt
/dev/sdc1       932G  242G  691G  26% /tmp

Solution

What happens to the actual content of /tmp when my hard drive is mounted?

Pretty much nothing. They're just hidden from view, not reachable via normal filesystem traversal.

Is it possible to perform r/w operations on the actual content of /tmp while the hard drive is mounted?

Yes. Processes that had open file handles inside your "original" /tmp will continue to be able to use them. You can also make the "reappear" somewhere else by bind-mounting / elsewhere.

# mount -o bind / /somewhere/else
# ls /somewhere/else/tmp  

---

Here's a little experiment you can run to get a better (I hope) feel for what is happening.

Note: This is not an attempt at being perfectly correct or an exhaustive description of what is really happening. Should be just accurate enough to give you the big picture though.

I created a user called me on my machine, and a random directory in his home, with a file in it:

me@home $ pwd
/home/me/tmp
me@home $ echo hello > some_file
me@home $ ls  
some_file
me@home $ cat some_file 
hello

At this point, nothing unusual - it's just a plain directory with a plain file. I leave that session open right as it is, with its cwd inside that test directory.

As root, I create a small filesystem and mount it over /home/me/tmp.

root@home # dd if=/dev/zero of=./fs bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00467318 s, 2.2 GB/s

root@home # mkfs -t ext2 ./fs 
mke2fs 1.42.12 (29-Aug-2014)
[... snip ...]
Writing superblocks and filesystem accounting information: done

root@home # mount ./fs /home/me/tmp

I then open a new terminal as me, and look around:

me@home #2 $ cd tmp
me@home #2 $ ls
lost+found
me@home #2 $ cat some_file
cat: some_file: No such file or directory
me@home #2 $ echo bye bye > some_file
-su: some_file: Permission denied

So, that file we created is clearly not there. The lost+found directory is indicative of the root of an ext filesystem. And I lost write permission, so it's clearly not the original directory.

Back to the first me session, let's look at how it sees the world:

me@home $ echo something else > other_file

No problem writing.

me@home $ cat some_file other_file 
hello
something else

Original file is still there, new file created without issue.

Huh? What's going on?

The first session entered the directory before it was overlayed by root mounting another filesystem on it. That mount action doesn't affect the original filesystem at all. The shell process has a perfectly valid handle to the directory in the original filesystem, and can continue interacting with it. It's sort of running around beneath the carpet mount point.

The second session entered the directory after the mount was laid down. So it sees the new, empty filesystem. And the sysadmin borked the permissions, so it can't use the requested space... lets fix that.

root@home # chown me:users /home/me/tmp

me@home #2 $ echo bye bye > some_file
me@home #2 $ ls 
lost+found  some_file
me@home #2 $ cat some_file 
bye bye

Can session 1 escape from under the rug? (It's getting musty.)

Sure! If session 1 moves back up the filesystem tree out of the mount, it will lose that handle to the inside and will follow the mount like everyone else.

me@home $ cd
me@home $ pwd
/home/me
me@home $ cd tmp
me@home $ cat some_file other_file
bye bye
cat: other_file: No such file or directory

Same view as session #2, we're back to normal.

But how do you know the files didn't disappear? No one's looking anymore!

That's one of the moments where bind mounts become handy. They let you mount an already mounted filesystem somewhere else.

me@home $ mkdir ~/bind

root@home # mount -o bind /home/me /home/me/bind

(Yes, you can bind-mount a filesystem "inside itself". Cool trick, eh?)

me@home $ ls bind/tmp
other_file  some_file
me@home $ cat bind/tmp/*
something else
hello

So they are indeed there, ready for action. It's simply that they're not visible/reachable at their original location, the mount hides them from normal directory traversals.

---

I encourage you to play around with this, it is really not complicated once you've understood the "trick" that is being played. And once you've Got It™, look into union filesystems for even more carpet pulling :-)

One note though: mounting over /tmp or /var (or any of the core OS directories) really isn't a good idea once the boot process is finished. A lot of applications leave state in those directories, and might get seriously confused if you play mount games around them.

Comments

[user]

This is a great answer - you have gone above and beyond what I asked of you. The idea of bind-mount is pretty cool too! Thanks for the detailed answer. Cheers.

[Dan Sheppard]

This is a very common way to mysteriously lose disk space. If a mount fails for whatever reason in a startup script, data can be written to the directory on the root filesystem. If a restart is then attempted, the mount can succeed and maybe nobody will notice (for example, if the filesystem contains tmp files or logs), except it will eat space, maybe a lot.

[Zan Lynx]

@DanSheppard That's one reason I like my mount points set chmod 000. Also why systemd fails the boot if critical mounts fail.

[jiggunjer]

Could you also -bind mount /home/me on /home/me instead of the "bind" folder? I.e. put another carpet on top of the carpet. Or would you have to unmount fs first?

[hliu]

@jiggunjer It seems union option can help.

[rottweilers_anonymous]

Thanks Mat, this was incredibly interesting and helpful

[Maryna Klokova]

Thank you so much!

[Ryan]

Thanks so much. I had a large 100G file hiding under a mount point which took me ages to find.