When would you use pivot_root over switch_root?

Question

I'm wanting to understand the Linux init process better in order to netboot a system over ceph rather than nfs.

In the process I've come across two forms of switching root. One called switch_root, and the other called pivot_root. These scripts being run from an in memory filesystem (initramfs) obtained via tftp using the pxe boot process.

When would you use one over the other? I've seen both used in some init script's placed in root.

Solution

I found a wonderful explanation here. However, let me try to put in a shorter format of what I understood in the answer.

Shorter Version

1. While the system boots, it needs an early userspace. It can be achieved using either initramfs or initrd.
2. initrd is loaded into ramdisk which is an actual FILE SYSTEM.
3. initramfs is not a file system.
4. For initrd pivot_root is used and for initramfs switch_root is used.

Longer Version

Now, to the detailed explanation of what I had put above.

While both an initramfs and an initrd serve the same purpose, there are 2 differences. The most obvious difference is that an initrd is loaded into a ramdisk. It consists of an actual filesystem (typically ext2) which is mounted in a ramdisk. An initramfs, on the other hand, is not a filesystem. It is simply a (compressed) cpio archive (of type newc) which is unpacked into a tmpfs. This has a side-effect of making the initramfs a bit more optimized and capable of loading a little earlier in the kernel boot process than an initrd. Also, the size of the initramfs in memory is smaller, since the kernel can adapt the size of the tmpfs to what is actually loaded, rather than relying on predefined ramdisk sizes, and it can also clean up the ram that was used whereas ramdisks tend to remain in use (due to details of the pivot_root implementation).

There is also another side-effect difference: how the root device (and switching to it) is handled. Since an initrd is an actual filesystem unpacked into ram, the root device must actually be the ramdisk. For an initramfs, there is a kernel "rootfs" which becomes the tmpfs that the initramfs is unpacked into (if the kernel loads an initramfs; if not, then the rootfs is simply the filesystem specified via the root= kernel boot parameter), but this interim rootfs should not be specified as the root= boot parameter (and there wouldn't be a way to do so, since there's no device attached to it). This means that you can still pass your real root device to the kernel when using an initramfs. With an initrd, you have to process what the real root device is yourself. Also, since the "real" root device with an initrd is the ramdisk, the kernel has to really swith root devices from one real device (the ramdisk) to the other (your real root). In the case of an initramfs, the initramfs space (the tmpfs) is not a real device, so the kernel doesn't switch real devices. Thus, while the command pivot_root is used with an initrd, a different command has to be used for an initramfs. Busybox provides switch_root to accomplish this, while klibc offers new_root.

Comments

[Daniel Alder]

I used pivot_root in the past for initramfs, switch_root didn't exist at that time. switch_root seems to be a convenience method ‎of pivot_root which does some more cleanup and also moves /proc /sys and /dev etc and not just the root itself

[TiCPU]

You can not use pivot_root on an initramfs rootfs, you will get Invalid Argument. You can only pivot real filesystems.

[Melab]

@TiCPU Then how would Linux exit early user space?

[erikbstack]

The provided solution seems to be wrong. Linus himself says that pivot_root() or chroot() will just change the current processes reference for /. So from my understanding it could be any path and has nothing to do with actual "disks".

[user2948306]

@Melab unix.stackexchange.com/questions/455223/…

[Terry Wang]

Wonderful explanation link is dead (to sourcemage wiki).