How to properly partition a LVM with two SSDs for maximum performance

Question

How would I partition two SSDs into a LVM for a Linux install? I have a 1TB Samsung SSD and then a 480GB NVMe SSD that's in the video card slot. I want to be able choose certain programs to install onto the 480GB NVMe so that they run at maximum speed since the read and write speeds will be higher. For example, if I install VirtualBox, I'll want it installed on the NVMe for maximum performance.

Should I be configuring my home partition to rest on the NVMe and install things in my home that I want to be at the best performance? Should I be making an /opt partition on the NVMe? How would i go about doing this?

Solution

For a workstation, it comes down to personal preference. I would likely create a /srv directory, then mount the NVMe's filesystem there. /opt is not typically for R/W storage. Virtualbox's performance is not based on where the application binary is installed, but instead where the disk image is stored, for example, on your NVMe drive is a good spot.

LVM doesn't really make sense for a single filesystem disk. It's not filesystem aware at all, so if you merge them into a single volume group (VG), you'd have to make separate logical volumes (LV) placed on specific physical volumes (PV). In a simple use case like storing VM images, I'd advise against it because it doesn't get you significant advantages unless you're doing disk passthrough to VMs. This is a lot more cumbersome to manage because you now have to manually allocate LVs for each VM by hand then manually create disk passthrough VMDKs, and if you want to store other filesystem data on the NVMe you'd need to create an LV for that too.

This answer assumes you're already installed to the SSD and are using the NVMe disk solely for high speed storage. If not, most installers will let you create arbitrary partitions and mount points as needed.

No LVM (eg. VDI files in the filesystem)

1. Create a partition and format it

You can use gparted to do this graphically. Select the device you want to modify, then run Device -> Create Partition Table, then Partition -> New, and finally Partition -> Format. The defaults are probably fine for each of these actions. Remember the filesystem type you choose in the last step.

2. Update fstab with the mount location

Use blkid or lsblk --fs to get the UUID number for your new filesystem. As root, open /etc/fstab in an editor like nano and add a line like the following

UUID=abcdef00-0000-0000-0000-000000000000   /srv   ext4 defaults 0 2

Replace the uuid with the one from your filesystem. Do not use quotes here. The filesystem type should be changed from ext4 to the type you selected in gparted.

Test mounting the filesystem.

sudo mount /srv
df -h

Don't provide extra options for mount because these should be detected from your fstab changes. If you need to provide options, then fstab is wrong and needs to be edited. If this is successful, df should show your NVMe device mounted at /srv. /srv will now be automatically mounted at boot.

Digitalocean has a detailed guide for creating and formatting a partition using the command line tools.

Now when you create virtual disks in vbox, place them in some subdirectory of /srv. You may need to change the permissions on directories to make that work (eg, sudo chown $(whoami) /srv to change the owner to yourself).

LVM (eg. VMDK disk passthrough)

1. Partition as above, but do not format

Resist the urge to use an unpartitioned disk. It's easier to make mistakes with it later when it has no partition information.

It's not strictly necessary, but you can set the Volume Type to Linux LVM.

2. Create Volume Group

Using the device from above run vgcreate to initialize a new volume group.

sudo vgcreate nvme /dev/sdb1

Replace /dev/sdb1 with the appropriate partition created above.

3. Create a LV for disk passthrough

If you wanted to create a 30GiB volume for a new VM named vm1, run lvcreate

sudo lvcreate -L30G -Cy -n vm1 nvme

Now use vboxmanage to create a VMDK file for use as a disk

sudo VBoxManage internalcommands createrawvmdk -filename $HOME/vm1.vmdx -rawdisk /dev/mapper/nvme-vm1

Device mapper names for LVM are in the format [vg name]-[lv name], so if you changed either above, change it appropriately here.

At this point, you may want to change the device permissions for this LV using udev.

There's a system-config-lvm tool for graphically managing LVM, but there's no graphical tool for creating the VMDK disk passthrough files that I'm aware of, and the VBoxManage gui can't do step 3 for you.

Comments

[davidgo]

LVM does offer advantages in a single disk setup, including things like snaphotting, the ability to shrink and increase non-contiguous partitions (and future proofing for if you get a bigger drive or second drive.)

[Andrew Domaszek]

There are a number of great reasons to use LVM, but it's not something I would recommend a user who wants to use vbox which I do not consider a production-grade virtualization platform. I'm particularly a fan of online LV disk migration with pvmove. In my experience, future-proofing with LVM trades management complexity in the period up to modification for a one-time future benefit that often never pays off. LVM is only slightly more complex to add later when making that transition to add new hardware. However, if the goal is to learn the linux disk subsystem, using LVM would be invaluable.