Best order of RAID, LVM and LUKS

Question

I am going to install a file server with Debian GNU/Linux 8.

I want to have software RAID-1 using mdadm as well as encrypted disks using LUKS. I also like the flexibility of having LVM.

There are different ways of doing this. One way is to: First set up RAID-1 using mdadm. Then set up LVM PVs to cover the /dev/mdX created. Then set up LVs and encrypt those using LUKS. Then format these with a filesystem, say ext4. This results in a layout like this:

RAID --> LVM --> LUKS --> ext4

Or I could to it in a different order:

RAID --> LUKS --> LVM --> ext4

Or maybe even:

LVM --> RAID --> LUKS --> ext4

And maybe even a different order altogether.

What are benefits and drawbacks for the different approaches? What gives best performance, security, maintainability, etc.?

Is there a "Best Way" of doing this?

Solution 1

First, the order of LUKS and LVM depends on if you want to have different LUKS passwords or other settings for different LVs. If say, you need to set up different passwords for different LVs, you definitely need to put LUKS on top of LVM. On the other hand, if all LVs share the same password and settings like keylen, you'd want to have LUKS below LVM, so that you don't have to deal with the overhead of having more than one LUKS partition (think what you would have to do when you need to change the password).

Second, you almost always want RAID to be the lowest level, so that when one disk dies, it can be swapped easily and transparently. If you were to set up RAID on top of LVM, you'd have to replace a PV when one disk dies, that would be a big pain in the neck. Also RAID on top of LVM would totally defeat the flexibility of LVM. You then probably need to set up the second layer of LVM on top of RAID again!

Therefore, because in most cases people just need to use a single password, this would suffice:

RAID --> LUKS --> LVM --> ext4

In some cases, you might need to use LVM to combine multiple RAID devices into a large volume, then you can do:

RAID --> LVM --> LUKS (--> LVM) --> ext4

Theoretically the order shouldn't affect performance much, if every layer is set up correctly, and in practice I haven't seen this setup has a particularly bad performance. The most important thing is probably the alignment:

1. make sure your partitions are 1MB aligned (very important for SSD);
2. for RAID layer, pick chunk size wisely;
3. for LVM, make sure you set --dataalignment to match the RAID chunk size (this might be helpful).

Also if on SSD, make sure you enable LUKS TRIM/DISCARD pass-through by adding rd.luks.options=discard to /etc/default/grub and discard to /etc/crypttab (These are what I do on Red Hat/Fedora Linux. Might be a little different on Debian.) LVM and RAID should automatically support discard if you use a new-ish kernel.

Of course, these are just general guidelines. If you have special needs, feel free to update your question or comment here.

Solution 2

If you want all RAID, LUKS and LVM I would recommend RAID -> LUKS -> LVM -> FS. RAID --> LVM --> LUKS --> LVM --> FS is no better than RAID -> LUKS -> LVM - to extend volumes just add more RAID -> LUKS devices to a volume group.

RAID --> LVM --> LUKS --> FS - encrypting only particular Logical volumes has a feature of not encrypting everything by default (may be seen as advantage or disadvantage) but it will make extending root FS easier.

Extending LUKS on top of logical volumes is a common source of problems when users extend/resize them in the wrong order. Having LUKS on the whole md RAID device will simplify resizing - add new md device, create LUKS on top of that, add device to /etc/crypttab (at least on Fedora and RHEL clones) and extend your volume group. If root FS is on the volume group, you will need to add another rd.luks.uuid entry to kernel cmdline (edit /etc/default/grub and regenerate grub.cfg.)

LUKS -> RAID is usually wrong - data will be encrypted multiple times, consuming more CPU cycles with no gain. There is also a chance of mistakenly replacing a failed disk with a new one without setting up LUKS when a disk fails.

Extending and shrinking:

When extending always go from the bottom of the stack, when shrinking from the top.

Example:

Extending RAID -> LVM -> LUKS -> FS (the first two step are optional if there is enough free space in the volume group):

1. Add new disks and create md RAID.
2. Add mdX device to volume group.
3. Extend logical volume.
4. Extend LUKS device.
5. Extend FS.

Shrinking RAID -> LVM -> LUKS -> FS:

1. Shrink FS.
2. Shrink LUKS device.
3. Shrink logical volume.

Comments

[Thomas]

Thanks for the quick reply. Good points! Can you comment on the performance differences if they are non-trivial?

[Yan Li]

Updated with some performance notes.

[davidgo]

It should also be pointed out that you don't always need to use MDADM (although I expect this would be the most typical solution). - LVM offers RAID (aka mirroring)

[Yan Li]

@davidgo Yes. I thought about mentioning that too but didn't do it, because I'm not sure if that's the right way to go. I haven't tried it personally (mostly I use higher levels of RAID like 5 or 6), and have read mixed feedback on LVM mirroring, such as mentioned by some answers in this question.

[davidgo]

@Yan, I agree. I too always use mdadm RAid (1)