SSH to decrypt encrypted LVM during headless server boot?

Question

When I installed Ubuntu 10.04 and, now, 10.10, I was offered the option of enabling "encrypted LVM" for my hard drive. After choosing that option, I am prompted for my password during boot to decrypt the LVM.

Now, I am thinking about setting up a headless server that runs Linux (not necessarily Ubuntu), but I am worried that since the server is headless I won't be able to decrypt it during startup. Would I be able to SSH in during boot to enter my password for the encrypted LVM? If so how do I set it up? Or is there another solution? Again this question is NOT specific to Ubuntu. Thanks.

Solution 1

For newer versions of ubuntu, for example, 14.04, I found a combination of @dragly and this blogposts' answers very helpful. To paraphrase:

1. (On server) Install Dropbear

   sudo apt-get install dropbear
   

2. (On server) Copy and assign permissions for root public/private key login

   sudo cp /etc/initramfs-tools/root/.ssh/id_rsa ~/.
   sudo chown user:user ~/id_rsa
   

remember to change user to your username on the server

3. (On client) Fetch private key from server

   scp [email protected]:~/id_rsa ~/.ssh/id_rsa_dropbear
   

4. (On client) Add an entry to ssh config

   Host parkia
       Hostname 192.168.11.111
       User root
       UserKnownHostsFile ~/.ssh/know_hosts.initramfs
       IdentityFile ~/.ssh/id_rsa_dropbear
   Remember to change _parkia_ to whatever you'd like to type `ssh my-box` to be.
   

5. (On server) Create this file at /etc/initramfs-tools/hooks/crypt_unlock.sh

6. (On server) Make that file executable

   sudo chmod +x /etc/initramfs-tools/hooks/crypt_unlock.sh
   

7. Update the initramfs

   sudo update-initramfs -u
   

8. Disable the dropbear service on boot so openssh is used after partition is decrypted

   sudo update-rc.d dropbear disable
   

You're done. Try it out. Check the blog post linked to above for instructions about how to configure the server with a static IP address if that is something you'd need to do.

Solution 2

A guide to do such a setup with BusyBox and Dropbear is shown in this blog post. early-ssh didn't work for me and is apparently not needed anymore.

I have summarized what you need to do in the following. For more details, have a look at the post above:

1. Install BusyBox and Dropbear on your server

   sudo apt-get install dropbear busybox
   

2. Update your initramfs on the server

   sudo update-initramfs -u
   

3. Copy the private key generated by dropbear to your client machine. You may have to copy this to a new dir and change ownership to do this. On your server do the following:

   sudo cp /etc/initramfs-tools/root/.ssh/id_rsa ~/.
   sudo chown user:user ~/id_rsa
   

   Remember to replace user with your username. Password logins don't seem to work.

4. Now you may transfer the private key with scp by calling the following on your client:

   scp [email protected]:~/id_rsa ~/.ssh/id_rsa_dropbear
   

5. Set up your client's ~/.ssh/config file for easy login. Open it up with a text editor and add the following:

   Host myremoteserver
       HostName my.remote.server
       User root
       UserKnownHostsFile ~/.ssh/known_hosts.initramfs
       IdentityFile ~/.ssh/id_rsa_dropbear
   

   Change the Host to whatever you like and HostName to the name of your server. Let the user be root. It appears to be the only accepted user in Dropbear. Save and close the file.

6. Restart your server and wait for the passphrase prompt. Give Dropbear a few seconds to detect and set up its internet connection. Connect to your server with the following command on your client:

   ssh myremoteserver # or any name you chose
   

7. When logged in, issue the following command on your server. See the blog post for details:

   pid=`ps | grep "/scripts/local-top/cryptroot" | cut -d " " -f 3`
   kill -9 $pid
   sleep 35
   /scripts/local-top/cryptroot
   pid=`ps | grep "/bin/sh" | cut -d " " -f 3`
   kill -9 $pid;
   

   It will take some time (30 seconds) before you get to type your passphrase. Type it in when prompted.

8. Close the connection by typing

   exit
   

9. Your server should now have unlocked its encrypted hard drive and boot as normal.

(A huge thanks to the original author of the blog post!)

Solution 3

I think early-ssh provides what you're searching for:

Early-ssh is a simple initramfs hook, which installs Dropbear SSH server into  
your initramfs, and starts it at boottime, so you will be able to do a lot of  
things remotely over SSH, before your root partition gets mounted, for example:

* unlocking LUKS encrypted crypto devices - 
  even your root can be an encrypted filesystem
* assembling/altering RAID arrays (mdadm)
* checking the root filesystem in read-write mode, 
  taking action in case of errors
* and so on...

There is already a .deb package available, so you're probably fine with Ubuntu.

Solution 4

Have a look at the cryptsetup readme for this in /usr/share/doc/cryptsetup/README.remote.gz (Ubuntu package cryptsetup). In there is a full guide to accomplish this. It is similar to dragly's answer, but I think this is a bit more elegant. (Dropbear formatted keys, passing the passphrase via a FIFO rather than a fragile shell script, etc.)

unlocking rootfs via ssh login in initramfs

You can unlock your rootfs on bootup from remote, using ssh to log in to the booting system while it's running with the initramfs mounted.

Setup

For remote unlocking to work, the following packages have to be installed before building the initramfs: dropbear busybox

The file /etc/initramfs-tools/initramfs.conf holds the configuration options used when building the initramfs. It should contain BUSYBOX=y (this is set as the default when the busybox package is installed) to have busybox installed into the initramfs, and should not contain DROPBEAR=n, which would disable installation of dropbear to initramfs. If set to DROPBEAR=y, dropbear will be installed in any case; if DROPBEAR isn't set at all, then dropbear will only be installed in case of an existing cryptroot setup.

The host keys used for the initramfs are dropbear_dss_host_key and dropbear_rsa_host_key, both located in /etc/initramfs-tools/etc/dropbear/. If they do not exist when the initramfs is compiled, they will be created automatically. Following are the commands to create them manually:

dropbearkey -t dss -f /etc/initramfs-tools/etc/dropbear/dropbear_dss_host_key
dropbearkey -t rsa -f /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key

As the initramfs will not be encrypted, publickey authentication is assumed. The key(s) used for that will be taken from /etc/initramfs-tools/root/.ssh/authorized_keys. If this file doesn't exist when the initramfs is compiled, it will be created and /etc/initramfs-tools/root/.ssh/id_rsa.pub will be added to it. If the latter file doesn't exist either, it will be generated automatically - you will find the matching private key which you will later need to log in to the initramfs under /etc/initramfs-tools/root/.ssh/id_rsa (or id_rsa.dropbear in case you need it in dropbear format). Following are the commands to do the respective steps manually:

To create a key (in dropbear format):

dropbearkey -t rsa -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear

To convert the key from dropbear format to openssh format:

/usr/lib/dropbear/dropbearconvert dropbear openssh \
    /etc/initramfs-tools/root/.ssh/id_rsa.dropbear \
    /etc/initramfs-tools/root/.ssh/id_rsa

To extract the public key:

dropbearkey -y -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear | \
    grep "^ssh-rsa " > /etc/initramfs-tools/root/.ssh/id_rsa.pub

To add the public key to the authorized_keys file:

cat /etc/initramfs-tools/root/.ssh/id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys

In case you want some interface to get configured using dhcp, setting DEVICE= in /etc/initramfs-tools/initramfs.conf should be sufficient. The initramfs should also honour the ip= kernel parameter. In case you use grub, you probably might want to set it in /boot/grub/menu.lst, either in the '# kopt=' line or appended to specific 'kernel' line(s). The ip= kernel parameter is documented in Documentation/nfsroot.txt in the kernel source tree.

Issues

Don't forget to run update-initramfs when you changed the config to make it effective!

Collecting enough entropy for the ssh daemon sometimes seems to be an issue. Startup of the ssh daemon might be delayed until enough entropy has been retrieved. This is non-blocking for the startup process, so when you are at the console you won't have to wait for the sshd to complete its startup.

Unlocking procedure

To unlock from remote, you could do something like this:

ssh -o "UserKnownHostsFile=~/.ssh/known_hosts.initramfs" \
    -i "~/id_rsa.initramfs" [email protected] \
    "echo -ne \"secret\" >/lib/cryptsetup/passfifo"

This example assumes that you have an extra known_hosts file "~/.ssh/known_hosts.initramfs" which holds the cryptroot system's host-key, that you have a file "~/id_rsa.initramfs" which holds the authorized-key for the cryptroot system, that the cryptroot system's name is "initramfshost.example.com", and that the cryptroot passphrase is "secret"

-- <[email protected]>, Wed, 30 Sep 2009

Thanks to jap for pointing this out to me on a different channel.

Solution 5

If you want to be able to boot unattended as well as remotely, you should also look at Mandos (which I and others have written):

Mandos is a system for allowing servers with encrypted root file systems to reboot unattended and/or remotely. See the intro manual page file for more information, including an FAQ list.

In short, the booting server gets the password over the network, in a secure fashion. See the README for details.

View more solutions

Comments

[Admin]

Also see: zless /usr/share/doc/cryptsetup/README.remote.gz

[Admin]

I think @Nate 's answer should be the accepted one: it (essentially, since an edit is required to reflect changes in the linked blog) uses public keys throughout instead of private ones.

[hpy]

Looks like this is exactly what I am looking for, thanks!

[Admin]

Do you have a link to a good tutorial or howto? I am stuck with early-ssh on my debian squeeze box.

[Gilles 'SO- stop being evil']

Thank you for your contributions, but note that 100% of your posts being almost-identical mentions of your product is borderline behavior, and you run a risk of being considered spam (I'd have flagged your posts if Mandos wasn't free software or you didn't have a history of non-Mandos posts on other sites).

[hpy]

Yes a tutorial would be great.

[Teddy]

@Gilles: Done now.

[gertvdijk]

I don't quite understand the reason for moving around with private keys. It should be sufficient to copy the public key of you on the client machine to the server as authorized key for the root server, right?

[dragly]

I'm sure it's possible to create the key pair on the client machine and only move the public key from there to the server, but if I recall correctly I think there were some problems finding a format that BusyBox would accept. So reusing the keys that were already on the server was the only option I got working.

[Gerharddc]

Any idea what I should do to get this working on Arch Linux?

[tcoolspy]

@Gerhman Check out the dropbear_initrd_encrypt package in the AUR for early-ssh support on Archlinux.

[joelpet]

This seems like a much better idea (being described in the offical docs and everything) than hackish ps-grepping. As a side note on the unlocking procedure provided though, one might want to be cautious about typing the passphrase directly on the command line as it will most likely end up in a shell history file somewhere. A possible solution is creating a tiny wrapper script prompting for the passphrase using read -s -p.

[hanetzer]

@Gerhman archwiki page: Remote unlocking of the root or other partition Haven't done it yet, but it looks interesting. Will have to check it out :)

[Frederick Nord]

note that there are issues with that approach in recent Ubuntu versions, i.e. bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/595648

[Gbo]

roosbertl.blogspot.de/2012/12/…

[Jonathan Y.]

The linked blog has added a reference to appending a client's public key to the server's /etc/initramfs-tools/root/.ssh/authorized_keys, even if it still goes through copying Dropbear's private key, which one can completely disregard. Following the rest of the instructions works for me, which means this should be the accepted answer (once it reflects that change), because it only uses public keys.

[ILMostro_7]

Great point! Though, "non-network" ways of getting in a headless server are mostly (only) relevant if the physical proximity between client/server is close; unless I'm overlooking some other possibility/feature of a Serial connection.

[tobiasBora]

I'm sorry I don't remember (and I'm not using it right now so can't really test). I'd say no like that, but not sure. But anyway, it's not hard to try I guess, let us know if you try.

[Dan Bolser]

Wow... Did this work?

[Fuad Ak]

It worked in 2015, but I'm not using this setup anymore.

[balasaheb barde]

@nsg what are you using now?