Hetzner root server full disk encryption with cryptsetup LUKS

6 July 2015 in Kizajournal

Updated 29 Dec 2015

This howto turns a Hetzner default Debian root server installation into a fully encrypted server with LUKS that can be unlocked remotely with the dropbear ssh server. If the server is rebooted, seized or a KVM switch is plugged into the wrong server, the data cannot be accessed. Likewise, your data will be lost forever* if you forget the passphrase. This works with Debian Jessie and Wheezy images. But read the note on Jessie below.

(* a long time)

Please note the date of this post. If it is older than a year, the defaults of the Hetzner install image might have changed. Paths and devices might be named differently. I assume the following (Hetzner default) disk layout. If you have configured your server differently, change the device names accordingly.

md0: Raid1 consisting of sda1, sdb1 as swap md1: Raid1 consisting of sda2, sdb2 as /boot md2: Raid1 consisting of sda3, sdb3 as the root filesystem

The server will not be reinstalled from scratch. All data is preserved (read below). You can encrypt an existing server without any config changes to the services it provides.

Debian Jessie

If you're using Jessie and onward, the init system will be systemd by default. Because of an unresolved bug with crypto luks + systemd + certain crypttab options, including keyscript, you'll have two options with this constellation. There are, of course, other more hackish approaches, but it is up to you to evaluate their safety.

Without systemd

Install the package sysvinit-core before you reboot. This removes systemd-sysv, which makes the system default to systemd as PID 1. Your system will continue to use the old sysvinit. Nothing else will change. Everything that worked in Wheezy will continue to do so.

With systemd

First copy /sbin/init to /sbin/init.sysv This will give an easy way back to sysvinit if thins go haywire. Install systemd-sysv to switch the default to systemd. Add a kernel parameter luks=no to the kernel command line in /etc/default/grub and run update-grub && update-initramfs -u. This will now tell systemd not to unlock the disk at boot, the old scripts in initramfs will take care of that.

There is one important note however: no other disk will be unlocked at boot time, only root. So if you have any other entry in crypttab, it will not be available to auto-mounting at boot and systemd will fail to bring up that mount point. That includes the encrypted swap further down in this guide. You will have to mount the rest of the paritions manually after each boot. If it's just swap, you could also remove it completely, 32GB or even 64GB in the current servers should be enough for almost every workload.

If the system now fails to boot, wait at least 1m30s, that's how long systemd tried to mount the missing devices until it gave up on my test system, pass the following kernel parameter on boot: init=/sbin/init.sysv. You can either do that with a LARA (KVM) console or edit your config in the rescue system. That will switch to sysvinit for this one boot.

Additional packages that must be present on your current system

Install these in your current installation before you proceed.

apt-get install cryptsetup apt-get install busybox dropbear

You should have cryptsetup installed before you install dropbear. There might be some strange side effects with the initramfs config otherwise. I have received such a report. At least it doesn't hurt.

Preparation

There is a complete description on how to generate the keys and unlock the disk in the cryptsetup documentation. It is installed in /usr/share/doc/cryptsetup/README.remote.gz. Configure dropbear following the readme before you proceed.

If you have configured it and had the private/public SSH keys generated automatically, do not forget to copy the private key or install your public key according to the readme.

Reboot into rescue system

Create a backup if you're working with important data!

Log into the robot and reboot the server into the (64bit) rescue system. At the time of this writing every tool was available in the rescue system.

Save your old system

The old system is moved aside while the encrypted disk is prepared. If you have more data than what fits onto the root filesystem, you need to move large data directories either to the backup FTP or restore them from a backup afterwards. In the rescue system you will have a root filesystem ramdisk which is half the size of the server's memory. So if you have a 32GB root server you will have ~16GB available.

mkdir /oldroot mount /dev/md2 /mnt rsync -a /mnt/ /oldroot/ umount /mnt

Your system is now mirrored into /oldroot.

Create the encrypted disk

Let's create the encrypted filesystem. This will destroy all data on your server! You might want to use a diceware password so you don't have to enter any special characters while booting the system. Example on xkcd. But don't use an online generator or the horse staple. :) Or for example: dd if=/dev/random bs=1 count=18 | openssl base64

The following command will create a LUKS device using the aes-xts-plain64 cipher with 512 bits key size and a number of key iterations that will take roughly 5 seconds to compute on your hardware. Meaning it will take 5 seconds to unlock the disk. Since servers are rarely rebooted, it does not hurt.

cryptsetup --cipher aes-xts-plain64 -s 512 --iter-time 5000 luksFormat /dev/md2

You can verify that everything went fine with the luksDump command. It should produce something like the following output:

# cryptsetup luksDump /dev/md2 LUKS header information for /dev/md2 Version: 1 Cipher name: aes Cipher mode: xts-plain64 Hash spec: sha1 Payload offset: 4096 MK bits: 512 MK digest: de ad be ef MK salt: de ad be ef de ea be ef MK iterations: 83750 UUID: 5f76b99e-0418-4fb5-a731-201004ede948 Key Slot 0: ENABLED Iterations: 336005 Salt: de ea be ef de ea be ef Key material offset: 8 AF stripes: 4000 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED

Once the LUKS device is created, activate it. This will ask for the passphrase you've just used.

cryptsetup luksOpen /dev/md2 root

The following step is optional but recommended. It fills the whole disk with random data and overwrites anything on the partition that may be recoverable. Even though it reads from /dev/zero, you are writing to the high level LUKS device which writes random encrypted noise to the disk. This might take a while. 2-4 hours with the current disk sizes. You can skip this step and create a large file on the filesystem after everything is finished with the same effect.

dd if=/dev/zero of=/dev/mapper/root bs=1M

Create a filesystem on the encrypted device. I will use ext4 with default options and set check interval to 6 months, error behavior to remount-ro and max mount count to 50.

mke2fs -t ext4 /dev/mapper/root tune2fs -i 6m -e remount-ro -c 50 /dev/mapper/root

You can now restore your data.

Reinstall the system

mkdir /newroot mount /dev/mapper/root /newroot rsync -a /oldroot/ /newroot/

Before we can changeroot into the new system we need some special file systems and the boot file system inside. These are needed so grub and initramfs reinstallation work.

mount /dev/md1 /newroot/boot mount --bind /dev /newroot/dev mount --bind /sys /newroot/sys mount --bind /proc /newroot/proc chroot /newroot

The fstab needs to be changed to reflect the new root file system. Edit the file and replace / (root) with the following:

/etc/fstab: /dev/mapper/root / ext4 defaults 0 2

To be able to boot from the encrypted file system we need a crypttab.

/etc/crypttab: # <target name> <source device> <key file> <options> root /dev/md2 none luks

Now all is set. We need to tell grub the new root device and regenerate the initramfs to include the encryption stuff. Run the following commands. They should automatically pick up /dev/mapper/root as the new root device.

update-initramfs -u update-grub grub-install /dev/sda grub-install /dev/sdb

IPv6

IPv6 will not be running after a reboot. The IPv4 networking will be brought up in the preboot ssh server. Once Debian boots the real system, configuring the network interfaces will fail, because eth0 was already configured with ip earlier and the network will be in a somewhat inconsistent state. I fixed this be forcing the network down and up at the end of the boot sequence. Add the following two lines to your /etc/rc.local:

/sbin/ifdown --force eth0 /sbin/ifup --force eth0

Reboot

You can now leave the chroot, reboot the server and hope for the best. :)

logout umount /newroot/boot umount /newroot/proc umount /newroot/sys umount /newroot/dev umount /newroot sync reboot

The system will now boot into the dropbear ssh server which you can connect to and enter the passphrase to unlock the disk. The system will then continue to boot. Once the server answers pings again, ssh into it. To enter the passphrase use the following command:

echo -n "YOUR SUPER SECRET PASSPHRASE" >/lib/cryptsetup/passfifo

The system will now disconnect your ssh. If not you may have used the wrong passphrase. My terminal was kind of broken afterwards and I had to restart the shell. You now should be able to connect normally.

If you have skipped the optional step (writing the disk full) you should do it now. The easiest way is to create a large file with dd.

dd if=/dev/zero of=foofile bs=1M

You should stop any databases or other software that does not like "No space left on device" before writing the disk full. <Insert Mysql bashing here>

Encrypted swap

The swap can also be encrypted with a random key at boot time. This works automagically. Add the following line into your /etc/crypttab. This sets up an encrypted swap device with a random key as /dev/mapper/swap everytime the system boots.

swap /dev/md0 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

And change the swap entry in your fstab into the following:

/dev/mapper/swap none swap sw 0 0

DISASTER!

Don't panic. After all you have backups!

If you cannot log in via dropbear at all, because you have forgotten to copy the generated private key to your system or did not put your public key onto the server, it can be fixed easily. Just boot back into the rescue system. You can edit the config and regenerate the initramfs with the same commands as used above.

If login does work, first check the passphrase. You never know, you might just have a typo. If it still does not work, reboot the system into the rescue mode. Try unlocking the disk manually.

cryptsetup luksOpen /dev/md2 root

If that does not work you might have mistyped it while creating the LUKS device. Either try some common typos or you will have to start over.

If you can successfully unlock the disk, mount it and check if you have a problem with the dropbear configuration. See the readme mentioned in the section "Preparation" for any errors you might have in your configuration.

What has changed now?

The encryption is completely transparent. The only user-visible things that have changed are the following devices:

/dev/md0 is now /dev/mapper/swap /dev/md2 is now /dev/mapper/root

Everything else, including disaster recovery from failed hard drives has not changed. The disks are still part of the underlying MD device and do not know anything about the encryption. The boot partition is still unencrypted.

How secure is it?

The encryption itself should be bullet proof as long as the passphrase is complex enough of course. But even if it is "i hate passwords" you should be fairly safe since we used 5 seconds worth of iterations. On Hetzner's EX40 with an i7-4770 it results in 1.5 million iterations and that is (was? see post date!) a very fast CPU.

The boot partition is unencrypted. If someone gained access to your robot account and boots the server into the rescue system he can, of course, modify the initramfs and install software that will compromise your system if you login. The robot logs these reboot requests so if you didn't request it, be very wary and do not log in. The encrypted data is most likely not compromised or modified. We used aes-xts-plain64 which does not suffer from an issue like the default cbc-essiv mode. See for example the description of the malleability attack.

You could probably make a copy of your boot partition which you can restore in case of an unexplained reboot or checksums of the files which you can compare while in rescue mode.

If you need more security than that, only systems with TPM can provide a secure boot path that prevents tampering.

Checklist

If you want, you can use the checklist below while you set up the server. All important steps are recorded, so you're unlikely to miss one.