Journal


Profile: Bishop's Crown

When I first sowed the Bishop's Crown seed in June this year I was more expecting it to become an indoor chili for the window sill. But it turns out they are even faster growers than the Fataliis and after only 1.5 months the plant is 75cm high. Not even a small pot prevented it from growing. It just became offended, letting its leaves hanging, and I had to put it into a bigger pot.

After a bit of research it turned out that these plants grow up to 2-3m high. They seem to grow much larger than your standard capsicum chinense plant. It even grew so fast that the trunk is kinda thin and it seems to have trouble getting enough water to the topmost leaves.

Environmental conditions

Hailing from the Carribean they should be accustomed to tropical climate. Interestingly 35°C behind a window glass (greenhouse like) seems to be almost uncomfortable and 40°C in direct sunlight is too much for them. They seem to like it much better in the shade and, in contrast to other chilis I have, need plenty of water. While others are fine with the soil completely dry for a day or two the Bishop's Crowns seem to prefer it mostly wet.


The Test: Carolina Reaper

The first Carolina Reaper chili was ready today. Compared to other red chilis they look more orangish and quite poisonous if you ask me. Ah well, no pain no gain. So I made a traditional chili con carne and spiced it up quite a bit with a finely chopped chili. The conclusion: they burn as hot as they look! I'd say the stories you've read elsewhere are definitely true. This is a weaponized chili pepper. If you enjoy pain or like to see others gasp for air while they try to eat your evil cuisine, give this one a try. :)


Hetzner root server full disk encryption with cryptsetup LUKS

Updated 29 Dec 2015

This howto turns a Hetzner default Debian root server installation into a fully encrypted server with LUKS that can be unlocked remotely with the dropbear ssh server. If the server is rebooted, seized or a KVM switch is plugged into the wrong server, the data cannot be accessed. Likewise, your data will be lost forever* if you forget the passphrase. This works with Debian Jessie and Wheezy images. But read the note on Jessie below.

(* a long time)

Please note the date of this post. If it is older than a year, the defaults of the Hetzner install image might have changed. Paths and devices might be named differently. I assume the following (Hetzner default) disk layout. If you have configured your server differently, change the device names accordingly.

md0: Raid1 consisting of sda1, sdb1 as swap md1: Raid1 consisting of sda2, sdb2 as /boot md2: Raid1 consisting of sda3, sdb3 as the root filesystem

The server will not be reinstalled from scratch. All data is preserved (read below). You can encrypt an existing server without any config changes to the services it provides.

Debian Jessie

If you're using Jessie and onward, the init system will be systemd by default. Because of an unresolved bug with crypto luks + systemd + certain crypttab options, including keyscript, you'll have two options with this constellation. There are, of course, other more hackish approaches, but it is up to you to evaluate their safety.

Without systemd

Install the package sysvinit-core before you reboot. This removes systemd-sysv, which makes the system default to systemd as PID 1. Your system will continue to use the old sysvinit. Nothing else will change. Everything that worked in Wheezy will continue to do so.

With systemd

First copy /sbin/init to /sbin/init.sysv This will give an easy way back to sysvinit if thins go haywire. Install systemd-sysv to switch the default to systemd. Add a kernel parameter luks=no to the kernel command line in /etc/default/grub and run update-grub && update-initramfs -u. This will now tell systemd not to unlock the disk at boot, the old scripts in initramfs will take care of that.

There is one important note however: no other disk will be unlocked at boot time, only root. So if you have any other entry in crypttab, it will not be available to auto-mounting at boot and systemd will fail to bring up that mount point. That includes the encrypted swap further down in this guide. You will have to mount the rest of the paritions manually after each boot. If it's just swap, you could also remove it completely, 32GB or even 64GB in the current servers should be enough for almost every workload.

If the system now fails to boot, wait at least 1m30s, that's how long systemd tried to mount the missing devices until it gave up on my test system, pass the following kernel parameter on boot: init=/sbin/init.sysv. You can either do that with a LARA (KVM) console or edit your config in the rescue system. That will switch to sysvinit for this one boot.

Additional packages that must be present on your current system

Install these in your current installation before you proceed.

apt-get install cryptsetup apt-get install busybox dropbear

You should have cryptsetup installed before you install dropbear. There might be some strange side effects with the initramfs config otherwise. I have received such a report. At least it doesn't hurt.

Preparation

There is a complete description on how to generate the keys and unlock the disk in the cryptsetup documentation. It is installed in /usr/share/doc/cryptsetup/README.remote.gz. Configure dropbear following the readme before you proceed.

If you have configured it and had the private/public SSH keys generated automatically, do not forget to copy the private key or install your public key according to the readme.

Reboot into rescue system

Create a backup if you're working with important data!

Log into the robot and reboot the server into the (64bit) rescue system. At the time of this writing every tool was available in the rescue system.

Save your old system

The old system is moved aside while the encrypted disk is prepared. If you have more data than what fits onto the root filesystem, you need to move large data directories either to the backup FTP or restore them from a backup afterwards. In the rescue system you will have a root filesystem ramdisk which is half the size of the server's memory. So if you have a 32GB root server you will have ~16GB available.

mkdir /oldroot mount /dev/md2 /mnt rsync -a /mnt/ /oldroot/ umount /mnt

Your system is now mirrored into /oldroot.

Create the encrypted disk

Let's create the encrypted filesystem. This will destroy all data on your server! You might want to use a diceware password so you don't have to enter any special characters while booting the system. Example on xkcd. But don't use an online generator or the horse staple. :) Or for example: dd if=/dev/random bs=1 count=18 | openssl base64

The following command will create a LUKS device using the aes-xts-plain64 cipher with 512 bits key size and a number of key iterations that will take roughly 5 seconds to compute on your hardware. Meaning it will take 5 seconds to unlock the disk. Since servers are rarely rebooted, it does not hurt.

cryptsetup --cipher aes-xts-plain64 -s 512 --iter-time 5000 luksFormat /dev/md2

You can verify that everything went fine with the luksDump command. It should produce something like the following output:

# cryptsetup luksDump /dev/md2 LUKS header information for /dev/md2 Version: 1 Cipher name: aes Cipher mode: xts-plain64 Hash spec: sha1 Payload offset: 4096 MK bits: 512 MK digest: de ad be ef MK salt: de ad be ef de ea be ef MK iterations: 83750 UUID: 5f76b99e-0418-4fb5-a731-201004ede948 Key Slot 0: ENABLED Iterations: 336005 Salt: de ea be ef de ea be ef Key material offset: 8 AF stripes: 4000 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED

Once the LUKS device is created, activate it. This will ask for the passphrase you've just used.

cryptsetup luksOpen /dev/md2 root

The following step is optional but recommended. It fills the whole disk with random data and overwrites anything on the partition that may be recoverable. Even though it reads from /dev/zero, you are writing to the high level LUKS device which writes random encrypted noise to the disk. This might take a while. 2-4 hours with the current disk sizes. You can skip this step and create a large file on the filesystem after everything is finished with the same effect.

dd if=/dev/zero of=/dev/mapper/root bs=1M

Create a filesystem on the encrypted device. I will use ext4 with default options and set check interval to 6 months, error behavior to remount-ro and max mount count to 50.

mke2fs -t ext4 /dev/mapper/root tune2fs -i 6m -e remount-ro -c 50 /dev/mapper/root

You can now restore your data.

Reinstall the system

mkdir /newroot mount /dev/mapper/root /newroot rsync -a /oldroot/ /newroot/

Before we can changeroot into the new system we need some special file systems and the boot file system inside. These are needed so grub and initramfs reinstallation work.

mount /dev/md1 /newroot/boot mount --bind /dev /newroot/dev mount --bind /sys /newroot/sys mount --bind /proc /newroot/proc chroot /newroot

The fstab needs to be changed to reflect the new root file system. Edit the file and replace / (root) with the following:

/etc/fstab: /dev/mapper/root / ext4 defaults 0 2

To be able to boot from the encrypted file system we need a crypttab.

/etc/crypttab: # <target name> <source device> <key file> <options> root /dev/md2 none luks

Now all is set. We need to tell grub the new root device and regenerate the initramfs to include the encryption stuff. Run the following commands. They should automatically pick up /dev/mapper/root as the new root device.

update-initramfs -u update-grub grub-install /dev/sda grub-install /dev/sdb

IPv6

IPv6 will not be running after a reboot. The IPv4 networking will be brought up in the preboot ssh server. Once Debian boots the real system, configuring the network interfaces will fail, because eth0 was already configured with ip earlier and the network will be in a somewhat inconsistent state. I fixed this be forcing the network down and up at the end of the boot sequence. Add the following two lines to your /etc/rc.local:

/sbin/ifdown --force eth0 /sbin/ifup --force eth0

Reboot

You can now leave the chroot, reboot the server and hope for the best. :)

logout umount /newroot/boot umount /newroot/proc umount /newroot/sys umount /newroot/dev umount /newroot sync reboot

The system will now boot into the dropbear ssh server which you can connect to and enter the passphrase to unlock the disk. The system will then continue to boot. Once the server answers pings again, ssh into it. To enter the passphrase use the following command:

echo -n "YOUR SUPER SECRET PASSPHRASE" >/lib/cryptsetup/passfifo

The system will now disconnect your ssh. If not you may have used the wrong passphrase. My terminal was kind of broken afterwards and I had to restart the shell. You now should be able to connect normally.

If you have skipped the optional step (writing the disk full) you should do it now. The easiest way is to create a large file with dd.

dd if=/dev/zero of=foofile bs=1M

You should stop any databases or other software that does not like "No space left on device" before writing the disk full. <Insert Mysql bashing here>

Encrypted swap

The swap can also be encrypted with a random key at boot time. This works automagically. Add the following line into your /etc/crypttab. This sets up an encrypted swap device with a random key as /dev/mapper/swap everytime the system boots.

swap /dev/md0 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

And change the swap entry in your fstab into the following:

/dev/mapper/swap none swap sw 0 0

DISASTER!

Don't panic. After all you have backups!

If you cannot log in via dropbear at all, because you have forgotten to copy the generated private key to your system or did not put your public key onto the server, it can be fixed easily. Just boot back into the rescue system. You can edit the config and regenerate the initramfs with the same commands as used above.

If login does work, first check the passphrase. You never know, you might just have a typo. If it still does not work, reboot the system into the rescue mode. Try unlocking the disk manually.

cryptsetup luksOpen /dev/md2 root

If that does not work you might have mistyped it while creating the LUKS device. Either try some common typos or you will have to start over.

If you can successfully unlock the disk, mount it and check if you have a problem with the dropbear configuration. See the readme mentioned in the section "Preparation" for any errors you might have in your configuration.

What has changed now?

The encryption is completely transparent. The only user-visible things that have changed are the following devices:

/dev/md0 is now /dev/mapper/swap /dev/md2 is now /dev/mapper/root

Everything else, including disaster recovery from failed hard drives has not changed. The disks are still part of the underlying MD device and do not know anything about the encryption. The boot partition is still unencrypted.

How secure is it?

The encryption itself should be bullet proof as long as the passphrase is complex enough of course. But even if it is "i hate passwords" you should be fairly safe since we used 5 seconds worth of iterations. On Hetzner's EX40 with an i7-4770 it results in 1.5 million iterations and that is (was? see post date!) a very fast CPU.

The boot partition is unencrypted. If someone gained access to your robot account and boots the server into the rescue system he can, of course, modify the initramfs and install software that will compromise your system if you login. The robot logs these reboot requests so if you didn't request it, be very wary and do not log in. The encrypted data is most likely not compromised or modified. We used aes-xts-plain64 which does not suffer from an issue like the default cbc-essiv mode. See for example the description of the malleability attack.

You could probably make a copy of your boot partition which you can restore in case of an unexplained reboot or checksums of the files which you can compare while in rescue mode.

If you need more security than that, only systems with TPM can provide a secure boot path that prevents tampering.

Checklist

If you want, you can use the checklist below while you set up the server. All important steps are recorded, so you're unlikely to miss one.

















Day 170

It's been some time, every plant made it. But with the weather being extremly bad most of the time they are kind of late this year. It's finally getting warmer and most of all sunny. I've given away some of them which leaves: 2x Carolina Reaper, 1x Trinidad Scorpion, 3x Habanero, 2x Ají dulce, 1x Fatalii. And a pot with two more plants that I forgot to label. Chili surprise. Should be a Fatalii and an Ají dulce, but who knows.

The fastest one is, as always, the Fatalii. It's the first one with a green and growing fruit. Now that the sun is out more often the Carolina Reapers also decided not to drop their petals anymore, but to start converting them into… well, let's call them flaming fruits of fiery doom.

I have also brought one more contender: Bishop's crown. I had them once as pickled chilies and they were absolutely fantastic. Extremly fruity and sweet, but rather mild. It's probably too late for this year, but other plants grew perfectly fine indoors during winter, so it may work out just fine.


Linux USB + Roccat Arvo keyboard = panic

Boom. I like boom… mostly. This must be the first, genuine, reproducible kernel panic I have had in a very long time! Some days ago I bought a Roccat Arvo gaming keyboard after my Logitech illuminated keyboard decided that it doesn't need the 'G' key anymore. Apparently a tiny plastic part broke that held the key in place. On ebay some sellers, more like scammers, sell "three key caps" for this exakt model. Turns out you get three random key caps. Hm, 105 keys, one broke, I get three random replacements. Do the math yourself. :)

Anyway, connecting the Arvo produced the following on my laptop:

input: ROCCAT ROCCAT Arvo as /devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2:1.0/0003:1E7D:30D4.000 2/input/input23 arvo 0003:1E7D:30D4.0002: input,hidraw1: USB HID v11.10 Keyboard [ROCCAT ROCCAT Arvo] on usb-0000:00:12.0-2/input0 arvo 0003:1E7D:30D4.0003: usb_submit_urb(ctrl) failed: -1 arvo 0003:1E7D:30D4.0003: timeout initializing reports input: ROCCAT ROCCAT Arvo as /devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2:1.1/0003:1E7D:30D4.0003/input/input24 arvo 0003:1E7D:30D4.0003: input,hidraw2: USB HID v11.10 Device [ROCCAT ROCCAT Arvo] on usb-0000:00:12.0-2/input1 arvo 0003:1E7D:30D4.0003: couldn't init struct arvo_device arvo 0003:1E7D:30D4.0003: couldn't install keyboard arvo: probe of 0003:1E7D:30D4.0003 failed with error -110

Seems some combinations of USB controllers and hid devices are bugged. Using the mode switch for the keyboard's numpad and then pressing some keys on the numpad result in an instant kernel panic in io_watchdog_func() [ohci_hcd].

On another system it works perfectly fine, without the USB error messages if it is plugged-in. At least it seems like the underlying bug is already known: https://bugzilla.kernel.org/show_bug.cgi?id=91511, http://www.spinics.net/lists/linux-usb/msg120049.html.

The solution!

What I tried is absurd and stupidly simple. Put an USB hub between computer and keyboard. All problems gone! At least until the bug is fixed which seems to be affecting a wide range of kernels. Happened with Debian's 3.16 - 4.0.5.

The real solution

This bug seems to have been fixed in kernel 4.1.4. USB: OHCI: Fix race between ED unlink and URB submission, commit 7d8021c967648accd1b78e5e1ddaad655cd2c61f upstream. Of course even Sid has 4.1.3 at the moment.


Day 37

After some more time almost all chilis have sprouted. Except for one Reaper and two Scorpions. So for this year I have:

  • 4x Habanero (actually 2 and one "dual mutant chili")
  • 3x Ají dulce
  • 3x Fatalii
  • 2x Carolina Reaper
  • 1x Scorpion

The LED lighting seemed to have worked quite OK. Now the days are getting longer and they should soon do fine without artificial lighting.


Day 17

The most likely final count for this year's chilis is:

  • 3x Ají dulce
  • 3x Fatalii
  • 2x Habanero
  • 2x Carolina Reaper
  • 1x Trinidad Scorpion

11 plants should be enough. Artificial lighting this year is an LED light. 5.5W, 385lm and 4000K. It's a "cold white" LED which is much "bluer" than the fluorescent lamp I used last year. But also brighter and consumes only 1/3 of its power. Let's see if it works better, but then the best light comes from outside anyway so they have a nice place on the window sill.


Day 13

More chilis! Interesting that they all sprout after roughly the same time. We now have the following list:

  1. Ají dulce: 3/3
  2. Habanero: 2/1 (2nd gen), 0/2 (original)
  3. Fatalii: 0/1 (2nd gen), 1/2 (original)
  4. Scorpion: 0/1 (2nd gen), 1/2 (original)
  5. Carolina Reaper: 1/3

Day 11

The first Chilis have sprouted. This year the first was a Habanero. Actually a Dual-Habanero as two seedlings sprouted from one seed. It's a second generation seed so it might have been mixed with something else… Habatalii or even a Ghost Habbi!

Number two is a Fatalii from the two year old original seeds. And third is an Ají dulce from new seeds I bought last year.


Chili Season '15

New year, new chilis. This year I started a bit earlier than last year, almost a month, because the fruits were a bit late last year. And this entry was created on Jan 10th rather than the 6th. This year's contenders are:

Trinidad Scorpion Butch T

Same as last year, “exceptionally hot”. To put it mildly. But they do have a lovely taste… for a moment. [Trinidad Scorpion Chili]

Habanero Red

Pure classic mexican habanero. At least you can eat them directly. [Habanero Chili]

Fatalii

Same yellow version like last year. They always produce so many fruits that you don't really need more than one plant. Also pretty hot (read: lava) and extremly fruity. Probably the best tasting chilis in the mix. [Fatalii Chili]

And there are two new chilis this year.

Ají dulce

These are supposed to be sweet and much milder to the point of not being hot at all. Still they retained their habanero like flavour from which they were grown. [Ají dulce]

Carolina Reaper

Hottest pepper in existence it probably melts your teeth and puts your tongue on fire. As to why they were called “Carolina Reaper”, one can only hope they live up to their name. They probably won't get much hotter than the Scorpions in these parts, but it's worth the shot. 2 mio SHU here we come! [Carolina Reaper]

Three plants each, I will probably give away four. Should be enough for 2015. They are all planted in their heated mini greenhouse again that worked great the last times. Temperatur during the day is around 28°C and 18°C during the night. Hopefully the first seeds will have germinated around Jan 16th.


← newer stuff older stuff →

Index: 1 - 2 - 3 - 4 - 5