Here's a strange thing. Seems like spf-tools-perl in jessie have some strange bugs. I never noticed, but after the update from wheezy SPF silently stopped working and the log was spammed with Warning: Unexpected error in SPF check. I traced this down to the invocation of /usr/bin/spfquery.mail-spf-perl which would abort with an error: unresolvable name: localhost at /usr/share/perl5/Mail/SPF/Server.pm line 225
The solution to all this madness, /etc/resolv.conf had an entry nameserver localhost. Changing that to 127.0.0.1 made the perl script and SPF checking work again. /etc/hosts does list 127.0.0.1 localhost, no idea why this broke.
Chili season 2016 has already started. I planted 15 about a week ago. This year I don't have an "extreme-only" mix, but variety. Of course no chili season can go without Reapers. So for this year we have:
- Carolina Reaper x3
- Trinidad Scorpion Butch T x3 (one of them is second generation, so maybe mixed up with something else)
- Cayenne Yellow x3, all of them second generation
- Bishop's Crown x3
- Purple Tiger x3
The Bishop's Crown grow extremly fast, and extremly big. They are also a bit fragile due to their size. Let's hope the first storm doesn't blow them away once they're outside. The list is sorted by heat. Complete meltdown on top and more like bell pepper on the bottom. But I'm still mostly looking forward to the Bishop's Crowns. They were extremly tasty without setting your mouth on fire. For some people, this may be a good thing. Probably not for us that grow chilies. :)
This year's harvest is probably less than 2014. Bad weather in the spring, less plants and experiments with soil which turned out to be suboptimal. Not all variants seem to like compressed cocos fibre. The stuff can hold a lot of water, but the plants contantly have "wet feet" and it dries out pretty fast. The turfless soil I tried this year is much better. I will use that again next year. Only downside: it's more annoying to transport. I could probably order it at amazon, but that really feels weird.
So, this year's leaderboards:
|Type||# of plants||# of fruits||Rank|
When I first sowed the Bishop's Crown seed in June this year I was more expecting it to become an indoor chili for the window sill. But it turns out they are even faster growers than the Fataliis and after only 1.5 months the plant is 75cm high. Not even a small pot prevented it from growing. It just became offended, letting its leaves hanging, and I had to put it into a bigger pot.
After a bit of research it turned out that these plants grow up to 2-3m high. They seem to grow much larger than your standard capsicum chinense plant. It even grew so fast that the trunk is kinda thin and it seems to have trouble getting enough water to the topmost leaves.
Hailing from the Carribean they should be accustomed to tropical climate. Interestingly 35°C behind a window glass (greenhouse like) seems to be almost uncomfortable and 40°C in direct sunlight is too much for them. They seem to like it much better in the shade and, in contrast to other chilis I have, need plenty of water. While others are fine with the soil completely dry for a day or two the Bishop's Crowns seem to prefer it mostly wet.
The first Carolina Reaper chili was ready today. Compared to other red chilis they look more orangish and quite poisonous if you ask me. Ah well, no pain no gain. So I made a traditional chili con carne and spiced it up quite a bit with a finely chopped chili. The conclusion: they burn as hot as they look! I'd say the stories you've read elsewhere are definitely true. This is a weaponized chili pepper. If you enjoy pain or like to see others gasp for air while they try to eat your evil cuisine, give this one a try. :)
Updated 29 Dec 2015
This howto turns a Hetzner default Debian root server installation into a fully encrypted server with LUKS that can be unlocked remotely with the dropbear ssh server. If the server is rebooted, seized or a KVM switch is plugged into the wrong server, the data cannot be accessed. Likewise, your data will be lost forever* if you forget the passphrase. This works with Debian Jessie and Wheezy images. But read the note on Jessie below.
(* a long time)
Please note the date of this post. If it is older than a year, the defaults of the Hetzner install image might have changed. Paths and devices might be named differently. I assume the following (Hetzner default) disk layout. If you have configured your server differently, change the device names accordingly.
md0: Raid1 consisting of sda1, sdb1 as swap md1: Raid1 consisting of sda2, sdb2 as /boot md2: Raid1 consisting of sda3, sdb3 as the root filesystem
The server will not be reinstalled from scratch. All data is preserved (read below). You can encrypt an existing server without any config changes to the services it provides.
If you're using Jessie and onward, the init system will be systemd by default. Because of an unresolved bug with crypto luks + systemd + certain crypttab options, including keyscript, you'll have two options with this constellation. There are, of course, other more hackish approaches, but it is up to you to evaluate their safety.
Install the package sysvinit-core before you reboot. This removes systemd-sysv, which makes the system default to systemd as PID 1. Your system will continue to use the old sysvinit. Nothing else will change. Everything that worked in Wheezy will continue to do so.
First copy /sbin/init to /sbin/init.sysv This will give an easy way back to sysvinit if thins go haywire. Install systemd-sysv to switch the default to systemd. Add a kernel parameter luks=no to the kernel command line in /etc/default/grub and run update-grub && update-initramfs -u. This will now tell systemd not to unlock the disk at boot, the old scripts in initramfs will take care of that.
There is one important note however: no other disk will be unlocked at boot time, only root. So if you have any other entry in crypttab, it will not be available to auto-mounting at boot and systemd will fail to bring up that mount point. That includes the encrypted swap further down in this guide. You will have to mount the rest of the paritions manually after each boot. If it's just swap, you could also remove it completely, 32GB or even 64GB in the current servers should be enough for almost every workload.
If the system now fails to boot, wait at least 1m30s, that's how long systemd tried to mount the missing devices until it gave up on my test system, pass the following kernel parameter on boot: init=/sbin/init.sysv. You can either do that with a LARA (KVM) console or edit your config in the rescue system. That will switch to sysvinit for this one boot.
Additional packages that must be present on your current system
Install these in your current installation before you proceed.
apt-get install cryptsetup apt-get install busybox dropbear
You should have cryptsetup installed before you install dropbear. There might be some strange side effects with the initramfs config otherwise. I have received such a report. At least it doesn't hurt.
There is a complete description on how to generate the keys and unlock the disk in the cryptsetup documentation. It is installed in /usr/share/doc/cryptsetup/README.remote.gz. Configure dropbear following the readme before you proceed.
If you have configured it and had the private/public SSH keys generated automatically, do not forget to copy the private key or install your public key according to the readme.
Reboot into rescue system
Create a backup if you're working with important data!
Log into the robot and reboot the server into the (64bit) rescue system. At the time of this writing every tool was available in the rescue system.
Save your old system
The old system is moved aside while the encrypted disk is prepared. If you have more data than what fits onto the root filesystem, you need to move large data directories either to the backup FTP or restore them from a backup afterwards. In the rescue system you will have a root filesystem ramdisk which is half the size of the server's memory. So if you have a 32GB root server you will have ~16GB available.
mkdir /oldroot mount /dev/md2 /mnt rsync -a /mnt/ /oldroot/ umount /mnt
Your system is now mirrored into /oldroot.
Create the encrypted disk
Let's create the encrypted filesystem. This will destroy all data on your server! You might want to use a diceware password so you don't have to enter any special characters while booting the system. Example on xkcd. But don't use an online generator or the horse staple. :) Or for example: dd if=/dev/random bs=1 count=18 | openssl base64
The following command will create a LUKS device using the aes-xts-plain64 cipher with 512 bits key size and a number of key iterations that will take roughly 5 seconds to compute on your hardware. Meaning it will take 5 seconds to unlock the disk. Since servers are rarely rebooted, it does not hurt.
cryptsetup --cipher aes-xts-plain64 -s 512 --iter-time 5000 luksFormat /dev/md2
You can verify that everything went fine with the luksDump command. It should produce something like the following output:
# cryptsetup luksDump /dev/md2 LUKS header information for /dev/md2 Version: 1 Cipher name: aes Cipher mode: xts-plain64 Hash spec: sha1 Payload offset: 4096 MK bits: 512 MK digest: de ad be ef MK salt: de ad be ef de ea be ef MK iterations: 83750 UUID: 5f76b99e-0418-4fb5-a731-201004ede948 Key Slot 0: ENABLED Iterations: 336005 Salt: de ea be ef de ea be ef Key material offset: 8 AF stripes: 4000 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED
Once the LUKS device is created, activate it. This will ask for the passphrase you've just used.
cryptsetup luksOpen /dev/md2 root
The following step is optional but recommended. It fills the whole disk with random data and overwrites anything on the partition that may be recoverable. Even though it reads from /dev/zero, you are writing to the high level LUKS device which writes random encrypted noise to the disk. This might take a while. 2-4 hours with the current disk sizes. You can skip this step and create a large file on the filesystem after everything is finished with the same effect.
dd if=/dev/zero of=/dev/mapper/root bs=1M
Create a filesystem on the encrypted device. I will use ext4 with default options and set check interval to 6 months, error behavior to remount-ro and max mount count to 50.
mke2fs -t ext4 /dev/mapper/root tune2fs -i 6m -e remount-ro -c 50 /dev/mapper/root
You can now restore your data.
Reinstall the system
mkdir /newroot mount /dev/mapper/root /newroot rsync -a /oldroot/ /newroot/
Before we can changeroot into the new system we need some special file systems and the boot file system inside. These are needed so grub and initramfs reinstallation work.
mount /dev/md1 /newroot/boot mount --bind /dev /newroot/dev mount --bind /sys /newroot/sys mount --bind /proc /newroot/proc chroot /newroot
The fstab needs to be changed to reflect the new root file system. Edit the file and replace / (root) with the following:
/etc/fstab: /dev/mapper/root / ext4 defaults 0 2
To be able to boot from the encrypted file system we need a crypttab.
/etc/crypttab: # <target name> <source device> <key file> <options> root /dev/md2 none luks
Now all is set. We need to tell grub the new root device and regenerate the initramfs to include the encryption stuff. Run the following commands. They should automatically pick up /dev/mapper/root as the new root device.
update-initramfs -u update-grub grub-install /dev/sda grub-install /dev/sdb
IPv6 will not be running after a reboot. The IPv4 networking will be brought up in the preboot ssh server. Once Debian boots the real system, configuring the network interfaces will fail, because eth0 was already configured with ip earlier and the network will be in a somewhat inconsistent state. I fixed this be forcing the network down and up at the end of the boot sequence. Add the following two lines to your /etc/rc.local:
/sbin/ifdown --force eth0 /sbin/ifup --force eth0
You can now leave the chroot, reboot the server and hope for the best. :)
logout umount /newroot/boot umount /newroot/proc umount /newroot/sys umount /newroot/dev umount /newroot sync reboot
The system will now boot into the dropbear ssh server which you can connect to and enter the passphrase to unlock the disk. The system will then continue to boot. Once the server answers pings again, ssh into it. To enter the passphrase use the following command:
echo -n "YOUR SUPER SECRET PASSPHRASE" >/lib/cryptsetup/passfifo
The system will now disconnect your ssh. If not you may have used the wrong passphrase. My terminal was kind of broken afterwards and I had to restart the shell. You now should be able to connect normally.
If you have skipped the optional step (writing the disk full) you should do it now. The easiest way is to create a large file with dd.
dd if=/dev/zero of=foofile bs=1M
You should stop any databases or other software that does not like "No space left on device" before writing the disk full. <Insert Mysql bashing here>
The swap can also be encrypted with a random key at boot time. This works automagically. Add the following line into your /etc/crypttab. This sets up an encrypted swap device with a random key as /dev/mapper/swap everytime the system boots.
swap /dev/md0 /dev/urandom swap,cipher=aes-cbc-essiv:sha256
And change the swap entry in your fstab into the following:
/dev/mapper/swap none swap sw 0 0
Don't panic. After all you have backups!
If you cannot log in via dropbear at all, because you have forgotten to copy the generated private key to your system or did not put your public key onto the server, it can be fixed easily. Just boot back into the rescue system. You can edit the config and regenerate the initramfs with the same commands as used above.
If login does work, first check the passphrase. You never know, you might just have a typo. If it still does not work, reboot the system into the rescue mode. Try unlocking the disk manually.
cryptsetup luksOpen /dev/md2 root
If that does not work you might have mistyped it while creating the LUKS device. Either try some common typos or you will have to start over.
If you can successfully unlock the disk, mount it and check if you have a problem with the dropbear configuration. See the readme mentioned in the section "Preparation" for any errors you might have in your configuration.
What has changed now?
The encryption is completely transparent. The only user-visible things that have changed are the following devices:
/dev/md0 is now /dev/mapper/swap /dev/md2 is now /dev/mapper/root
Everything else, including disaster recovery from failed hard drives has not changed. The disks are still part of the underlying MD device and do not know anything about the encryption. The boot partition is still unencrypted.
How secure is it?
The encryption itself should be bullet proof as long as the passphrase is complex enough of course. But even if it is "i hate passwords" you should be fairly safe since we used 5 seconds worth of iterations. On Hetzner's EX40 with an i7-4770 it results in 1.5 million iterations and that is (was? see post date!) a very fast CPU.
The boot partition is unencrypted. If someone gained access to your robot account and boots the server into the rescue system he can, of course, modify the initramfs and install software that will compromise your system if you login. The robot logs these reboot requests so if you didn't request it, be very wary and do not log in. The encrypted data is most likely not compromised or modified. We used aes-xts-plain64 which does not suffer from an issue like the default cbc-essiv mode. See for example the description of the malleability attack.
You could probably make a copy of your boot partition which you can restore in case of an unexplained reboot or checksums of the files which you can compare while in rescue mode.
If you need more security than that, only systems with TPM can provide a secure boot path that prevents tampering.
If you want, you can use the checklist below while you set up the server. All important steps are recorded, so you're unlikely to miss one.
It's been some time, every plant made it. But with the weather being extremly bad most of the time they are kind of late this year. It's finally getting warmer and most of all sunny. I've given away some of them which leaves: 2x Carolina Reaper, 1x Trinidad Scorpion, 3x Habanero, 2x Ají dulce, 1x Fatalii. And a pot with two more plants that I forgot to label. Chili surprise. Should be a Fatalii and an Ají dulce, but who knows.
The fastest one is, as always, the Fatalii. It's the first one with a green and growing fruit. Now that the sun is out more often the Carolina Reapers also decided not to drop their petals anymore, but to start converting them into… well, let's call them flaming fruits of fiery doom.
I have also brought one more contender: Bishop's crown. I had them once as pickled chilies and they were absolutely fantastic. Extremly fruity and sweet, but rather mild. It's probably too late for this year, but other plants grew perfectly fine indoors during winter, so it may work out just fine.
Boom. I like boom… mostly. This must be the first, genuine, reproducible kernel panic I have had in a very long time! Some days ago I bought a Roccat Arvo gaming keyboard after my Logitech illuminated keyboard decided that it doesn't need the 'G' key anymore. Apparently a tiny plastic part broke that held the key in place. On ebay some sellers, more like scammers, sell "three key caps" for this exakt model. Turns out you get three random key caps. Hm, 105 keys, one broke, I get three random replacements. Do the math yourself. :)
Anyway, connecting the Arvo produced the following on my laptop:
input: ROCCAT ROCCAT Arvo as /devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2:1.0/0003:1E7D:30D4.000 2/input/input23 arvo 0003:1E7D:30D4.0002: input,hidraw1: USB HID v11.10 Keyboard [ROCCAT ROCCAT Arvo] on usb-0000:00:12.0-2/input0 arvo 0003:1E7D:30D4.0003: usb_submit_urb(ctrl) failed: -1 arvo 0003:1E7D:30D4.0003: timeout initializing reports input: ROCCAT ROCCAT Arvo as /devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2:1.1/0003:1E7D:30D4.0003/input/input24 arvo 0003:1E7D:30D4.0003: input,hidraw2: USB HID v11.10 Device [ROCCAT ROCCAT Arvo] on usb-0000:00:12.0-2/input1 arvo 0003:1E7D:30D4.0003: couldn't init struct arvo_device arvo 0003:1E7D:30D4.0003: couldn't install keyboard arvo: probe of 0003:1E7D:30D4.0003 failed with error -110
Seems some combinations of USB controllers and hid devices are bugged. Using the mode switch for the keyboard's numpad and then pressing some keys on the numpad result in an instant kernel panic in io_watchdog_func() [ohci_hcd].
On another system it works perfectly fine, without the USB error messages if it is plugged-in. At least it seems like the underlying bug is already known: https://bugzilla.kernel.org/show_bug.cgi?id=91511, http://www.spinics.net/lists/linux-usb/msg120049.html.
What I tried is absurd and stupidly simple. Put an USB hub between computer and keyboard. All problems gone! At least until the bug is fixed which seems to be affecting a wide range of kernels. Happened with Debian's 3.16 - 4.0.5.
The real solution
This bug seems to have been fixed in kernel 4.1.4.
USB: OHCI: Fix race between ED unlink and URB submission, commit 7d8021c967648accd1b78e5e1ddaad655cd2c61f upstream. Of course even Sid has 4.1.3 at the moment.
After some more time almost all chilis have sprouted. Except for one Reaper and two Scorpions. So for this year I have:
- 4x Habanero (actually 2 and one "dual mutant chili")
- 3x Ají dulce
- 3x Fatalii
- 2x Carolina Reaper
- 1x Scorpion
The LED lighting seemed to have worked quite OK. Now the days are getting longer and they should soon do fine without artificial lighting.
The most likely final count for this year's chilis is:
- 3x Ají dulce
- 3x Fatalii
- 2x Habanero
- 2x Carolina Reaper
- 1x Trinidad Scorpion
11 plants should be enough. Artificial lighting this year is an LED light. 5.5W, 385lm and 4000K. It's a "cold white" LED which is much "bluer" than the fluorescent lamp I used last year. But also brighter and consumes only 1/3 of its power. Let's see if it works better, but then the best light comes from outside anyway so they have a nice place on the window sill.
Index: 1 - 2 - 3 - 4 - 5