Notice: Undefined offset: 1 in /usr/local/src/wordpress/wp-content/themes/montezuma/includes/parse_php.php on line 79

HowTo: Build an Encrypted ZFS Array ~ Part 1 ~ Encryption

~~   Forward   ~~


The Zettabyte File System is an advanced filesystem which was developed by Sun Microsystems and is now owned by Oracle, and although it has always been open-source, its CDDL license is incompatible with GPL and so it will not be included in the Linux kernel.   Now that ZFSonLinux is stable though, it is available as a DKMS package.

With this article, we are going to set up a ZFS array of multiple disks, which will be assembled to appear as one volume, for use as /home, or /media/backups, or other functions where massive data storage is required.   In addition each of the disks comprising our array will be encrypted, and the data will be compressed for better storage efficiency and throughput.   Now this may look long, but I am documenting everything and I’ve made every effort to make it easy.

ZFS is 128-bit, meaning it is very scalable. (e.g. 16 exabyte limit).   With this incomparable scalability, it has a number of features that no other file system has:

  • Data integrity   —   State-of-the-art unsurpassed data reliability (including silent corruption detection and automatic data repair) from end to end, thanks to transparent checksums in parent block pointers.   Data integrity is a high priority in ZFS because recent research shows that none of the currently widespread file systems –such as UFS, Ext3-4, XFS, JFS, or NTFS– nor even hardware RAID provide sufficient protection against such problems.
  • Storage pools   —   Traditional file systems reside on single devices, and thus require a volume manager to bind more than one device, whereas ZFS is built as virtual storage pools called ‘zpools’.   A zpool can be comprised of multiple files, hard drive partitions, or entire drives, with the latter being the recommended usage.   These zpools can be configured non-redundantly (similar to RAID0), with two or more devices mirrored (RAID1), as a RAID-Z group of three or more devices (similar to RAID5), or as a RAID-Z2 group of four or more devices (similar to RAID6).   RAID-Z is a data-protection technology to reduce the block overhead in mirroring.
  • ZFS eliminates the RAID5 write hole, which for conventional technology requires battery-powered drives or UPS technology to keep RAID arrays consistent in case of power failure.
  • ZFS has online array reconstruction/reassemble, in a fraction of the time usually required to reconstruct an array, thanks to reconstruction dependent on contents rather than on disk topology.
  • Online check (scrub) and online resilver of faulty drives, to verify and preserve the validity of on-disk data in the case of silent disk data corruption.
  • Automatic deduplication and compression of data, selectable per volume or filesystem according to administrative policy.
  • Dynamic striping across all devices to maximize throughput.
  • Copy-on-write design makes most disk writes sequential.
  • Multiple block sizes, automatically chosen to match workload.
  • Explicit I/O priority with deadline scheduling.
  • Globally optimal I/O sorting and aggregation.
  • Multiple independent prefetch streams with automatic length and stride detection for maximum streaming performance.
  • Unlimited, instantaneous read/write snapshots.
  • Parallel, constant-time directory operations.
  • ~~   Installation   ~~

    All very cool, but let’s get to the good stuff.   This HowTo is Debian-centric.   Caution:   Sometimes command-lines wrap below, because of the width of the page.

    Installation these days could hardly be easier;   it’s the later stuff where you need me.   First we have to add the zfsonlinux repository to our repertoire.   Edit /etc/apt/sources.list and add:

    # ZFS on Linux
    deb wheezy-daily main

    # apt-get update
    # apt-get install zfs-dkms zfsutils

    Although it is possible to set your whole boot disk to ZFS, there is a Debian bug which makes it inadvisable. (boot to single-user mode fails)   Notice that we are installing the DKMS version of ZFS;   this means that whenever we do an apt-get dist-upgrade, the spl and zfs kernel modules will be automatically recompiled for the new kernel.

    ~~   Disk Encryption   ~~

    ZFSonLinux does not yet have encryption support, and so we are going to encrypt in the most technically-advisable way.   It is automatic and completely reliable.   We will use the built-in LUKS encryption to set up each disk individually, and assume we have four disks which will comprise our ZFS array.   You can add more later.   For best results with ZFS these disks should be the same size, the same brand, and approximately the same model (i.e. 4TB, Western Digital, Greens or Reds or ?)

    There are many different encryption algorithms, and we want to use the fastest and best one…   but which is it?
    # cryptsetup benchmark /dev/sdd
    Algorithm   |   Key   |   Encryption   |   Decryption

    aes-xts      256b      2415.2 MiB/s      2417.0 MiB/s
    aes-xts      512b      1899.9 MiB/s      1908.0 MiB/s

    AES is a very good cipher;   Blowfish is a very bad one.   If certain AES ciphers appear to have (e.g. tenfold) higher throughput, these are probably the ones with hardware support in the CPU, and are definitely the best choice.   In my case, aes-xts is the fastest of the best, and a 512bit encryption key is my choice because this is for backups and I don’t care as much about speed as security.   If you decide not to specify a cipher, you can check what the default ciphers are with:
    # cryptsetup –help |tail

    Let’s encrypt the disks:
    # modprobe dm_mod
    # cryptsetup –cipher aes-xts-plain64 –key-size 512 luksFormat /dev/sdb
    # cryptsetup –cipher aes-xts-plain64 –key-size 512 luksFormat /dev/sdc
    # cryptsetup –cipher aes-xts-plain64 –key-size 512 luksFormat /dev/sdd
    # cryptsetup –cipher aes-xts-plain64 –key-size 512 luksFormat /dev/sde

    (Assuming these are the disks for the array)

    Check it:
    # cryptsetup luksDump /dev/sdb

    Key Slot 0: ENABLED

    So, of the 8 slots where we can put encryption keys on/for that disk, the first is occupied by our password.   But it’s a hassle to enter a password for every single encrypted disk on boot, so we are going to make unlocking automatic.   You can set an encrypted disk to unlock using any binary file as its key.   In my case I wrote a nice little plasmoid which gives a graph of network traffic called KDevMon, and it’s in /usr/local/bin.   I know I will always have it and likely won’t be updated, so I make that the key for the second slot of each encrypted drive.   This way, any drive can be unlocked with the binary file on boot, or if something goes wrong I can also unlock it with my password.   Now, this binary file resides on my boot disk, and the boot disk is encrypted, so my ZFS array can’t be opened unless the boot disk is opened, so I enter one password for my boot disk, and all the other drives are automatically opened.   Completely secure.   If something ever goes wrong, I can still enter the password for any drive.

    Add a binary file:
    # cryptsetup luksAddKey /dev/sdb /usr/local/bin/kdevmon
    # cryptsetup luksAddKey /dev/sdc /usr/local/bin/kdevmon
    # cryptsetup luksAddKey /dev/sdd /usr/local/bin/kdevmon
    # cryptsetup luksAddKey /dev/sde /usr/local/bin/kdevmon

    Check it:
    # cryptsetup luksDump /dev/sdb
    … and you’ll now see that both keyslot 0 and 1 are occupied. (password and binary-file keys)

    This info is stored in the crypto header on each drive, which is not protected by ZFS data integrity, so it’s a good idea to back up the crypto headers;   to somewhere on our encrypted root drive.

    # cryptsetup luksHeaderBackup /dev/sdb –header-backup-file /home/{USER}/dl/cryptheader-sdb.bak.img
    # cryptsetup luksHeaderBackup /dev/sdc –header-backup-file /home/{USER}/dl/cryptheader-sdc.bak.img
    # cryptsetup luksHeaderBackup /dev/sdd –header-backup-file /home/{USER}/dl/cryptheader-sdd.bak.img
    # cryptsetup luksHeaderBackup /dev/sde –header-backup-file /home/{USER}/dl/cryptheader-sde.bak.img
    # chmod 700 /home/{USER}/dl/cryptheader-sd*

    If you ever need to restore a crypto header:
    # cryptsetup luksHeaderRestore /dev/sdb –header-backup-file /home/{USER}/dl/cryptheader-sdb.bak.img

    And last we need to instruct the system about these crypto drives, so it will set them up automatically.   It’s a good idea to use UUIDs here, as drives can get different assignments on every boot.   For our ZFS array we want to keep them in order every time.

    What are the blockIDs on this system?
    # blkid
    /dev/sdb:      UUID=”ce9e9fc4-cc74-406a-883b-f0dfb8b6f182″      TYPE=”crypto_LUKS”
    /dev/sdc:      UUID=”0e4237ae-3c46-41f3-b0e0-f1c81f36d792″      TYPE=”crypto_LUKS”
    /dev/sdd:      UUID=”f6ca74e8-23fd-4d38-ba4a-47e67a15ef76″      TYPE=”crypto_LUKS”
    /dev/sde:      UUID=”a2e38c27-a967-47b9-b208-3745030c8443″      TYPE=”crypto_LUKS”

    Edit /etc/crypttab and put in it:

    sdb_crypt      UUID=ce9e9fc4-cc74-406a-883b-f0dfb8b6f182      /usr/local/bin/kdevmon      luks
    sdc_crypt      UUID=0e4237ae-3c46-41f3-b0e0-f1c81f36d792      /usr/local/bin/kdevmon      luks
    sdd_crypt      UUID=f6ca74e8-23fd-4d38-ba4a-47e67a15ef76      /usr/local/bin/kdevmon      luks
    sde_crypt      UUID=a2e38c27-a967-47b9-b208-3745030c8443      /usr/local/bin/kdevmon      luks

    Now on every boot, the system will automatically unlock those drives, and make them available in their unencrypted incantation at /dev/mapper/sd?_crypt.   Treat them here just like any other raw disk.

    Warning:   With LUKS-encrypted drives, Debian doesn’t know what drivers to load at boot.   This will cause a hang when it asks for the drive password, and your mouse light will go out.   To prevent this awful situation, edit /etc/initramfs-tools/modules and add:


    … then # update-initramfs -u -k all

    Reboot for the changes to take effect.
    # ls -al /dev/mapper/sd*
    lrwxrwxrwx 1 root root 7 Aug 16 09:37 sdb -> ../dm-2
    lrwxrwxrwx 1 root root 7 Aug 16 09:37 sdc -> ../dm-1
    lrwxrwxrwx 1 root root 7 Aug 16 09:37 sdd -> ../dm-4
    lrwxrwxrwx 1 root root 7 Aug 16 09:37 sde -> ../dm-3

    Success!   On to Build an Encrypted ZFS Array – Part 2 – The Array.

    ,'after' => '

    ') )