Installing Arch Linux with ZFS on a USB stick

Posted on 2019-03-28
In this small tutorial we're going to install Arch Linux with ZFS on a USB stick. This can be useful if you're running a homelab NAS and you want to boot of the USB stick and only use your drives for ZFS. It can also be used as a diagnostic or administration tool for dealing with ZFS.

I am going to use the ext4 filesystem for the USB stick. The Arch Linux wiki recommends disabling the journal when using ext4 on a USB stick, but since all decent flash devices perform internal wear leveling I think the benefit of running with the journal outweighs the risk of premature wearing out the USB stick.

I am going to use the BIOS setup and not UEFI in this tutorial.

If you're doing the installation from an existing Arch Linux installation on your computer you need the Arch install scripts:

# pacman -S arch-install-scripts

Otherwise just boot of the Arch Linux installation media of your choice (can be another USB stick) and install from there. Make sure you don't mix up the devices in case you're also using an USB stick as the installation media.

If you're running of the Arch Linux installation media, begin by setting up your keyboard:

# loadkeys KEYMAP

Where "KEYMAP" corresponds to your keyboard layout. For a list of all the available keymaps, use the command:

# localectl list-keymaps

In my case I am using a Danish keyboard layout so I am using the following command:

# loadkeys dk

If you find that your keymap is represented with and without a "latin" version, like in "dk" and "dk-latin", then the latin version is the one that enables dead keys while the one without the latin part doesn't.

Dead keys are those keyboard keys that do not type anything until you hit the key twice or a combination of two keys. Tildes and umlauts are like this by default under plain Linux. This is the default behavior for these keys under Microsoft Windows as well.

Also remember that the above keyboard setup might change if you use a graphical window system (such as Xfce4 or Gnome for example) because such systems often has their own keyboard layout setup. The above setup is for usage in the console without X Window System running.

Next, locate your USB device:

# fdisk -l

In my case it's the "sdc" device.

Then partition the disk. If you're not comfortable using fdisk then cfdisk is a nice partitioning tool.

# cfdisk /dev/sdc

Note: Remember to make the root filesystem bootable!

Create the ext4 filesystem:

# mkfs.ext4 /dev/sdc1

Use the following command instead you want to do it with a disabled journal:

# mkfs.ext4 -O "^has_journal" /dev/sdc1

Then mount the USB stick:

# mount /dev/sdc1 /mnt

Then install the base package group and set everything up (these are the basic steps the Arch Linux wiki illustrates with explanations):

# pacstrap /mnt base
# genfstab -U /mnt >> /mnt/etc/fstab
# arch-chroot /mnt
# ln -sf /usr/share/zoneinfo/Europe/Copenhagen /etc/localtime
# hwclock --systohc
# passwd root

Uncomment "en_US.UTF-8" UTF-8 and other needed locales in /etc/locale.gen, and generate them with:

# locale-gen

Create the locale.conf file and set the LANG variable accordingly:

/etc/locale.conf
LANG=en_US.UTF-8

Set the keyboard layout:

/etc/vconsole.conf
KEYMAP=dk

I am going to install some additional packages, besides from git and base-devel you don't need these (and you can use nano instead of Vim if you prefer that):

# pacman -S vim mc cryptsetup openssh grub git
# pacman -S --needed base-devel

I enable SSH in order to be able to log in to the computer booting of the USB stick. This is handy if the computer is a homelab NAS running of the USB stick (otherwise you don't need this):

# systemctl enable sshd

Install additional users and setup the network (you don't need to install additional users if you just want to use the USB stick as a diagnostic tool, however it makes using makepkg later a lot easier):

# useradd --create-home foo
# passwd foo

I prefer to run with systemd-networkd, so I'll set that up (you can see what name your network device has with the ip a command):

# vim /etc/systemd/network/wired.network
[Match]
Name=enp*

[Network]
DHCP=yes

Then enable the network and setup a name server:

# systemctl enable systemd-networkd.service
# echo "nameserver 0.0.0.0" > /etc/resolv.conf (use whatever nameserver you use instead of 0.0.0.0)

Then setup GRUB:

# grub-install --target=i386-pc /dev/sdc --recheck
# grub-mkconfig -o /boot/grub/grub.cfg

I had some problems with GRUB not being able find the "root device" after using the USB stick on another computer than the one on which I installed the stick. This shouldn't be happening since the system is using UUID, which is a mechanism to give each filesystem a unique identifier. These identifiers are generated by filesystem utilities (e.g. mkfs.) when the device gets formatted and are designed so that collisions are unlikely. However, I found other users with similar problems: https://superuser.com/questions/769047/unable-to-find-root-device-on-a-fresh-archlinux-install

The solution is, as described in the link, to edit /etc/mkinitcpio.conf and change the HOOKS line so that block comes before autodetect.

HOOKS="base udev block autodetect modconf filesystems keyboard fsck"

Then run mkinitcpio to regenerate the initramfs:

# mkinitcpio -p linux

Then it's time to install ZFS.

I am using the zfs-dkms package from Arch Linux AUR and running makepkg as the foo user I setup before:

# pacman -S dkms linux-headers
# su foo
$ cd /home/foo
$ git clone https://aur.archlinux.org/zfs-utils.git
$ cd zfs-utils
$ makepkg -s
$ cd ..
$ git clone https://aur.archlinux.org/spl-dkms.git
$ cd spl-dkms
$ makepkg -s
$ cd ..
$ git clone https://aur.archlinux.org/zfs-dkms.git
$ cd zfs-dkms
$ makepkg -s
$ cd ..
$ exit
# pacman -U zfs-utils/zfs-utils-0.7.13-1-x86_64.pkg.tar.xz
# pacman -U spl-dkms/spl-dkms-0.7.13-1-any.pkg.tar.xz
# pacman -U zfs-dkms/0.7.13-1-x86_64.pkg.tar.xz

If you get the error:

==> Verifying source file signatures with gpg...
    zfs git repo ... FAILED (unknown public key 6AD860EED4598027)
==> ERROR: One or more PGP signatures could not be verified!

Then manually import the key as explained in One or more PGP signatures could not be verified and Use a key server

$ gpg --recv-keys 6AD860EED4598027

You can add an IgnorePkg entry to pacman.conf to prevent these packages from upgrading when doing a regular update. Since the AUR is unsupported any packages you install are your responsibility to update, not pacman's. If packages in the official repositories are updated, you will need to rebuild any AUR packages that depend on those libraries.

# vim /etc/pacman.conf
# IgnorePkg=zfs-dkms zfs-utils spl-dkms

For ZFS to live by its "zero administration" namesake, the zfs daemon must be loaded at startup. A benefit to this is that it is not necessary to mount the zpool in /etc/fstab The zfs daemon can import and mount zfs pools automatically. The daemon mounts the zfs pools reading the file /etc/zfs/zpool.cache. For each pool you want automatically mounted by the zfs daemon execute:

# zpool set cachefile=/etc/zfs/zpool.cache <pool>

Enable the service so it is automatically started at boot time:

# systemctl enable zfs.target

Note: Beginning with ZOL version 0.6.5.8 the ZFS service unit files have been changed so that you need to explicitly enable any ZFS services you want to run. See https://github.com/archzfs/archzfs/issues/72 for more information.

In order to mount zfs pools automatically on boot you need to enable the following services and targets:

# systemctl enable zfs-import-cache
# systemctl enable zfs-mount
# systemctl enable zfs-import.target

We're done. Exit the chroot and un-mount the USB stick:

# exit
# umount /mnt

You should now have a working USB stick with Arch Linux and ZFS that you can use as a boot device or a diagnostic tool.

That's it.

Feel free to email me any suggestions or comments.