In the last section of this series I discussed using ZFS snapshots, ZFS send and using other interesting features ZFS has to offer. In this post I discuss setting up a arch system using a ZFS pool as a root filesystem

I originally posted this on June 23rd 2016, since then I have learned a fair bit about setting up a system on ZFS, the following set up is my revised configuration as of September 2nd 2017.

Part Two - Installation Link to heading

Pre-Install Setup Link to heading

Boot partitions Link to heading

I would recomend using a dedicated partition wheather using BIOS or UEFI. While you are supposed to be able to store your kernel and any other boot images on a ZFS dataset when using a BIOS bootloader like GRUB, in my experience I have had more luck using a boot partition.

ZFS Setup Link to heading

Pool Type and Redundancy Link to heading

ZFS has several different levels of redundancy based on the number of discs that are used and in what setting they are configured. They are different than traditional RAID levels.

A crude description of the different levels:

  • Stripe
    • No redundancy, similar to RAID0.
    • Default pool creation mode.
    • Minimum of 3 discs.
    • One disc can be lost without pool failure.
    • One of the discs is redundant and will not provide storage.
  • RAIDZ2
    • Minimum of 4 discs.
    • Two discs can be lost without pool failure.
    • Two of the discs are redundant and will not provide storage.
  • RAIDZ3
    • Minimum of 5 discs.
    • Three discs can be lost without pool failure.
    • Three of the discs are redundant and will not provide storage.
  • Mirror - Minimum of 2 discs.
    • Half of the discs can be lost without pool failure.
    • Storage from half of the discs are used as redundancy and will not provide storage.
    • Provides the best performance and flexibility at a cost of storage space.

Disk Setup Link to heading

It’s recommended to use the disk id names as recommended by the ZOL project. The identification and partition number of each drive can be found with:

[root]# ls /dev/disk/by-id/

Note: In the following post I will be using the above two sandisk SSDs as an example.

Once the disc IDs are known, the pool can be created. While the disks can be partitioned manually in GPT or MBR, it is not necessary to partition entire drives before creating the pool as ZFS will partition the drives itself as Solaris Root (bf00) in the creation of a new pool.

ZFS has issues being booted off of. The simplest option is to partition the boot partition in another format as would be done with a regular install.

I do not use a swap partition and thus have no experience using one; however, the Arch wiki explains the process of sending one up if necessary.

Pool Creation Link to heading

After deciding on a pool type and getting the disc IDs, a pool can be created with the zpool create command.

The syntax is:

zpool create [-fnd] [-o property=value] ... \
              [-O file-system-property=value] ... \
              [-m mountpoint] [-R root] ${POOL_NAME} ${DISK}	...


  • -f - Force.
  • -n - Display creation but don’t create pool.
  • -d - Do not enable any features unless specified.
  • -o - Set a pool property.
  • -O - Set a property on root filesystem.
  • -m - Mountpoint.
  • -R - Set an alternate root location.

First, probe for ZFS on the system, there should be no output from modprobe.

[root]# modprobe zfs

Then the zpool create command can be used to create a new pool.

When creating a pool ashift=12 will specify advanced format disks, this will force 4096 size blocksize. Here I create a mirrored pool named ‘vault’ with my two SSD’s.

[root]# zpool create -f -o ashift=12 vault mirror \
                ata-SanDisk_SDSSDXPS480G_152271401093 \

Test the pool was created successfully with zpool status.

[root]# zpool status
pool: vault
state: ONLINE
scan: scrub repaired 0 in 0h8m with 0 errors on Mon Jun 13 00:08:39 2016

NAME                                       STATE     READ WRITE CKSUM
vault                                      ONLINE       0     0     0
  mirror-0                                 ONLINE       0     0     0
    ata-SanDisk_SDSSDXPS480G_152271401093  ONLINE       0     0     0
    ata-SanDisk_SDSSDXPS480G_154501401266  ONLINE       0     0     0

errors: No known data errors

Properties Link to heading

There are many properties that can be set on an entire pool or specific data sets. A property that will almost always be wanted on the entire pool is compression. ZFS uses “LZ4” compression which is a great compromise between performance and amount of compression.

Set compression on

[root]# zfs set compression=on vault

By default atime=on is enabled, it can be turned off to increase performance, or set to relatime which is the default on many Linux filesystems.

In recording access time relatime is a good compromise, it will record access time but much more infrequently than the default atime=on setting.

[root]# zfs set atime=on vault
[root]# zfs set relatime=on vault

Dataset Creation Link to heading

Datasets are similar to partitions; however, they come with many benefits partitions do not have. In addition to being hierarchically organizable, they do not require a specific quota. All of the datasets will share space in a given pool. These qualities in datasets mean that they can be used extensively without repercussions.

ZFS has the ability to manage datasets itself, or to tell the datasets to fall back to the system controlled legacy management where datasets will be managed with the fstab.

I have found using legacy management works best. If legacy mounting fails, which it does in certain circumstances I use ZFS managed mounting.

I’ll use a few variables to represent different locations in the pool for datasets.

  • SYS_ROOT=vault/sys - The location of any systems on the pool. Alternatively Set to the root dataset, e.g. vault, if you plan on only ever running a single system on the ZFS pool.
  • DATA_ROOT=vault/data - System shared data.
Key Datasets Link to heading

In order to create a setup that may be used with boot environments the root filesystem will be contained inside an additional ‘ROOT’ dataset. When used with boot environments this will allow the root filesystem to be cloned between environments while sharing any datasets that are not inside ‘ROOT’. It is not necessary to use boot environments but there is no downside to creating a system that is compatible with them for the future.

At minimum the following datasets should be created inside the root pool, ${SYS_ROOT}/${SYSTEM_NAME}/ROOT, and ${SYS_ROOT}/${SYSTEM_NAME}/ROOT/default. Most people will also want a separate ${SYS_ROOT}/${SYSTEM_NAME}/home dataset that is not contained within the boot environment ‘ROOT’ data set.

    • Will contain boot environments.
    • The dataset we will be using as the filesystem root ‘default’ will reside within.
    • Will not be mounted using property mountpoint=none.
  • ${SYS_ROOT}/${SYSTEM_NAME}/ROOT/default
    • The root dataset boot environment, can be named anything but ‘default’ is the convention for the initial boot environment.
    • Uses ZFS mounting with property mountpoint=/.
  • ${SYS_ROOT}/${SYSTEM_NAME}/home
    • The home dataset.
    • Does not go within ${SYS_ROOT}/${SYSTEM_NAME}/ROOT so that it is shared between boot environments.
    • Uses legacy mounting with property mountpoint=legacy.
    • Will be mounted at /home.

For example, my current system’s boot environment which will be mounted to /:


In this configuration it makes it easy to dual boot multiple systems off of a single ZFS pool. To create a new system just add a new dataset under vault/sys, and set it up as normal. This should even work dual booting Linux and FreeBSD.

Create the ${SYS_ROOT}/${SYSTEM_NAME}/ROOT/default and ${SYS_ROOT}/${SYSTEM_NAME}/home datasets and set their mount points.

[root]# SYS_ROOT=vault/sys; SYSTEM_NAME=chin
[root]# zfs create -o mountpoint=none -p ${SYS_ROOT}/${SYSTEM_NAME}
[root]# zfs create -o mountpoint=none ${SYS_ROOT}/${SYSTEM_NAME}/ROOT
[root]# zfs create -o mountpoint=/ ${SYS_ROOT}/${SYSTEM_NAME}/ROOT/default
[root]# zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home
Additional Datasets Link to heading

It is not necessary to create any additional datasets; however, since there is virtually no cost to using them and doing so gives the ability to manipulate properties for each dataset individually.

I set the properties following the tuning recommendations in various places including the Arch wiki:

canmount=off Datasets Link to heading

Set /var, /var/lib and /usr to canmount=off meaning they’re not mounted and are only there to create the ZFS dataset structure. This will put their data in the boot environment dataset. Their properties will be inherited by any child datasets.

Properties for /var:

  • xattr=sa - Stores data in inodes and can increase performance.

Create the datasets:

[root]# zfs create -o canmount=off -o mountpoint=/var -o xattr=sa ${SYS_ROOT}/${SYSTEM_NAME}/var
[root]# zfs create -o canmount=off -o mountpoint=/var/lib ${SYS_ROOT}/${SYSTEM_NAME}/var/lib
[root]# zfs create -o canmount=off -o mountpoint=/var/lib/systemd ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/systemd
[root]# zfs create -o canmount=off -o mountpoint=/usr ${SYS_ROOT}/${SYSTEM_NAME}/usr
System Datasets Link to heading

The rest of the datasets will be independent from the boot environment and will not change between boot environments.

I keep some datasets like /var/cache’s’ dataset seperate to avoid having to snapshot and backup their data. I also keep /var/log ’s’ dataset seperate so the logs are always available as well as the datasets for my containers and VMs.

NOTE: For the following for loop work in zsh, you need to make zsh split over whitespace with set -o shwordsplit

[root]# SYSTEM_DATASETS='var/lib/systemd/coredump var/log var/log/journal var/lib/lxc var/lib/lxd var/lib/machines var/lib/libvirt var/cache usr/local'
[root]# SYS_ROOT=vault/sys; SYSTEM_NAME=chin;
[root]# for ds in ${SYSTEM_DATASETS}; do zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/${ds}; done

Create and turn on posixacls for systemd-journald’s /var/log/journal dataset.

[root]# zfs create -o mountpoint=legacy -o acltype=posixacl ${SYS_ROOT}/${SYSTEM_NAME}/var/log/journal
User Datasets Link to heading

I create extensive user datasets outside the boot environment.

[root]# USER_DATASETS='john john/local john/config john/cache'
[root]# for ds in ${USER_DATASETS}; do zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/${ds}; done

I create ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share and set it to canmount=off just to create the ZFS dataset structure.

[root]# zfs create -o mountpoint=/home/john/.local/share -o canmount=off ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share
[root]# zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share/Steam
Storage Datasets Link to heading

I’ll be creating my data directories under vault/data, they exist outside the different systems and are shared between them.

[root]# DATA_ROOT=vault/data;
[root]# zfs create -o mountpoint=none ${DATA_ROOT}
DATA_DATASETS='Books Computer Personal Pictures University Workspace Reference'
[root]# for ds in ${DATA_DATASETS}; do zfs create -o mountpoint=legacy ${DATA_ROOT}/${ds}; done
Final Structure Link to heading

So my system ends up as.

[root]# zfs list -o name | grep -E 'chin|data'

vault/data                                   768K   860G    96K  none
vault/data/Books                              96K   860G    96K  legacy
vault/data/Computer                           96K   860G    96K  legacy
vault/data/Personal                           96K   860G    96K  legacy
vault/data/Pictures                           96K   860G    96K  legacy
vault/data/Reference                          96K   860G    96K  legacy
vault/data/University                         96K   860G    96K  legacy
vault/data/Workspace                          96K   860G    96K  legacy
vault/sys/chin                              1.97M   860G    96K  none
vault/sys/chin/ROOT                          192K   860G    96K  none
vault/sys/chin/ROOT/default                   96K   860G    96K  /
vault/sys/chin/home                          672K   860G    96K  legacy
vault/sys/chin/home/john                     576K   860G    96K  legacy
vault/sys/chin/home/john/cache                96K   860G    96K  legacy
vault/sys/chin/home/john/config               96K   860G    96K  legacy
vault/sys/chin/home/john/local               288K   860G    96K  legacy
vault/sys/chin/home/john/local/share         192K   860G    96K  /home/john/.local/share
vault/sys/chin/home/john/local/share/Steam    96K   860G    96K  legacy
vault/sys/chin/usr                           192K   860G    96K  /usr
vault/sys/chin/usr/local                      96K   860G    96K  legacy
vault/sys/chin/var                           864K   860G    96K  /var
vault/sys/chin/var/cache                      96K   860G    96K  legacy
vault/sys/chin/var/lib                       576K   860G    96K  /var/lib
vault/sys/chin/var/lib/lxc                    96K   860G    96K  legacy
vault/sys/chin/var/lib/lxd                    96K   860G    96K  legacy
vault/sys/chin/var/lib/machines               96K   860G    96K  legacy
vault/sys/chin/var/lib/libvirt                96K   860G    96K  legacy
vault/sys/chin/var/lib/systemd               192K   860G    96K  /var/lib/systemd
vault/sys/chin/var/lib/systemd/coredump       96K   860G    96K  legacy
vault/sys/chin/var/log                        96K   860G    96K  legacy

User Delegation Link to heading

As of zfsonlinux 0.7.0 ZFS delegation using zfs allow works on linux. I delegate all datasets under ${SYS_ROOT}/${SYSTEM_NAME}/home/john to my user ‘john’ giving the abiity to snapshot and create datasets.

zfs allow john create,mount,mountpoint,snapshot ${SYS_ROOT}/${SYSTEM_NAME}/home/john

Checking permissions shows john’s permissions.

zfs allow ${SYS_ROOT}/${SYSTEM_NAME}/home/john
---- Permissions on vault/sys/chin/home/john -------------------------
Local+Descendent permissions:
        user john create
[root@chin ~]# zfs allow john snapshot ${SYS_ROOT}/${SYSTEM_NAME}/home/john
[root@chin ~]# zfs allow ${SYS_ROOT}/${SYSTEM_NAME}/home/john
---- Permissions on vault/sys/chin/home/john -------------------------
Local+Descendent permissions:
        user john create,snapshot

Available options:

NAME             TYPE           NOTES
allow            subcommand     Must also have the permission that is
                                being allowed
clone            subcommand     Must also have the 'create' ability and
                                ability in the origin file system
create           subcommand     Must also have the 'mount' ability
destroy          subcommand     Must also have the 'mount' ability
hold             subcommand     Allows adding a user hold to a snapshot
mount            subcommand     Allows mount/umount of ZFS datasets
promote          subcommand     Must also have the 'mount' and 'promote'
                                ability in the origin file system
receive          subcommand     Must also have the 'mount' and 'create'
release          subcommand     Allows releasing a user hold which
                                might destroy the snapshot
rename           subcommand     Must also have the 'mount' and 'create'
                                ability in the new parent
rollback         subcommand
send             subcommand
share            subcommand     Allows sharing file systems over NFS or
                                SMB protocols
snapshot         subcommand
groupquota       other          Allows accessing any groupquota@...
groupused        other          Allows reading any groupused@... property
userprop         other          Allows changing any user property
userquota        other          Allows accessing any userquota@...
userused         other          Allows reading any userused@... property
aclinherit       property
aclmode          property
atime            property
canmount         property
casesensitivity  property
checksum         property
compression      property
copies           property
dedup            property
devices          property
exec             property
logbias          property
mlslabel         property
mountpoint       property
nbmand           property
normalization    property
primarycache     property
quota            property
readonly         property
recordsize       property
refquota         property
refreservation   property
reservation      property
secondarycache   property
setuid           property
shareiscsi       property
sharenfs         property
sharesmb         property
snapdir          property
utf8only         property
version          property
volblocksize     property
volsize          property
vscan            property
xattr            property
zoned            property

Prepare Pool Link to heading

With the datasets created, the pool can be configured.

As a precaution and to prevent later issues, unmount the pool and all datasets.

[root]# zfs umount -a

With the pool ready should be exported. This is a necessary step to prevent problems with importing.

[root]# zpool export vault

Setup Installation Link to heading

Import the pool to the location where the installation will be done, /mnt.

[root]# zpool import -d /dev/disk/by-id -R /mnt vault

The root dataset should be mounted to /mnt, check with zfs mount.

An important cache file was created with the pool. Copy it into the new system.

[root]# cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

If this cache does not exist, create one.

[root]# zpool set cachefile=/etc/zfs/zpool.cache vault

The datasets can now be mounted. If there are any non ZFS data sets such as a boot partition, or swap, it should be mounted normally.

Create the mount points and mount the non ZFS managed datasets and boot partition. Replace xY for the boot partition. Repeat for all your datasets.

[root]# mkdir /mnt/boot
[root]# mount /dev/sdxY /mnt/boot
[root]# mkdir /mnt/home
[root]# mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home /mnt/home
[root]# mkdir /mnt/usr/local
[root]# mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/usr/local /mnt/usr/local
[root]# # Repeat...

With all datasets successfully mounted, legacy datasets can be added to the new fstab. To start with, an fstab can be generated, it will need to be edited to remove any non legacy datasets.

[root]# genfstab -U -p /mnt >> /mnt/etc/fstab

The fstab should contain any partitions or datasets the final system needs, including swap if used.

Edit the mirrorlist to your desired location.

[root]# nano /etc/pacman.d/mirrorlist

Install Link to heading

With everything set up the installation can finally be started.

Install the base system

[root]# pacstrap -i /mnt base base-devel

Configure Ramdisk Link to heading

The mkinitcpio will need some different hooks.

If no separate datasets are used the following hooks should be in the mkinitcpio in a specific order. fsck is not needed with ZFS and should only be there if ext3 or ext4 are used.

Make sure keyboard comes before ZFS so that recovery can be done using the keyboard if necessary..

[root]# nano /mnt/etc/mkinitcpio.conf
# ...
HOOKS="base udev autodetect modconf block keyboard zfs filesystems"
# ...

If a separate data set is used for /usr the ‘usr’ hook should be enabled. I have also found the ‘shutdown’ hook is also needed to make /var unmount properly on shutdown.

# ...
HOOKS="base udev autodetect modconf block keyboard zfs usr filesystems shutdown"
# ...

Enter Chroot Link to heading

The install can now be chrooted into.

[root]# arch-chroot /mnt /bin/bash

Setup ZFS Repositories Link to heading

I find using the archzfs repository is the easiest way to install ZFS. If it is preferable, ZFS can also be compiled from source using the AUR, but the archzfs repo has ZFS pre-compiled making it an simple install.

Before proceeding with the install, the ZFS repositories need to be added.

Add the archzfs repository to /etc/pacman.conf. The archzfs repository should be listed first so that is it is the preferred server. Place it above all other mirrors.

[root]# nano /etc/pacman.conf
Server =$repo/x86_64

# Other repositories...

Next sign the repository key. Confirm it is correct by checking the Arch unofficial user repositories listing before using.

[root]# pacman-key -r 5E1ABF240EE7A126
[root]# pacman-key --lsign-key 5E1ABF240EE7A126

Install ZFS Link to heading

Now ZFS can be installed, there are a few options from the archzfs repository:

I was originally using the git packages but after running into a problem I switched over to the zfs-linux repository which is the ZOL release version. Unless you are very concerned with staying on the extreme bleeding edge I would recommend using the zfs-linux repository.

Update the mirrors and install ZFS.

[root]# pacman -Syyu
[root]# pacman -S  zfs-linux

Install System Link to heading

At this point the system can be installed as usual. Proceed through until the point where the bootloader would normally be configured.

Bootloader Link to heading

EFI Bootloader Link to heading

My preferred bootloader for its simplicity is ‘gummiboot’, now called ‘systemd-boot’. When using an EFI system it is what is recommended by the Arch wiki, and what i’d recommend. It will already be installed on Arch by default.

Install systemd-boot to wherever the esp is mounted, /boot generally.

[root]# bootctl --path=/boot install

Make the bootloader entry. When using ZFS the extra parameter zfs=<root dataset> must be added to the list of options. Other than that, bootloader parameters should be the same as a normal install.

[root]# nano /boot/loader/entries/arch.conf
title     Arch Linux
linux     /vmlinuz-linux
initrd    /initramfs-linux.img
options   zfs=vault/sys/chin/ROOT/default rw

If you decide to go with a different bootloader, the setup should be the same as normal except for adding zfs=<root dataset> to the options

BIOS Bootloader Link to heading

If you have a BIOS system you will want to use grub.

After installing grub, run (replace sdx with your drive you’re booting from)

[root]# grub-install --target=i386-pc /dev/sdx

Setup a custom boot entry

# /etc/grub.d/40_custom

exec tail -n +3 $0

set timeout=2
set default=0

# (0) Arch Linux
menuentry "Arch Linux" {
    linux /vmlinuz-linux zfs=vault/ROOT/default rw
    initrd /intel-ucode.img /initramfs-linux.img

After editing run

[root]# grub-mkconfig -o /boot/grub/grub.cfg

You might get the following output

Installing for i386-pc platform.
grub-install: error: failed to get canonical path of `/dev/ata-SAMSUNG_SSD_830_Series_S0VVNEAC702110-part2'.

A workaround is to symlink the expected partition to the id

[root]# ln -s /dev/sda2 /dev/ata-SAMSUNG_SSD_830_Series_S0VVNEAC702110-part2

Clean Up Link to heading

Once finishing everything necessary to finish installation, it is important to export a pool properly before restarting. Failing to do so can result in the pool not importing at boot.

Exit out of the install.

[root]# exit

Export Pool Link to heading

After exiting out of the install, unmount any normal partitions, followed by any ZFS datasets. The command zfs unmount -a should take care of unmounting all of the ZFS datasets however if the pool doesn’t want to export they may need to be unmounted by hand.

[root]# umount /mnt/boot
[root]# zfs umount -a

Now the pool can be exported.

[root]# zpool export vault

First Tasks Link to heading

The system should start up normally for the first boot; however, a few tasks are necessary to make sure the system continues to boot properly.

Set the cache file.

[root]# zpool set cachefile=/etc/zfs/zpool.cache vault

To make sure pools are imported automatically, enable

[root]# systemctl enable

If your datasets refuse to automount on boot you may have to play around with switching from legacy mounting to ZFS managed mounting, or vise versa. You may also have to enable certain units such as zfs-import-cache and zfs-mount.

Due to problems with the machines host ID being unavailable to the system at boot, the initramfs image needs to be adjusted to store the host ID. The easiest way to do this is to run a program which will save the host ID into the image. The alternative is to pass the host ID to the bootloader in an additional option with spl.spl_hostid=0x<hostid>.

You can generate a hostid with zgenhostid

[root]# zgenhostid $(hostid)

Now that the system will properly remember it’s host ID, the initramfs should be regenerated

[root]# mkinitcpio -p linux

Problems Link to heading

That should conclude process of setting up ZFS on Arch Linux. Make sure the system boots properly and that all datasets are mounted at boot. If some datasets do not seem to be mounting properly make sure the properties are set right.

ZFS properties can be queried with zfs get <property> <dataset> so the home dataset can be checked with:

[root]# zfs get mountpoint vault/sys/chin/home

A property can be set with zfs set <property> <dataset>.

Follow-up Link to heading

Following getting an installation working, there are plenty of features to play with in in ZFS which I get into in part 3 of this series, Arch Linux on ZFS - Part 3: Backups, Snapshots and Other Features. A few ow these key features to take look at are:

  • snapshots - Take atomic snaphots of a system that can be used as a source of backup or saved and rolled back to in an emergency.
  • rollback - Revert a dataset back to the state it was in. Can be useful for reverting system breaking changes.
  • send and recieve - Systems built directly into ZFS for sending and recieving a stream of data. Can be used in combination with snapshots to send a stream of data over SSH and do incremental backups.

All of the code used in this post is available on my github. I have split the code up into three parts, the code used to setup before the chroot, the code used in the chroot, and the code used after reboot. The scripts are not runnable, but they are a good reference.

Updated on June 1st, 2018