In the last section of this series I explained how to embed ZFS on the Arch Linux install media so that an installation can be completed from the ISO. While in this section I will go through the installation of Arch Linux using ZFS as the root filesystem.
Part Two - Installation
- Pre-Install Setup
- Clean Up
Note: The commands used in this post are available on my github.
I would recomend using a dedicated partition wheather using BIOS or UEFI. While you are supposed to be able to store your kernel and any other boot images on a ZFS dataset when using a BIOS bootloader like GRUB, in my experience I have had more luck using a boot partition.
Pool Type and Redundancy
ZFS has several different levels of redundancy based on the number of discs that are used and in what setting they are configured. They are different than traditional RAID levels.
A crude description of the different levels:
- No redundancy, similar to RAID0.
- Default pool creation mode.
- Minimum of 3 discs.
- One disc can be lost without pool failure.
- One of the discs is redundant and will not provide storage.
- Minimum of 4 discs.
- Two discs can be lost without pool failure.
- Two of the discs are redundant and will not provide storage.
- Minimum of 5 discs.
- Three discs can be lost without pool failure.
- Three of the discs are redundant and will not provide storage.
- Mirror - Minimum of 2 discs.
- Half of the discs can be lost without pool failure.
- Storage from half of the discs are used as redundancy and will not provide storage.
- Provides the best performance and flexibility at a cost of storage space.
It’s recommended to use the disk id names as recommended by the ZOL project. The identification and partition number of each drive can be found with:
Note: In the following post I will be using the above two sandisk SSDs as an example.
Once the disc IDs are known, the pool can be created. While the disks can be partitioned manually in GPT or MBR, it is not necessary to partition entire drives before creating the pool as ZFS will partition the drives itself as
Solaris Root (bf00) in the creation of a new pool.
ZFS has issues being booted off of. The simplest option is to partition the boot partition in another format as would be done with a regular install.
I do not use a swap partition and thus have no experience using one; however, the Arch wiki explains the process of sending one up if necessary.
After deciding on a pool type and getting the disc IDs, a pool can be created with the
zpool create command.
The syntax is:
-n- Display creation but don’t create pool.
-d- Do not enable any features unless specified.
-o- Set a pool property.
-O- Set a property on root filesystem.
-R- Set an alternate root location.
First, probe for ZFS on the system, there should be no output from
zpool create command can be used to create a new pool.
When creating a pool
ashift=12 will specify advanced format disks, this will force 4096 size blocksize. Here I create a mirrored pool named ‘vault’ with my two SSD’s.
Test the pool was created successfully with
There are many properties that can be set on an entire pool or specific data sets. A property that will almost always be wanted on the entire pool is compression. ZFS uses “LZ4” compression which is a great compromise between performance and amount of compression.
Set compression on
atime=on is enabled, it can be turned off to increase performance, or set to
relatime which is the default on many Linux filesystems.
In recording access time
relatime is a good compromise, it will record access time but much more infrequently than the default
Datasets are similar to partitions; however, they come with many benefits partitions do not have. In addition to being hierarchically organizable, they do not require a specific quota. All of the datasets will share space in a given pool. These qualities in datasets mean that they can be used extensively without repercussions.
ZFS has the ability to manage datasets itself, or to tell the datasets to fall back to the system controlled legacy management where datasets will be managed with the fstab.
I have found using legacy management works best. If legacy mounting fails, which it does in certain circumstances I use ZFS managed mounting.
In order to create a setup that may be used with boot environments the root filesystem will be contained inside an additional ‘
ROOT’ dataset. When used with boot environments this will allow the root filesystem to be cloned between environments while sharing any datasets that are not inside ‘
ROOT’. It is not necessary to use boot environments but there is no downside to creating a system that is compatible with them for the future.
At minimum the following data sets should be created inside the root pool,
vault/ROOT/default. Most people will also want a separate
vault/home dataset that is not contained within the boot environment ‘
ROOT’ data set.
- Will contain boot environments.
- The dataset we will be using as the filesystem root ‘
default’ will reside within.
- Will not be mounted using property
- The root dataset boot environment, can be named anything but ‘default’ is the convention for the initial boot environment.
- Uses legacy mounting with property
- Will be mounted at
- The home dataset.
- Does not go within
vault/ROOTso that it is shared between boot environments.
- Uses ZFS mounting with property
- Will be mounted at
vault/home datasets and set their mount points.
It is not necessary to create any additional datasets; however, since there is virtually no cost to using them and doing so gives the ability to manipulate properties for each dataset individually, it can make sense to use them for certain key directories such as
I set the properties following the tuning recommendations in various places including the Arch wiki:
sync=disabled- Disabling sync will increase performance by ignoring sync requests.
devices=off- Prevent use of device nodes for security.
setuid=off- Can help prevent privilege-escalation attacks.
mountpoint=/tmp- Use ZFS mounting. Create dataset for
Since we are using a custom dataset for /tmp, mask (disable) systemd’s automatic tmpfs-backed
xattr=sa- Stores data in inodes and can increase performance.
mountpoint=legacy- Use legacy mounting.
Create dataset for
mountpoint=legacy- Use legacy mounting.
With the datasets created, the pool can be configured.
As a precaution and to prevent later issues, unmount the pool and all datasets.
The legacy datasets should all be added to the fstab.
vault/tmp will be mounted by ZFS and do not need to be in the fstab.
The dataset that is going to be used as ‘root’ and booted from needs to have the bootfs property set.
With the pool ready it cannot be exported. This is a necessary step to prevent problems with importing.
Import the pool to the location where the installation will be done,
Unmount zfs mounted datasets if they were automatically mounted
Mount the root dataset
An important cache file was created with the pool. Copy it into the new system.
If this cache does not exist, create one.
The datasets can now be mounted. If there are any non ZFS data sets such as a boot partition, it should be mounted normally.
Create the mount points.
Mount the non ZFS managed datasets and boot partition. Replace xY for the boot partition
Mount the ZFS managed datasets.
Check everything is successfully mounted
With all datasets successfully mounted, legacy datasets can be added to the new fstab. To start with, an fstab can be generated, it will need to be edited to remove any non legacy datasets.
The fstab will look similar to our earlier fstab except it should include the boot partition, and any other partitions or datasets the final system needs including swap if used.
Since my system does not use swap, only the three legacy data sets and a boot partition were needed in my fstab
Edit the mirrorlist to your desired location.
With everything set up the installation can finally be started.
Install the base system
The mkinitcpio will need some different hooks.
If no separate datasets are used the following hooks should be in the mkinitcpio in a specific order. fsck is not needed with ZFS and should only be there if ext3 or ext4 are used.
Make sure keyboard comes before ZFS so that recovery can be done using the keyboard if necessary..
If a separate data set is used for
/usr the ‘usr’ hook should be enabled. I have also found the ‘shutdown’ hook is also needed to make
/var unmount properly on shutdown.
The install can now be chrooted into.
Setup ZFS Repositories
I find using the archzfs repository is the easiest way to install ZFS. If it is preferable, ZFS can also be compiled from source using the AUR, but the archzfs repo has ZFS pre-compiled making it an simple install.
Before proceeding with the install, the ZFS repositories need to be added.
archzfs repository to
archzfs repository should be listed first so that is it is the preferred server. Place it above all other mirrors.
Next sign the repository key. Confirm it is correct by checking the Arch unofficial user repositories listing before using.
Now ZFS can be installed, there are a few options from the archzfs repository:
zfs-linux-git- Packages tracking the zfsonlinux master branch. Recompiled on each kernel release.
zfs-linux- ZOL release packages. Correspond to specific version releases.
zfs-linux-lts- Arch lts linux kernel with ZOL release packages. For people concerned with stability.
I was originally using the git packages but after running into a problem I switched over to the
zfs-linux repository which is the ZOL release version. Unless you are very concerned with staying on the extreme bleeding edge I would recommend using the
Update the mirrors and install ZFS.
At this point the system can be installed as usual. Proceed through until the point where the bootloader would normally be configured.
My preferred bootloader for its simplicity is ‘gummiboot’, now called ‘
systemd-boot’. When using an EFI system it is what is recommended by the Arch wiki, and what i’d recommend. It will already be installed on Arch by default.
systemd-boot to wherever the esp is mounted,
Make the bootloader entry. When using ZFS the extra parameter
zfs=<root dataset> must be added to the list of options. Other than that, bootloader parameters should be the same as a normal install.
If you decide to go with a different bootloader, the setup should be the same as normal except for adding
zfs=<root dataset> to the options
If you have a BIOS system you will want to use grub.
After installing grub, run (replace sdx with your drive you’re booting from)
Setup a custom boot entry
After editing run
You might get the following output
A workaround is to symlink the expected partition to the id
Once finishing everything necessary to finish installation, it is important to export a pool properly before restarting. Failing to do so can result in the pool not importing at boot.
Exit out of the install.
After exiting out of the install, unmount any normal partitions, followed by any ZFS datasets. The command
zfs unmount -a should take care of unmounting all of the ZFS datasets however if the pool doesn’t want to export they may need to be unmounted by hand.
Now the pool can be exported.
The system should start up normally for the first boot; however, a few tasks are necessary to make sure the system continues to boot properly.
Set the cache file.
To make sure pools are imported automatically, enable
If your datasets refuse to automount on boot you may have to play around with switching from legacy mounting to ZFS managed mounting, or vise versa. You may also have to enable certain units such as
Due to problems with the machines host ID being unavailable to the system at boot, the initramfs image needs to be adjusted to store the host ID. The easiest way to do this is to run a program which will save the host ID into the image. The alternative is to pass the host ID to the bootloader in an additional option.
You can clone the file from my github.
Or copy it to a file in the new system.
Compile the program, give it execute permissions, and execute it.
Now that the system will properly remember it’s host ID, the initramfs should be regenerated
That should conclude process of setting up ZFS on Arch Linux. Make sure the system boots properly and that all datasets are mounted at boot. If some datasets do not seem to be mounting properly make sure the properties are set right.
I have had trouble having the
home dataset and
tmp dataset mounted by the the fstab and have had better success mounting them with ZFS. If they are not being mounted make sure their property is set to
ZFS properties can be queried with
zfs get <property> <dataset> so the
home dataset and
tmp dataset can be checked with
A property can be set with
zfs set <property> <dataset>.
Following getting an installation working, there are plenty of features to play with in in ZFS which I get into in part 3 of this series, Arch Linux on ZFS - Part 3: Backups, Snapshots and Other Features. A few ow these key features to take look at are:
- snapshots - Take atomic snaphots of a system that can be used as a source of backup or saved and rolled back to in an emergency.
- rollback - Revert a dataset back to the state it was in. Can be useful for reverting system breaking changes.
- send and recieve - Systems built directly into ZFS for sending and recieving a stream of data. Can be used in combination with snapshots to send a stream of data over SSH and do incremental backups.
All of the code used in this post is available on my github. I have split the code up into three parts, the code used to setup before the chroot, the code used in the chroot, and the code used after reboot. The scripts are not runnable, but they are a good reference.