In previous posts I went though setting up a custom Arch Linux install image with ZFS pre-installed, and then installing ZFS on Arch linux. Once finished the install, the fun part comes; playing with all the interesting features ZFS has to offer. This post will walk through using and setting up some of the great futures ZFS has, and how to use them on Linux.

Part Three - Backups, Snapshots and Other Features

Note: For the following post I work with an example pool similar to my own system which looks like the following.

[root]# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
vault                327G   103G    96K  none
vault/ROOT          4.48G   103G    96K  none
vault/ROOT/default  4.48G   103G  2.35G  legacy
vault/home           253G   103G  94.1G  /home
vault/tmp            414M   103G  36.2M  /tmp
vault/usr           11.7G   103G  3.27G  legacy
vault/var           10.1G   103G  5.00G  legacy

Snapshots

Snapshots are one of the best features of ZFS. They work in an intelligent manner and unlike some other filesystems that offer snapshots, they are atomic and instant.

Taking Snapshots Manually

Taking a snapshot of a dataset, for example my home dataset ‘vault/home’, is as easy as:

[root]# zfs snapshot [email protected]

The part following the ‘@’ will be the name of the snapshot.

A recursive snapshot can be taken that will include all child datasets by using the ‘-r’ flag. To take a snapshot of an entire pool and it’s children datasets:

[root]# zfs snapshot -r [email protected]

The snapshots can be listed with zfs list -t snapshot

[root]# zfs list -t snapshot
NAME                                       USED  AVAIL  REFER  MOUNTPOINT
[email protected]                           0      -    96K  -
[email protected]              0      -  2.35G  -
[email protected]                  4.91M      -  94.1G  -
[email protected]                       0      -  3.27G  -
[email protected]                   4.00M      -  5.00G  -

Snapshots can also be viewed by looking In the hidden directory ‘.zfs/snapshots/’. This folder will exist under every dataset.

This can be a great way to look for, and restore old files. If you delete something you need, instead of rolling back an entire snapshot, you can open up ‘.zfs/snapshots/’, find the snapshot you’re looking for, and restore your file.

ZFS Snapshots

ZFS Rollback

If you decide to roll back to an existing snapshot, the syntax is very similar to taking a snapshot.

[root]# zfs rollback [email protected]

The ‘-r’ option can be confusing, seeing as when taking snapshots it means to take a recursive snapshot, you might assume the same here. In fact, the ‘-r’ option will destroy any snapshot between the one you want to roll back to, and the existing dataset. If you’ve taken snapshots after the snapshot you want to roll back to, it will probably be necessary to use this flag.

[root]# zfs rollback -r [email protected]

Sending Snapshots With ZFS Send and Recieve

One of the great features that goes along well with snapshots is zfs send and zfs receive. zfs send and recieve streams snapshots to to another location or file, and can act similar to rsync. The fact that they can be streamed means they can be piped to another disk, another file for cold storage, or best of all, sent over SSH.

If you have access to another box with ZFS within reach of SSH, you can send entire datasets incrementally, and receive them on the other end replicating them into an identical dataset. With zfs send and receive you can even save snapshots, making it a perfect backup solution.

[root]# zfs send [email protected]_snapshot | ssh user zfs recv tank/backups/home

If you do not have another machine with set of eyes on it to back up to, you can pipe ZFS send into a file, and back up that file. Later on you can pipe the file back into ZFS receive, restoring an old pool.

[root]# zfs send vault  | gzip > zfs_backup.gz

Snapshot Tools

Because the snapshots are so small and effectively take up no space until something changes, they really come to shine when used routinely. While it is possible to set up a simple backup system this way, there are numerous tools available to take care of the difficult parts.

ZFS Auto Snapshot

One of the simplest of these is a shell script made by the ZFS on Linux guys, zfs-auto-snapshot, an “Automatic Snapshot Service for Linux”.

The Arch Linux package zfs-auto-snapshot-git, comes with systemd units and is simple to use requiring almost no setup.

While it can be customized by hand, several pre-configured systemd units come with the Arch package that will take snapshots every 15 minutes, hourly, daily, or monthly and we’ll take care of the cleanup process. It snapshot all datasets, in every pool. Datasets can be excluded by setting the property ‘com.sun:auto-snapshot=false

This is the unit for the daily configuration, it will prefix all snapshots with znap and keep 31 snapshots at a time.

[root]# cat /usr/lib/systemd/system/zfs-auto-snapshot-daily.service
[Unit]
Description=ZFS daily snapshot service

[Service]
ExecStart=/bin/zfs-auto-snapshot --skip-scrub --prefix=znap --label=daily --keep=31 //

Setting up recurring snapshots on a daily basis is as simple as starting a few services:

[root]# systemctl enable zfs-auto-snapshot-daily.timer
[root]# systemctl start zfs-auto-snapshot-daily.timer

You will probably want to exclude your ‘/tmp’ directory.

[root]# zfs set com.sun:auto-snapshot=false vault/tmp

ZnapZend

ZnapZend is a more advanced tool that not only takes snapshots, it will also send them over SSH to another computer and replicate datasets. After giving ZnapZend a dataset, source, and destination, it will do the rest. You can also give it specifications on how long to keep hourly, daily, monthly, and yearly snapshots. If you do not have a server to send your backups to you are also able to send them to another location on your computer.

The configuration is simple. For each dataset you want to replicate, you specify a configuration, and then enable the znapzend daemon which will take the snapshots and run the backups.

On my computer I have everything replicated every 15 minutes to my other server.

[root]# znapzendzetup create --recursive --tsformat='%Y-%m-%d-%H%M%S' \
    SRC '1d=>15min,7d=>1h,30d=>4h,90d=>1d' shard/home \
    DST:lilan '1d=>15min,7d=>1h,30d=>4h,90d=>1d,1y=>1w,10y=>1month' \
    [email protected]:tank/replication/Switchblade/home

The status of your replication can be checked with znapzendztatz.

[root]# znapzendztatz
USED    LAST SNAPSHOT       DATASET
 828M   2016-08-29-010000   shard/home
 905M   2016-08-29-010000   [email protected]:tank/replication/Switchblade/home
14.1M   2016-08-29-010000   vault/ROOT/default
22.8M   2016-08-29-010000   [email protected]:tank/replication/Switchblade/default
 144M   2016-08-29-010000   vault/usr
 232M   2016-08-29-010000   [email protected]:tank/replication/Switchblade/usr
 609M   2016-08-29-010000   vault/var
 712M   2016-08-29-010000   [email protected]:tank/replication/Switchblade/var

Scrub

Occasionally ZFS should scan your datasets looking for and fixing any corruption it finds, this is called a scrub. Scrubbing is simple and should be run every few weeks.

To run a scrub on a pool run ‘zpool scrub ${pool}’.

On Arch Linux this can be run with a systemd service and timer. The package systemd-zpool-scrub includes a simple service file that will run a scrub weekly.

The service file zpool-scrub@.service:

[Unit]
Description=Scrub ZFS Pool
Requires=zfs.target
After=zfs.target

[Service]
Type=oneshot
ExecStartPre=-/usr/bin/zpool scrub -s %i
ExecStart=/usr/bin/zpool scrub %i

and timer zpool-scrub@.timer:

[Unit]
Description=Scrub ZFS pool weekly

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=timers.target

These should be run for each pool:

[root]# systemctl enable [email protected]
[root]# systemctl start [email protected]

ZED: The ZFS Event Daemon

The ZFS Event Daemon, or ZED, keeps track of currents events on your system and can do useful things such as shooting you an email when a scrub starts, or when your pool becomes unhealthy.

ZFS events can be seen with zpool events. As you can see here, recently a scrub was started on my system.

[root]# zpool events
TIME                           CLASS
Aug 28 2016 22:53:35.983333543 resource.fs.zfs.statechange
Aug 28 2016 22:53:36.106666894 ereport.fs.zfs.config.sync
Aug 28 2016 22:53:36.136666898 ereport.fs.zfs.config.sync
Aug 28 2016 22:53:38.856334836 resource.fs.zfs.statechange
Aug 28 2016 22:53:39.416334823 ereport.fs.zfs.config.sync
Aug 28 2016 22:53:39.523001487 ereport.fs.zfs.config.sync
Aug 29 2016 00:00:13.049477101 ereport.fs.zfs.scrub.start
Aug 29 2016 00:00:13.052810465 ereport.fs.zfs.scrub.start
Aug 29 2016 00:00:24.022911412 ereport.fs.zfs.scrub.finish
Aug 29 2016 00:16:40.849409154 ereport.fs.zfs.scrub.finish

I like to know that these scrubs are happening, and that they haven’t found anything that needs to be pursued. Assuming you have email setup on your system, setting the daemon up to email you is very easy. If you have not yet configured email, I recommend checking out SSMTP. Rather than being a mail server, SSMTP simply send out mail, and is quite easy to set up.

The ZFS Event Daemon has a few pre written configuration files, and can be set up in its simplest form by just uncommenting a few lines.

Edit the configuration file /etc/zfs/zed.d/zed.rc

[root]# nano /etc/zfs/zed.d/zed.rc

At minimum you will need to uncomment ZED_EMAIL_ADDR="root".

##
# zed.rc
#
# This file should be owned by root and permissioned 0600.
##

##
# Email address of the zpool administrator for receipt of notifications;
#   multiple addresses can be specified if they are delimited by whitespace.
# Email will only be sent if ZED_EMAIL_ADDR is defined.
# Disabled by default; uncomment to enable.
#
ZED_EMAIL_ADDR="root"

and start ZED.

[root]# systemctl start zfs-zed.service
[root]# systemctl enable zfs-zed.service

You may also want to define your mail program.

##
# Name or path of executable responsible for sending notifications via email;
#   the mail program must be capable of reading a message body from stdin.
# Email will only be sent if ZED_EMAIL_ADDR is defined.
#
ZED_EMAIL_PROG="mail"

If you want to receive email no matter the state of your pool, you’ll want to set ZED_NOTIFY_VERBOSE=1.

##
# Notification verbosity.
#   If set to 0, suppress notification if the pool is healthy.
#   If set to 1, send notification regardless of pool health.
#
ZED_NOTIFY_VERBOSE=1

I haven’t personally played with them, but there are some interesting options like setting a hard drive to be replaced when errors are detected.

##
# Replace a device with a hot spare after N checksum errors are detected.
# Disabled by default; uncomment to enable.
#
#ZED_SPARE_ON_CHECKSUM_ERRORS=10

##
# Replace a device with a hot spare after N I/O errors are detected.
# Disabled by default; uncomment to enable.
#
#ZED_SPARE_ON_IO_ERRORS=1

When a scrub finishes you’ll receive a message similar to this for a healthy pool.

Received: by Archon (sSMTP sendmail emulation); Mon, 29 Aug 2016 19:07:05 -0700
From: "root" <[email protected]>
Date: Mon, 29 Aug 2016 19:07:05 -0700
To: root
Subject: ZFS scrub.finish event for vault on Archon

ZFS has finished a scrub:

   eid: 8
 class: scrub.finish
  host: Archon
  time: 2016-08-29 19:07:05-0700
  pool: vault
 state: ONLINE
  scan: scrub repaired 0 in 0h19m with 0 errors on Mon Aug 29 19:07:05 2016
config:

	NAME                                       STATE     READ WRITE CKSUM
	vault                                      ONLINE       0     0     0
	  mirror-0                                 ONLINE       0     0     0
	    ata-SanDisk_SDSSDXPS480G_152271401093  ONLINE       0     0     0
	    ata-SanDisk_SDSSDXPS480G_154501401266  ONLINE       0     0     0

errors: No known data errors

If you’re not happy with the default email you can change what will be emailed to you by editinh the file /etc/zfs/zed.d/scrub.finish-notify.sh.

Conclusion

There are many interesting features built into ZFS and I have only touched on a few of them. ZFS is a great piece of enterprise grade software and we owe a debt of gratitude to the programmers who I have made it so freely available, on multiple platforms.