How To Use ZFS on Linux for Your Home Directory

By | 2015/12/22

Here is how to setup two hard disks in a zpool mirror on Linux for use as a home directory!


Some ZFS points to know:

– A zpool is a collection of disks. This guide will use two disks destructively. Two disks are recommended as a minimum though you can have a zpool of one disk.
– Use two identical disks, or at least two that are the same exact size.
– Make sure you have ~8GB of ram minimum. 16GB or more is better.
– You cannot partition or format a drive zfs with gparted, or similar. To add a disk to a zpool, one gives zfs the entire disk.
– 64bit only. The zfsonlinux.org FAQ state that this should be used on 64bit Linux only.
– Don’t put data directly into the zpool. Use ZFS datasets instead.
– This is not a backup. Backup your data to another disk or other backup solution.
– Only do this if you know what you are doing.


This guide

This guide assumes the boot or / is on a separate disk, such as an SSD. /home will be a zpool mirror of two new disks.

Notes and instructions here are taken from the zfsonlinux.org FAQ and Aaron Toponce’s ZFS on Debian guide.


Step 1: Install ZFS on Linux

From the following page, setup the repository and install the ZFS for your distribution following the instructions indicated:

http://zfsonlinux.org/

For snapshots, install the following:

https://github.com/zfsonlinux/zfs-auto-snapshot

Note: On Debian, I had to edit the file /etc/cron.d/zfs-auto-snapshot and make the path line at the top look like the following:

PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin"

See this github link.


Step 2: Locate the correct disk names

Create your pool using whole disks: When running zpool create use whole disk names. This will allow ZFS to automatically partition the disk to ensure correct alignment. It will also improve interoperability with other ZFS implementations which honor the wholedisk property.

For small pools (less than 10 disks) use the /dev/disk/by-id/ identifier for each disk.

I’ve identified my two disks I will use in the zpool mirror from an $ ls /dev/disk/by-id/ and locating my disks I want to use.

ata-TOSHIBA_MD04ACA400_95J3KNMLFSAA

ata-TOSHIBA_MD04ACA400_Y4NHK7CJFSBA

Step 3: Advanced format drives: option ashift=12

For now, take note of this option we will be using when first creating the zpool on Advanced Format (4K) drives.

Improve performance by setting ashift=12: You may be able to improve performance for some workloads by setting ashift=12. This tuning can only be set when the pool is first created and it will result in a decrease of capacity. For additional detail on why you should set this option when using Advanced Format drives see section on the zfsonlinux.org FAQ.


Step 4: Create the zpool

Note: THIS IS DESTRUCTIVE! All data on these disks will then be gone!

sudo zpool create -o ashift=12 zpool0 mirror ata-TOSHIBA_MD04ACA400_95J3KNMLFSAA ata-TOSHIBA_MD04ACA400_Y4NHK7CJFSBA

Step 5: Set Compression on and dedupe off

From Aaron Toponce’s guide:

“Compression is disabled by default. This doesn’t make much sense with today’s hardware. ZFS compression is extremely cheap, extremely fast, and barely adds any latency to the reads and writes. In fact, in some scenarios, your disks will respond faster with compression enabled than disabled. A further benefit is the massive space benefits.”

Here I am setting my compression to use lz4 – this is optional.

Deduplication is typically disabled for small home systems. Here I am turning it off. I would advise to read over this doc to become familiar with the cost of deduplication.

sudo zfs set compression=on zpool0
sudo zfs set compression=lz4 zpool0
sudo zfs set dedup=off zpool0

Step 6: Mount your home directory

There is no need (nor can one) use /etc/fstab with zfs. Instead, just set the mountpoint with zfs and this will be mounted at boot by zfs.

Here we are making a dataset inside zpool0 called home.

sudo zfs create zpool0/home
sudo zfs set mountpoint=/home zpool0/home
sudo zfs mount zpool0/home

Step 7: scrub

It is important to scrub zfs weekly. Create the file /etc/cron.weekly/zfsscrub with the following contents, and sudo chmod +x the file.

#!/bin/sh
/sbin/zpool scrub zpool0

Step 8: Set snapshot directory visible

The snapshot directory will be /home/.zfs/

sudo zfs set snapdir=visible zpool0/home

Step 9: Limit ARC cache memory usage

Create or edit the existing file /etc/modprobe.d/zfs.conf and limit the arc cache to 4GB. Otherwise, ZFS will use all system ram available which is ok for a file server, but not desirable for a workstation.

# /etc/modprobe.d/zfs.conf
# Limit arc to 4GB
options zfs zfs_arc_max=4294967296
Done!

This page is an excellent command reference guide for zfs.

Here are a few random good commands to know:

stmiller@brahms:~$ sudo zfs list
NAME          USED  AVAIL  REFER  MOUNTPOINT
zpool0       1.61T  1.90T    96K  /zpool0
zpool0/home  1.61T  1.90T  1.61T  /home

stmiller@brahms:~$ sudo zfs get all zpool0/home
NAME         PROPERTY              VALUE                  SOURCE
zpool0/home  type                  filesystem             -
zpool0/home  creation              Mon Dec 21 18:43 2015  -
zpool0/home  used                  1.61T                  -
zpool0/home  available             1.90T                  -
zpool0/home  referenced            1.61T                  -
zpool0/home  compressratio         1.08x                  -
zpool0/home  mounted               yes                    -
zpool0/home  quota                 none                   default
zpool0/home  reservation           none                   default
zpool0/home  recordsize            128K                   default
zpool0/home  mountpoint            /home                  local
zpool0/home  sharenfs              off                    default
zpool0/home  checksum              on                     default
zpool0/home  compression           lz4                    inherited from zpool0
zpool0/home  atime                 on                     default
zpool0/home  devices               on                     default
zpool0/home  exec                  on                     default
zpool0/home  setuid                on                     default
zpool0/home  readonly              off                    default
zpool0/home  zoned                 off                    default
zpool0/home  snapdir               visible                local
zpool0/home  aclinherit            restricted             default
zpool0/home  canmount              on                     default
zpool0/home  xattr                 on                     default
zpool0/home  copies                1                      default
zpool0/home  version               5                      -
zpool0/home  utf8only              off                    -
zpool0/home  normalization         none                   -
zpool0/home  casesensitivity       sensitive              -
zpool0/home  vscan                 off                    default
zpool0/home  nbmand                off                    default
zpool0/home  sharesmb              off                    default
zpool0/home  refquota              none                   default
zpool0/home  refreservation        none                   default
zpool0/home  primarycache          all                    default
zpool0/home  secondarycache        all                    default
zpool0/home  usedbysnapshots       0                      -
zpool0/home  usedbydataset         1.61T                  -
zpool0/home  usedbychildren        0                      -
zpool0/home  usedbyrefreservation  0                      -
zpool0/home  logbias               latency                default
zpool0/home  dedup                 off                    inherited from zpool0
zpool0/home  mlslabel              none                   default
zpool0/home  sync                  standard               default
zpool0/home  refcompressratio      1.08x                  -
zpool0/home  written               1.61T                  -
zpool0/home  logicalused           1.74T                  -
zpool0/home  logicalreferenced     1.74T                  -
zpool0/home  filesystem_limit      none                   default
zpool0/home  snapshot_limit        none                   default
zpool0/home  filesystem_count      none                   default
zpool0/home  snapshot_count        none                   default
zpool0/home  snapdev               hidden                 default
zpool0/home  acltype               off                    default
zpool0/home  context               none                   default
zpool0/home  fscontext             none                   default
zpool0/home  defcontext            none                   default
zpool0/home  rootcontext           none                   default
zpool0/home  relatime              on                     temporary
zpool0/home  redundant_metadata    all                    default
zpool0/home  overlay               off                    default

stmiller@brahms:~$ sudo zpool status zpool0
  pool: zpool0
 state: ONLINE
  scan: none requested
config:

        NAME                                     STATE     READ WRITE CKSUM
        zpool0                                   ONLINE       0     0     0
          mirror-0                               ONLINE       0     0     0
            ata-TOSHIBA_MD04ACA400_95J3KNMLFSAA  ONLINE       0     0     0
            ata-TOSHIBA_MD04ACA400_Y4NHK7CJFSBA  ONLINE       0     0     0

errors: No known data errors

Special nod to TeamCO,