Skip to main content

Storage & Backups

Create zpool10 Storage Pool

The host's rpool (default ZFS pool used by Proxmox) can be used for templates and whatnot but I want a big pool for storing all my data. So I created a ZFS RAIDZ2 pool using 6x 10TB HDDs.

This is the command I used to build my ZFS pool.

# zpool create -o ashift=12 -o failmode=continue -O compression=lz4 -O xattr=sa -O atime=off \
-O encryption=aes-256-gcm -O keyformat=passphrase \
-m /storage/zpool10 zpool10 raidz2 \
/dev/disk/by-id/ata-WDC_WD100EMAZ-00WJTA0_JEKH3DVZ \
/dev/disk/by-id/ata-WDC_WD100EMAZ-00WJTA0_2YK148SD \
/dev/disk/by-id/ata-WDC_WD100EMAZ-00WJTA0_JEKH8RWZ \
/dev/disk/by-id/ata-WDC_WD100EMAZ-00WJTA0_JEK6ESAN \
/dev/disk/by-id/ata-WDC_WD100EMAZ-00WJTA0_JEK53ZHN \
/dev/disk/by-id/ata-WDC_WD100EMAZ-00WJTA0_2YK0HL0D
    ashift=12 to use 4k blocks failmode=continue to let us keep reading if a drive goes bad compression=lz4 save save space and increase speed xattr=sa be more efficient for linux attributes encryption=aes-256-gcm use fast/secure encryption algorithm keyformat=passphrase unlock with a passphrase

    Setup ZFS Scrub (Data Integrity)

    Automate ZFS scrubbing so the data integrity on disks is actively monitored, repaired if necessary, and I'm alerted if there is a problem with my disks.

    Create systemd Service/Timer (source)

    Create a simple systemd servcie template for scrubbing ZFS pools.

    # /etc/systemd/system/zpool-scrub@.service
    + [Unit]
    + Description=Scrub ZFS Pool
    + Requires=zfs.target
    + After=zfs.target
    +
    + [Service]
    + Type=oneshot
    + ExecStartPre=-/usr/sbin/zpool scrub -s %I
    + ExecStart=/usr/sbin/zpool scrub %I
    

    Then create a systemd timer template for periodically running that service. I am running the scrub weekly, but semi-monthly or monthly would almost certainly be ok too.

    # /etc/systemd/system/zpool-scrub@.timer
    + [Unit]
    + Description=Scrub ZFS pool weekly
    +
    + [Timer]
    + OnCalendar=weekly
    + Persistent=true
    +
    + [Install]
    + WantedBy=timers.target
    

    Enable ZFS Scrub

    systemctl daemon-reload
    systemctl enable --now zpool-scrub@rpool.timer
    systemctl enable --now zpool-scrub@zpool10.timer
    

    Setup Storage Layout

    I wanted to logically device my storage pool up into datasets based on their intended usage. This allows me to tweak their paramters if needed and have different snapshot/backup policies.

      zpool10/backups local backups of rpool zpool10/downloads landing zone of downloads zpool10/downloads/incomplete landing zone for incomplete downloads (recordsize=16k for bittorrent) zpool10/media storage for audio/tv/movies zpool10/proxmox additional storage for proxmox zpool10/proxmox/backups backup for proxmox containers/vms zpool10/services storage for services (possibly databases, so use recordsize=16k)
      zfs create zpool10/backups
      zfs create zpool10/downloads
      zfs create -o recordsize=16K zpool10/downloads/incomplete
      zfs create zpool10/media
      zfs create zpool10/proxmox
      zfs create zpool10/proxmox/backups
      zfs create -o recordsize=16K zpool10/services
      

      Setup Sanoid/Syncoid (Data Backup)

      Run Sanoid for automating snapshots and Syncoid for remote backups. Unfortunately this isn't available in repositories so you have to build it yourself. However the author makes it fairly simple.

      Install (source)

      apt-get install debhelper libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuffer
      sudo git clone https://github.com/jimsalterjrs/sanoid.git
      cd sanoid
      ln -s packages/debian .
      dpkg-buildpackage -uc -us
      apt install ../sanoid_*_all.deb
      

      Configure Sanoid

      I want to take hourly snapshots of both of my ZFS pools because sometimes I am not as careful or thoughtful as I should be about what I am doing at any given moment. But I don't want to snapshot zpool/backups because it is a backup destination that will likely already have snapshots (rpool snapshots are stored there for example) and I also don't want to snapshot zpool/downloads because there is nothing important under there and it is likely to change frquently.

      # /etc/sanoid/sanoid.conf
      + [rpool]
      + 	recursive = yes
      +         frequently = 0
      +         hourly = 36
      +         daily = 30
      +         monthly = 3
      +         yearly = 0
      +         autosnap = yes
      +         autoprune = yes
      +
      + [zpool10]
      +         recursive = yes
      +         frequently = 0
      +         hourly = 24
      +         daily = 30
      +         monthly = 1
      +         yearly = 0
      +         autosnap = yes
      +         autoprune = yes
      +
      + [zpool10/backups]
      +         recursive = yes
      +         hourly = 24
      +         autosnap = no
      +         autoprune = yes
      +
      + [zpool10/downloads]
      +         recursive = yes
      +         hourly = 24
      +         autosnap = no
      +         autoprune = yes
      

      Maybe this is a sin, but I'd like my snapshots to be in local time so I don't have to do the (admittedly simple) conversion in my head.

      # /usr/lib/systemd/system/sanoid.service
        [Service]
      - Environment=TZ=UTC
      + Environment=TZ=EST
      

      Eanble Sanoid

      systemctl daemon-reload
      systemctl enable --now sanoid.service
      

      Configure Syncoid

      Backup rpool to zpool10/backups

      Right now rpool is just running on a single non-redundant 512GB SSD disk. Even though it is only used for Proxmox (config, templates, ISOs) this isn't great practice and I'll work on this in the future. But in the meantime I am backing up everything on a daily timer to my main ZFS pool so I could recover very quickly if the SSD dies.

      # /etc/systemd/system/rpool-backup.timer
      + [Unit]
      + Description=Backup rpool daily
      +
      + [Timer]
      + OnCalendar=daily
      + Persistent=true
      +
      + [Install]
      + WantedBy=timers.target
      
      # /etc/systemd/system/rpool-backup.service
      + [Unit]
      + Description=Use syncoid to backup rpool to zpool10/backups/rpool
      + Requires=zfs.target
      + After=zfs.target
      +
      + [Service]
      + Type=oneshot
      + ExecStart=/usr/sbin/syncoid --recursive rpool zpool10/backups/rpool
      
      Backup zpool10/services offsite

      All my docker containers store their configuration and data under the zpool10/services dataset. It is imperative this is backed up offsite so if anything catastrophic ever happens I don't lose anything important and can get back up and running as quickly as I can download my backup.

      # /etc/systemd/system/zpool10-services-backup.timer
      + [Unit]
      + Description=Backup zpool10/services daily
      +
      + [Timer]
      + OnCalendar=daily
      + Persistent=true
      +
      + [Install]
      + WantedBy=timers.target
      
      # /etc/systemd/system/zpool10-services-backup.service
      + [Unit]
      + Description=Use syncoid to backup zpool10/services to backedup.swigg.net:bpool/zpool10/services
      + Requires=zfs.target
      + After=zfs.target
      +
      + [Service]
      + Type=oneshot
      + ExecStart=/usr/sbin/syncoid --recursive zpool10/services root@backedup.swigg.net:bpool/zpool10/services
      

      Enable Syncoid

      systemctl daemon-reload
      systemctl enable --now rpool-backup.timer