# mini

# Physical Hardware

## Basic Components

### Compute

![](https://www.amd.com/system/files/82446-raven-am4-ryzen-3-pib-left-facing-1260x709.png)#### Processor

[AMD Ryzen 3 3200G](https://en.wikichip.org/wiki/amd/ryzen_3/3200g) - Economical CPU that has 4 cores and integrated graphics so a PCIe slot can be saved since a GPU isn't necessary.

<span class="vertical-table" id="bkmrk-cores-%2F-threads-4-%2F-"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Cores / Threads</td><td>4 / 4</td></tr><tr><td>Base Frequency</td><td>3.6 GHz</td></tr><tr><td>Burst Frequency</td><td>4.0 GHz</td></tr><tr><td>Cache</td><td>4MB L3 Cache</td></tr><tr><td>TDP</td><td>65W</td></tr></tbody></table>

</span>![](https://www.vortez.net/articles_file/40128_asrock_b450m_pro4_front_box.jpg)### Motherboard

[Asrock B450M Steel Legend](https://www.asrock.com/mb/amd/b450%20steel%20legend/index.asp) - Small form factor mATX board that is being reused from a prototyping project. Being an mATX board it is very limited in expansion.

<span class="vertical-table" id="bkmrk-manufacturer-asrock-"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>**Manufacturer**</td><td>Asrock</td></tr><tr><td>**Model**</td><td>B450M Steel Legend</td></tr><tr><td>**CPU**</td><td>AMD AM4 Socket</td></tr><tr><td>**Chipset**</td><td>AMD Promontory B450</td></tr><tr><td>**Memory**</td><td>- 4x DDR4 DIMM supporting up to 64GB
- Dual channel memory architecture

</td></tr><tr><td>**Display**</td><td>- 1x HDMI 1.4
- 1x DisplayPort 1.2

</td></tr><tr><td>**Networking**</td><td>- 1x Realtek RTL8111H 1GbE LAN

</td></tr><tr><td>**Expansion**</td><td>- 1x PCIe 3.0 x16
- 1x PCIe 2.0 x16
- 1x PCIe 2.0 x1

</td></tr><tr><td>**Storage**</td><td>- 1x M.2 PCIe x4/x2
- 1x M.2 SATA x4/x2
- 4x SATA3

</td></tr><tr><td>**USB**</td><td>- 1x USB 3.1 Gen 2 (Type-C)
- 1x USB 3.1 Gen 2 (Type-A)
- 4x USB 3.1 Gen 1 (Type-A)
- 2x USB 2.0

</td></tr></tbody></table>

</span>### Memory

<span class="vertical-table" id="bkmrk-slot-1-corsair-venge"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Slot 1 ![](https://alfawazit.com/wp-content/uploads/2020/06/CMK16GX4M1A2400C16-Gallery-VENG-LPX-BLK-03.png)</td><td>Corsair Vengeance LPX 8GB DDR4 2666MHz (1x8GB)   
- 2Rx8 Dual Rank
- CAS Latency 16
- timing 16-18-18-35
- 1.2V

</td></tr><tr><td>Slot 2 ![](https://alfawazit.com/wp-content/uploads/2020/06/CMK16GX4M1A2400C16-Gallery-VENG-LPX-BLK-03.png)</td><td>Corsair Vengeance LPX 8GB DDR4 2666MHz (1x8GB)   
- 2Rx8 Dual Rank
- CAS Latency 16
- timing 16-18-18-35
- 1.2V

</td></tr><tr><td>Slot 3 ![](https://alfawazit.com/wp-content/uploads/2020/06/CMK16GX4M1A2400C16-Gallery-VENG-LPX-BLK-03.png)</td><td>Corsair Vengeance LPX 8GB DDR4 2666MHz (1x8GB)   
- 2Rx8 Dual Rank
- CAS Latency 16
- timing 16-18-18-35
- 1.2V

</td></tr><tr><td>Slot 4 ![](https://alfawazit.com/wp-content/uploads/2020/06/CMK16GX4M1A2400C16-Gallery-VENG-LPX-BLK-03.png)</td><td>Corsair Vengeance LPX 8GB DDR4 2666MHz (1x8GB)   
- 2Rx8 Dual Rank
- CAS Latency 16
- timing 16-18-18-35
- 1.2V

</td></tr></tbody></table>

</span>![](https://www.fractal-design.com/wp-content/uploads/2019/06/Define-Mini-C_1.jpg)### Case

[Fractal Design - Define Mini C (Blackout)](https://www.fractal-design.com/products/cases/define/define-mini-c/black/) - A fantastic case with an attractive minimalistic design that in a mATX form factor.

<span class="vertical-table" id="bkmrk-manufacturer-fractal"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>**Manufacturer**</td><td>Fractal Design</td></tr><tr><td>**Model**</td><td>Define Mini C (Blackout)</td></tr><tr><td>**Features**</td><td>- Sound dampening panels
- Excellent build quality
- 2x 3.5" Drive Bays
- 2x 2.5" Drive Bays

</td></tr></tbody></table>

</span>### Storage

<table id="bkmrk-%23-capacity-interface"><thead><tr><th>**\#**</th><th>**Capacity**</th><th>**Interface**</th><th>**Type**</th><th>**Manufacturer &amp; Model**</th><th>**Speed**</th></tr></thead><tbody><tr><td>1x ![](https://shop.westerndigital.com/content/dam/store/en-us/assets/products/internal-storage/wd-black-sn750-nvme-ssd/gallery/without-heatsink/wd-black-sn750-nvme-ssd-Noheatsink2.png.thumb.1280.1280.png)</td><td>256GB</td><td>NVMe</td><td>SSD</td><td>Western Digital Black WDS256G1X0C</td><td>PCIe 3.0 x2</td></tr><tr><td>2x ![](https://ssdcomparison.com/wp-content/uploads/2015/04/Crucial-MX100-Vs-M500.png)</td><td>512GB</td><td>SATA</td><td>SDD</td><td>Crucial MX100</td><td>SATA3 6.0Gb/s</td></tr><tr><td>1x ![](https://cdn.cnetcontent.com/syndication/feeds/wd/inline-content/92/0519E12CA7ABBD8CF2D927E60BC1553957DDFB1C_WDBLUEPCDESKTOPHARDDRIVE1TB_gallery.png)</td><td>1TB</td><td>SATA</td><td>HDD</td><td>Western Digital WD10EZEX</td><td>SATA3 6.0Gb/s</td></tr><tr><td>1x ![](https://5.imimg.com/data5/SELLER/Default/2021/2/TS/GU/XU/58564276/cctv-hard-disk-500x500.png)</td><td>1TB</td><td>SATA</td><td>HDD</td><td>Seagate ST1000DM003</td><td>SATA3 6.0Gb/s</td></tr></tbody></table>

### Cooling

<span class="vertical-table" id="bkmrk-cpu-amd-wraith-spire"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>CPU ![](https://avadirect-freedomusainc1.netdna-ssl.com/Pictures/500/11927135_3.png)</td><td>AMD Wraith Spire</td></tr><tr><td>Case (Front) ![](https://cdn-reichelt.de/bilder/web/xxl_ws/E201/NF-A14-PWM_04.png)</td><td>[Noctua NF-A14 PWM 140mm](https://noctua.at/en/products/fan/nf-a14-pwm)</td></tr><tr><td>Case (front) ![](https://img.gigatron.rs/img/products/large/image5a54b5d0d3252.png)</td><td>[Noctua NF-F12 PWM 120mm](https://noctua.at/en/products/fan/nf-f12-pwm)</td></tr><tr><td>Case (rear) ![](https://img.gigatron.rs/img/products/large/image5a54b5d0d3252.png)</td><td>[Noctua NF-F12 PWM 120mm](https://noctua.at/en/products/fan/nf-f12-pwm)</td></tr></tbody></table>

</span>![](https://images.evga.com/products/gallery//png/220-G2-0550-Y2_XL_4.png)### Power Supply

<span class="vertical-table" id="bkmrk-manufacturer-evga-mo"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Manufacturer</td><td>EVGA</td></tr><tr><td>Model</td><td>[SuperNOVA 550 G2](https://www.evga.com/products/product.aspx?pn=120-GP-0850-X1)</td></tr><tr><td>Features</td><td>- 550W
- fully module

</td></tr></tbody></table>

</span>### UPS

n/a

## Add-On Cards

---

<span class="vertical-table" id="bkmrk-pci-3.0-x16-radeon-r"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>PCI 3.0 x16 ![](https://bookstack.swigg.net/uploads/images/gallery/2021-04/scaled-1680-/gigabyte-radeon560.png)</td><td>[Radeon RX 560 Gaming OC 4G (rev. 1.0)](https://www.gigabyte.com/us/Graphics-Card/GV-RX560GAMING-OC-4GD-rev-10)- PCIe Gen3 x8

</td></tr><tr><td>PCI 3.0 x1 ![](https://bookstack.swigg.net/uploads/images/gallery/2021-04/scaled-1680-/inatek-quad-usb.png)</td><td>[Inateck 4 Ports PCIe to USB 3.0](https://www.inateck.com/inateck-kt4004-4-port-usb-3-0-pci-e-express-card-for-mac-pro.html)- PCIe Gen3 x1

</td></tr><tr><td>PCI 2.0 x16 ![](https://bookstack.swigg.net/uploads/images/gallery/2021-04/scaled-1680-/10gtek-single-sfp.png)</td><td>[10Gtek Intel 82599ES SFP+ PCIe x8](https://www.amazon.com/gp/product/B01LZRSQM9)- PCIe Gen3 x8
- SFP+ 10GbE port
- SR-IOV

</td></tr></tbody></table>

</span>

# Host Configuration



# Base Install

## Operating System
----------------------
[Proxmox Virtual Environment](https://www.proxmox.com/en/) 6.x

## Configuration

<p class="callout info">Proxmox configuration has been transitioned to being automated by an <a href="https://gitlab.swigg.net/dustins/ansible/-/blob/master/proxmox.yml">Ansible Role</a></p>

# Networking (out-of-date)

### Configuration

Because I don't want my main management interface to ever change names, I explicitly give it a name based on its MAC address.

```diff
# /etc/systemd/network/10-management-net.link
+ [Match]
+ MACAddress=70:85:c2:fe:4c:b7
+ 
+ [Link]
+ Name=man0
```

### Bridges

| Master | Bridge | IP Address  | Gateway  | Description |
|--------|--------|-------------|----------|-------------|
| man0   | vmbr0  | 10.0.2.5/21 | 10.0.2.1 | Main Interface (slower Realtek NIC) |
| enp6s0 | --     | --          | --       | Intel 10GbE SFP+ (used for PCI passthrough)

# Common Software

### Install fail2ban

This blocks connections that make repeated failed attempts to authenticate. SSH is covered by default which is what I am interested in, and I'll add additional config to similarly block too many repeated auth failures against the Proxmox web interface.

```bash
apt install fail2ban
```

```diff
# /etc/fail2ban/jail.local
+ [proxmox]
+ enabled = true
+ port = https,http,8006
+ filter = proxmox
+ logpath = /var/log/daemon.log
+ maxretry = 3
+ # 1 hour
+ bantime = 3600
```

### Install sysfsutils
Sysfs is a virtual file system in Linux kernel 2.5+ that provides a tree of system devices. This package provides the program 'systool' to query it: it can list devices by bus, class, and topology.
 
In addition this package ships a configuration file /etc/sysfs.conf which allows one to conveniently set sysfs attributes at system bootup (in the init script etc/init.d/sysfsutils).

```bash
apt install sysfsutils
```


### Install Netdata Monitoring

Install [Netadata](https://www.netdata.cloud/) so that I can get a detailed view of system metrics. It will also be used as a datasource for [LXC / Conception / Prometheus](https://bookstack.swigg.net/books/blackboxhermz/page/lxc-conception-prometheus) so I can look at metrics over a larger timeframe.

```bash
apt update
apt install curl
bash <(curl -Ss https://my-netdata.io/kickstart.sh)
```

# Setup PCI Passthrough

See [PCI Passthrough](https://bookstack.swigg.net/books/how-to-guides/page/pci-passthrough) for more detail as to why I am doing these things.

Proxmox doesn't need a GPU, so blacklist the GPU and prepare it to be passed for a guest machine.

### Enable Kernel Modules

```diff
# /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
+ vfio_pci
+ vfio
+ vfio_iommu_type1
+ vfio_virqfd
```

### Bind `vfio-pci` Driver to Devices

```diff
# /etc/modprobe.d/vfio.conf
+ # AMD Radeon RX 560 [1002:67ff,1002:aae0]
+ alias pci:v00001002d000067FFsv00001458sd000022FFbc03sc00i00 vfio-pci
+ alias pci:v00001002d0000AAE0sv00001458sd0000AAE0bc04sc03i00 vfio-pci
+ 
+ options vfio-pci ids=1002:aae0,1002:67ff disable_vga=1
```

### Rebuild `initramfs`

The `initramfs` needs to be rebuilt to reflect the changes I just did.

```bash
update-initramfs -u
```

### Update Bootloader

Proxmox uses `systemd-boot` as the bootloader so I have to make sure to update the boot entries

#### Update Kernel Parameters

```diff
# /etc/kernel/cmdline
- root=ZFS=rpool/ROOT/pve-1 boot=zfs
+ root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=on video=efifb:off pcie_acs_override=multifunction
```

#### Rebuild Bootloader Options

```bash
pve-efiboot-tool refresh
```

# Storage & Backups (out-of-date)

### Setup ZFS Scrub (Data Integrity)

Automate [ZFS scrubbing](https://wiki.archlinux.org/index.php/ZFS#Scrubbing) so the data integrity on disks is actively monitored, repaired if necessary, and I'm alerted if there is a problem with my disks.

#### Create systemd Service/Timer ([source](https://github.com/lnicola/systemd-zpool-scrub))

Create a simple systemd servcie template for scrubbing ZFS pools.

```diff
# /etc/systemd/system/zpool-scrub@.service
+ [Unit]
+ Description=Scrub ZFS Pool
+ Requires=zfs.target
+ After=zfs.target
+
+ [Service]
+ Type=oneshot
+ ExecStartPre=-/usr/sbin/zpool scrub -s %I
+ ExecStart=/usr/sbin/zpool scrub %I
```

Then create a systemd timer template for periodically running that service. I am running the scrub weekly, but semi-monthly or monthly would almost certainly be ok too.

```diff
# /etc/systemd/system/zpool-scrub@.timer
+ [Unit]
+ Description=Scrub ZFS pool weekly
+
+ [Timer]
+ OnCalendar=weekly
+ Persistent=true
+
+ [Install]
+ WantedBy=timers.target
```

#### Enable ZFS Scrub

```bash
systemctl daemon-reload
systemctl enable --now zpool-scrub@rpool.timer
```

### Setup Sanoid/Syncoid (Data Backup)

Run [Sanoid](https://github.com/jimsalterjrs/sanoid) for automating snapshots and Syncoid for remote backups. Unfortunately this isn't available in repositories so you have to build it yourself. However the author makes it fairly simple.

#### Install ([source](https://github.com/jimsalterjrs/sanoid))

```bash
apt-get install build-essential debhelper dpkg-buildpackage libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuffer
sudo git clone https://github.com/jimsalterjrs/sanoid.git
cd sanoid
ln -s packages/debian .
dpkg-buildpackage -uc -us
apt install ../sanoid_*_all.deb
```

#### Configure Sanoid

I want to take hourly snapshots of both of my ZFS pools because sometimes I am not as careful or thoughtful as I should be about what I am doing at any given moment.

```diff
# /etc/sanoid/sanoid.conf
+ [template_proxmox]
+         frequently = 0
+         hourly = 24
+         daily = 7
+         weekly = 4
+         monthly = 1
+         yearly = 0
+         autosnap = yes
+         autoprune = yes
+ 
+ [rpool]
+         use_template = template_proxmox
+         process_children_only = yes
+         recursive = yes
+ 
+ [rpool/ROOT]
+         use_template = rpool           
+         process_children_only = yes
+         recursive = yes
+ 
+ [rpool/data]
+         use_template = template_proxmox
+         weekly = 1
+         monthly = 1
+         process_children_only = yes
+         recursive = yes
```

Maybe this is a sin, but I'd like my snapshots to be in local time so I don't have to do the (admittedly simple) conversion in my head.

```diff
# /usr/lib/systemd/system/sanoid.service
  [Service]
- Environment=TZ=UTC
+ Environment=TZ=EST
```

#### Configure Syncoid

I haven't decided where I want to replicate to yet so I haven't configured syncoid yet.

# VM / macOSAMD

## Description

This VM is for running macOS via dedicated hardware so I have something faster than my laptop.

## Configuration

### Resources

<table id="bkmrk-hostname-cpu-memory-"><thead><tr><th>Hostname</th><th>CPU</th><th>Memory</th></tr></thead><tbody><tr><td>MiMac</td><td>12 vCPU</td><td>16GB</td></tr></tbody></table>

### Storage

<table id="bkmrk-disk-controller-size"><thead><tr><th>Disk</th><th>Controller</th><th>Size</th><th>Purpose</th></tr></thead><tbody><tr><td>local-zfs:vm-100-disk-1</td><td>ide0</td><td>1M</td><td>NVRAM</td></tr><tr><td>local-zfs:vm-100-disk-0</td><td>virtio</td><td>200MB</td><td>EFI boot loader</td></tr><tr><td>local-zfs:vm-100-disk-2</td><td>virtio</td><td>1TB</td><td>boot disk</td></tr></tbody></table>

### PCI Passthrough

<table id="bkmrk-name-bdf-settings-fr"><thead><tr><th>Name</th><th>BDF</th><th>Settings</th></tr></thead><tbody><tr><td>Fresco USB 3.0 Controller</td><td>04:00</td><td>n/a</td></tr><tr><td>AMD RX 560 GPU</td><td>08:00</td><td>pcie=1,x-vga=1</td></tr><tr><td>Intel 10GbE SFP+ NIC</td><td>06:00</td><td>n/a</td></tr></tbody></table>

### Networking

#### Interfaces

<table id="bkmrk-id-name-bridge-ip-ad"><thead><tr><th>ID</th><th>Name</th><th>Bridge</th><th>IP Address</th><th>Comments</th></tr></thead><tbody><tr><td>eno0</td><td>n/a</td><td>vmbr0</td><td>10.0.2.5/21 (DHCP)</td><td>1GbE</td></tr><tr><td>eno1</td><td>n/a</td><td>--</td><td>10.0.10.5/24 (manual)</td><td>10GbE</td></tr></tbody></table>

#### Configuration

Because we want macOS to route all traffic destined for [blackbox.hermz](https://bookstack.swigg.net/books/blackboxhermz) over our 10GbE network we need to adjust our routing table to redirect all traffic to [LXC / Routeman](https://bookstack.swigg.net/books/blackboxhermz/page/lxc-routeman) first.

Knowing that blackbox.hermz has ip addresses `10.0.2.2`, and `10.0.2.3` and all of its services run with `10.0.4.x` we can easily setup some updated routes.

```bash
# 10.0.2.2 (main interface) and 10.0.2.3 (admin interface)
ip route add 10.0.2.2/31 dev en1 via 10.0.10.6

# 10.0.4.x (services running on blackbox)
ip route add 10.0.4.0/24 dev en1 via 10.0.10.6

```

This works perfectly except it isn't persistent.