# Project Router

# Introduction: Novice to Network Admin

## Introduction

If you're just looking to get started reading about how I setup everything you can skip down to the [Goals](#goals) section or go straight to [Guest Setup](https://bookstack.swigg.net/books/project-router/page/guest-setup) to get started.

## Background

From when I first started using computers as a kid I treated all things related to networking as a black box. I had a rudimentary understanding of IP addresses but had no real idea how data got from my computer to a server other than the high level concept of *"my computer is sending data to that address"*. It is similar to how most of us don't really know how the US Postal Service works. We have a vague notion that we drop a letter in a mailbox to be picked up and then *"my letter gets delivered to the address on the envelope."* In reality how mail gets picked up, sorted, tracked, bundled, routed, and delivered is much more involved than we ever think about. Up until about three years ago my knowledge hadn't progressed much past knowing [network packets](https://en.wikipedia.org/wiki/Network_packet) existed and for a computer to get on the Internet it had to have an IP address, a subnet mask (not that I really understood what this was), and have a router address.

Like most computer users I never delved into networking. I'd plug my router (always an [Apple AirPort](https://en.wikipedia.org/wiki/AirPort) into the modem and go with mostly the defaults. I used *Linux* for years but strictly as a server for web apps I programmed and never ventured very far off that path.

That all started to change in 2018 when I decided to build a server (really just a fast PC at the time) to play around with. It also coincided with me starting a new job that put me adjacent to some networking topics that started to peak my interest. Pretty soon I was self hosting a several applications accessible from the Internet and wanted to take the plunge into setting up a [DMZ](https://en.wikipedia.org/wiki/DMZ_%28computing%29) to try and keep my local network safe from the ever increasing number of things I was making publically accessible. Looking into how best to cordon off my applications I saw that putting public applications on a different subnet and often a [VLAN](https://en.wikipedia.org/wiki/Virtual_LAN) is the best practice. But my AirPort didn't support that so I went searching for a router/firewall that would work better. The general consensus at the time seemed to be to use a [Unifi Security Gateway](https://www.ui.com/unifi-routing/usg/) or run [pfSense](https://www.pfsense.org/) on an old PC. Not having a PC to run it on I first tried virtualizing my router by running it as a virtual machine on my "server." For a newbie this was more complicated than necessary and had the risk that if not done correctly could expose my internal network to the Internet. I liked *pfSense* but I quickly found out the downside of running your gateway to the Internet on your server is that when your server has a problem you likely won't have the Internet to fix it which is quite frustrating.

So I bought a mini PC ([Protecli Vault](https://protectli.com/vault-2-port/)) and started running pfSense on there and was happy for a year. I continued to self-host more applications and eventually got to self-hosting DNS with [Pi-hole](https://pi-hole.net/). Pretty soon I ran into my old problem of when the server is down there is no DNS and so the Internet pretty much stops working. I briefly considered buying buyin a [Raspberry Pi](https://www.raspberrypi.org/) which would have been a great solution but I decided to treat the mini PC as my "network infrastructure server" and instead of just running pfSense on there I'd use [Proxmox VE](https://www.proxmox.com/en/proxmox-ve) and virtualize both *pfSense* and *Pi-hole* the same way I had been virtualizing Pi-hole on my server.

This worked great except I eventually noticed that my maximum download speeds were slower than they had been when only *pfSense* was running on the min PC. After a bunch of testing and [attempted workarounds](https://bookstack.swigg.net/books/linux/page/bridge-zero-copy-transmit) I realized I <s>needed</s> *wanted* a new plan. So once again I bought a [mini PC](https://bookstack.swigg.net/books/edge/page/physical-hardware) that was pretty much just a more powerful version of the one I already had to try and fix the problem. I was disappointed to see that my upgrade helped but wasn't enough to overcome the overhead that came along with virutalizing *pfSense*. Additionally I was starting to push into things that *pfSense* didn't support yet like using [Wireguard](https://www.wireguard.com/).

The next step was to investigate if a *Linux* based firewall would perform better while virtualized. The answer turns out to be no since both *pfSense* and *Linux* both implement *Virtio* network drivers that work very similarly and the real problem seems to just be the result of the additional layers a packet has to travel up through. Each packet coming in must be processed by the hypervisor kernel then redirected to the virtualized router kernel and if the packet was destined somewhere else had to go through the same layers in reverse to exit. That seems to be jus too much for a device with modest CPU and memory performance.

Then it dawned on me that I could get around those layers of virtualization by using the same containerization I had been using for servers I had virtualized to run all my self-hosted projects. I prefer to run my guest machines as [LXC (Linux Container)](https://en.wikipedia.org/wiki/LXC) guests instead of virtual machines. A *LXC* guest uses the same *Linux* [Kernel](https://en.wikipedia.org/wiki/Kernel_(computing)) as the host Operating System so there are no additional layers of virtualization overhead to deal with. Pretty neat!

When I was looking at *Linux* based firewalls I came across [VyOS](https://vyos.io) which is based on [Debian](https://www.debian.org/) Linux and allowed me to peek behind the curtain of the types of tools you use for a *Linux* firewall. The seed of an idea had been planted that would use all the knowledge I gained over the last 3 years tinkering with my home network-- *Linux*, *Proxmox*, *Wireguard*, and virtualization.

I could run a *LXC* guest with *Debian* that would have zero virtualization overhead and provide all the functionality I needed. Best of all I'd be able to customize the firewall/router because it is just a *Linux* machine!

So I made a list of features I had been using on *pfSense* and *VyOS* to see what I'd have to implement.

## Goals

- Basic *Linux* install in *LXC* guest
- Firewall protection of my local network from the Internet
- VLAN separation for added isolation between my subnets and the Internet
- [DHCP](https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol) to provide each subnet with IPv4 address assignment and local DNS resolution
- [Recursive DNS](https://en.wikipedia.org/wiki/Public_recursive_name_server) for added security, privacy and removing reliance on external entities
- IPv6 stack support ([DHCPv6](https://en.wikipedia.org/wiki/Public_recursive_name_server), [Router Advertisements (NDP)](https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol), [Prefix Delegation](https://en.wikipedia.org/wiki/Prefix_delegation))
- *Wireguard* support

## Extras

- Create bonded Ethernet interface to remove bandwidth bottleneck between router and switch
- Add intrusion prevension system like [Snort](https://www.snort.org/)
- Add bandwidth monitor and graphing

# LXC Guest Setup

As I discussed in [Introduction: Novice to Network Admin](https://bookstack.swigg.net/books/project-router/page/introduction-novice-to-network-admin) the goal is to run a router/firewall inside a *LXC* guest so there is little to no overhead when routing packets. So I created an **unpriviledged** *LXC* guest with a *Debian 10* template.

## Resources

The [mini PC](https://bookstack.swigg.net/books/edge) this will run on isn't a powerhouse but should provide more than enough resources and have a few spare cycles leftover for something useful like running *Pi-hole*.

<span class="vertical-table" id="bkmrk-cores-unlimited-%284-c"><table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Cores</td><td>*unlimited* (4 cores)</td></tr><tr><td>Memory</td><td>2048 MiB</td></tr><tr><td>Swap</td><td>512 MiB</td></tr><tr><td>Root Disk</td><td>2 GB</td></tr></tbody></table>

</span>## Networking

<table id="bkmrk-device-id-name-physi"><thead><tr><th>Device</th><th>ID</th><th>Name</th></tr></thead><tbody><tr><td>physical</td><td>net0</td><td>eth0</td></tr><tr><td>virtio</td><td>net1</td><td>eth1</td></tr></tbody></table>

Because there is some overhead with using an [Ethernet Bridge](https://en.wikipedia.org/wiki/Bridging_(networking)) I only wanted to use one where it made the most sense. Since the Ethernet connection from the modem will only ever talk directly to this *LXC* guest I am "passing" one of the physical Ethernet interfaces from *Proxmox* to this *LXC* guest. This makes it unavailable to the host and allows the *LXC* guest direct access to it similar to how [PCI(e) Passthrough](https://pve.proxmox.com/wiki/PCI(e)_Passthrough) would work on a virtual machine.

This can be accomplished with *Proxmox*/*LXC* configuration similar to what is shown below.

```diff
# /etc/pve/lxc/100.conf
net1: name=eth1,bridge=vmbr0,hwaddr=D6:A9:67:D5:66:22,type=veth
+ lxc.net.0.type: phys
+ lxc.net.0.link: enp1s0
+ lxc.net.0.name: eth0

```

<p class="callout info">Be careful to not reuse the same index for `lxc.net.\[index\]` and `net\[index\]` values or the guest will fail to boot.</p>

## Operating System

I didn't have to do much to the system itself other than making sure the timezone was correct and that it was up to date.

```bash
$ dpkg-reconfigure tzdata
$ apt update
$ apt upgrade

```

# Initial Network Setup

## Configure Interfaces

I need Internet access to download all the packages necessary so I setup *DHCP* on the WAN connection `eth0`.

<p class="callout info">Setting all the local network interfaces to `manual` and not providing any addresses prevents any accidental routing before everything is secured.</p>

```diff
# /etc/network/interfaces
    auto eth0
    iface eth0 inet dhcp
+     
+   auto eth1
+     iface eth1 inet manual
+     
+    auto eth1.8
+    iface eth1.8 inet manual
+            vlan-raw-device eth1
+            
+    auto eth1.9
+    iface eth1.9 inet manual
+            vlan-raw-device eth1

```

Then I restart the networking service to apply the changes and create the new interfaces.

```bash
$ systemctl restart networking

```

## Initial Security

I won't have SSH access allowed from the Internet when I am done but in the interim I want to install `fail2ban`. It doesn't hurt to have running even once the firewall is fully setup and provides just one more layer of defense.

```bash
$ apt install fail2ban

```

# DNS: Recursive DNS

## Option 1: Unbound

```bash
$ apt install unbound

```

Recursive DNS can sometimes sacrifice speed for security so the `unbound` server is going to be limited to only serve DNS requests from loopback addresses. Everyone else will have to go through a DNS caching server (*dnsmasq*) that I'll setup later to perform DNS queries.

```diff
# /etc/unbound/unbound.conf.d/local.conf
+ server:
+ 	interface: 127.0.0.1
+ 	interface: ::1
+ 	access-control: 127.0.0.1 allow
+ 	access-control: ::1 allow

```

```bash
$ systemctl restart unbound

```

## Option 2: Public Recursive Name Server

I don't have to do anything since [dnsmasq](https://en.wikipedia.org/wiki/Dnsmasq) will be setup to query a public recursive DNS server like *Cloudflare*'s `1.1.1.1` and `1.0.0.1`.

# Logging in LXC

## Logging

One problem I ran into is that access to kernel logging is limited or unavailable from inside of a LXC container. For some usecases (like *netfilter*'s `LOG` action) any logging that happens in a LXC container will be blackholed and not recorded anywhere without [a change](https://bookstack.swigg.net/books/linux/page/netfilteriptable-logging) on the host. Most often the solution to these permission/security problems is to find a way to allow access to these things from userspace.

### ulogd2

I solved the *netfilter* `LOG` problem by simply using *ulogd2* to replace kernel logging with userspace logging. After installing and configuring *ulogd2* I just replaced any references to `LOG` with `NFLOG` in my *netfilter/iptables* rules. Don't worry if this doesn't make sense right now I'll talk about this more in the [Firewall Setup](https://bookstack.swigg.net/books/project-router/page/firewall-setup) section.

> ulogd is a userspace logging daemon for netfilter/iptables related logging. This includes per-packet logging of security violations, per-packet logging for accounting, per-flow logging and flexible user-defined accounting.

#### Installation

```bash
apt install ulogd2

```

#### Configuration

To get the output I wanted I had to edit the *ulogd2* config…

```diff
# /etc/ulogd2.conf
- stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU
+ #stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU
...
+ stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,firewall:LOGEMU
+ stack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,firewall:LOGEMU
+ stack=log3:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,firewall:LOGEMU
+ 
+ [firewall]
+ file="/var/log/ulog/firewall.log"
+ sync=1

```

## Connection Tracking

Similarly to *netfilter* logging connection tracking in a LXC container is more limited due to not having access to the host's `/proc/` filesystem. But I can install *conntrack* to provide a way to see connection tracking from userspace.

### conntrack

> The conntrack utilty provides a full featured userspace interface to the Netfilter connection tracking system that is intended to replace the old /proc/net/ip\_conntrack interface. This tool can be used to search, list, inspect and maintain the connection tracking subsystem of the Linux kernel.

#### Installation

```bash
apt install conntrack

```

# IPv4



# Firewall Setup

## Install Shorewall

To manage `nftables/iptables` I decided to go with [Shorewall](https://shorewall.org/) since it is easy to configure and very mature. At some point I may look into switching to [FireHol](https://firehol.org/) since it looks even simpler to configure but I wanted something I knew I'd be able to make do everything I needed.

I started by installing _shorewall_ as my firewall, _shorewall-doc_ which includes examples, and _shorewall-init_ which can lockdown the system at boot before _Shorewall_ has had a chance to configure the firewall.

```bash
# apt install shorewall shorewall-doc shorewall-init
```

Then I update the _shorewall_ configuration to reflect that I'm using _ulogd2_ for logging and that I want IPv4 forwarding enabled when _shorewall_ starts.

```diff
# /etc/shorewall/shorewall.conf
- LOG_LEVEL="info"
+ LOG_LEVEL="NFLOG(1,0,1)"
...
- LOGFILE=/var/log/messages
+ LOGFILE=/var/log/firewall.log
...
- IP_FORWARDING=Keep
+ IP_FORWARDING=Yes
```

All my configuration files are adapted from the examples that _shorewall-doc_ makes available under `/usr/share/doc/shorewall/examples`.

Setting up the zones is pretty self-explanitory. The only addition I made is I have a `warp` zone which I will use later when I am setting up my VPN.

```diff
# /etc/shorewall/zones
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-zones"
+ #
+ # See http://shorewall.net/manpages/shorewall-zones.html for more information
+ ###############################################################################
+ #ZONE   TYPE    OPTIONS                 IN                      OUT
+ #                                       OPTIONS                 OPTIONS
+ fw      firewall
+ wan     ipv4
+ lan     ipv4
+ dmz     ipv4
+ warp    ipv4
```

Setting up the interfaces and assiging them zones is also pretty self-explanatory.

```diff
# /etc/shorewall/interfaces
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-interfaces"
+ #
+ # See http://shorewall.net/manpages/shorewall-interfaces.html for more information
+ ###############################################################################
+ ?FORMAT 2
+ ###############################################################################
+ #ZONE		INTERFACE       OPTIONS
+ wan		WAN_IF			tcpflags,dhcp,nosmurfs,routefilter,logmartians,sourceroute=0,physical=eth0
+ lan		LAN_IF			tcpflags,dhcp,nosmurfs,routefilter,logmartians,physical=eth1
+ dmz		DMZ_IF			tcpflags,dhcp,nosmurfs,routefilter,logmartians,physical=eth1.8
+ warp		WARP_IF			tcpflags,dhcp,nosmurfs,routefilter,logmartians,physical=eth1.9
```

My real `/etc/shorewall/policy` file is less liberal than what is shown below (`lan` being allowed to access whatever it wants) but I wanted to show a reasonably secure policy that allowed me to have a very simple `/etc/shorewall/rules` config below.

```diff
# /etc/shorewall/policy
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-policy"
+ #
+ # See http://shorewall.net/manpages/shorewall-policy.html for more information
+ ###############################################################################
+ #SOURCE	DEST		POLICY		LOGLEVEL	RATE    CONNLIMIT
+ 
+ $FW		all			ACCEPT
+ lan		all			ACCEPT
+ dmz		$FW,wan		ACCEPT
+ warp		$FW			ACCEPT
+ 
+ wan		all			DROP		$LOG_LEVEL
+ # THE FOLLOWING POLICY MUST BE LAST
+ all		all			REJECT		$LOG_LEVEL
```

Because my example policy is pretty open, my rules in this example are pretty sparse.

```diff
# /etc/shorewall/rules
+ #------------------------------------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-rules"
+ #
+ # See http://shorewall.net/manpages/shorewall-rules.html for more information
+ ######################################################################################################################################################################################################
+ #ACTION         SOURCE          DEST            PROTO   DEST    SOURCE          ORIGINAL        RATE          USER/    MARK    CONNLIMIT       TIME            HEADERS         SWITCH          HELPER
+ #                                                       PORT    PORT(S)         DEST            LIMIT         GROUP
+ ?SECTION ALL
+ ?SECTION ESTABLISHED
+ ?SECTION RELATED
+ ?SECTION INVALID
+ ?SECTION UNTRACKED
+ ?SECTION NEW
+ 
+ #       Don't allow connection pickup from the net
+ Invalid(DROP)   wan             all             tcp
+ 
+ DNS(ACCEPT)     all!wan,warp    $FW
+ DNS(ACCEPT)     $FW,dmz         lan:10.0.1.2
+ 
+ Web(ACCEPT)     dmz             $FW
+ Web(DNAT)       wan             dmz:10.0.8.2
```

Lastly is the magic that allows private addresses to access the Internet by masquerading them all as my one public IPv4 address I am assigned. The following just says all traffic heading out of `WAN_IF` (`eth0`) coming from a private IP range should be [masqueraded](https://en.wikipedia.org/wiki/Network_address_translation).

```diff
# /etc/shorewall/snat
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-snat"
+ #
+ # See http://shorewall.net/manpages/shorewall-snat.html for more information
+ ###########################################################################################################################################
+ #ACTION                 SOURCE                  DEST            PROTO   PORT    IPSEC   MARK    USER    SWITCHORIGDEST PROBABILITY
+ MASQUERADE              10.0.0.0/8,\
+                         169.254.0.0/16,\
+                         172.16.0.0/12,\
+                         192.168.0.0/16          WAN_IF
```

Now that I have everything configured it might be wise to run `shorewall check` just to make sure I didn't have any typos.

I hooked _shorewall_ into the boot process to make sure the system is secure during boot by enabling _shorewall-init.service_ and _shorewall.service_. First I told _shorewall-init_ that it needs to account for _shorewall_ when it runs.

```diff
# /etc/default/shorewall-init
- PRODUCTS=""
+ PRODUCTS="shorewall"
```

Then I simply told those services to start at boot.

```bash
# systemctl enable shorewall
# systemctl enable shorewall-init
```

## Modify Interfaces

Now that _Shorewall_ will secure everything at bootup it is safe to update `/etc/networking/interfaces` and add their IPv4 addresses.

```diff
# /etc/networking/interfaces
 auto eth1
- iface eth1 inet manual
+ iface eth1 inet static
+         address 10.0.1.1/21
  
  auto eth1.8
- iface eth1.8 inet manual
+ iface eth1.8 inet static
          vlan-raw-device eth1
+         address 10.0.8.1/24
  
  auto eth1.9
- iface eth1.9 inet manual
+ iface eth1.9 inet static
          vlan-raw-device eth1
+         address 10.0.9.1/24
```

Now if I reboot the system all my interfaces will come up configured and the system will be protected by _nftables/iptables_ configured by _Shorewall_.

<p class="callout info">Be sure to sanity check the configuration so Shorewall doesn't block SSH access if that is needed.</p>

```bash
# reboot
```

# DHCP and DNS Cache

## Install dnsmasq

I decided to use [dnsmasq](https://thekelleys.org.uk/dnsmasq/doc.html) since it can fulfull multiple roles as both a DHCP and DNS cache. I'll first configure it for IPv4 and then later add in the few extra IPv6 lines needed.

### Setup DHCP

The following can look complicated but that is just becuase there are a ton of [MAC Addresses](https://en.wikipedia.org/wiki/MAC_address) and [IP Addresses](https://en.wikipedia.org/wiki/IP_address) mixed throughout. If you look closely you can see that there are only four types of lines.

1. `no-dhcp-interface=eth0,lo` prevents DHCP binding on our loopback address and `eth0` which is the interface facing the Internet.

2. `dhcp-range=` declares a start and stop address and lease lifetime for each subnet. I am also setting an optional tag for each so I can target them later if I want.

3. `dhcp-option=` allows me to set specific DHCP options. The `tag:` allows me to target addresses matching a specific tag. I am overriding the default DNS servers because I want `lan` and `dmz` to use my _Pi-hole_ server and `warp` should use a public DNS server since any device on that subnet is routed through a VPN tunnel so it doesn't have local network access.

4. `dhcp-host=` defines what IP addresses and hostnames get assigned to which network device with a specific MAC address

```diff
# /etc/dnsmasq.d/dhcp.conf
+ no-dhcp-interface=eth0,lo
+ 
+ dhcp-range=set:lan,10.0.5.1,10.0.7.254,12h
+ dhcp-range=set:dmz,10.0.8.1,10.0.8.254,12h
+ dhcp-range=set:warp,10.0.9.1,10.0.9.254,5m
+ 
+ dhcp-option=tag:lan,option:dns-server,10.0.1.2
+ dhcp-option=tag:lan,option:dns-server,10.0.1.2
+ dhcp-option=tag:warp,option:dns-server,1.1.1.1,1.0.0.1
+ 
+ # LAN - network infrastructure
+ dhcp-host=aa:af:57:f3:4e:90,10.0.1.2,pihole			# pihole
+ dhcp-host=b4:fb:e4:8f:f9:74,10.0.1.3,unifi-switch-8	# unifi-switch-8
+ 
+ # LAN  - proxmox
+ dhcp-host=e0:d5:5e:63:fe:30,10.0.3.2,blackbox			# blackbox
+ dhcp-host=70:85:c2:fe:4c:b7,10.0.3.3,mini				# mini
+ dhcp-host=6e:91:84:4a:74:f1,10.0.3.4,backup			# backup
+ 
+ # LAN - assigned devices
+ dhcp-host=d0:a6:37:ed:8c:7f,10.0.4.4,silverbook		# silverbook
+ dhcp-host=82:13:00:9c:c7:00,10.0.4.5,thunderbolt		# thunderbolt
+ dhcp-host=34:36:3b:7f:18:1e,10.0.4.8,jess				# jess
+ dhcp-host=96:64:5F:1C:A6:2C,10.0.5.6,refuge			# refuge
+ dhcp-host=7A:BC:46:D1:A3:1B,10.0.5.9,unifi			# unifi
+ 
+ # DMZ - assigned devices
+ dhcp-host=62:59:92:A7:1D:F1,10.0.8.5,bitcoin			# bitcoin
+ dhcp-host=32:cc:fb:a3:1a:57,10.0.8.2,contained		# contained
```

### Setup DNS Caching

Everything here is commented with an explanation of what it does. The only thing slightly interesting is I have two `server=` parameters pointing to the IPv4 loopback addresses which is where _Unbound_ is listening. If _Unbound_ wasn't being used I'd either remove `no-resolv` and use the system namesevers or change the `server=` parameters to point to a [public recursive name sever](https://en.wikipedia.org/wiki/Public_recursive_name_server).

```diff
# /etc/dnsmasq.d/dns.conf
+ # Add the domain to simple names (without a period) in /etc/hosts in the same way as for DHCP-derived names.
+ expand-hosts
+ 
+ # Log the results of DNS queries handled by dnsmasq.
+ log-queries
+ 
+ # Do not listen on the specified interface.
+ except-interface=eth0,lo
+ 
+ # Accept DNS queries only from hosts whose address is on a local subnet, ie a subnet for which an interface exists on the server.
+ local-service
+ 
+ # Dnsmasq binds the address of individual interfaces, allowing multiple dnsmasq instances, but if new interfaces or addresses appear, it automatically listens on those
+ bind-dynamic
+ 
+ # Return answers to DNS queries from /etc/hosts and --interface-name which depend on the interface over which the query was received.
+ localise-queries
+ 
+ # All reverse lookups for private IP ranges (ie 192.168.x.x, etc) which are not found in /etc/hosts or the DHCP leases file are answered with "no such domain"
+ bogus-priv
+ 
+ # Later versions of windows make periodic DNS requests which don't get sensible answers from the public DNS and can cause problems by triggering dial-on-demand links.
+ filterwin2k
+ 
+ # Enable code to detect DNS forwarding loops
+ dns-loop-detect
+ 
+ # Reject (and log) addresses from upstream nameservers which are in the private ranges.
+ stop-dns-rebind
+ 
+ # Exempt 127.0.0.0/8 and ::1 from rebinding checks.
+ rebind-localhost-ok
+ 
+ # Tells dnsmasq to never forward A or AAAA queries for plain names, without dots or domain parts, to upstream nameservers.
+ domain-needed
+ 
+ # Specifies DNS domains for the DHCP server.
+ domain=hermz.io
+ 
+ # Don't read /etc/resolv.conf. Get upstream servers only from the command line or the dnsmasq configuration file.
+ no-resolv
+
+ server=127.0.0.1
+ server=::1
```

### Resolve Static Clients

One problem I ran into was that static clients never use DHCP so the DHCP server doesn't register their hostname with their intended IP address. To work around this limitation I just added those entries to the `/etc/hosts` file since by default _dnsmasq_ will resolve using those entries too.

```diff
# /etc/hosts
+ 10.0.1.1        ember
+ 10.0.1.2        pihol
+ 10.0.1.3        unifi-switch-8
+ 10.0.3.2        blackbox
+ 10.0.3.3        mini
+ 10.0.3.4        backup
+ 10.0.3.5        edge
 
  # --- BEGIN PVE ---
```

### Reboot

Now that _dnsmasq_ is fully configured I just restart it using _systemctl_

```bash
# systemctl restart dnsmasq.service
```

# IPv6



# IPv6 Intro

## Refresher

For a quick crash course into IPv6 checkout my [IPv6 Quick Explainer](https://bookstack.swigg.net/books/guides/page/ipv6-quick-explainer) guide.

## Why Did I Setup IPv6?

Beyond just being good to know because it'll be what we're all using sooner than later there are a few practical advantages of IPv6 over IPv4. Most important to me though is being able to have IP addresses that don't have to be masqueraded by the router. This has several knock-on effects I appreciate.

### No Need for Hairpin NAT

I don't have to masquerade IP addresses which means that when I access a device from my LAN I can use the same IP address that is used when people access a device from the WAN. I don't have to setup a hacky [Hairpin NAT](https://wiki.mikrotik.com/wiki/Hairpin_NAT) or necessarily use [Split-horizen DNS](https://en.wikipedia.org/wiki/Split-horizon_DNS) to just have everything work. The less janky configurations I have to create and maintain to paper over problems of IPv4 the better.

### Fine-grained DNS Control

Because each device can have a publically routable address I can setup subdomains to **actually** point to different addresses. As an example I can have `wireguard.swigg.net` point to my router IP address for VPN access while `*.swigg.net` can point to my server IP address I am running in my DMZ. With IPv4 I had to have them both point to my router public IP address and then use some sort of proxy to forward based on hostname plus do something janky like above.

# Firewall Setup

## Install Shorewall6

Configuring _Shorewall_ for IPv6 is nearly identical to how I did it for IPv4. The biggest different is I can skip most things related to _masquerading_ since that is less often necessary in the world of IPv6.

The only changes that need to be made is installing and configuring _shorewall6_. I am not going to go over everything again since it is nearly identical to [Firewall Setup](https://bookstack.swigg.net/books/project-router-work-in-progress/page/firewall-setup) under IPv4 but pay close attention to the path is now `/etc/shorewall6`

```bash
# apt install shorewall6
```

```diff
# /etc/shorewall6/shorewall.conf
- LOG_LEVEL="info"
+ LOG_LEVEL="NFLOG(1,0,1)"
...
- LOGFILE=/var/log/messages
+ LOGFILE=/var/log/firewall.log
...
- IP_FORWARDING=Keep
+ IP_FORWARDING=Yes
```

```diff
# /etc/shorewall6/zones
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-zones"
+ #
+ # See http://shorewall.org/manpages/shorewall-zones.html for more information
+ ###############################################################################
+ #ZONE   TYPE    OPTIONS                 IN                      OUT
+ #                                       OPTIONS                 OPTIONS
+ fw      firewall
+ wan     ipv4
+ lan     ipv4
+ dmz     ipv4
+ warp    ipv4
```

```diff
# /etc/shorewall6/interfaces
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall6-interfaces"
+ #
+ # See http://shorewall.org/manpages/shorewall-interfaces.html for more information
+ ###############################################################################
+ ?FORMAT 2
+ ###############################################################################
+ #ZONE		INTERFACE	OPTIONS
+ wan		WAN_IF		tcpflags,dhcp,forward=1,accept_ra=2,sourceroute=0,physical=eth0
+ lan		LAN_IF		tcpflags,dhcp,forward=1,physical=eth1
+ dmz		DMZ_IF		tcpflags,dhcp,forward=1,physical=eth1.8
+ warp		WARP_IF		tcpflags,dhcp,forward=1,physical=eth1.9
```

```diff
# /etc/shorewall6/policy
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-policy"
+ #
+ # See http://shorewall.net/manpages/shorewall-policy.html for more information
+ ###############################################################################
+ #SOURCE	DEST		POLICY		LOGLEVEL	RATE    CONNLIMIT
+ 
+ $FW		all			ACCEPT
+ lan		all			ACCEPT
+ dmz		$FW,wan		ACCEPT
+ warp		$FW			ACCEPT
+ 
+ wan		all			DROP		$LOG_LEVEL
+ # THE FOLLOWING POLICY MUST BE LAST
+ all		all			REJECT		$LOG_LEVEL
```

```diff
# /etc/shorewall6/rules
+ #------------------------------------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-rules"
+ #
+ # See http://shorewall.net/manpages/shorewall-rules.html for more information
+ ######################################################################################################################################################################################################
+ #ACTION			SOURCE          DEST            PROTO   DEST    SOURCE          ORIGINAL        RATE          USER/    MARK    CONNLIMIT       TIME            HEADERS         SWITCH          HELPER
+ #                                                       PORT    PORT(S)         DEST            LIMIT         GROUP
+ ?SECTION ALL
+ ?SECTION ESTABLISHED
+ ?SECTION RELATED
+ ?SECTION INVALID
+ ?SECTION UNTRACKED
+ ?SECTION NEW
+ 
+ #       Don't allow connection pickup from the net
+ Invalid(DROP)		wan             all             tcp
+ 
+ DNS(ACCEPT)		all!wan,warp    $FW
+ DNS(ACCEPT)		$FW,dmz         lan:2001:db8:2fa3:4848::9a57:cec2
+ 
+ Web(ACCEPT)		dmz             $FW
+ Web(ACCEPT)		wan             dmz:2001:db8:2fa3:4848:66:1cb:59a7:bbe1
```

At this point I just have an empty `/etc/shorewall6/snat` configuration because IPv6 doesn't need masqueraded to access the Internet.

```diff
# /etc/shorewall/snat
+ #------------------------------------------------------------------------------
+ # For information about entries in this file, type "man shorewall-snat"
+ #
+ # See http://shorewall.org/manpages/shorewall-snat.html for more information
+ ###########################################################################################################################################
+ #ACTION                 SOURCE                  DEST            PROTO   PORT    IPSEC   MARK    USER    SWITCH  ORIGDEST        PROBABILITY
```

Just like before it might be wise to run `shorewall6 check` just to make sure I didn't have any typos.

I already enabled _shorewall-init.service_ to secure the system during boot so to hook in _shorewall6_ I just needed to edit its configuration and then enable _shorewall6.service_ to start at boot like I already did for _shorewall.service_ and _shorwall-init.service_.

```diff
# /etc/default/shorewall-init
- PRODUCTS="shorewall"
+ PRODUCTS="shorewall shorewall6"
```

Then I told it to start at boot.

```bash
# systemctl enable shorewall6
```

## Reboot

It isn't strictly neccessary to reboot but I just prefer to see my system as it would be after it starts up.

```bash
# reboot
```

# Prefix Delegation

I'd recommend reading about [Prefix Delegation](https://en.wikipedia.org/wiki/Prefix_delegation) to get a better understanding of it but the gist is that using DHCPv6 it is possible to request a "prefix" where any IPv6 address starting with that will be routed to the router. Then the router can use that to configure clients on the network to each have a unique address instead of the router only one (as in IPv4) and having to share it using a hack like masquerading.

## Install A Client

There are a few different DHCPv6 clients you can use that support _Prefix Delegation_ but I decided to go with _wide-dhcpv6-client_. I also tried _dhcpcd_ but found the configuration syntax to be a little uglier.

```bash
# apt install wide-dhcpv6-client
```

The config below is doing a few different things that I'll list but you can read about all the possible [dhcp6c.conf](https://manpages.debian.org/testing/wide-dhcpv6-client/dhcp6c.conf.5.en.html) configuration.

```diff
# /etc/wide-dhcpv6/dhcp6c.conf 
+ 
+ interface eth0 {
+ #  send rapid-commit;
+   send ia-na 0;
+   send ia-pd 1;
+ };
+ 
+ id-assoc na 0 {
+ 
+ };
+  
+ id-assoc pd 1 {
+   prefix ::/60 infinity;
+   
+   prefix-interface eth1 {
+     sla-id 0;
+     sla-len 4;
+     ifid 1;
+   };
+ 
+   prefix-interface eth1.8 {
+     sla-id 1;
+     sla-len 4;
+     ifid 1;
+   };
+ 
+   prefix-interface eth1.9 {
+     sla-id 2;
+     sla-len 4;
+     ifid 1;
+   };
+ };
```

- **Lines 3-7** use `eth0` to request a "normal address" with `ia-na` and a delegrated prefix range with `ia-pd`
- **Lines 9-11** are required and correspond to the "normal addresss" I asked for, but there is no extra configuration needed
- **Lines 13 and 14** are the start of the prefix delegation block and I am specifying I want a prefix that gives me a subnet of `/60`. I know _Comcast_ will give me a `/60` which is 295,147,905,179,352,825,856 (two hundred ninety five quintillion, one hundred forty seven quadrillion, nine hundred five trillion, one hundred seventy nine billion, three hundred fifty two million, eight hundred twenty five thousand, eight hundred fifty six) so that should be more than enough.
- **Lines 16-20** and the other similar blocks just specify what slice of the prefix I was delegated that I want applied to each interface. The lines `sla-id` (Site-Level Aggregation identifier) is just an index to a what is basically an IPv4 subnet, `sla-len` is the size of that subnet (I requested a /60 so if our SLA is /4 then I end up with /64 which is ideal for an IPv6 subnet), and `ifid` just defines that I want the first address available in our subnet assigned to our interface. So if the IPv6 addresses assigned to this interface was `2601:1833:a3a:100::/64` the interface would have `2601:1833:a3a:100::1/64` assigned to it.

The next step was just to enable and run the service.

```bash
# systemctl enable --now wide-dhcpv6-client
```

Then I was able to verify that I had publicaly accessible IPv6 addresses.

```bash
# ip -6 addr
1: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2001:6020:ae3:1022:a4d3:f031:fb7e:e629/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::2b0:c9ff:fe79:cd77/64 scope link
       valid_lft forever preferred_lft forever
2: eth1@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2601:1833:a3a:100::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::d45a:67ff:fec6:6688/64 scope link
       valid_lft forever preferred_lft forever
3: eth1.8@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2601:1833:a3a:101::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::d45a:67ff:fec6:6688/64 scope link
4: eth1.9@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2601:1833:a3a:102::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::d45a:67ff:fec6:6688/64 scope link
       valid_lft forever preferred_lft forever
...
```

Exactly like I hoped, I can see that using `ia-nd` to request a "normal address" for `eth0` resulted in `2001:6020:ae3:1022:a4d3:f031:fb7e:e629/128` being assigned as my routers public IPv6 address. It also looks like the prefix delegation worked since `eth1`, `eth1.8`, `eth1.9` all have the same prefix with incrementing SLA identifiers that you can see represented by `100`, `101`, `102` in their addresses.

# DHCP and SLAAC

I already setup _dnsmasq_ for IPv4 and so there is very little that needs to be done to add IPv6 support.

I just needed to add `dhcp-range` lines for each subnet. I am tagging them the same as before and using the `::,constructor:<interface>` syntax to tell _dnsmasq_ to determine the the prefix the DHCPv6 range should be valid over from the [GUAs (Global Unicast Addresse)](https://www.omnisecu.com/tcpip/ipv6/global-unicast-ipv6-addresses.php) (publically routable IPs) on each interface. These were assigned in the previous section ([Prefix Delegation](https://bookstack.swigg.net/books/project-router-work-in-progress/page/prefix-delegation)) by _wide-dhcpv6-client_. Declaring `ra-stateless` configures _dnsmasq_ to use [SLAAC](https://en.wikipedia.org/wiki/IPv6#Stateless_address_autoconfiguration_(SLAAC)) to automatically configure clients in this prefix.

```diff
# /etc/dnsmasq.d/dhcp.conf
+ dhcp-range=set:lan,::,constructor:eth1,ra-stateless,12h
+ dhcp-range=set:dmz,::,constructor:eth1.8,ra-stateless,12h
+ dhcp-range=set:warp,::,constructor:eth1.9,ra-stateless,5m
```

Then I enabled router advertisements so _dnsmasq_ will broadcast information to any potential clients on the subnet.

```diff
# /etc/dnsmasq.d/router-advertisements.conf
+ enable-ra
```

# Virtual Private Networking



# Wireguard

I had two goals I wanted to accomplish with VPNs.

1. I need to redirect all outbound traffic from a specific subnet through a VPN so any client on that subnet would have its privacy protected by the VPN.

2. Allow me to VPN into my home network from somehwere else and have access to everything as if I was sitting on my computer at home.

Both of them could have been accomlished with any VPN most likely but I went with [WireGuard](https://bookstack.swigg.net/books/networking/page/wireguard) since it is a simple and fast VPN whose setup is similar to SSH so it was inuitive for me to setup.

## Host Setup

To use _Wireguard_ inside of a LXC container the host has to have _Wireguard_ installed since _LXC_ guests are run with the kernel of the host system. _Wireguard_ was first mainlined into the _Linux_ kernel in version 5.6 so with kernel versions using 5.6 or later it is already built in. Anything before 5.6 that doesn't specifically have _Wireguard_ backported in will need to use kernel modules to get it working. [Wireguard.com](https://www.wireguard.com/install/) has detailed instructions on how to install it on nearly any platform. Since I am using _Proxmox_ as my host it was already backported into the kernel.

## Guest Setup

Additionally I needed the `wireguard-tools` package in the _LXC_ guest and `resolvconf` so DNS can be configured properly.

```bash
# echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/buster-backports.list
# apt update
# apt install --no-install-recommends wireguard-tools
# apt install resolvconf
```

# Route Subnet Through Wireguard Interface

Funneling all traffic from an Ethernet interface through a _Wireguard_ interface is relatively easy once I became familar with how packets flow through _Linux_. I mostly just needed to modify my _Wireguard_ `*.conf` file to add the `Table`, `PostUp`, and `PreDown` parameters.

<p class="callout info">I also needed to setup IP masquerading of outgoing traffic on my <em>Wireguard</em> interface. See below for instructions.</p>

## Create Interface

Creating the configuration file is a bit out of the scope of this document. A VPN provider that supports _Wireguard_ will likely just provide a pre-built configuration file. But I also have a [brief overview of](https://bookstack.swigg.net/books/networking/page/wireguard) how you'd make one.

```ini
# /etc/wireguard/warp.conf
[Interface]
PrivateKey = ****
Address = 10.10.20.59/19, 2a03:4012:4021:80af::1f3c/64
DNS = 10.10.0.1, 2a03:4012:4021:80af::1
Table = 9
PostUp = ip rule add iif eth1.9 lookup 9; ip -6 rule add iif eth1.9 lookup 9
PreDown = ip rule del iif eth1.9 lookup 9; ip -6 rule del iif eth1.9 lookup 9

[Peer]
PublicKey = T28Qn5VFzT4wiwEPd7DscwcP3Rsmq23QcnjH1N5G/wc=
Endpoint = wireguard.vpn-provider.example:51820
AllowedIPs = 0.0.0.0/0, ::0/0...
```

**Line 5:** All rules/routes should be applied to a custom route table `9`. I could have also named my custom route table by running `echo "9 warp" > /etc/iproute2/rt_tables` and then say `Table = warp` for improved readability.

**Line 6:** Adds rules for IPv4 and IPv6 that all traffic coming in interface `eth1.9` should use custom route table `9`. Because I defined a peer with `AllowedIPs = 0.0.0.0/0, ::0/0` a default route will be setup on custom route table `9` that redirects all traffic to the _Wireguard_ interface. If I named my custom route like shown above I could have said `lookup warp` inplace of `lookup 9`.

**Line 7:** Just the inverse of line 5 to clean up after myself when taking down the _Wireguard_ interface.

## Setup IP Masquerading

> IP Masquerading is a technique that hides an entire IP address space, usually consisting of private IP addresses, behind a single IP address in another, usually public address space.

Source: [Wikipedia](https://en.wikipedia.org/wiki/Network_address_translation)

### Configuration

The easiest way to set this up are to append some _netfilter_ rules to the `PostUp` and `PreDown` parameters.

```ini
...
PostUp = ...; iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PreDown = ...; iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

[Peer]
...
```

Although this works fine there is a risk of the _iptables/netfilter_ rules getting squashed by _Shorewall_ if it is restarted while the _Wireguard_ interface exists. It is best to have _Shorewall_ setup the masquerading by making a simple declaration in `/etc/shorewall/snat`. I've included the other _Shorewall_ configuration files that would be necessary to make this setup work.

First I define the `wg` zone…

```diff
# /etc/shorewall/zones
  #ZONE   TYPE    OPTIONS                 IN                      OUT
  #                                       OPTIONS                 OPTIONS
  warp    ipv4
+ wg      ipv4
```

Then I define the interface `WG_IF` and put it in the `wg` zone…

```diff
# /etc/shorewall/interfaces
  #ZONE	INTERFACE	OPTIONS
  warp	WARP_IF		tcpflags,nosmurfs,routefilter=2,logmartians,physical=eth1.9
+ wg	WG_IF		physical=wg0
```

This tells Shorwall to masquerade all IPs going out on `WG_IF`…

```diff
# /etc/shorewall/snat
  #ACTION		SOURCE		DEST
+ MASQUERADE	0.0.0.0/0	WG_IF
```

Then I allow the `warp` zone to send packets to the `wg` zone. The `warp` zone isn't allowed to send packets to any other subnet or the `wan`. This prevents any data/privacy spills from happening if the _Wireguard_ interface ever goes down. It is always best to fail into a state that protects security and privacy.

```diff
# /etc/shorewall/policy
  #SOURCE	DEST		POLICY          LOGLEVEL        RATE    CONNLIMIT
- warp		$FW			ACCEPT          $LOG_LEVEL
+ warp		$FW,wg		ACCEPT          $LOG_LEVEL
```

# Remote Access

Allowing remote access is just a matter of setting up a new _Wireguard_ interface, allowing incoming traffic to that interface, and making sure the firewall allows that traffic to connect to the rest of the network.

## Create Interface

```bash
# cd /etc/wireguard
# umask 077
# wg genkey | tee guard.key | wg pubkey > guard.pub
# printf "[Interface]\PrivateKey = %s\n" `cat guard.key`
```

Then I modified my file to finish configuring the interface and allow a `[Peer]` for my laptop. 

```diff
# /etc/wireguard/guard.conf
[Interface]
PrivateKey = ****
+ Address = 10.0.2.1/28, 2001:db8:2ebf:2::1/64
+ ListenPort = 51820
+
+ [Peer]
+ PublicKey = Iz5ceR0+tCN3BLTWehZxSplzdbABRT8geqifFxubHUA=
+ AllowedIPs = 10.0.2.4/32, 2001:db8:2ebf:1::4/128
+ PresharedKey = ***
```

**Line 4:** Sets an IPv4 and IPv6 address for this interface. These will be the servers IPs on each virtual subnet.

**Line 5:** Sets the port to listen to for this interface. It is just the default _Wirgaurd_ port and I'll allow traffic through the firewall for it soon.

**Line 7-10:** Declare a peer, define the public key to use when communicating and validaing any connections, set what IPs the peer is allowed to use on each virtual subnet, and configure a pre-shared key for additional secuirty.

<p class="callout info">A preshard key can be generated by running <code>wg genpsk</code> and must be the same on both the <code>[Peer]</code> block on the server and the <code>[Interface]</code> block on the client.</p>

## Firewall Configuration

First I had to declare a new interface and since I want it to be as if I was sitting on my laptop at home, I put it in the `lan` zone.

```diff
# /etc/shorewall/interfaces
...
 #ZONE	INTERFACE	OPTIONS
...
 wg	WGAZSE1_IF	tcpflags,nosmurfs,routefilter,logmartians,physical=wgazse1
+ lan	WGGUARD_IF	tcpflags,nosmurfs,routefilter,logmartians,physical=wgguard
```

```diff
# /etc/shorewall/interfaces
...
 #ZONE	INTERFACE	OPTIONS
...
 wg	WGAZSE1_IF	tcpflags,nosmurfs,routefilter,logmartians,physical=wgazse1
+ lan	WGGUARD_IF	tcpflags,forward=1,physical=wgguard
```

For outside clients to connect I need to add a rule that allows them to connect to the firewall on port 51820.

```diff
# /etc/shorewall[6]/rules
+ ACCEPT          wan,lan         $FW             udp             51820
```

The last step is to once again setup masquerading so traffic from clients on the _Wireguard_ subnet appear to be originating from the `wgguard` interface which is in the `lan` zone.

```diff
# /etc/shorewall/snat
+ MASQUERADE		10.0.2.0/28				WAN_IF,LAN_IF,DMZ_IF
```

```diff
# /etc/shorewall6/snat
+ MASQUERADE		fde9:2375:2ebf:2::/64	WAN_IF,LAN_IF,DMZ_IF
```

# Bonded Interface

work in progress

# Network Intrusion Detection

work in progress

# Traffic Graphing/Monitoring

work in progress