Skip to main content

LXC GPU Access

Giving a LXC container GPU access allows you to use a GPU in a guest while it is still available to the host machine. This is a big advantage over virtual machines where only a single host or guest can have access to a GPU at one time.

Determine Device Major/Minor Numbers

To allow a container access to the device you'll have to know the devices major/minor numbers. This can be found easily enough by running ls -l in /dev/. As an example to pass through the integated UHD 630 GPU from an Core i7 8700k you would first list /dev/dri

root@blackbox:~# ls -l /dev/dri
total 0
drwxr-xr-x 2 root root         80 May 12 21:54 by-path
crw-rw---- 1 root video  226,   0 May 12 21:54 card0
crw-rw---- 1 root render 226, 128 May 12 21:54 renderD128

From that you can see the major device number is 226 and the minors are 0 and 128.

Provide LXC Access

In our configuration file you'd then add lines to allow it access to that device, and then also bind the devices.

# /etc/pve/lxc/*.conf
+ lxc.cgroup.devices.allow: c 226:* rwm
+ lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file,mode=0666
+ lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

Allow unprivileged Containers Access

In the example above we saw that card0 and renderD128 have their groups set to video and render. You'll need to explicitly map the group IDs from the host system to the guest container using ID maps.

First you need to give root permission to map the group IDs.

# /etc/subgid
+ root:44:1
+ root:108:1

Then you'll need to create the ID mapping. The first line maps user IDs 0-65536 to IDs starting at 100,000. Then we map the IDs before the one we care about (0-44)

# /etc/pve/lxc/*.conf
  lxc.cgroup.devices.allow: c 226:* rwm
  lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file,mode=0666
  lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
+ lxc.idmap: u 0 100000 65536
+ lxc.idmap: g 0 100000 44
+ lxc.idmap: g 44 44 1
+ lxc.idmap: g 45 100045 60
+ lxc.idmap: g 106 108 1
+ lxc.idmap: g 107 100107 65429

Ensure IOMMU Is Activated

First step of this process is to make sure that your hardware is even capable of this type of virtualization. You need to have a motherboard, CPU, and BIOS that has an IOMMU controller and supports Intel-VT-x and Intel-VT-d or AMD-v and AMD-vi. Some motherboards use different terminology for these, for example they may list AMD-v as SVM and AMD-vi as IOMMU controller.

Update Bootloader

Update Kernel Parameters

**NOTE** Be sure to replace intel_iommu=on with amd_iommu=on if you're running on AMD instead of Intel.

Grub2
# /etc/default/grub
- GRUB_CMDLINE_LINUX_DEFAULT="quiet"
+ GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt
Systemd
# /etc/kernel/cmdline
- root=ZFS=rpool/ROOT/pve-1 boot=zfs
+ root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

Rebuild Bootloader Options

Grub
update-grub
systemd-boot
bootctl update
Proxmox
pve-efiboot-tool refresh

Enable Virtual Functions

cat /sys/class/net/enp10s0f0/device/sriov_totalvfs
7

To enable virtual functions you just echo the number you want to sriov_numvfs in sysfs...

echo 4 > /sys/class/net/enp10s0f0/device/sriov_numvfs

Make Persistent

Sysfs is a virtual file system in Linux kernel 2.5+ that provides a tree of system devices. This package provides the program 'systool' to query it: it can list devices by bus, class, and topology.

In addition this package ships a configuration file /etc/sysfs.conf which allows one to conveniently set sysfs attributes at system bootup (in the init script etc/init.d/sysfsutils).

apt install sysfsutils

Configure sysfsutils

To make these changes persistent, you need to update /etc/sysfs.conf so that it gets set on startup.

echo "class/net/eth2/device/sriov_numvfs = 4" >> /etc/sysfs.conf
Back to top