Home A - Z
Page Index

An Otter Wiki

Blame

421654 admin 2024-10-16 02:14:17
pve host install guide
1 
# Use Nvidia GPU on Proxmox K3s LXC
2
3
# Proxmox Host
4
This guide pertains to installing Nvidia drivers on K3s running on a privileged LXC. Make sure that you install the same Nvidia driver version on the LXC that you installed on the Proxmox (PVE) host.
5
6
7
Install Nvidia drivers on the PVE host. Make note of the driver version you install you'll need to install the same version on the LXC.
8
9
10
Instructions taken from this [guide](https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation):
11
```
12
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
13
14
apt update
15
apt install dkms libc6-dev proxmox-default-headers --no-install-recommends
16
17
wget -O NVIDIA-Linux-x86_64-550.120.run https://us.download.nvidia.com/XFree86/Linux-x86_64/550.120/NVIDIA-Linux-x86_64-550.120.run
18
chmod +x NVIDIA-Linux-x86_64-550.120.run
19
./NVIDIA-Linux-x86_64-550.120.run --no-nouveau-check --dkms
20
```
21
22
23
You must add the following udev rules to create the Nvidia devices on the PVE host:
24
```
25
cat << EOF > /etc/udev/rules.d/70-nvidia.rules
26
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
27
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
28
EOF
29
```
30
31
32
Once the Nvidia devices exist you need to obtain their major device numbers (e.g. 195 and 236):
33
```
34
root@pve-media:~# ls -la /dev/nvid*
35
crw-rw-rw- 1 root root 195, 0 Sep 27 19:40 /dev/nvidia0
36
crw-rw-rw- 1 root root 195, 255 Sep 27 19:40 /dev/nvidiactl
37
crw-rw-rw- 1 root root 236, 0 Sep 27 19:40 /dev/nvidia-uvm
38
crw-rw-rw- 1 root root 236, 1 Sep 27 19:40 /dev/nvidia-uvm-tools
39
40
/dev/nvidia-caps:
41
total 0
42
drwxr-xr-x 2 root root 80 Sep 27 19:40 .
43
drwxr-xr-x 20 root root 5060 Sep 27 20:08 ..
44
cr-------- 1 root root 239, 1 Sep 27 19:40 nvidia-cap1
45
cr--r--r-- 1 root root 239, 2 Sep 27 19:40 nvidia-cap2
46
```
47
48
49
Next you must add to following to the LXC config:
50
51
`/etc/pve/lxc/<lxc_id>.conf`
52
53
```
54
mp1: /usr/lib/modules,mp=/usr/lib/modules
55
lxc.cgroup2.devices.allow: c 195:* rw
56
lxc.cgroup2.devices.allow: c 236:* rw
57
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
58
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
59
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
60
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
61
```
62
63
64
These lines perform the following:
65
- Mount host kernel headers so gpu-operator helm chart on K3s can build Nvidia drivers
66
- Create cgroup2 allowist entries for the major device numbers of the Nvidia devices
67
- Passthrough Nvidia devices through mounts
68
69
70
# References
71
https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation
72
https://github.com/UntouchedWagons/K3S-NVidia
73
https://theorangeone.net/posts/lxc-nvidia-gpu-passthrough/
74
https://forum.proxmox.com/threads/sharing-gpu-to-lxc-container-failed-to-initialize-nvml-unknown-error.98905/