Home A - Z
Page Index

An Otter Wiki

Blame

c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
1 
# Install Nvidia GPU on Proxmox K3s LXC
421654 admin 2024-10-16 02:14:17
pve host install guide
2
This guide pertains to installing Nvidia drivers on K3s running on a privileged LXC. Make sure that you install the same Nvidia driver version on the LXC that you installed on the Proxmox (PVE) host.
3
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
4
## Installing Nvidia Drivers on the Proxmox Host
5
Make note of the driver version you install as you'll need to install the same version later on the K3s LXC. Use the following instructions to install the Nvidia driver on your Proxmox host ([Proxmox official docs](https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation)):
421654 admin 2024-10-16 02:14:17
pve host install guide
6
```
7
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
8
9
apt update
10
apt install dkms libc6-dev proxmox-default-headers --no-install-recommends
11
12
wget -O NVIDIA-Linux-x86_64-550.120.run https://us.download.nvidia.com/XFree86/Linux-x86_64/550.120/NVIDIA-Linux-x86_64-550.120.run
13
chmod +x NVIDIA-Linux-x86_64-550.120.run
14
./NVIDIA-Linux-x86_64-550.120.run --no-nouveau-check --dkms
15
```
16
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
17
You must also add the following udev rules to create the Nvidia devices on the Proxmox host:
421654 admin 2024-10-16 02:14:17
pve host install guide
18
```
19
cat << EOF > /etc/udev/rules.d/70-nvidia.rules
20
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
21
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
22
EOF
23
```
24
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
25
Reboot your Proxmox host to load the Nvidia driver and create the udev devices.
421654 admin 2024-10-16 02:14:17
pve host install guide
26
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
27
## Configuring the LXC
28
Once the driver is loaded and the Nvidia devices exist you need to obtain their major device numbers. Run the following on the Proxmox host to get your numbers. In my case they are 195 and 236:
421654 admin 2024-10-16 02:14:17
pve host install guide
29
```
30
root@pve-media:~# ls -la /dev/nvid*
31
crw-rw-rw- 1 root root 195, 0 Sep 27 19:40 /dev/nvidia0
32
crw-rw-rw- 1 root root 195, 255 Sep 27 19:40 /dev/nvidiactl
33
crw-rw-rw- 1 root root 236, 0 Sep 27 19:40 /dev/nvidia-uvm
34
crw-rw-rw- 1 root root 236, 1 Sep 27 19:40 /dev/nvidia-uvm-tools
35
36
/dev/nvidia-caps:
37
total 0
38
drwxr-xr-x 2 root root 80 Sep 27 19:40 .
39
drwxr-xr-x 20 root root 5060 Sep 27 20:08 ..
40
cr-------- 1 root root 239, 1 Sep 27 19:40 nvidia-cap1
41
cr--r--r-- 1 root root 239, 2 Sep 27 19:40 nvidia-cap2
42
```
43
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
44
Next you must add to following to the LXC config changing your device numbers as needed:
421654 admin 2024-10-16 02:14:17
pve host install guide
45
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
46
Edit:
421654 admin 2024-10-16 02:14:17
pve host install guide
47
`/etc/pve/lxc/<lxc_id>.conf`
48
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
49
50
Add:
51
421654 admin 2024-10-16 02:14:17
pve host install guide
52
```
53
mp1: /usr/lib/modules,mp=/usr/lib/modules
54
lxc.cgroup2.devices.allow: c 195:* rw
55
lxc.cgroup2.devices.allow: c 236:* rw
56
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
57
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
58
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
59
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
60
```
61
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
62
These lines perform the following for your LXC (in order):
63
- Mount host kernel headers so gpu-operator helm chart on K3s can build the Nvidia drivers
421654 admin 2024-10-16 02:14:17
pve host install guide
64
- Create cgroup2 allowist entries for the major device numbers of the Nvidia devices
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
65
- Passthrough Nvidia devices to LXC
421654 admin 2024-10-16 02:14:17
pve host install guide
66
67
68
# References
69
https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation
70
https://github.com/UntouchedWagons/K3S-NVidia
71
https://theorangeone.net/posts/lxc-nvidia-gpu-passthrough/
72
https://forum.proxmox.com/threads/sharing-gpu-to-lxc-container-failed-to-initialize-nvml-unknown-error.98905/