Home A - Z
Page Index

An Otter Wiki

Blame

c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
1 
# Install Nvidia GPU on Proxmox K3s LXC
0ca852 admin 2024-10-16 14:51:15
update description
2
Guide for installing Nvidia drivers on a Proxmox privileged LXC to enable their use in K3s pods.
421654 admin 2024-10-16 02:14:17
pve host install guide
3
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
4
## Installing Nvidia Drivers on the Proxmox Host
5
Make note of the driver version you install as you'll need to install the same version later on the K3s LXC. Use the following instructions to install the Nvidia driver on your Proxmox host ([Proxmox official docs](https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation)):
421654 admin 2024-10-16 02:14:17
pve host install guide
6
```
7
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
8
9
apt update
10
apt install dkms libc6-dev proxmox-default-headers --no-install-recommends
11
12
wget -O NVIDIA-Linux-x86_64-550.120.run https://us.download.nvidia.com/XFree86/Linux-x86_64/550.120/NVIDIA-Linux-x86_64-550.120.run
13
chmod +x NVIDIA-Linux-x86_64-550.120.run
14
./NVIDIA-Linux-x86_64-550.120.run --no-nouveau-check --dkms
15
```
16
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
17
You must also add the following udev rules to create the Nvidia devices on the Proxmox host:
421654 admin 2024-10-16 02:14:17
pve host install guide
18
```
19
cat << EOF > /etc/udev/rules.d/70-nvidia.rules
20
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
21
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
22
EOF
23
```
24
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
25
Reboot your Proxmox host to load the Nvidia driver and create the udev devices.
421654 admin 2024-10-16 02:14:17
pve host install guide
26
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
27
## Configuring the LXC
28
Once the driver is loaded and the Nvidia devices exist you need to obtain their major device numbers. Run the following on the Proxmox host to get your numbers. In my case they are 195 and 236:
421654 admin 2024-10-16 02:14:17
pve host install guide
29
```
30
root@pve-media:~# ls -la /dev/nvid*
31
crw-rw-rw- 1 root root 195, 0 Sep 27 19:40 /dev/nvidia0
32
crw-rw-rw- 1 root root 195, 255 Sep 27 19:40 /dev/nvidiactl
33
crw-rw-rw- 1 root root 236, 0 Sep 27 19:40 /dev/nvidia-uvm
34
crw-rw-rw- 1 root root 236, 1 Sep 27 19:40 /dev/nvidia-uvm-tools
35
36
/dev/nvidia-caps:
37
total 0
38
drwxr-xr-x 2 root root 80 Sep 27 19:40 .
39
drwxr-xr-x 20 root root 5060 Sep 27 20:08 ..
40
cr-------- 1 root root 239, 1 Sep 27 19:40 nvidia-cap1
41
cr--r--r-- 1 root root 239, 2 Sep 27 19:40 nvidia-cap2
42
```
43
0ca852 admin 2024-10-16 14:51:15
update description
44
Next you must add to following to the LXC config changing your device numbers as needed.
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
45
0ca852 admin 2024-10-16 14:51:15
update description
46
Edit: `/etc/pve/lxc/<lxc_id>.conf`
421654 admin 2024-10-16 02:14:17
pve host install guide
47
```
48
mp1: /usr/lib/modules,mp=/usr/lib/modules
49
lxc.cgroup2.devices.allow: c 195:* rw
50
lxc.cgroup2.devices.allow: c 236:* rw
51
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
52
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
53
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
54
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
55
```
56
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
57
These lines perform the following for your LXC (in order):
58
- Mount host kernel headers so gpu-operator helm chart on K3s can build the Nvidia drivers
421654 admin 2024-10-16 02:14:17
pve host install guide
59
- Create cgroup2 allowist entries for the major device numbers of the Nvidia devices
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
60
- Passthrough Nvidia devices to LXC
421654 admin 2024-10-16 02:14:17
pve host install guide
61
62
63
# References
64
https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation
65
https://github.com/UntouchedWagons/K3S-NVidia
66
https://theorangeone.net/posts/lxc-nvidia-gpu-passthrough/
67
https://forum.proxmox.com/threads/sharing-gpu-to-lxc-container-failed-to-initialize-nvml-unknown-error.98905/