Home A - Z
Page Index

An Otter Wiki

Blame

c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
1 
# Install Nvidia GPU on Proxmox K3s LXC
0ca852 admin 2024-10-16 14:51:15
update description
2
Guide for installing Nvidia drivers on a Proxmox privileged LXC to enable their use in K3s pods.
421654 admin 2024-10-16 02:14:17
pve host install guide
3
186689 admin 2024-10-16 15:18:33
nvidia lxc install
4
## Software
5
- Proxmox v8.2.2
6
- Debian LXC v12.7
7
- K3s v1.30.5
8
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
9
## Installing Nvidia Drivers on the Proxmox Host
186689 admin 2024-10-16 15:18:33
nvidia lxc install
10
Make note of the driver version you install as you'll need to install the same version later on the K3s LXC. Use the following instructions to install the Nvidia driver on your Proxmox host ([Proxmox official guide](https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation)):
421654 admin 2024-10-16 02:14:17
pve host install guide
11
```
12
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
13
14
apt update
15
apt install dkms libc6-dev proxmox-default-headers --no-install-recommends
16
17
wget -O NVIDIA-Linux-x86_64-550.120.run https://us.download.nvidia.com/XFree86/Linux-x86_64/550.120/NVIDIA-Linux-x86_64-550.120.run
18
chmod +x NVIDIA-Linux-x86_64-550.120.run
19
./NVIDIA-Linux-x86_64-550.120.run --no-nouveau-check --dkms
20
```
21
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
22
You must also add the following udev rules to create the Nvidia devices on the Proxmox host:
421654 admin 2024-10-16 02:14:17
pve host install guide
23
```
24
cat << EOF > /etc/udev/rules.d/70-nvidia.rules
25
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
26
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
27
EOF
28
```
29
186689 admin 2024-10-16 15:18:33
nvidia lxc install
30
Reboot your Proxmox host to load the Nvidia driver and create the udev devices. You can verify the drivers are working with `nvidia-smi`:
31
```
32
root@pve-media:~# nvidia-smi
33
Wed Oct 16 11:17:11 2024
34
+-----------------------------------------------------------------------------------------+
35
| NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 |
36
|-----------------------------------------+------------------------+----------------------+
37
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
38
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
39
| | | MIG M. |
40
|=========================================+========================+======================|
41
| 0 NVIDIA GeForce GTX 1080 Off | 00000000:01:00.0 Off | N/A |
42
| 0% 35C P8 16W / 210W | 0MiB / 8192MiB | 0% Default |
43
| | | N/A |
44
+-----------------------------------------+------------------------+----------------------+
45
46
+-----------------------------------------------------------------------------------------+
47
| Processes: |
48
| GPU GI CI PID Type Process name GPU Memory |
49
| ID ID Usage |
50
|=========================================================================================|
51
| No running processes found |
52
+-----------------------------------------------------------------------------------------+
53
```
421654 admin 2024-10-16 02:14:17
pve host install guide
54
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
55
## Configuring the LXC
56
Once the driver is loaded and the Nvidia devices exist you need to obtain their major device numbers. Run the following on the Proxmox host to get your numbers. In my case they are 195 and 236:
421654 admin 2024-10-16 02:14:17
pve host install guide
57
```
58
root@pve-media:~# ls -la /dev/nvid*
59
crw-rw-rw- 1 root root 195, 0 Sep 27 19:40 /dev/nvidia0
60
crw-rw-rw- 1 root root 195, 255 Sep 27 19:40 /dev/nvidiactl
61
crw-rw-rw- 1 root root 236, 0 Sep 27 19:40 /dev/nvidia-uvm
62
crw-rw-rw- 1 root root 236, 1 Sep 27 19:40 /dev/nvidia-uvm-tools
63
64
/dev/nvidia-caps:
65
total 0
66
drwxr-xr-x 2 root root 80 Sep 27 19:40 .
67
drwxr-xr-x 20 root root 5060 Sep 27 20:08 ..
68
cr-------- 1 root root 239, 1 Sep 27 19:40 nvidia-cap1
69
cr--r--r-- 1 root root 239, 2 Sep 27 19:40 nvidia-cap2
70
```
71
0ca852 admin 2024-10-16 14:51:15
update description
72
Next you must add to following to the LXC config changing your device numbers as needed.
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
73
0ca852 admin 2024-10-16 14:51:15
update description
74
Edit: `/etc/pve/lxc/<lxc_id>.conf`
421654 admin 2024-10-16 02:14:17
pve host install guide
75
```
76
mp1: /usr/lib/modules,mp=/usr/lib/modules
77
lxc.cgroup2.devices.allow: c 195:* rw
78
lxc.cgroup2.devices.allow: c 236:* rw
79
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
80
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
81
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
82
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
83
```
84
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
85
These lines perform the following for your LXC (in order):
86
- Mount host kernel headers so gpu-operator helm chart on K3s can build the Nvidia drivers
421654 admin 2024-10-16 02:14:17
pve host install guide
87
- Create cgroup2 allowist entries for the major device numbers of the Nvidia devices
c9a46a admin 2024-10-16 14:48:50
update k3s nvidia
88
- Passthrough Nvidia devices to LXC
421654 admin 2024-10-16 02:14:17
pve host install guide
89
186689 admin 2024-10-16 15:18:33
nvidia lxc install
90
### Installing Nvidia Driver on LXC
91
For the LXC you are going to install the same Nvidia driver version but instead with the `--no-kernel-module` option as the LXC shares the same kernel as your Proxmox host:
92
```
93
wget -O NVIDIA-Linux-x86_64-550.120.run https://us.download.nvidia.com/XFree86/Linux-x86_64/550.120/NVIDIA-Linux-x86_64-550.120.run
94
chmod +x NVIDIA-Linux-x86_64-550.120.run
95
./NVIDIA-Linux-x86_64-550.120.run --no-kernel-module
96
```
97
98
### Installing the Nvidia Container Toolkit
99
Next you need to install the Nvidia container toolkit. Start by running the following to add the repository to Apt ([Nvidia official guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)):
100
```
101
apt install -y gpg curl
102
103
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg --yes \
104
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
105
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
106
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
107
```
108
109
Then install the container toolkit:
110
```
111
apt update
112
apt install -y nvidia-container-runtime
113
```
114
115
Debian is not officially supported so we have to create a soft link for `ldconfig` so the [nvidia-container-cli](https://github.com/NVIDIA/nvidia-container-toolkit/issues/147) can find it:
116
```
117
ln -s /sbin/ldconfig /sbin/ldconfig.real
118
```
119
120
Reboot the LXC and verify the drivers are loaded and working with `nvidia-smi`:
121
```
122
root@k3s-media:~# nvidia-smi
123
Wed Oct 16 11:15:20 2024
124
+-----------------------------------------------------------------------------------------+
125
| NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 |
126
|-----------------------------------------+------------------------+----------------------+
127
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
128
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
129
| | | MIG M. |
130
|=========================================+========================+======================|
131
| 0 NVIDIA GeForce GTX 1080 Off | 00000000:01:00.0 Off | N/A |
132
| 0% 35C P8 16W / 210W | 0MiB / 8192MiB | 0% Default |
133
| | | N/A |
134
+-----------------------------------------+------------------------+----------------------+
135
136
+-----------------------------------------------------------------------------------------+
137
| Processes: |
138
| GPU GI CI PID Type Process name GPU Memory |
139
| ID ID Usage |
140
|=========================================================================================|
141
| No running processes found |
142
+-----------------------------------------------------------------------------------------+
143
```
421654 admin 2024-10-16 02:14:17
pve host install guide
144
145
# References
186689 admin 2024-10-16 15:18:33
nvidia lxc install
146
- https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Preparation
147
- https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
148
- https://github.com/UntouchedWagons/K3S-NVidia
149
- https://theorangeone.net/posts/lxc-nvidia-gpu-passthrough/
150
- https://forum.proxmox.com/threads/sharing-gpu-to-lxc-container-failed-to-initialize-nvml-unknown-error.98905/