If you have done everything correctly [K3s should automatically detect the Nvidia container runtime](https://docs.k3s.io/advanced#nvidia-container-runtime-support) when the service is started. You can verify this by running:
This will create a bunch of pods in the `gpu-operator` namespace that will build the Nvidia drivers. You will see some of these pods restarting, this is normal as some pods are dependant on others to complete. Overall the build process should take a couple of minutes. If it's taking longer than 10 minutes you likely have an issue and should look at the logs of the `gpu-operator-node-feature-discovery-worker` pods (This is how I figure out you need to mount the Proxmox host kernel headers on the LXC as the pod couldn't find the kernel modules).
+
+
You can verify that everything is working correctly by spinning up a pod that uses the GPU:
+
```
+
cat << EOF > gpu-benchmark-pod.yaml
+
---
+
apiVersion: v1
+
kind: Pod
+
metadata:
+
name: nbody-gpu-benchmark
+
namespace: default
+
spec:
+
restartPolicy: OnFailure
+
runtimeClassName: nvidia
+
containers:
+
- name: cuda-container
+
image: nvcr.io/nvidia/k8s/cuda-sample:nbody
+
args: ["nbody", "-gpu", "-benchmark"]
+
resources:
+
limits:
+
nvidia.com/gpu: 1
+
env:
+
- name: NVIDIA_VISIBLE_DEVICES
+
value: all
+
- name: NVIDIA_DRIVER_CAPABILITIES
+
value: all
+
EOF
+
+
kubectl apply -f gpu-benchmark-pod.yaml
+
```
+
+
GPU successfully identified and benchmarked:
+
```
+
root@k3s:~# k logs nbody-gpu-benchmark
+
Run "nbody -benchmark [-numbodies=<numBodies>]" to measure performance.
+
-fullscreen (run n-body simulation in fullscreen mode)
+
-fp64 (use double precision floating point values for simulation)
+
-hostmem (stores simulation data in host memory)
+
-benchmark (run benchmark to measure performance)
+
-numbodies=<N> (number of bodies (>= 1) to run in simulation)
+
-device=<d> (where d=0,1,2.... for the CUDA device to use)
+
-numdevices=<i> (where i=(number of CUDA devices > 0) to use for simulation)
+
-compare (compares simulation results running once on the default GPU and once on the CPU)
+
-cpu (run n-body simulation on the CPU)
+
-tipsy=<file.bin> (load a tipsy model file for simulation)
+
+
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
+
+
> Windowed mode
+
> Simulation data stored in video memory
+
> Single precision floating point simulation
+
> 1 Devices used for simulation
+
GPU Device 0: "Pascal" with compute capability 6.1
+
+
> Compute 6.1 CUDA device: [NVIDIA GeForce GTX 1080]
+
20480 bodies, total time for 10 iterations: 14.370 ms
+
= 291.883 billion interactions per second
+
= 5837.668 single-precision GFLOP/s at 20 flops per interaction
+
```
+
+
## Final Notes
+
When running pods that utilize your GPU you will only be able to see the processes by running `nvidia-smi` on your Proxmox host. Running `nvidia-smi` in the LXC will not show any GPU processes even if they are running.
