Running K3s on Proxmox LXC
Background
I only have a couple servers so in my case dedicating an entire system for Kubernetes was not an option. The choice was then left between using a VM or LXC. While using a VM would probably be the best choice in most scenarios, I had a unique requirement of needing to access a ZFS pool across both my Kubernetes single node cluster and other LXC containers on my Proxmox host.
One major advantage of LXCs is I can manage all my ZFS pools on my Proxmox host and then mount specific host paths on my containers. This allows me to share a single ZFS pool with multiple LXCs all while not having to worry about NFS and the potential performance implications.
Software
- Proxmox v7.3
- Debian LXC v11.6
- K3s v1.25.6
Proxmox Configuration
After you have created a new privileged LXC (uncheck Unprivileged container in GUI) append the follwing to your container config:
/etc/pve/lxc/<ct-id>.conf
lxc.apparmor.profile: unconfined lxc.cap.drop: lxc.mount.auto: "proc:rw sys:rw" lxc.cgroup2.devices.allow: c 10:200 rwm
LXC Configuration
Kubernetes also requires access to /dev/kmsg. This can be done by creating a soft link of /dev/console to /dev/kmsg:
ln -s /dev/console /dev/kmsg
Some people recommend putting this command in your /etc/rc.local to ensure it is run on boot. I prefer to add it as an override config to my K3s systemd service. This means it is only run when the K3s service is started:
/etc/systemd/system/k3s.service.d/override.conf
[Service] ExecStartPre=-/bin/ln -s /dev/console /dev/kmsg
Note the use of ExecStartPre=- the - tells systemd to still start the service even if the command fails. This ensures that you can restart k3s.service because when the ln command is executed a second time it will fail on restart as the link already exists.
K3s
Now that the service override exists you can simply download and run the K3s installation script:
curl -sfL https://get.k3s.io | sh -
When the installation script starts k3s.service systemd will ensure the ln command is run before Kubernetes is actually started.