Blame
| 04589f | admin | 2024-10-14 22:01:23 | 1 | # Running K3s on Proxmox LXC |
| 2 | ||||
| 3 | ## Background |
|||
| 4 | I only have a couple servers so in my case dedicating an entire system for Kubernetes was not an option. The choice was then left between using a VM or LXC. While using a VM would probably be the best choice in most scenarios, I had a unique requirement of needing to access a ZFS pool across both my Kubernetes single node cluster and other LXC containers on my Proxmox host. |
|||
| 5 | ||||
| 6 | One major advantage of LXCs is I can manage all my ZFS pools on my Proxmox host and then [mount specific host paths on my containers](https://pve.proxmox.com/wiki/Linux_Container#_bind_mount_points). This allows me to share a single ZFS pool with multiple LXCs all while not having to worry about NFS and the potential performance implications. |
|||
| 7 | ||||
| 8 | ## Software |
|||
| 9 | - Proxmox v7.3 |
|||
| 10 | - Debian LXC v11.6 |
|||
| 11 | - K3s v1.25.6 |
|||
| 12 | ||||
| 13 | ## Proxmox Configuration |
|||
| 14 | After you have created a new privileged LXC (uncheck `Unprivileged container` in GUI) append the follwing to your container config: |
|||
| 15 | ||||
| 16 | `/etc/pve/lxc/<ct-id>.conf` |
|||
| 17 | ``` |
|||
| 18 | lxc.apparmor.profile: unconfined |
|||
| 19 | lxc.cap.drop: |
|||
| 20 | lxc.mount.auto: "proc:rw sys:rw" |
|||
| 21 | lxc.cgroup2.devices.allow: c 10:200 rwm |
|||
| 22 | ``` |
|||
| 23 | ||||
| 24 | ## LXC Configuration |
|||
| 25 | Kubernetes also requires access to `/dev/kmsg`. This can be done by creating a soft link of `/dev/console` to `/dev/kmsg`: |
|||
| 26 | ``` |
|||
| 27 | ln -s /dev/console /dev/kmsg |
|||
| 28 | ``` |
|||
| 29 | ||||
| 30 | Some people recommend putting this command in your `/etc/rc.local` to ensure it is run on boot. I prefer to add it as an override config to my K3s systemd service. This means it is only run when the K3s service is started: |
|||
| 31 | ||||
| 32 | `/etc/systemd/system/k3s.service.d/override.conf` |
|||
| 33 | ``` |
|||
| 34 | [Service] |
|||
| 35 | ExecStartPre=-/bin/ln -s /dev/console /dev/kmsg |
|||
| 36 | ``` |
|||
| 45a274 | admin | 2024-10-14 22:01:45 | 37 | |
| 04589f | admin | 2024-10-14 22:01:23 | 38 | Note the use of `ExecStartPre=-` the `-` tells systemd to still start the service even if the command fails. This ensures that you can restart `k3s.service` because when the `ln` command is executed a second time it will fail on restart as the link already exists. |
| 45a274 | admin | 2024-10-14 22:01:45 | 39 | |
| 04589f | admin | 2024-10-14 22:01:23 | 40 | |
| 41 | ## K3s |
|||
| 42 | Now that the service override exists you can simply download and run the [K3s installation script](https://docs.k3s.io/quick-start): |
|||
| 43 | ``` |
|||
| 44 | curl -sfL https://get.k3s.io | sh - |
|||
| 45 | ``` |
|||
| 46 | ||||
| 47 | When the installation script starts `k3s.service` systemd will ensure the `ln` command is run before Kubernetes is actually started. |
|||
| 48 | ||||
| 49 | ||||
| 50 | # References |
|||
| 51 | - https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 |
|||
| 52 | - https://davegallant.ca/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/ |