K3s 1.31 and the race for edge ARM control

K3s resource efficiency and edge deployment

K3s is a CNCF-certified Kubernetes distribution that is not a fork. It was donated to the CNCF in June 2020 and passes Sonobuoy conformance tests. Any valid Kubernetes manifest works on K3s because it maintains API compatibility. The distribution packages the control plane into a single binary under 100MB, which includes the containerd runtime, Flannel CNI, Traefik, and a Klipper load balancer. I find this single-binary approach more efficient for edge, IoT, and CI/CD workloads than standard Kubernetes. Standard Kubernetes requires 2 vCPUs and 2 GB of RAM for control plane components alone, whereas K3s agent nodes can run on 512 MB of RAM. For a server node, K3s requires 2 vCPUs and 2 GB of RAM. The official documentation reports that a 2 vCPU and 4 GB server handles up to 350 agents, while a 4 vCPU and 8 GB configuration manages 900 agents, and a three-server setup can reach 1,200 agents. K3s is suitable for x86, ARM64, ARMv7, and S390X architectures. It uses SQLite as the default datastore for single-node deployments, but supports etcd, MySQL, or PostgreSQL for high-availability needs. K3s also uses Spegel, an embedded distributed container image registry mirror, to help nodes pull images from each other when uplink speeds are low. K3s removes alpha and beta components to reduce the attack surface and memory footprint. K3s remains the most widely deployed Kubernetes distribution on Turing Pi clusters. It targets resource-constrained environments like older laptops, IoT devices, and industrial appliances. This distribution bundles Traefik and the local-path provisioner by default to simplify deployment. This approach allows for fast local development cycles on modest hardware. I find its ARM64 support particularly mature for users running Raspberry Pi or Nvidia Jetson boards. You probably already know that managing SD cards on Raspberry Pi requires care because etcd’s write intensity can damage them.

Comparison of k0s and MicroK8s

k0s provides a zero-dependency, single-binary installation developed by Team Lens and Mirantis. It is a modular architecture that targets both production and development environments while following CNCF principles. The control plane components run as ordinary processes supervised by the k0s binary, which means users cannot schedule workloads onto controller nodes. k0s uses kine to translate the etcd API into SQL, mirroring the data handling strategy used by K3s. The k0s binary is statically compiled with no OS-level dependencies besides the kernel. k0s provides Konnectivity, CoreDNS, Metrics Server, and containerd with runc. It uses Kube-router as the default CNI and Calico as a preconfigured alternative. k0s requires 1 vCPU and 1 GB of RAM for a controller, while workers need 0.5 vCPU and 1 GB of RAM. For larger workloads, a configuration with 200 workers and 20,000 pods uses approximately 3.3 GB of RAM. MicroK8s is developed by Canonical and installs via Snap on Ubuntu 22.04. It is a cross-platform tool that works on Linux, MacOS, and Windows. MicroK8s includes an add-on system to enable features like DNS, ingress, and storage with a single command. High availability in MicroK8s uses the dqlite datastore, where voters replicate the database and standby nodes replicate without voting. In the event that a leader disappears ungracefully, the cluster elects a new leader in up to 5 seconds.

MicroK8s is not a portable choice for fleets that forbid snapd. I find the snap-managed lifecycle creates friction for operators who want more control over updates and service management. While MicroK8s simplifies the setup for Ubuntu users, it feels more specialized than its competitors. Does the MicroK8s add-on model scale as well as the K3s bundled approach in highly disconnected environments?

Selecting the right distribution for the workload

RKE2 serves organizations that must meet CIS Kubernetes Benchmarks or FIPS 140-2 requirements. It is a sibling to K3s that emphasizes security and hardening through components scanned with Trivy in the build pipeline. RKE2 is a hybrid that inherits usability from K3s and alignment from RKE1. RKE2 is also known as RKE Government and targets the U.S. Federal Government sector. RKE2 recently replaced NGINX with Traefik as the default ingress controller for all new clusters starting with version 1.36. RKE2 bundles four primary CNI plugins: Canal, Cilium, Calico, and Flannel. K3s is the best fit for most ARM homelabs and edge sites because of its extensive community support and documentation. k0s is the best option for operators who want a cleaner foundation with minimal system intrusion. MicroK8s works well for developers who use Ubuntu and need a feature-rich local cluster via add-ons. Several hybrid architectures use both K3s and K8s together. For instance, Home Depot manages its site locations using K3s clusters administered through Rancher. Edge K3s clusters are managed from a central control plane running on K8s or RKE2. This parity between K3s and K8s allows developers to test manifests on K3s before promoting them to production on enterprise clusters. I recommend K3s for any team managing remote IoT clusters on limited hardware.