k_actuallykasi.
← Back to the notebook
Kubernetes4 Nov 20266 min read

Kubernetes on One Box, Part 2: Preparing Debian Nodes for kubeadm

One idempotent Ansible playbook for kernel modules, sysctls, swap, time sync, containerd 2.x and held Kubernetes packages.

k_Written by Kasi · Engineering notes

In part 1 we recorded the versions, networks and API endpoint as Ansible data. This part turns three plain Debian 13 VMs into machines kubeadm can use. Everything happens in one playbook, node-preparation.yml, which is safe to run as often as you like.

It stops before kubeadm init. Creating the cluster is a separate, deliberate step in part 3.

What kubeadm needs from a node

 ┌───────────── each node ─────────────┐
 │ kubelet  ──CRI socket──▶ containerd │──▶ runc ──▶ containers
 │   │  (both use systemd cgroups)     │
 │   └─ kubeadm writes its config      │
 │                                     │
 │ kernel: overlay, br_netfilter,      │
 │         ip_forward, no swap         │
 │ clock: systemd-timesyncd            │
 │ name:  k8s-api.home.arpa → .10      │
 └─────────────────────────────────────┘
  • Kernel: overlay for container filesystems, br_netfilter so bridged Pod traffic passes through iptables, and IP forwarding so the node can route Pod traffic.
  • No swap: kubelet’s default policy expects swap to be off.
  • Time sync: etcd and TLS both rely on clocks agreeing.
  • A container runtime: containerd, speaking CRI to kubelet.
  • Matching cgroup drivers: on a systemd host, kubelet and containerd must both use the systemd driver.
  • kubeadm, kubelet, kubectl at the pinned version, held.

Before you start

On the admin machine, install ansible-core (2.21 was used here), OpenSSH and Python. No extra collections are needed; everything is ansible.builtin.

Accept each guest’s host key once, interactively, so Ansible can run with strict host-key checking:

ssh -J kasi@homelab kasi@192.168.122.10
ssh -J kasi@homelab kasi@192.168.122.11
ssh -J kasi@homelab kasi@192.168.122.12

Make sure Ansible can reach all three:

cd k8s/ansible
ansible-inventory -i inventory/k8s-nodes.yml --graph
ansible -i inventory/k8s-nodes.yml k8s_cluster -m ping     # three "pong"s

Run it

ansible-playbook -i inventory/k8s-nodes.yml --syntax-check node-preparation.yml
ansible-playbook -i inventory/k8s-nodes.yml node-preparation.yml --limit k8s_cluster

Run it a second time. A correct playbook reports changed=0 on every node, because every task checks before it acts.

--check --diff is a useful preview, but it can’t simulate everything: it doesn’t download the containerd tarball, so it stops at the unpack step.

The rest of this post goes through what the playbook does, stage by stage, with the equivalent manual commands so you can see what’s underneath.

Stage 0: Make sure it’s the right machine

Before changing anything, the playbook checks that:

  • the hostname is one of the three expected nodes, and the address matches node_ip;
  • the node runs Debian 13 on x86_64, with roughly the RAM the VM was given;
  • /etc/machine-id is valid and different on every node. Cloned VMs can share an ID, which confuses kubelet and DHCP.

It then adds the API name to /etc/hosts and checks name resolution:

echo '192.168.122.10 k8s-api.home.arpa' | sudo tee -a /etc/hosts
getent hosts deb.debian.org k8s-api.home.arpa

If containerd is already installed, the playbook also confirms it’s healthy before touching it.

Stage 1: Kernel modules and sysctls

printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/kubernetes.conf
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes-node.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

The files under /etc/modules-load.d and /etc/sysctl.d make the settings survive a reboot. The modprobe and sysctl --system calls apply them now.

Check it worked:

lsmod | grep -E 'overlay|br_netfilter'
sysctl net.bridge.bridge-nf-call-iptables net.ipv4.ip_forward   # both = 1

Stage 2: Turn swap off for good

sudo swapoff --all
sudo sed -E -i 's|^(\s*[^#\s]\S*\s+\S+\s+swap\s+.*)$|# disabled for Kubernetes: \1|' /etc/fstab

swapoff turns swap off now. Commenting out the swap line in /etc/fstab keeps it off after a reboot.

Check it worked: swapon --show prints nothing.

Stage 3: Time sync

sudo apt-get install -y systemd-timesyncd
sudo systemctl enable --now systemd-timesyncd

Check it worked: timedatectl shows System clock synchronized: yes.

Stage 4: containerd 2.3.6

Debian’s packaged containerd isn’t the version we pinned, so install the upstream release tarball and verify its checksum:

cd /var/cache
sudo curl -LO https://github.com/containerd/containerd/releases/download/v2.3.6/containerd-2.3.6-linux-amd64.tar.gz
echo 'df806bb9b86653ebc94822a7076642ff3c3614324dbc1b511bdf75ac1669b762  containerd-2.3.6-linux-amd64.tar.gz' | sha256sum -c
sudo tar -C /usr/local -xzf containerd-2.3.6-linux-amd64.tar.gz
sudo apt-get install -y runc          # the low-level runtime containerd calls

Generate the config from the binary you just installed, so it matches that version exactly, and switch on systemd cgroups:

sudo mkdir -p /etc/containerd
/usr/local/bin/containerd config default \
  | sed 's/SystemdCgroup = false/SystemdCgroup = true/' \
  | sudo tee /etc/containerd/config.toml >/dev/null

Then install a systemd unit at /etc/systemd/system/containerd.service:

[Unit]
Description=containerd container runtime
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity

[Install]
WantedBy=multi-user.target

Delegate=yes lets containerd manage the cgroups of the containers it starts. KillMode=process means restarting containerd doesn’t kill running containers.

sudo systemctl daemon-reload
sudo systemctl enable --now containerd

Check it worked:

containerd --version                                  # … v2.3.6 …
sudo ctr plugins ls | grep 'io.containerd.grpc.v1.*cri'   # status "ok"

The version is the third field of containerd --version. In 2.x the module path ends in /v2, so match on that field rather than on the whole string.

Stage 5: kubeadm, kubelet and kubectl at exact versions

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubeadm=1.36.5-1.1 kubelet=1.36.5-1.1 kubectl=1.36.5-1.1
sudo apt-mark hold kubeadm kubelet kubectl

The signed-by keyring trusts the Kubernetes key for this repository only, not for every repository on the system.

Check it worked:

dpkg-query -W kubeadm kubelet kubectl     # all 1.36.5-1.1
apt-mark showhold                         # kubeadm kubectl kubelet

Stage 6: kubelet settings that kubeadm won’t overwrite

kubeadm regenerates kubelet’s main config on init, join and upgrade, so put your own settings somewhere it leaves alone: a drop-in directory.

sudo mkdir -p /etc/kubernetes/kubelet.conf.d
cat <<'EOF' | sudo tee /etc/kubernetes/kubelet.conf.d/50-graceful-shutdown.conf
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 20s
shutdownGracePeriodCriticalPods: 10s
EOF

echo 'KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --config-dir=/etc/kubernetes/kubelet.conf.d' \
  | sudo tee /etc/default/kubelet

sudo systemctl enable kubelet          # enabled, but not started yet

/etc/default/kubelet sets kubelet’s cgroup driver to systemd, to match containerd. It also points kubelet at the drop-in directory.

The graceful-shutdown drop-in matters on a control plane. With kubelet’s default grace period of 0s, a poweroff stops every container at once. etcd exits first, and the API server then hangs until systemd kills it about 90 seconds later. With a grace period, kubelet holds a shutdown inhibitor and stops ordinary Pods first, then critical ones such as etcd and the API server. The node powers off in seconds.

kubelet is enabled but left stopped. Until kubeadm init or join gives it a config, it would just restart in a loop. If kubelet is already running (on a node that’s been bootstrapped), the playbook restarts it so the drop-in takes effect. Otherwise it leaves it stopped.

Check it worked: systemctl is-enabled kubelet says enabled, and systemctl is-active kubelet says inactive (before bootstrap).

What the playbook deliberately leaves alone

It doesn’t change guest IP addresses, host networking, VM disks or SSH. It doesn’t touch an existing Node Exporter. It never runs kubeadm init, join or reset, and it never reboots anything. The containerd tarball stays in /var/cache so reruns don’t download it again.

Tying it together

After this playbook:

  • the kernel will pass and forward Pod traffic;
  • containerd serves CRI at /run/containerd/containerd.sock, using systemd cgroups;
  • kubelet is installed, configured for systemd cgroups and graceful shutdown, and waiting for a cluster config;
  • every node resolves k8s-api.home.arpa to the control plane.

That is exactly what kubeadm expects. In part 3, a separate, guarded playbook creates the cluster.

← Part 1Part 3 →
— END OF NOTE —Explore more notes ↗