k_actuallykasi.
← Back to the notebook
Kubernetes4 Nov 20267 min read

Kubernetes on One Box, Part 3: kubeadm init, Cilium and Guard Rails

Create the cluster with a playbook that only runs when you mean it: kubeadm init once, Cilium via Helm, worker joins, and a private kubeconfig.

k_Written by Kasi · Engineering notes

The nodes are prepared (part 2). Now we create the cluster: initialize the control plane, install Cilium as the network plugin, join the two workers, and copy an admin kubeconfig to the Mac.

This lives in its own playbook, cluster-bootstrap.yml. Node preparation can be rerun forever. Cluster creation can’t: kubeadm init mints the cluster’s CA and identity. So this playbook refuses to run unless you explicitly ask it to, and it never resets anything.

The order of operations

 k8s-cp-01                                   workers
 ─────────                                   ───────
 1. render kubeadm config (once)
 2. install pinned Helm
 3. kubeadm init (once)
 4. helm install cilium 1.20.2
 5. kubeadm token create ──── join cmd ────▶ 6. kubeadm join (once each)
 7. wait: nodes Ready, Cilium Ready, CoreDNS Ready
 8. confirm control-plane taint
 9. copy admin.conf ──────────────────────▶ admin Mac (~/.kube, mode 0600)

Each step is guarded by “does this already exist?”, so a second run changes nothing.

Before you run it

  • Part 2’s playbook has run, and a second run reported changed=0.
  • The route check from part 1 found no overlap with 10.244.0.0/16 or 10.96.0.0/12.
  • The versions, the API name and the node addresses in inventory are the ones you want.

Run it

From k8s/ansible:

ansible-playbook -i inventory/k8s-nodes.yml --syntax-check cluster-bootstrap.yml
ansible-playbook -i inventory/k8s-nodes.yml cluster-bootstrap.yml \
  -e bootstrap_authorized=true -e vpn_cidrs_verified=true

Without both flags it stops at its first task:

- name: Require explicit live-bootstrap authorization
  ansible.builtin.assert:
    that:
      - bootstrap_authorized | default(false) | bool
      - not (rebuild_authorized | default(false) | bool)
      - vpn_cidrs_verified | bool

It also insists the inventory is exactly the three expected nodes. There’s no reset or rebuild path in this playbook at all. Rebuilding a node is a separate, manual decision.

What follows is what the playbook does, with the manual equivalent of each step.

Step 1: Render the kubeadm config once

The template from part 1 is rendered on the control plane as /etc/kubernetes/kubeadm-config.yaml, with mode 0600. It is never overwritten. On later runs, the playbook reads it back and confirms the version, endpoint, address, CRI socket and CIDRs still match inventory. If they don’t, it stops and leaves it to you.

Rendered for k8s-cp-01:

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "192.168.122.10"
  bindPort: 6443
nodeRegistration:
  criSocket: "unix:///run/containerd/containerd.sock"
  kubeletExtraArgs:
    - name: node-ip
      value: "192.168.122.10"
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: "v1.36.5"
controlPlaneEndpoint: "k8s-api.home.arpa:6443"
apiServer:
  certSANs:
    - "k8s-api.home.arpa"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  dnsDomain: cluster.local

Check it: sudo kubeadm config validate --config /etc/kubernetes/kubeadm-config.yaml

Step 2: Install a pinned Helm on the control plane

Cilium is installed from an OCI Helm chart, so the control plane needs Helm. Pin it and verify the checksum, like containerd:

curl -LO https://get.helm.sh/helm-v4.2.1-linux-amd64.tar.gz
echo '479dca836e5b45e8bd222400c5591b0e3a647378f03ff96597180db97c17fdae  helm-v4.2.1-linux-amd64.tar.gz' | sha256sum -c
tar -xzf helm-v4.2.1-linux-amd64.tar.gz
sudo install -m 0755 linux-amd64/helm /usr/local/bin/helm

If a different Helm version is already installed, the playbook stops rather than replacing it.

Step 3: Initialize the control plane

sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml

This generates the cluster CA and certificates, writes the static Pod manifests for etcd, the API server, the controller manager and the scheduler, starts kubelet, and writes /etc/kubernetes/admin.conf. The playbook uses that file as its guard: if admin.conf exists, the cluster exists, and init is skipped.

The task runs with no_log: true. kubeadm init prints a join token and a certificate key, and those shouldn’t end up in logs.

Check it worked:

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get --raw=/readyz    # ok
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n kube-system get pods

The node will show NotReady until there’s a network plugin. That’s next.

Step 4: Install Cilium

The values file, k8s/bootstrap/cilium-values.yaml:

kubeProxyReplacement: false        # keep kube-proxy; Cilium is the CNI only
tunnelProtocol: vxlan              # overlay between nodes on UDP 8472
operator:
  replicas: 1                      # small cluster: one operator is enough
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList: [10.244.0.0/16]
    clusterPoolIPv4MaskSize: 24    # each node gets its own /24
rollOutCiliumPods: true
hubble:
  enabled: false

How the pieces fit:

  • IPAM. Cilium’s operator gives each node a /24 from the Pod CIDR, for example 10.244.1.0/24 for one worker and 10.244.2.0/24 for the other. Pods on a node get addresses from that node’s block.
  • VXLAN. Pod-to-Pod traffic between nodes is wrapped in UDP 8472 and sent over the node network (192.168.122.0/24). The LAN never sees Pod addresses.
  • kube-proxy is kept. Service virtual IPs (10.96.x.x) are still handled by kube-proxy’s rules. Cilium can replace kube-proxy with eBPF, but that’s a separate change for later.

Install with the pinned chart version:

sudo helm upgrade --install cilium oci://quay.io/cilium/charts/cilium \
  --version 1.20.2 --namespace kube-system \
  --values /etc/kubernetes/cilium-values.yaml \
  --kubeconfig /etc/kubernetes/admin.conf \
  --wait --timeout 10m

The playbook first runs helm list to check for an existing cilium release. At the pinned version, it skips the install. At any other version, it stops, because Cilium upgrades are their own planned change.

Check it worked:

sudo helm list -n kube-system --kubeconfig /etc/kubernetes/admin.conf   # cilium-1.20.2, deployed
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n kube-system \
  exec ds/cilium -c cilium-agent -- cilium-dbg status --brief          # OK

Step 5: Join the workers

For each worker, /etc/kubernetes/kubelet.conf existing means it has already joined. If either worker hasn’t, the control plane creates a fresh, short-lived join command. The playbook keeps it only in memory:

# on k8s-cp-01
sudo kubeadm token create --print-join-command

# on each worker that hasn't joined, paste what it printed:
sudo kubeadm join k8s-api.home.arpa:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

The join goes to k8s-api.home.arpa:6443, which the worker resolves through the /etc/hosts entry from part 2. The --discovery-token-ca-cert-hash lets the worker confirm it’s talking to this cluster’s CA before it trusts anything it receives.

If both workers have already joined, no token is created at all.

Step 6: Wait until everything is actually up

kubectl wait --for=condition=Ready nodes --all --timeout=10m
kubectl -n kube-system wait --for=condition=Ready pods -l k8s-app=cilium   --all --timeout=10m
kubectl -n kube-system wait --for=condition=Ready pods -l k8s-app=kube-dns --all --timeout=10m

Then confirm the control plane still has its taint, which keeps ordinary workloads off the 3 GiB control-plane node:

kubectl get node k8s-cp-01 -o jsonpath='{.spec.taints}'
# → includes node-role.kubernetes.io/control-plane, effect NoSchedule

Don’t remove this taint on a cluster this small.

Step 7: Get a kubeconfig onto your Mac, privately

The playbook copies admin.conf to ~/.kube/homelab-admin.conf on the controller with mode 0600. It never prints the file, and it never overwrites an existing copy. To do it by hand:

scp -o ProxyJump=kasi@homelab \
  kasi@192.168.122.10:/etc/kubernetes/admin.conf ~/.kube/homelab-admin.conf
chmod 600 ~/.kube/homelab-admin.conf

admin.conf is root-owned on the node, so you may need to sudo cp it to your home directory there first. Treat it like a root password: keep it out of Git and out of CI artifacts.

The node network isn’t routable from the Mac, so reach the API through an SSH tunnel. There are two ways to do it.

Option A. Use the real name, with a hosts entry:

ssh -N -J kasi@homelab -L 6443:192.168.122.10:6443 kasi@192.168.122.10
# /etc/hosts on the Mac:   127.0.0.1  k8s-api.home.arpa
export KUBECONFIG=~/.kube/homelab-admin.conf

Option B. Use a spare local port, with no /etc/hosts change:

ssh -f -N -o ExitOnForwardFailure=yes -J kasi@homelab \
  -L 16443:192.168.122.10:6443 kasi@192.168.122.10
alias k='kubectl --kubeconfig ~/.kube/homelab-admin.conf \
  --server https://127.0.0.1:16443 --tls-server-name k8s-api.home.arpa'

--tls-server-name connects to 127.0.0.1 but checks the certificate against k8s-api.home.arpa, so TLS verification stays fully on.

Check it worked, from the Mac:

kubectl get nodes -o wide
#   k8s-cp-01, k8s-worker-01, k8s-worker-02 — Ready, v1.36.5, containerd://2.3.6
kubectl -n kube-system get ds cilium kube-proxy           # 3/3 each
kubectl -n kube-system get deploy coredns cilium-operator # 2/2 and 1/1

Rerunning

Run the same command again whenever you like. With the cluster in place, every step finds its guard and skips: there’s no init, no join, no token and no Cilium change. The run should report changed=0.

What you have now

  • A kubeadm cluster at k8s-api.home.arpa:6443: one tainted control plane and two workers.
  • Cilium 1.20.2 providing Pod networking over VXLAN, with kube-proxy handling Services and CoreDNS answering at 10.96.0.10.
  • An admin kubeconfig on the Mac, reached over SSH, with nothing exposed to the internet.

In part 4 we use it: Pods talking across nodes, Services, DNS, a NetworkPolicy, and draining a worker.

← Part 2Part 4 →
— END OF NOTE —Explore more notes ↗