k_actuallykasi.
← Back to the notebook
Kubernetes4 Nov 20269 min read

Kubernetes on One Box, Part 4: Networking, Policy and Drains, Hands-On

Watch the new cluster work: Pods talking across nodes, Services and DNS, a Cilium-enforced NetworkPolicy, and draining a worker safely.

k_Written by Kasi · Engineering notes

The cluster from part 3 is up. This part uses a tiny throwaway workload to show, step by step, how the pieces connect:

  • Pod-to-Pod traffic, on the same node and across the VXLAN overlay;
  • Services and cluster DNS;
  • a NetworkPolicy that Cilium actually enforces;
  • draining a worker and watching Pods move.

Everything happens in one namespace, which you delete at the end.

The workload

All the containers use agnhost, the test image from Kubernetes’ own e2e suite. It’s small, it ships with curl and dig, and it can run as a tiny HTTP server. Pin it by digest:

registry.k8s.io/e2e-test-images/agnhost:2.66.1@sha256:6e2eef87d0dc77e070a7549bc5fdfa290c0fefe3359eb33c2ad1eda692fe66a4

            k8s-worker-01                         k8s-worker-02
   ┌──────────────────────────────┐     ┌──────────────────────────────┐
   │ echo-xxxx   10.244.1.x:8080  │     │ echo-yyyy   10.244.2.x:8080  │
   │ client-w1   (access=allowed) │     │ client-w2   (access=allowed) │
   │ client-denied                │     │                              │
   └──────────────────────────────┘     └──────────────────────────────┘
                 Service "echo" (ClusterIP :80) → both echo Pods

1. A namespace with the strictest Pod Security level

apiVersion: v1
kind: Namespace
metadata:
  name: cluster-validation
  labels:
    pod-security.kubernetes.io/enforce: restricted

Under the restricted level, every Pod must run as non-root, drop all capabilities and use the default seccomp profile. That’s a good habit for anything you deploy.

2. echo: two replicas, one on each worker

apiVersion: apps/v1
kind: Deployment
metadata: { name: echo, namespace: cluster-validation }
spec:
  replicas: 2
  selector: { matchLabels: { app.kubernetes.io/name: echo } }
  strategy:
    rollingUpdate: { maxSurge: 0, maxUnavailable: 1 }
  template:
    metadata: { labels: { app.kubernetes.io/name: echo } }
    spec:
      automountServiceAccountToken: false
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          nodeTaintsPolicy: Honor
          labelSelector: { matchLabels: { app.kubernetes.io/name: echo } }
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: echo
          image: registry.k8s.io/e2e-test-images/agnhost:2.66.1@sha256:6e2eef87d0dc77e070a7549bc5fdfa290c0fefe3359eb33c2ad1eda692fe66a4
          args: ["netexec", "--http-port=8080"]     # GET /hostname returns the Pod name
          ports: [{ name: http, containerPort: 8080 }]
          readinessProbe: { httpGet: { path: /healthz, port: http } }
          resources:
            requests: { cpu: 10m, memory: 16Mi }
            limits:   { cpu: 100m, memory: 64Mi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
---
apiVersion: v1
kind: Service
metadata: { name: echo, namespace: cluster-validation }
spec:
  selector: { app.kubernetes.io/name: echo }
  ports: [{ name: http, port: 80, targetPort: http }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: echo, namespace: cluster-validation }
spec:
  minAvailable: 1
  selector: { matchLabels: { app.kubernetes.io/name: echo } }

What each part does:

  • topologySpreadConstraints with DoNotSchedule puts exactly one replica on each worker. nodeTaintsPolicy: Honor leaves the tainted control plane out of the calculation.
  • The PodDisruptionBudget keeps at least one echo Pod running during a drain, so evictions happen one at a time.
  • maxSurge: 0 matters later. When you restart the Deployment, Pods are replaced one at a time with no extras, so the spread rule can put them back on separate workers.

3. Three idle client Pods to run commands from

client-w1 is pinned to worker-01 and client-w2 to worker-02, both labelled access=allowed. client-denied sits on worker-01 without that label. Each one just runs agnhost pause:

apiVersion: v1
kind: Pod
metadata:
  name: client-w1
  namespace: cluster-validation
  labels: { app.kubernetes.io/name: client, access: allowed }
spec:
  nodeSelector: { kubernetes.io/hostname: k8s-worker-01 }
  # …same securityContext, image and small resources as echo…
  containers:
    - name: client
      image: registry.k8s.io/e2e-test-images/agnhost:2.66.1@sha256:6e2eef87d0dc77e070a7549bc5fdfa290c0fefe3359eb33c2ad1eda692fe66a4
      args: ["pause"]

Apply everything:

kubectl apply -f namespace.yaml -f echo.yaml -f clients.yaml
kubectl -n cluster-validation rollout status deploy/echo
kubectl -n cluster-validation get pods -o wide

Check it worked: in the NODE column, one echo Pod should be on each worker. The IP column shows each node’s own /24, such as 10.244.1.x on one worker and 10.244.2.x on the other. That’s Cilium’s per-node IPAM from part 3.

Pod-to-Pod: same node and across nodes

Put the echo Pod IPs in variables:

NS=cluster-validation
kubectl -n $NS get pods -l app.kubernetes.io/name=echo \
  -o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName

From client-w1 on worker-01, call both echo Pods by IP:

kubectl -n $NS exec client-w1 -- curl -s http://<echo-ip-on-worker-01>:8080/hostname
kubectl -n $NS exec client-w1 -- curl -s http://<echo-ip-on-worker-02>:8080/hostname

Each call returns the name of the Pod you addressed. Getting any reply isn’t enough; it has to be the right Pod.

  • The first call stays on one node: Pod → Cilium → Pod.
  • The second crosses nodes. Cilium wraps the packet in VXLAN (UDP 8472) and sends it over 192.168.122.0/24 to worker-02, where it’s unwrapped and delivered.

Repeat from client-w2 to see both directions.

Services: one stable address for both Pods

kubectl -n $NS get svc echo          # note the CLUSTER-IP, e.g. 10.97.x.x
for i in 1 2 3 4 5 6; do
  kubectl -n $NS exec client-w1 -- curl -s http://echo/hostname; echo
done

The replies alternate between the two Pod names. The ClusterIP isn’t assigned to any interface. kube-proxy has programmed rules on every node that rewrite traffic for the ClusterIP to one of the Service’s ready endpoints. To see which endpoints it’s using:

kubectl -n $NS get endpointslices -l kubernetes.io/service-name=echo -o wide

DNS: how http://echo resolved

kubectl -n $NS exec client-w1 -- cat /etc/resolv.conf
#   nameserver 10.96.0.10
#   search cluster-validation.svc.cluster.local svc.cluster.local cluster.local
kubectl -n $NS exec client-w1 -- dig +short kubernetes.default.svc.cluster.local   # 10.96.0.1
kubectl -n $NS exec client-w1 -- dig +short echo.cluster-validation.svc.cluster.local   # the ClusterIP
kubectl -n $NS exec client-w1 -- dig +short +search echo                               # same ClusterIP

kubelet gives every Pod 10.96.0.10, the CoreDNS Service, as its resolver, plus a search path that starts with the Pod’s own namespace. That’s why the short name echo works inside cluster-validation.

NetworkPolicy: Cilium enforcing who can connect

Without any policy, every Pod can reach every other Pod. Confirm that client-denied can reach echo now:

kubectl -n $NS exec client-denied -- curl -s --max-time 5 http://echo/hostname   # works

Now apply a policy that admits only clients labelled access=allowed, and only on TCP 8080:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: echo-allow-labelled-clients
  namespace: cluster-validation
spec:
  podSelector:
    matchLabels: { app.kubernetes.io/name: echo }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app.kubernetes.io/name: client, access: allowed }
      ports: [{ protocol: TCP, port: 8080 }]

kubectl apply -f network-policy.yaml

Once any policy selects the echo Pods, they switch to default-deny for ingress. Only what the policy allows gets through.

Check it worked:

kubectl -n $NS exec client-w1     -- curl -s --max-time 5 http://echo/hostname   # still works
kubectl -n $NS exec client-denied -- curl -s --max-time 5 http://echo/hostname   # hangs, then exits 28
kubectl -n $NS exec client-denied -- dig +short echo.cluster-validation.svc.cluster.local   # DNS still fine

Exit code 28 is curl’s timeout. Cilium drops denied packets silently rather than rejecting them, so from the client a denial looks like a timeout. DNS keeps working because the policy only selects the echo Pods, not CoreDNS.

You can also see enforcement from Cilium’s side. Find the Cilium agent on a worker that runs an echo Pod:

kubectl -n kube-system get pods -l k8s-app=cilium -o wide
kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- cilium-dbg endpoint list
#   the echo endpoint's POLICY (ingress) ENFORCEMENT column now reads "Enabled"

kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- cilium-dbg metrics list \
  | grep 'cilium_drop_count_total' | grep 'Policy denied'
#   the INGRESS counter rises each time client-denied tries again

Remove the policy and client-denied gets through again:

kubectl delete -f network-policy.yaml

Draining a worker

kubectl drain is how you take a node out of service for maintenance: it cordons the node so nothing new lands there, then evicts its Pods so their controllers recreate them elsewhere.

Before draining, there are three things to check.

1. The other worker has room. Compare what the remaining worker can hold with what’s already requested there:

kubectl describe node k8s-worker-02 | sed -n '/Allocatable:/,/Events:/p'

It needs enough memory and CPU for its own Pods plus everything coming off the drained node. On this cluster, worker-01 has about 3.7 GiB allocatable and worker-02 about 4.7 GiB. That’s plenty for test Pods, but not for every future workload at once.

2. Bare Pods have to go first. drain won’t evict a Pod that has no controller, unless you add --force, and then the Pod is simply gone. The client Pods are bare, so delete them yourself:

kubectl -n $NS delete pod -l app.kubernetes.io/name=client

3. Nothing else critical is running. Once CI runners live on this cluster, pause CI before you drain.

Now drain:

kubectl drain k8s-worker-01 --ignore-daemonsets --timeout=5m

--ignore-daemonsets leaves the per-node agents (cilium, cilium-envoy, kube-proxy) where they are, because they belong on every node. The PDB makes drain evict echo Pods one at a time, each replacement Ready before the next eviction.

Check it worked:

kubectl get nodes                                # k8s-worker-01  Ready,SchedulingDisabled
kubectl -n $NS get pods -o wide                  # both echo Pods now on k8s-worker-02
kubectl -n $NS get endpointslices -l kubernetes.io/service-name=echo -o wide
kubectl get pods -A --field-selector=status.phase=Pending   # should be empty

If a replacement Pod sits in ContainerCreating for a while, the other worker is probably pulling its image for the first time. Give it a minute.

Bring the node back:

kubectl uncordon k8s-worker-01

Uncordoning doesn’t move anything back. Spread constraints only apply when a Pod is scheduled, so both echo Pods stay on worker-02. Restart the Deployment to rebalance:

kubectl -n $NS rollout restart deploy/echo
kubectl -n $NS rollout status deploy/echo
kubectl -n $NS get pods -o wide        # one per worker again

Do the same for k8s-worker-02. Never drain the control plane on this cluster: it’s the only one.

Know the limits of one box

A drain moves Pods between VMs. All three VMs share one host, one SSD and one network link. If the host goes down, so does everything. Check how much real headroom you have:

# on the host
free -m
cat /proc/pressure/memory
virsh --connect qemu:///system list --all
ps -eo rss=,args= | grep '[q]emu-system'     # actual memory each VM is using

# on each node
free -m
df -h / /var/lib/containerd /var/lib/kubelet

The VMs are given 12 GiB out of about 15 GiB usable on the host. Host swap doesn’t count as VM capacity. Also remember that the scheduler only counts requests: a Pod with no requests counts as zero, even if it uses a lot.

Clean up

kubectl get nodes                                    # make sure no worker is still cordoned
kubectl delete namespace cluster-validation

The whole picture

Across the four parts:

  1. Decisions as data: versions, CIDRs, the API name and ports, in one Ansible file.
  2. Node preparation: kernel settings, containerd with systemd cgroups, held kubeadm packages, graceful shutdown.
  3. Bootstrap: a guarded kubeadm init, Cilium over VXLAN, token-based joins, and a private kubeconfig.
  4. Hands-on:
    • Pods get per-node addresses and reach each other across the overlay;
    • Services and DNS put stable names in front of them;
    • Cilium enforces policy;
    • drains move workloads within the limits of a single box.

From here the cluster is ready for platform pieces: Argo CD for GitOps, and Actions Runner Controller for running CI on the cluster itself.

← Part 3
— END OF NOTE —Explore more notes ↗