From CNI zero to CNI hero

Doug Smith and Tomofumi (Tomo) Hayashi work on OpenShift networking at Red Hat, maintain Multus CNI and sit in the Kubernetes Network Plumbing Working Group. Tomo also wrote reference plugins such as static and the route-override plugin, and Doug wrote Whereabouts, a cluster-wide IPAM plugin. In this one-hour tutorial from KubeCon + CloudNativeCon Europe 2024, Tomo walks through how CNI works end to end, then Doug runs a live lab on a kind cluster.

The core claim: CNI looks big, but the contract is small. A plugin reads environment variables and a JSON config on stdin, does its work, prints a JSON result on stdout and exits with an exit code. Everything else is detail around that.

What CNI does for a pod

Run ip a inside a pod and you’ll see lo and eth0 with an IPv4 (and maybe IPv6) address. CNI put those there. When a pod is created, the CNI plugin creates the interface inside the pod’s network namespace, assigns the address and applies anything else the network needs: nftables or iptables rules, MTU changes and so on. When the pod is deleted, CNI removes the address and interface again.

A pod is not a container. It’s a shared context: a set of Linux namespaces. Several containers in a pod each get their own PID and mount namespaces, but they all share one network namespace. That network namespace is what CNI configures, and it’s why CNI runs only at two moments: pod creation and pod deletion.

CNI is a plugin architecture. The CNCF CNI project ships reference plugins (macvlan, ipvlan, bridge, the host-local IPAM plugin and more), and vendors ship their own, such as Calico.

Who calls the plugin

Kubernetes never runs a CNI plugin directly. The chain goes through the kubelet and the container runtime:

1. kubectl create sends the Pod object to the API server.
2. The kubelet on the chosen node sees it and sends a RunPodSandbox gRPC call over the CRI (Container Runtime Interface).
3. The container runtime (containerd or CRI-O) starts the pod sandbox, which creates the Linux namespaces, including the network namespace.
4. The runtime calls libcni, a library from the CNI project compiled into the runtime.
5. libcni reads the config from the CNI config directory (/etc/cni/net.d) and execs the plugin binary named in it, from the CNI binary directory (/opt/cni/bin).
6. The plugin creates the interface and returns a result. The kubelet picks up the pod’s IP through the CRI PodSandboxStatus call.

So the two things you configure on a node are the CNI config and the CNI binary.

The CNI config

The config is JSON, not YAML. CNI is an independent project that predates Kubernetes and serves other orchestrators too, so it never adopted Kubernetes conventions.

• File names end in .conflist (for example 10-flannel.conflist). Older deployments use .conf, which is deprecated in recent spec versions.
• If the directory holds several configs, containerd and CRI-O use only one: the first in ASCII sort order. ASCII order isn’t quite alphabetical, so check it.
• The format is set by the CNI specification, which has gone through 0.1.0, 0.2.0, 0.3.x, 0.4.0 and 1.0.0 (released in 2021). The syntax is similar across versions but not identical, so a version mismatch is a common source of trouble.

A simple config:

{
  "cniVersion": "1.0.0",
  "name": "mynet",
  "plugins": [
    {
      "type": "ipvlan",
      "master": "eth1",
      "ipam": {
        "type": "host-local",
        "subnet": "10.1.1.0/24"
      }
    }
  ]
}

The nesting carries three kinds of configuration:

ipam is delegation: ipvlan creates the interface without an address and calls host-local, which allocates the IP.

The plugin contract: inputs and outputs

Inputs

• Environment variables carry the per-pod parameters. The most important is CNI_COMMAND: ADD means create the interface, DEL means remove it. The others are CNI_CONTAINERID, CNI_NETNS (the path to the pod’s network namespace), CNI_IFNAME (usually eth0), CNI_ARGS (pod name, namespace and UID from Kubernetes) and CNI_PATH.
• stdin carries the network config: the contents of your .conflist entry, as JSON.

Outputs

• Exit code: 0 for success, non-zero for failure.
• stderr: an error message. The kubelet and runtime capture it, so you can read it with journalctl.
• stdout: on success, the CNI result object, again JSON:

{
  "cniVersion": "1.0.0",
  "interfaces": [
    { "name": "eth0", "mac": "0a:58:0a:01:01:03", "sandbox": "/var/run/netns/cni-1234" }
  ],
  "ips": [
    { "address": "10.1.1.3/24", "gateway": "10.1.1.1", "interface": 0 }
  ]
}

A result can list several interfaces, so each IP carries an interface index. Here "interface": 0 means 10.1.1.3 belongs to eth0. The runtime reads this result and the kubelet records the pod IP from it.

Plugin chaining and prevResult

plugins is a list, so a config can run several plugins in sequence. With ipvlan followed by tuning, ipvlan creates eth0 and assigns the address, then tuning adjusts that interface, for example setting sysctls such as net.core.somaxconn or turning on promiscuous mode.

Plugins in a chain run one after another, never in parallel. libcni rewrites the stdin config for every plugin after the first: it injects the previous plugin’s output as a prevResult field. That’s how tuning knows which interface to work on without being told in its own config.

Capabilities and runtimeConfig

Some settings come from the pod, not the node. Pods can carry annotations like kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth, and something has to turn those into shaping on the interface. That’s the job of capabilities and runtimeConfig.

A Calico node config, for example, chains three plugins:

1. calico creates the interface and assigns the IP.
2. portmap declares "capabilities": {"portMappings": true}.
3. bandwidth declares "capabilities": {"bandwidth": true}.

When the kubelet creates a pod with bandwidth annotations, it passes those values to libcni as runtime config. libcni checks each plugin’s declared capabilities and injects a matching runtimeConfig block into that plugin’s stdin. Just before bandwidth runs, its config holds both the prevResult from earlier plugins and something like "runtimeConfig": {"bandwidth": {"ingressRate": ..., "egressRate": ...}}.

Capabilities and runtimeConfig aren’t in SPEC.md. They’re an optional convention documented in CONVENTIONS.md in the CNI repo.

Writing your own plugin

The job is the contract above: parse the environment and stdin config, create the interface in the pod’s network namespace (usually through the Linux netlink API), and print the result JSON. Simple to state; the details take care.

Talking to Kubernetes. CNI isn’t integrated with Kubernetes, so a plugin gets no API access by default. If it needs Pod objects, it needs its own credentials (a service account or kubeconfig), and you have to keep them current as certificates expire.

DEL must not fail. The spec says DEL should not return an error, even when something goes wrong, because runtimes aren’t built to handle a failed delete. Runtimes may also call DEL more than once for the same pod. The first call removes the interface; the second finds nothing to remove and still has to exit 0.

The result changes shape by version. The plugin must print the result in the format of the cniVersion it was given:

Debugging CNI on a node

Doug’s checklist, from running into all of these:

The readiness signal is a blunt instrument. Writing a config file marks the node ready even if the plugin can’t actually work yet, which is exactly what happens with Flannel in the lab below. On OpenShift, Doug and Tomo’s operators run checks that the network is really up before they write the config.

Try it yourself

The lab is in Doug’s repo, dougbtv/cni-hero-hands-on, with a recording on asciinema. You need Docker, kind and kubectl. On kind, each node is a Docker container, so docker exec into it stands in for SSHing to a host.

1. Create a cluster without a CNI

git clone https://github.com/dougbtv/cni-hero-hands-on
cd cni-hero-hands-on
kind create cluster --name cni-demo --config kind-config.yaml

kind-config.yaml sets networking.disableDefaultCNI: true, so the cluster comes up like a vanilla one, with a control-plane node and a worker.

How to check it worked: kubectl get nodes lists cni-demo-control-plane and cni-demo-worker as NotReady, and the CNI config directory is empty:

kubectl get nodes
docker exec -it cni-demo-worker ls -la /etc/cni/net.d/

2. Install Flannel

kubectl create -f https://github.com/flannel-io/flannel/releases/download/v0.24.3/kube-flannel.yml

How to check it worked: kubectl get nodes now shows both nodes Ready, and the worker has a config and a binary:

docker exec -it cni-demo-worker ls -la /etc/cni/net.d/   # 10-flannel.conflist
docker exec -it cni-demo-worker ls -la /opt/cni/bin/     # flannel

3. Find out why pods still don’t start

The nodes say Ready, but the CoreDNS pods stay in ContainerCreating. Read the events:

kubectl get pods -n kube-system
kubectl describe pod -n kube-system <coredns-pod-name>

The events show failed to delegate add: failed to find plugin "bridge" in path [/opt/cni/bin]. Flannel delegates the interface work to the reference bridge plugin, which isn’t installed. You can see the same error in the node’s logs:

docker exec -it cni-demo-worker bash -c 'journalctl -u kubelet | grep -i error; journalctl -u containerd | grep -i error'

4. Install the reference CNI plugins

kubectl create -f reference-cni-plugins.yml

This DaemonSet tolerates every taint and uses hostNetwork: true, so it can run before pod networking works. Its init container copies the reference plugins into each node’s /opt/cni/bin.

How to check it worked: docker exec -it cni-demo-worker ls /opt/cni/bin now lists bandwidth, bridge and the other reference plugins, and kubectl get pods -n kube-system shows CoreDNS Running. Start the sample pod and look inside:

kubectl create -f sample-pod.yml
kubectl exec -it sample-pod -- ip a

You should see eth0 with an address in Flannel’s 10.244.0.0/16 range. Then remove it with kubectl delete pod sample-pod.

5. Write a CNI plugin in bash

Open a shell on the worker node:

docker exec -it cni-demo-worker /bin/bash

Create the “binary”. It logs every variable it gets, logs stdin, and prints a valid but fake result:

cat >/opt/cni/bin/dummy <<'EOF'
#!/bin/bash
logit () {
  echo "$1" >> /tmp/cni-inspect.log
}

logit "-------------- CNI call for $CNI_COMMAND on $CNI_CONTAINERID"
logit "CNI_COMMAND: $CNI_COMMAND"
logit "CNI_CONTAINERID: $CNI_CONTAINERID"
logit "CNI_NETNS: $CNI_NETNS"
logit "CNI_IFNAME: $CNI_IFNAME"
logit "CNI_ARGS: $CNI_ARGS"
logit "CNI_PATH: $CNI_PATH"
logit "-- cni config"
stdin=$(cat /dev/stdin)
logit "$stdin"
# Just a valid response with fake info.
echo '{
  "cniVersion": "0.3.1",
  "interfaces": [
      {
          "name": "eth0",
          "sandbox": "'"$CNI_NETNS"'"
      }
  ],
  "ips": [
      {
          "version": "4",
          "address": "192.0.2.22/24",
          "gateway": "192.0.2.1",
          "interface": 0
      }
  ]
}'
EOF
chmod +x /opt/cni/bin/dummy

Give it a config that sorts before 10-flannel.conflist, so it takes over without removing Flannel’s:

cat >/etc/cni/net.d/00-dummy.conf <<EOF
{
    "cniVersion": "0.3.1",
    "name": "my_dummy_network",
    "type": "dummy"
}
EOF
exit

"type": "dummy" is what makes libcni exec /opt/cni/bin/dummy.

6. Watch it run

kubectl create -f sample-pod.yml
docker exec -it cni-demo-worker tail -n 9 /tmp/cni-inspect.log

How to check it worked: the log shows an ADD call with CNI_NETNS pointing at /var/run/netns/cni-..., CNI_IFNAME: eth0, CNI_ARGS holding K8S_POD_NAMESPACE=default;K8S_POD_NAME=sample-pod and the config JSON from stdin.

Now compare what Kubernetes believes with what the pod has:

kubectl get pod sample-pod -o wide          # IP column: 192.0.2.22
kubectl exec -it sample-pod -- ip a          # only lo, no eth0

The API server shows 192.0.2.22 because that’s what the plugin printed. The pod only has a loopback, because the plugin never created anything. The IP you see in kubectl is whatever the CNI result said.

7. Restore Flannel and clean up

kubectl delete pod sample-pod
docker exec -it cni-demo-worker rm /etc/cni/net.d/00-dummy.conf
kubectl create -f sample-pod.yml

How to check it worked: kubectl get pod sample-pod -o wide shows a 10.244.x.x address, and kubectl exec -it sample-pod -- ip a shows eth0 with that same address. When you’re done, run kind delete cluster --name cni-demo.

Q&A notes

Windows. The basics carry over. Some reference plugins (macvlan, ipvlan) are Linux-only, and Windows has a smaller set. Find the CNI binary and config directories from the kubelet configuration on the Windows node, then follow the same steps, with the bash plugin rewritten in PowerShell.

DHCP leases. CNI runs only on add and delete, so the dhcp IPAM plugin relies on a separate DHCP daemon running on each node, typically as a DaemonSet. The daemon handles lease renewals for the life of the pod, and the plugin asks it for or releases a lease on ADD and DEL.

Where to read more

• cni.dev renders all the CNI docs, including every reference plugin’s parameters. Only three or four fields are common to every plugin; the rest is per plugin.
• containernetworking/cni holds libcni, debug tools and the two documents to read next: SPEC.md and CONVENTIONS.md.
• containernetworking/plugins holds the reference plugins: bridge, macvlan, ipvlan, IPAM plugins like host-local, dhcp and static, and chained plugins such as tuning, portmap and bandwidth.

How it ties together

• The kubelet asks the container runtime for a pod sandbox over CRI; the runtime creates the network namespace and calls libcni.
• libcni picks one config file, the first in ASCII order in /etc/cni/net.d, and execs the binary named by each plugin’s type from /opt/cni/bin.
• Every plugin follows the same contract: environment variables + stdin JSON in; exit code, stderr and a result JSON out.
• Chaining runs plugins in order, passing each one the last result as prevResult. Capabilities let libcni inject per-pod runtimeConfig from Kubernetes annotations.
• DEL must succeed and tolerate repeats, and the result must match the requested cniVersion.
• The pod IP in the Kubernetes API is just what the plugin reported, and node readiness is just the presence of a config file. Both are worth remembering when debugging.

Sources: the talk on YouTube and the hands-on repo.