Cilium and eBPF: Kubernetes networking without iptables

eBPF in one section

The Linux kernel controls CPU, memory, storage and networking, and applications reach it through system calls. Changing what the kernel does has traditionally meant modifying it or loading a kernel module, which is slow and risky.

eBPF (extended Berkeley Packet Filter) lets you load small sandboxed programs into the running kernel. A verifier checks each program is safe before it runs, so it cannot crash the machine. Programs attach to events such as a packet arriving, a system call being made or a function being entered, and can read data, change behaviour, or be replaced on the fly with no reboot.

Typical uses are networking, security enforcement, observability and tracing. XDP is a related hook that runs a program at the earliest point a packet reaches the network driver, which is why it is used for very fast load balancing and filtering.

Cilium as the CNI

A Container Network Interface (CNI) plugin gives each pod an IP address and connects it to the rest of the cluster. Cilium is a CNI whose data path is implemented in eBPF, and which adds identity-aware security, load balancing, encryption and observability on top. It runs as an agent on every node, with an operator handling cluster-wide tasks. In recent versions the Cilium CLI installs and checks it:

cilium install
cilium status --wait
cilium connectivity test

Pod IPs are handed out from ranges per node. In the default cluster-scope mode Cilium's operator manages the pool itself. In Kubernetes host-scope mode each node takes its range from the Node object's podCIDR.

Routing modes

Mode	How it works	Trade-offs
Encapsulation (default)	Pod traffic is wrapped in VXLAN or Geneve tunnels between nodes.	Works on almost any network with only node-to-node connectivity. Adds a header, so the effective MTU shrinks and there is a little CPU cost.
Native routing	Pod traffic is routed as ordinary packets with no tunnel.	Lower overhead and easier to debug, but the underlying network has to know how to route pod CIDRs (cloud routes, BGP or a flat network).

Whenever you use encapsulation, check the MTU. A tunnel header on a full-size packet means fragmentation or drops if the pod MTU is not reduced to match.

Identity-based policy

Traditional firewalls match IP addresses, which is a poor fit when pods come and go constantly. Cilium assigns each group of pods with the same labels a numeric security identity, and policy is written against identities, not addresses. Each pod is a Cilium endpoint that carries its identity, and that identity travels with the packet.

Policies work at three layers:

• Layer 3: who may talk to whom (labels, namespaces, CIDRs, or named entities such as world, host and remote-node).
• Layer 4: which ports and protocols.
• Layer 7: which HTTP methods and paths, or DNS names, are allowed.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: api-read-only }
spec:
  endpointSelector:
    matchLabels: { app: api }
  ingress:
  - fromEndpoints:
    - matchLabels: { app: frontend }
    toPorts:
    - ports: [{ port: "8080", protocol: TCP }]
      rules:
        http:
        - method: GET
          path: "/v1/.*"

Use CiliumClusterwideNetworkPolicy for rules that apply to the whole cluster. Before enforcing a new policy, switch the endpoint to policy audit mode so denied traffic is logged but not dropped. That lets you find mistakes without causing an outage.

Replacing kube-proxy

kube-proxy implements Services by writing iptables rules on each node. iptables evaluates rules as a list, so lookup time grows with the number of Services, and updates rewrite large chunks of the table. On big clusters that costs latency and CPU.

Cilium can replace kube-proxy completely, handling ClusterIP, NodePort and LoadBalancer Services with eBPF hash-map lookups that stay fast no matter how many Services exist. It also handles the case where a pod calls a Service and is directed to a backend, without a chain of NAT rules.

Encryption and service mesh

• Transparent encryption of pod-to-pod traffic between nodes uses IPsec or WireGuard, with no change to applications.
• Mutual TLS authenticates workloads to each other using their identities.
• Cilium's sidecarless service mesh does much of the work of a mesh, such as L7 policy, ingress and Gateway API support, in the node's data path, so no proxy container is injected into every pod.

Observability with Hubble

Hubble is Cilium's observability layer. Because eBPF already sees every packet and every policy decision, Hubble can show you the real network flows without instrumenting applications.

hubble observe --namespace prod --verdict DROPPED
hubble observe --from-pod prod/frontend --protocol http

The Hubble UI draws a live service map of who talks to whom. Both the Cilium agent and Hubble expose Prometheus metrics, so flows, drops and DNS or HTTP rates can be graphed and alerted on in Grafana.

Cluster Mesh

Cluster Mesh joins several Kubernetes clusters into one network. Pods in different clusters can reach each other directly, policies can span clusters, and a global service can load-balance across backends in every cluster, which gives you cross-cluster failover. Each cluster needs a unique name and ID, and non-overlapping pod CIDRs when using native routing. Clusters share state through a small key-value store, and clustermesh-apiserver with KVStoreMesh caches remote state locally so a remote cluster being down does not break local operation.