← Back to blog
Networking4 min read

Cilium and eBPF on bare-metal Kubernetes: networking without a cloud load balancer

kubernetesciliumebpfnetworkingon-prem

When you build Kubernetes in a cloud, type: LoadBalancer is a solved problem — the cloud hands you an IP and you're done. On bare metal you don't get that. No cloud controller, no external IP, just raw servers, a switch and your own network. I built a production on-prem platform on RKE2, and here's what actually works.

eBPF and why it matters

eBPF lets you run small sandboxed programs directly inside the Linux kernel — no kernel patching, no kernel modules. For networking this means you can process packets in the kernel data path instead of chaining dozens of iptables rules through userspace logic. Cilium is a CNI built exactly on this.

Cilium replaces kube-proxy

Classic kube-proxy programs service routing through iptables. With thousands of Services that becomes a linearly long rule chain the kernel walks on every connection — an O(n) lookup that hurts at scale. Cilium replaces this with eBPF hash maps: backend lookup is effectively O(1) and programming the rules is orders of magnitude faster.

An important detail from practice: the mode is configured via kubeProxyReplacement: true (or false). The old values strict, partial and disabled were removed in Cilium 1.16. If you're migrating from two-year-old guides you'll hit a validation error. true means the agent bails out on startup if the kernel lacks the required features — which is exactly what you want to see on production immediately, not silently.

Kube-proxy replacement isn't just a performance thing — it's a prerequisite for L2 Announcements below.

The bare-metal problem: where does the external IP come from?

Without a cloud, a type: LoadBalancer Service stays <pending> forever. Nothing assigns an IP. Historically you solved this with MetalLB. Today, if Cilium is already running with kube-proxy replacement, you don't need another component — Cilium does it itself.

LB IPAM handles IP assignment. You define a CiliumLoadBalancerIPPool with a range from your network, and Cilium automatically assigns an IP to every LoadBalancer Service from it. Choose the pool carefully — the addresses must live in the same L2 segment as the nodes and must not collide with DHCP or existing static IPs. This is the most common source of mysterious conflicts.

L2 Announcements handles reachability. An assigned IP is useless if the switch doesn't know about it. A CiliumL2AnnouncementPolicy tells Cilium to answer ARP (IPv4) and NDP (IPv6) queries for Service VIPs. Watch out — both externalIPs and loadBalancerIPs default to false, so an empty policy announces nothing. And don't forget --devices (Helm), or Cilium won't know which interface to ARP on.

L2 Announcements gotchas

This cost me the most time:

  • Only one node ever responds. For a given Service IP exactly one node holds the ARP, chosen via a Kubernetes lease (cilium-l2announce-<namespace>-<service> in kube-system). It behaves like keepalived — no load balancing in front of the cluster, one node is the north/south gateway.
  • Failover has timing. It's governed by leaseDuration (default 15s), leaseRenewDeadline (5s) and leaseRetryPeriod (2s). When a leader dies the outage window is on the order of seconds, not milliseconds. If you expect sub-second failover, this isn't BGP.
  • Switch and ARP cache. A gratuitous ARP is sent on failover, but some switches hold the old MAC in cache longer than they should. A brief blip after a leader switch is normal.
  • externalTrafficPolicy: Local is incompatible. Packets drop on nodes without a backing pod. Keep Cluster unless you know exactly what you're doing.

KubeVIP for an HA control plane

LB IPAM covers application services, but the control plane needs its own highly available VIP — one stable endpoint for the API server across multiple masters. For that I use KubeVIP in ARP mode. A leader is elected via a Kubernetes lease, takes the VIP and sends a gratuitous ARP. When a master dies, the next leader inherits the VIP — the same pattern as L2 Announcements.

KubeVIP also does BGP mode, where every node advertises the VIP as a /32 route, giving you ECMP and routable failover across L3 boundaries. If you have BGP-capable top-of-rack routers, that's better. I stayed on ARP — simpler, no dependency on the network team, and perfectly sufficient for a control plane.

Network policies and Hubble

Once Cilium is running you get CiliumNetworkPolicy for free. On top of standard L3/L4 policies it also does L7 — allow only GET /api, for example, and drop the rest of the HTTP. Native Kubernetes can't do that.

And Hubble is the reason you'd want Cilium even for nothing else. It's an observability layer over the eBPF data path — you see in real time which pod talks to which, what a policy drops and why. Debugging "why is this connection failing" turns from guessing into looking. When writing network policies it's invaluable.

Wrap-up

On bare metal you get a fully working type: LoadBalancer without a cloud and essentially without extra components: Cilium with kube-proxy replacement, LB IPAM for IP assignment, L2 Announcements for ARP, and KubeVIP for an HA control plane. The biggest traps aren't in the config — they're in how your switch and its ARP cache behave. Keep Hubble handy, and expect L2 failover to be seconds, not instant.