How eBPF is changing container networking

“No man is an island,” wrote John Donne. But containers certainly are. They are created as islands in and of themselves, packaged with all they need to operate independently—code, dependencies, and runtime. This autonomy is great for security but not so great for networking.

Just as a person needs community, distributed workloads depend on connections between containers to function as a whole. “In the world of distributed computing, everything goes over the network, making it the critical component for applications to work and work together,” says Bill Mulligan, head of community at Isovalent, the company behind Cilium, an open-source project that provides networking software for Kubernetes and other cloud-native environments.

However, containers have historically required virtual networking to connect workloads, relying on networks composed of software-based network adapters, switches, firewalls, load balancers, etc. that lacked in efficiency, usability, and programmability. For once, the answer is not to abstract—as virtual networks had abstracted from physical networks—but to step down a level deeper into the stack.

eBPF, a technology that allows custom sandboxed code to run in the Linux kernel, opens a doorway to implementing networking inside the operating system, where efficiency, visibility, and control all can be increased. And it is turning out to be the answer to modern software networking in general and container networking in particular.

“Traditional container networking stacks primarily relied on iptables and sidecar proxies to manage network traffic,” says Taranvir Singh, research manager at IDC. “These approaches can become inefficient at large scale due to added complexity and processing overhead.” In contrast, eBPF can process packets and enforce policies directly in the kernel, bringing more resource efficiency and granular control, he says.

Used to create networking capabilities at the kernel level, eBPF is disrupting the status quo by bringing little to no overhead for container networking—without requiring changes to network configurations. “It allows us to build tooling that compensates for the isolation,” says Liz Rice, chief open source officer at Isovalent.

Container networking before eBPF

Before eBPF, container networking lacked a clear approach. “Networking has often been perceived as a somewhat opaque aspect of workloads, particularly within Kubernetes,” says Shane Utt, senior principal software engineer at Red Hat.

Legacy workarounds for container networking brought operational obstacles. “Traditional container networking relies heavily on tools like iptables for managing network traffic, which can become inefficient as the number of containers grows,” says Lin Sun, director of open source at Solo.io, who holds steering positions on the Istio project and the Cloud Native Computing Foundation’s technical oversight committee (TOC).

Traditional practices also incurred duplication and overhead. By design, containers are isolated from each other and the server, requiring a virtual Ethernet connection and a network stack on both sides of the connection. This redundancy adds unnecessary overhead and latency, says Isovalent’s Rice. “With eBPF, we can bypass parts of these superfluous networking stacks,” she says.

Lastly, networking hinged on the deployment environment. “Before eBPF, people were adding performance and flexibility gains through things like kernel bypass (DPDK) or hardware acceleration,” says Isovalent’s Mulligan. “They provided gains but killed the dream of white box networking since performance became highly dependent on where things were running.”

Nowadays, most deployments are migrating to plugins based on the Container Network Interface (CNI) standard. “This standard essentially allows for the deployment of host-level binaries that configure the network according to their specific requirements and preferences, utilizing various underlying technologies,” Utt says. Popular CNI plugins include Flannel, Calico, Weave, and, most recently, the eBPF-based Cilium.

Container networking with eBPF

eBPF is revolutionary because it works at the kernel level. Even though containers on the same host have their own isolated view of user space, says Rice, all containers and the host share the same kernel. Applying networking, observability, or security features here makes them instantly available to all containerized applications with little overhead. “A container doesn’t even need to be restarted, or reconfigured, for eBPF-based tools to take effect,” says Rice.

Because eBPF operates at the kernel level to implement network policies and operations such as packet routing, filtering, and load balancing, it’s better positioned than other cloud-native networking technologies that work in the user space, says IDC’s Singh. ”This helps eBPF to tap into many potential benefits and provide faster packet processing times with low resource overhead, a crucial networking requirement of modern application workloads.” This closeness to the infrastructure brings the added benefit of deep observability without additional monitoring proxies.

“eBPF has proven to improve inter-container networking, and it is particularly helpful in improving network efficiency,” says Solo.io’s Sun. Impressively, Seznam.cz reported a 72x reduction in CPU usage by switching to eBPF-powered load balancing technology. Mulligan predicts the performance benefits of eBPF will become table stakes as companies stretch the scalability of systems with higher-density clusters to support AI training.

The pace of kernel innovation using eBPF is inherently quickened because that work now bypasses the official kernel development timeline, which can take a few years. Not all kernel PRs become merged, either. “Adding functionality to the kernel on the fly adds a lot of programmability to a system that has been slow to innovate (also for good reason) for decades,” says Mulligan.

Others agree that eBPF is helping to advance development at a faster pace. Networking innovations used to be tethered to hardware life cycles on the scale of years. But the software-defined networking (SDN) revolution brought this down to weeks, according to Dave Tucker, senior principal software engineer at Red Hat. “As SDN matured, we were disrupted again by eBPF, so there’s still plenty to be excited about.”

Running eBPF on hardware

eBPF also enables nearly universal use. “Because eBPF is available on every Linux system, its benefits are available to everyone, enabling fast and flexible container networking everywhere,” says Isovalent’s Mulligan. To his point, about 96% of web servers run Linux. Android, which uses the Linux kernel, is the world’s most popular operating system, with 47% of the market share worldwide.

New strategies aim to offload eBPF processing to the hardware, further optimizing networking and benefiting security. As Rice explains, eBPF can be processed on a network interface card (NIC), the hardware component installed locally on computers to enable network connectivity. This uses eXpress Data Path (XDP), a high-performance framework for handling packets.

By offloading packet processing to the device itself, you can efficiently mitigate “ping of death” security vulnerabilities, in which hackers try to exploit a bug in the operating system kernel by sending malformed or malicious packets. Discarding risky code before it reaches the kernel’s networking stack inherently improves isolation and security.

Projects using eBPF for networking

eBPF has made its way into several open-source and commercial projects for container-based networking, security, and observability. Chief among these is Cilium, a graduated Cloud Native Computing Foundation (CNCF) project that has become popular for cloud-native networks.

Rice says Cilium’s graduated status with CNCF and its inclusion in projects like OpenTelemetry and modern security qualifications make it a de facto standard for handling network policies. “You have to pick a CNI, and the standard one people pick these days is Cilium.” Confluent and eBay are two companies that have shared their journeys adopting Cilium for this purpose.

Netkit, a recent improvement to Cilium, enhances the underlying kernel to speed up connections in container networks. Netkit uses eBPF to switch packets from the host to the container or host network, dramatically reducing network overhead for running containers, says Rice. “Netkit makes container networking as fast as host networking, eliminating the virtualization overhead,” adds Mulligan.

With Cilium, eBPF can be leveraged to dynamically apply universal network policies across clusters. For instance, a policy could dictate containers can only connect with other containers of the same type, denying access to others. “We have network security in Cilium,” says Rice. “Now, we’re extending that to control network security wherever your workloads are running.”

“Projects like Cilium have done a phenomenal job at bringing the power of eBPF to Kubernetes networking without fundamentally changing the Kubernetes network model,” says Red Hat’s Tucker. That said, this model may have to be revisited to ensure interoperability as more projects adopt eBPF, he says. “We’ll see more focus on standardization, the same way that CNI emerged for container networking.”

Beyond Cilium, Red Hat’s Utt points to LoxiLB, which uses eBPF to increase the speed and programmability of Kubernetes load balancing, while Sun points to Tigera’s Project Calico, a container networking and security solution that offers a pluggable eBPF data plane. There’s also Prosimo, which uses eBPF to enhance data plane capabilities in its multi-cloud networking platform.

“eBPF is expanding so far beyond its roots in networking to observability, tracing, profiling, security, and so many more,” says Mulligan, citing the growing number of eBPF case studies. “eBPF brings innovation back into the kernel in so many areas, some that might not have been able to make major changes or updates in years or even decades.”

eBPF caveats and concerns

While eBPF presents opportunities for substantial networking performance improvements, these benefits come at the cost of increased complexity, says Utt. “In the short term, eBPF will enhance performance, but this will be accompanied by significant operational challenges in managing containerized workloads.” This could even make container networking on Kubernetes more opaque, he adds.

Proponents behind service meshes like Istio, which uses sidecars to proxy traffic, aren’t fully onboard with eBPF for all use cases. Replacing iptable rules with eBPF has been found to improve latency or throughput by 5% to 10%—a fairly minor improvement for such a significant overhaul. “Running programs in the kernel doesn’t mean it’s always faster,” says Solo.io’s Sun.

“eBPF comes with overhead and complexity that should not be overlooked, such as kernel requirements, which often require newer kernels, additional privileges to run the eBPF programs, and difficulty debugging and troubleshooting when things go wrong,” says Sun. A limited pool of eBPF expertise is available for such troubleshooting, adding to the hesitation. “It is reasonable for service mesh projects to continue using and recommending iptables rules,” she says.

Meta’s use of Cilium netkit across millions of containers shows eBPF’s growing usage and utility. Yet, upgrades to eBPF-based networking won’t happen overnight. Migrations will likely be gradual because network devices cannot be easily swapped at run time. For instance, ByteDance is rolling out netkit across its vast container network, but will retain traditional virtual Ethernet (veth) devices during the transition.

“It makes sense to have a transition period where both virtual devices are supported on existing clusters until the veth-based containers/pods gradually phase out,” says Daniel Borkmann, co-creator of both eBPF and Cilium, founding engineer at Isovalent, and active Linux kernel contributor. To support netkit on Cilium-managed Kubernetes clusters, he recommends applying a per-node configuration. Newly joined nodes can use netkit while older nodes continue using veth until they are fully phased out, he says.

Applying eBPF in observability and security

In addition to networking, eBPF is being tapped for security, observability, and other purposes. Since most of these use cases involve data retrieval, not state changes, they are arguably simpler and easier to enact than networking, says Utt. “It has been a game changer and truly inspiring to witness the growth of eBPF in these kinds of use cases,” says Utt, who contributes to Bpfman, a universal loader for all eBPF programs on a given system.

Others also anticipate great future momentum in this area. “I see eBPF playing an important role in observability, security, and compliance, probably more than networking,” says Sun, who notes the many observability and security or compliance-related eBPF projects populating the CNCF landscape, like Kepler, Pixie, and KubeArmor. Most are at the “sandbox” level, meaning they’re in the early stages and not yet widely adopted, signaling room for growth.

One eBPF-based security project that is maturing quickly is Tetragon, a Cilium sub-project. Tetragon monitors processes, system calls, and network and file activity and filters events against policies directly in the Linux kernel. It can spot behavioral irregularities that might signal nefarious activity, thus informing cybersecurity efforts, and even step in to block policy violations, such as a suspicious file open, as they happen.

“eBPF is not limited to networking,” says Singh, who agrees its programmatic capabilities can benefit multiple use cases across security and observability. “Further integrations can be done with cloud-provider IaaS network services, which can extend eBPF benefits across the whole network stack,” he adds.

Ultimately, eBPF could bring a standard networking layer to heterogeneous networks, such as those involving legacy databases running on virtual machines across multiple clouds.

eBPF and the Game of Life

Containers used to face a significant trade-off. Architecting dynamic networked container ecosystems was possible, yet it incurred dramatic overhead. eBPF and the projects being built on it are changing that. “Cilium continues to push the envelope and is a leader when it comes to the frontier of eBPF networking technologies and moving the Linux kernel forward to support us all,” says Red Hat’s Utt.

“I strongly believe that eBPF strengthens container networking, security, and observability functionality, while simplifying system development and execution” says IDC’s Singh. “These gains lead to wider adoption and improved infrastructure resiliency and responsiveness.”

And the limits of eBPF continue to be pushed. Impressively, Isovalent’s Rice recently demoed that eBPF can run the Game of Life, the famous cellular automaton invented by mathematician John Conway. Given simple initial conditions, Life produces a complex society of “living” organisms.

The ability to run Life is a litmus test for eBPF’s ability to run heterogeneous programs. “It proves that there aren’t limitations for eBPF,” says Rice.

It’s still too early to tell whether eBPF will usurp all previous approaches to networking. “eBPF still has to prove its way of revolutionizing container networking beyond replacing iptables for larger numbers of containers,” says Solo.io’s Sun, reiterating the small performance gains for service meshes and minimal eBPF expertise in the market. “These factors combined made eBPF-related projects still relatively early in the adoption phase.”

While some holdouts remain—due to upfront implementation challenges, a general lack of eBPF expertise, and deep investments in legacy networking—the momentum behind eBPF is undeniable. With strong tooling and industry-wide optimism, eBPF is well on its way to becoming the prevailing model for cloud networking, connecting those individual container “islands” to, as Donne would say, “a piece of the continent.”