Secure Azure Kubernetes with Advanced Container Networking Services

Microsoft’s Azure Kubernetes Service is designed to take care of much of the day-to-day management of Kubernetes, simplifying a lot of the complexity that goes with building and running large-scale, cloud-native applications. With Azure taking much of the strain, you’re able to concentrate on building out your code and containers.

AKS mainly focuses on the basics of running Kubernetes, but as the platform has grown, users have needed more complex services to offer deeper insights into applications and more advanced security at a much lower level. Kubernetes’ modular nature helps here, with support for sidecars and service meshes to extend the platform and add necessary functionality without requiring changes to your applications.

Introducing ACNS

Advanced Container Networking Services (ACNS) bundles open source tools to add observability and security to AKS. Microsoft handles configuration, simplifying the process of integrating tools such as Grafana and Prometheus with Cilium-enabled nodes using Hubble and eBPF for deep-level integration with the underlying container host systems. Alternatively, you can build nodes around Microsoft’s own Retina Kubernetes observability tool.

For now, the tool supports two linked services: Advanced Network Observability and the new Fully Qualified Domain Name (FQDN) filtering service with an associated DNS proxy. Both take advantage of Azure support for extended Berkeley Packet Filters, with kernel-level access to both Windows and Linux networking.

The observability tools give you real-time data on network performance inside your AKS clusters, helping spot performance problems early, with visualization tools that show interconnections and how services interact. Meanwhile, the DNS tools work with Azure networking to help deploy a set of networking policies that control who and what has access to your service, while at the same time reducing the risk of DNS-based outages. If you’re using Retina for observability, you’ll need to switch to Cilium to use the networking tools in Advanced Container Network Services.

Using domain names to filter access to services makes sense when you’re working with orchestrator-driven environments such as Kubernetes. If you’re using IP addresses to filter, you need to continually update your access tables as nodes are added to an application or are removed. By focusing on domain names, it’s easier to control which hosts have access to which services, inside and outside AKS. The resulting access control tables are easier to read than lists of IP addresses, so secops engineers can quickly identify errors and risks.

Although ACNS is still in preview, it adds enough new features to be worth considering as part of your AKS infrastructure. With a mix of security and platform tools, it’s likely to become an essential component of an Azure cloud-native environment.

Getting started with ACNS domain filtering

Setup requires either of Azure’s command lines: working with Bash in Cloud Shell or the Azure CLI in a desktop terminal session. You need to be logged into the Azure environment you’re configuring if you’re using the CLI. As ACNS is a preview service, you will need the Azure CLI preview extension to enable the necessary feature flags to use the ACNS preview.

Once you have created a resource group for your AKS cluster, you’re ready to set up the cluster and add ACNS services. As part of the AKS flags needed, your cluster must include the appropriate data plane, Cilium if you’re using FQDN’s filtering. Simply add the --enable-acns flag to the aks create command, and you’re ready to go, with support for both observability and FQDN filtering.

If you only want to use the FQDN filtering feature, simply replace --enable-acns with --enable-fqdn-policy. You can add ACNS support to existing AKS clusters by using the same flags with aks update.

The FQDN policies are managed by the Cilium agent running in your node. DNS requests made from a pod are redirected by Cilium to the ACNS DNS proxy. If the request is allowed, the proxy makes a DNS request, gets the results, and passes a valid response to the pod. If the agent blocks a request, no query is made and no response is delivered.

Although ACNS builds on Cilium capabilities, it doesn’t support the complete feature set. You can configure other network policies, but they will be blocked by AKS for now. That does imply potential future expansions for the service, but Microsoft will need to evaluate and test these additional features with its AKS Kubernetes implementation, as well as develop any required supporting features such as the DNS Proxy in this ACNS release. Cilium will cache resolved IP addresses to speed up operations, though the cache is relatively small and will expire the oldest addresses quickly if you’re making a lot of short-lived connections.

Filtering domain names in Cilium is a Layer 7 feature, controlling access to external resources. This gives it priority over other policies, so a DNS block will block access to an endpoint even if other policies would allow your pod to connect. Cilium offers a range of DNS filtering options, configured using either YAML or JSON.

Using ACNS for network observability

ACNS’s observability tools help deliver the network information you need to get the most out of a cloud-native architecture, using either Cilium or Retina to extract metrics from your service hosts and map them to nodes and pods. Data is stored in Kubernetes’ Prometheus format, and you can use a preconfigured Grafana dashboard to visualize the data.

There’s support for Hubble to trace flows within your environment, exploring communications between pods and generating a service connection graph. If you want to use Hubble’s DNS tool, you need to use it with Retina, as it’s not supported with Cilium.

ACNS does not include the Hubble visualization tools. If you want to use them, you will need to implement your own servers, either on-premises or in Azure. If you take the Azure option, this will add to the costs associated with your application, on top of the ACNS usage charges.

Advanced Container Networking Services is part of AKS, but it does come with additional costs. As it’s $0.025 per node per hour, the larger and more complex the application, the more expensive. Your costs will fluctuate as AKS scales up and down, adding and removing nodes in response to demand.

Microsoft has done a lot to simplify installing Kubernetes networking tools in its managed AKS environment. A single command will install Cilium, set up eBPF probes, and configure a Hubble environment.

However, that’s only part of the story, and there’s a need for another layer in ACNS to do the same for configuration, guiding users to the settings that help secure and optimize their applications. Presets don’t need to be complex, perhaps a form to list approved domains and then generate the Cilium configuration for you, or a list of suggested observability configurations based on your application architecture.

For now, you need to put in the work, learn another YAML dialect, and spend the time to get the right probes and the right configurations. It’s not strictly wasted time, as you do end up with improved security and essential performance data, but it would be nice if you could use that time to work with ACNS results to deliver a better Kubernetes application for your users.