networking | Ronak Nathani

How a Kubernetes Pod Gets an IP Address

Fri, 21 Aug 2020 13:04:27 -0700

One of the core requirements of the Kubernetes networking model is that every pod should get its own IP address and that every pod in the cluster should be able to talk to it using this IP address. There are several network providers (flannel, calico, canal, etc.) that implement this networking model.

As I started working on Kubernetes, it wasn’t completely clear to me how every pod is assigned an IP address. I understood how various components worked independently, however, it wasn’t clear how these components fit together. For instance, I understood what CNI plugins were, however, I didn’t know how they were invoked. So, I wanted to write this post to share what I have learned about various networking components and how they are stitched together in a kubernetes cluster for every pod to receive an IP address.

There are various ways of setting up networking in kubernetes and various options for a container runtime. For this post, I will use Flannel as the network provider and Containerd as the container runtime. Also, I am going to assume that you know how container networking works and only share a very brief overview below for context.

Some Background Concepts

Container Networking: A Very Brief Overview

There are some really good posts explaining how container networking works. For context, I will go over a very high level overview here with a single approach that involves linux bridge networking and packet encapsulation. I am skipping details here as container networking deserves a blog post of itself. Some of the posts that I have found to be very educational in this space are linked in the references below.

Containers on the same host

One of the ways containers running on the same host can talk to each other via their IP addresses is through a linux bridge. In the kubernetes (and docker) world, a veth (virtual ethernet) device is created to achieve this. One end of this veth device is inserted into the container network namespace and the other end is connected to a linux bridge on the host network. All containers on the same host have one end of this veth pair connected to the linux bridge and they can talk to each other using their IP addresses via the bridge. The linux bridge is also assigned an IP address and it acts as a gateway for egress traffic from pods destined to different nodes.

Containers on different hosts

One of the ways containers running on different hosts can talk to each other via their IP addresses is by using packet encapsulation. Flannel supports this through vxlan which wraps the original packet inside a UDP packet and sends it to the destination.

In a kubernetes cluster, flannel creates a vxlan device and some route table entries on each of the nodes. Every packet that’s destined for a container on a different host goes through the vxlan device and is encapsulated in a UDP packet. On the destination, the encapsulated packet is retrieved and the packet is routed through to the destined pod.

NOTE: This is just one of the ways how networking between containers can be configured.

What Is CRI?

CRI (Container Runtime Interface) is a plugin interface that allows kubelet to use different container runtimes. Various container runtimes implement the CRI API and this allows users to use the container runtime of their choice in their kubernetes installation.

What is CNI?

CNI project includes a spec to provide a generic plugin-based networking solution for linux containers. It also consists of various plugins which perform different functions in configuring the pod network. A CNI plugin is an executable that follows the CNI spec and we’ll discuss some plugins in the post below.

Assigning Subnets To Nodes For Pod IP Addresses

If all pods are required to have an IP address, it’s important to ensure that all pods across the entire cluster have a unique IP address. This is achieved by assigning each node a unique subnet from which pods are assigned IP addresses on that node.

Node IPAM Controller

When nodeipam is passed as an option to the kube-controller-manager’s --controllers command line flag, it allocates each node a dedicated subnet (podCIDR) from the cluster CIDR (IP range for the cluster network). Since these podCIDRs are disjoint subnets, it allows assigning each pod a unique IP address.

A kubernetes node is assigned a podCIDR when the node first registers with the cluster. To change the podCIDR allocated to nodes in a cluster, nodes need to be de-registered and then re-registered with any configuration changes first applied to the kubernetes control plane. podCIDR for a node can be listed using the following command.

$ kubectl get no <nodeName> -o json | jq '.spec.podCIDR'
10.244.0.0/24

Kubelet, Container Runtime and CNI Plugins - how it’s all stitched together

When a pod is scheduled on a node, a lot of things happen to start up a pod. In this section, I’ll only focus on the interactions that relate to configuring network for the pod.

Once a pod is scheduled on the node, the following interactions result in configuring the network and starting the application container.

Ref: Containerd cri plugin architecture

Interactions between Container Runtime and CNI Plugins

Every network provider has a CNI plugin which is invoked by the container runtime to configure network for a pod as it’s started. With containerd as the container runtime, Containerd CRI plugin invokes the CNI plugin. Every network provider also has an agent that’s installed on each of the kubernetes node to configure pod networking. When the network provider agent is installed, it either ships with the CNI config or it creates one on the node which is then used by the CRI plugin to figure out which CNI plugin to call.

The location for the CNI config file is configurable and the default value is /etc/cni/net.d/<config-file>. CNI plugins need to be shipped on every node by the cluster administrators. The location for CNI plugins is configurable as well and the default value is /opt/cni/bin.

In case of containerd as the container runtime, path for CNI configuration and CNI plugin binaries can be specified under [plugins."io.containerd.grpc.v1.cri".cni] section of the containerd config.

Since we are referring to Flannel as the network provider here, I’ll talk a little about how Flannel is set up. Flanneld is the Flannel daemon and is typically installed on a kubernetes cluster as a daemonset with install-cni as an init container. The install-cni container creates the CNI configuration file - /etc/cni/net.d/10-flannel.conflist - on each node. Flanneld creates a vxlan device, fetches networking metadata from the apiserver and watches for updates on pods. As pods are created, it distributes routes for all pods across the entire cluster and these routes allow pods to connect to each other via their IP addresses. For details on how flannel works, I recommend the linked references below.

The interactions between Containerd CRI Plugin and CNI plugins can be visualized as follows:

As described above, kubelet calls the Containerd CRI plugin in order to create a pod and Containerd CRI plugin calls the CNI plugin to configure network for the pod. The network provider CNI plugin calls other base CNI plugins to configure the network. The interactions between CNI plugins are described below.

Interactions Between CNI Plugins

There are various CNI plugins that help configure networking between containers on a host. For this post, we will refer to 3 plugins.

Flannel CNI Plugin

When using Flannel as the network provider, the Containerd CRI plugin invokes the Flannel CNI plugin using the CNI configuration file - /etc/cni/net.d/10-flannel.conflist.

$ cat /etc/cni/net.d/10-flannel.conflist
{
"name": "cni0",
"plugins": [
{
"type": "flannel",
"delegate": {
"ipMasq": false,
"hairpinMode": true,
"isDefaultGateway": true
}
}
]
}

The Fannel CNI plugin works in conjunction with Flanneld. When Flanneld starts up, it fetches the podCIDR and other network related details from the apiserver and stores them in a file - /run/flannel/subnet.env.

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

The Flannel CNI plugin uses the information in /run/flannel/subnet.env to configure and invoke the bridge CNI plugin.

Bridge CNI Plugin

Flannel CNI plugin calls the Bridge CNI plugin with the following configuration:

{
"name": "cni0",
"type": "bridge",
"mtu": 1450,
"ipMasq": false,
"isGateway": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.0.0/24"
}
}

When Bridge CNI plugin is invoked for the first time, it creates a linux bridge with the "name": "cni0" specified in the config file. For every pod, it then creates a veth pair - one end of the pair is in the container’s network namespace and the other end is connected to the linux bridge on the host network. With Bridge CNI plugin, all containers on a host are connected to the linux bridge on the host network.

After configuring the veth pair, Bridge plugin invokes the host-local IPAM CNI plugin. Which IPAM plugin to use can be configured in the CNI config CRI plugin uses to call the flannel CNI plugin.

Host-local IPAM CNI plugins

The Bridge CNI plugin calls the host-local IPAM CNI plugin with the following configuration:

{
"name": "cni0",
"ipam": {
"type": "host-local",
"subnet": "10.244.0.0/24",
"dataDir": "/var/lib/cni/networks"
}
}

Host-local IPAM (IP Address Management) plugin returns an IP address for the container from the subnet and stores the allocated IP locally on the host under the directory specified under dataDir - /var/lib/cni/networks/<network-name=cni0>/<ip>. /var/lib/cni/networks/<network-name=cni0>/<ip> file contains the container ID to which the IP is assigned.

When invoked, the host-local IPAM plugin returns the following payload

{
"ip4": {
"ip": "10.244.4.2",
"gateway": "10.244.4.3"
},
"dns": {}
}

Summary

Kube-controller-manager assigns a podCIDR to each node. Pods on a node are assigned an IP address from the subnet value in podCIDR. Because podCIDRs across all nodes are disjoint subnets, it allows assigning each pod a unique IP address.

Kubernetes cluster administrator configures and installs kubelet, container runtime, network provider agent and distributes CNI plugins on each node. When network provider agent starts, it generates a CNI config. When a pod is scheduled on a node, kubelet calls the CRI plugin to create the pod. In containerd’s case, Containerd CRI plugin then calls the CNI plugin specified in the CNI config to configure the pod network. And all of this results in a pod getting an IP address.

It took me a while to understand all the interactions and the details involved. I hope this helped you in improving your understanding of how kubernetes works. If you think I got something wrong, please let me know via twitter or email me at hello@ronaknathani.com. If you’d like to discuss something in this post or anything else, feel free to reach out. I’d love to hear from you!

References

Container Networking

How Flannel Works

CRI and CNI

Kubernetes NodePort and iptables rules

Wed, 08 Jul 2020 14:36:33 -0700

I have been working on kubernetes over the last few months and having fun learning about the underlying systems. When I started using kubernetes services, I wanted to learn about the iptables rules that kube-proxy creates to enable them, however, I didn’t exactly know where to start. While there are some really good posts explaining kubernetes networking and how the concept of services works, I couldn’t easily find a starting point to explore these iptables except reading the code or trying to make sense of the output of iptables-save command (which I can’t).

In this post, I share some of what I have learned by digging a little deeper into the iptables rules for NodePort type services and share answers to the following questions I came across while working on kube-proxy:

What happens when a non-kubernetes process starts using a port that’s allocated as a NodePort to a service?
Does the service endpoint continue to route traffic to pods if kube-proxy (configured in iptables mode) process dies on the node?

Some background

Kubernetes allows a few ways to expose applications to the world outside the kubernetes cluster through the concept of services. In our setup at work, we expose a few of our apps through NodePort type services. Also, kube-proxy is the kubernetes component that powers the services concept and we run it in iptables mode (default).

A little about NodePort

NodePort is one of the ways of exposing Kubernetes Services to the world outside the kubernetes cluster. As per Kubernetes docs,

If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767). Each node proxies that port (the same port number on every Node) into your Service.

This means that if we have a service with NodePort 30000, a request to <kubernetes-node-ip>:30000 will get routed to our app. Under normal circumstances kube-proxy binds and listens on all NodePorts to ensure these ports stay reserved and no other processes can use them.

err…a non-kubernetes is using the NodePort!!

A few weeks back at work, one of our kubernetes nodes was rebooted and as it was added back to the cluster, we saw errors in kube-proxy logs - bind: address already in use:

I0707 20:57:38.648179 1 proxier.go:701] Syncing iptables rules
E0707 20:57:38.679876 1 proxier.go:1072] can't open "nodePort for default/azure-vote-front:" (:30450/tcp), skipping this nodePort: listen tcp :30450: bind: address already in use

This error meant that a port allocated as a NodePort to a service was already in use by another process. We found out that there was a non-kubernetes process using this NodePort as a client port to connect to a remote server. This happened due of a race condition post node reboot. This other process started using the NodePort before kube-proxy could bind and listen on the port to reserve it.

Will the NodePort on this node still route traffic to the pods?

This was the immediate question that popped up in our heads as the NodePort allocated to a service was now being used by a non-kubernetes process. One of my colleagues, Alex Dudko pointed out that kube-proxy creates iptables rules in the PREROUTING chain in nat table.

Because of these rules, the answer to the above question is - Yes! Traffic sent to the NodePort on this node will still correctly be routed to the backend pods it targets.

The NodePort would also continue to work if another process was listening on the port as a server and not just using it as a client. Though, would our HTTP GET requests reach this server? More on this later.

Kube-proxy iptables rules

In iptables mode, kube-proxy creates iptables rules for kubernetes services which ensure that the request to the service gets routed (and load balanced) to the appropriate pods.

These iptables rules also help answer the second question mentioned above. As long as these iptables rules exist, requests to services will get routed to the appropriate pods even if kube-proxy process dies on the node. Endpoints for new services won’t work from this node, however, since kube-proxy process won’t create the iptables rules for it.

Rules in the PREROUTING chain

As outlined in this flow chart, there are rules in the PREROUTING chain of the nat table for kubernetes services. As per iptables rules evaluation order, rules in the PREROUTING chain are the first ones to be consulted as a packet enters the linux kernel’s networking stack.

Rules created by kube-proxy in the PREROUTING chain help determine if the packet is meant for a local socket on the node or if it should be forwarded to a pod. It is these rules that ensure that the request to <kubernetes-node-ip>:<NodePort> continue to get routed to pods even if the NodePort is in use by another process.

Below is an over-simplified diagram that demonstrates this:

In rest of this post, I walk through a setup that reproduces the above scenario and examine the iptables rules that make this work.

Setting up a Kubernetes cluster

I created a kubernetes cluster using Azure Kubernetes Service .

Creating pods and a NodePort Service

From the AKS docs, I also created a deployment for an app with a frontend and a backend. I updated the azure-vote-front to type: NodePort service.

azure-vote-front pod in the cluster

$ kubectl get po -l app=azure-vote-front --show-labels -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
azure-vote-front-5bc759676c-x9bwg 1/1 Running 0 24d 10.244.0.11 aks-nodepool1-22391620-vmss000000 <none> <none> app=azure-vote-front,pod-template-hash=5bc759676c

azure-vote-front service targeting the above pod

$ kubectl get svc azure-vote-front -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
azure-vote-front NodePort 10.0.237.3 <none> 80:30450/TCP 24d app=azure-vote-front

The above output shows that the port 30450 is assigned as the NodePort to our service. And we can send HTTP requests to the app from host network:

$ curl -IX GET localhost:30450
HTTP/1.1 200 OK
Server: nginx/1.13.7
Date: Wed, 08 Jul 2020 00:02:08 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 950
Connection: keep-alive
$ curl -sSL localhost:30450 | grep title
<title>Azure Voting App</title>

Examining iptables rules for a NodePort service

Note: AKS doesn’t allow ssh’ing into an AKS worker node directly, instead, a pod needs to be spun up to ssh into the worker node. Instructions here.

Let’s look at the iptables rules for the azure-vote-front service.

After ssh’ing into the kubernetes worker node, let’s look at the PREROUTING chain in the nat table. (PREROUTING chain exists in raw, nat and mangle tables, however, kube-proxy only creates PREROUTING chain rules in nat table)

$ sudo iptables -t nat -L PREROUTING | column -t
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL

There’s a KUBE-SERVICES chain in the target that’s created by kube-proxy. Let’s list the rules in that chain.

$ sudo iptables -t nat -L KUBE-SERVICES -n | column -t
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.202.188 /* kube-system/healthmodel-replicaset-service: cluster IP */ tcp dpt:25227
KUBE-SVC-KA44REDDIFNWY4W2 tcp -- 0.0.0.0/0 10.0.202.188 /* kube-system/healthmodel-replicaset-service: cluster IP */ tcp dpt:25227
KUBE-MARK-MASQ udp -- !10.244.0.0/16 10.0.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- 0.0.0.0/0 10.0.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- 0.0.0.0/0 10.0.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.190.160 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:80
KUBE-SVC-XGLOHA7QRQ3V22RZ tcp -- 0.0.0.0/0 10.0.190.160 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:80
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.195.6 /* kube-system/metrics-server: cluster IP */ tcp dpt:443
KUBE-SVC-LC5QY66VUV2HJ6WZ tcp -- 0.0.0.0/0 10.0.195.6 /* kube-system/metrics-server: cluster IP */ tcp dpt:443
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.129.119 /* default/azure-vote-back: cluster IP */ tcp dpt:6379
KUBE-SVC-JQ4VBJ2YWO22DDZW tcp -- 0.0.0.0/0 10.0.129.119 /* default/azure-vote-back: cluster IP */ tcp dpt:6379
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.237.3 /* default/azure-vote-front: cluster IP */ tcp dpt:80
KUBE-SVC-GHSLGKVXVBRM4GZX tcp -- 0.0.0.0/0 10.0.237.3 /* default/azure-vote-front: cluster IP */ tcp dpt:80
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- 0.0.0.0/0 10.0.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

The last terget in the KUBE-SERVICES chain is the KUBE-NODEPORTS chain. Since the service we created is of type NodePort, let’s list the rules in KUBE-NODEPORTS chain.

$ sudo iptables -t nat -L KUBE-NODEPORTS -n | column -t
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/azure-vote-front: */ tcp dpt:30450
KUBE-SVC-GHSLGKVXVBRM4GZX tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/azure-vote-front: */ tcp dpt:30450

We can see that the above targets are for packets destined to our NodePort 30450. The comments also show the namespace/pod name - default/azure-vote-front. For requests originating from outside the cluster and destined to our app running as a pod, the KUBE-MARK-MASQ rule marks the packet to be altered later in the POSTROUTING chain to use SNAT (source network address translation) to rewrite the source IP as the node IP (so that other hosts outside the pod network can reply back).

Since KUBE-MARK-MASQ target is to MASQUERADE packets later, let’s follow the KUBE-SVC-GHSLGKVXVBRM4GZX chain to further examine our service.

$ sudo iptables -t nat -L KUBE-SVC-GHSLGKVXVBRM4GZX -n | column -t
Chain KUBE-SVC-GHSLGKVXVBRM4GZX (2 references)
target prot opt source destination
KUBE-SEP-QXDNOBCCLOXLV7LV all -- 0.0.0.0/0 0.0.0.0/0
$ sudo iptables -t nat -L KUBE-SEP-QXDNOBCCLOXLV7LV -n | column -t
Chain KUBE-SEP-QXDNOBCCLOXLV7LV (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.11 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp to:10.244.0.11:80

Here we see the DNAT (Destination Network Address Translation) target is used to rewrite the destination of the packet destined to port 30450 to our pod 10.244.0.11:80. We can verify the podIP and containerPort of our pod as follows:

$ kubectl get po azure-vote-front-5bc759676c-x9bwg -o json | jq -r '[.status.podIP, .spec.containers[0].ports[0].containerPort | tostring] | join(":")'
10.244.0.11:80

Verify kube-proxy is listening on NodePort

Under normal circumstances kube-proxy binds and listens on all NodePorts to ensure these ports stay reserved and no other processes can use them. We can verify this on the above kubernetes node:

$ sudo lsof -i:30450
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
hyperkube 11558 root 9u IPv6 95841832 0t0 TCP *:30450 (LISTEN)
$ ps -aef | grep -v grep | grep 11558
root 11558 11539 0 Jul02 ? 00:06:37 /hyperkube kube-proxy --kubeconfig=/var/lib/kubelet/kubeconfig --cluster-cidr=10.244.0.0/16 --feature-gates=ExperimentalCriticalPodAnnotation=true --v=3

kube-proxy is listening on NodePort 30450.

Create a non-kubernetes process use the NodePort

Now let’s kill kube-proxy process and start a server that listens on this NodePort instead.

$ kill -9 11558
$ python -m SimpleHTTPServer 30450 &
[1] 123578
$ sudo lsof -i:30450
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 123578 azureuser 3u IPv4 95854834 0t0 TCP *:30450 (LISTEN)
$ ps -aef | grep -v grep | grep 123578
azureus+ 123578 85107 0 00:49 pts/5 00:00:00 python -m SimpleHTTPServer 30450

Where will the requests to NodePort be routed - pod or the non-kubernetes process?

While the python process is listening on port 30450, also allocated as a NodePort to our kubernetes service, the iptables rules in the PREROUTING chain will route all requests to port 30450 to our pod. We can verify this as below:

$ curl -I -XGET localhost:30450
HTTP/1.1 200 OK
Server: nginx/1.13.7
Date: Wed, 08 Jul 2020 00:53:09 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 950
Connection: keep-alive

As we send a GET request to port 30450, we receive a response from an nginx server hosting Azure Voting App, like we saw when kube-proxy was listening on port 30450.

$ curl -sSL localhost:30450 | grep title
<title>Azure Voting App</title>

Summary

When kube-proxy is used in iptables mode, routing requests to services continues to work for existing services even when the kube-proxy process dies on the node
Kube-proxy binds and listens (on all k8s nodes) to all ports allocated as NodePorts to ensure these ports stay reserved and no other processes can use them
Even if a process starts using NodePort, iptables rules (because they are in PRESOUTING chain) ensure that the traffic sent to the NodePort gets routed to the pods

References

@thockin’s tweet, describing kube-proxy iptables rules in a flow chart, is an excellent way to holistically follow a packet’s path through various iptables rules before it reaches the destined pod.
Kubernetes Networking Demystified: A Brief Guide is a great post explaining how kubernetes services and kubernetes networking work.