Ronak Nathani

Kubernetes Has No "Done": Modeling Rollout Progress and Success

Fri, 19 Jun 2026 11:00:00 -0400

“When should a rollout be considered successful?”

The answer is that Kubernetes gives you the signals to decide, not the decision itself. A controller is not an operation that completes. It is an always-converging loop.

To a user, a rollout is a concrete event: “I deployed version X. Did it succeed or fail?” The controller does not see it that way. It moves current state towards desired state and keeps reconciling whenever they drift apart. The moments when they match are snapshots, not an ending.

A release pipeline cannot wait on “the controller is converging” forever. At some point it needs an answer for this specific rollout. Is it making progress? Did it reach the point where the next stage can safely begin? Or has it failed?

Platforms built on Kubernetes need to synthesize this answer. In practice, the contract has two parts:

Progress: is the rollout still moving towards desired state? It tells an orchestrator whether to keep waiting, nothing more.
Verdict: has the rollout reached a terminal outcome, succeeded or failed, that a pipeline can branch on?

Progress tells the platform whether to keep waiting. The verdict tells the pipeline whether it can continue. Success means the rollout reached the desired state and held it. Failure means progress stopped or the platform observed a terminal failure.

What Is Progress, Really?

In a converging system, progress means that the current state is moving towards the desired state. For a rolling update (and most other cases), it usually comes down to these observable transitions:

More replicas have been created from the new pod template.
Old replicas have been deleted.
More replicas from the new pod template have become available.

As long as one of these transitions keeps happening, the rollout is making progress. It may be slow, but it is not stuck. Each observed transition resets the clock on progress. The question is not how long the rollout has taken in total, but how long it has been since anything meaningful changed. For Deployment-style rolling updates, the pace of a rollout is shaped by maxSurge and maxUnavailable. What the platform watches for is whether any of these transitions is still happening at all.

In an always-converging system, progress is the primitive you can define. Failure is derived from the absence of progress.

For a discrete rollout result, the platform eventually has to stop waiting and emit a failed verdict for that generation. The controller may keep reconciling, but “still reconciling” is no longer useful information for the upstream system waiting on a verdict.

Conditions are how the platform exposes that contract:

Progress

type:Converged

status:Unknown

observedGeneration:N

reason:Progressing

message:progress observed

Success

type:Converged

status:True

observedGeneration:N

reason:DesiredStateObserved

message:desired state observed

Failure

type:Converged

status:False

observedGeneration:N

reason:NoProgressWithinBudget

message:no progress within budget

Each condition is scoped to the rollout generation the upstream system is evaluating.

Kubernetes Deployments expose a similar idea through the Progressing condition.

The exact reason names are less important than the shape of the contract. The controller exposes a generation-scoped condition that upstream systems can trust.

That does not mean the Converged condition becomes sticky once it reaches True or False. It should reflect the controller’s current view of reality. When the controller observes new state, the condition should change with it. The terminal verdict is for the upstream system that needed a decision at that point in the rollout.

A subtle but important detail: any orchestration system reading a condition needs to make sure it is looking at status for the current spec. When a condition carries observedGeneration¹, it should match the resource metadata.generation. Otherwise the system may be reading a stale status capturing the result of a previous rollout.

The Progress Deadline Defines Failure, but It Is Not the Right User Knob

Kubernetes exposes this idea through progressDeadlineSeconds on Deployment objects. It is the amount of time the Deployment controller waits for new progress before setting the Progressing condition to False with reason ProgressDeadlineExceeded.

The concept is a useful primitive for the platform, but not the right user-facing knob. What users need is for the platform to apply a progress budget: how long the platform should wait without meaningful movement before classifying the rollout as failed for orchestration purposes.

That budget is rarely something an application team can set well on its own. The application contributes important inputs, but it does not own the whole equation. A good budget combines what the workload declares with what the platform guarantees.

Some parts come from the workload:

Startup timeout: how long the container may take to start.
Readiness behavior: how long the container may take before it can serve traffic.
minReadySeconds: how long a pod should remain ready before counting as available.

Other parts come from the platform:

Scheduling latency: how long it takes to bind a pod to a node.
Pool autoscaling: how long it takes to add capacity, if needed, when the rollout triggers a scale-up.
Node provisioning: how long newly requested machines take to join the cluster.
Image distribution: image size, pull concurrency, and node cache state.

When we built a CRD for our workloads, exposing this as a user-configurable field seemed natural. Kubernetes has one, so ours should too. But that field made a bad claim: that application teams could choose the progress deadline correctly, even though they only had visibility into part of it.

The failure mode is concrete. If the progress deadline is shorter than a container takes to start up, the rollout fails even though nothing is broken. A validation webhook can catch obvious mistakes, like a deadline shorter than the configured startup and readiness probes, but it cannot model the platform side of the budget.

A better contract is to take the workload-owned inputs and let the platform calculate the deadline from the SLOs it holds for each phase of the rollout path. In practice you can start with percentile bounds for each phase and add them up: scheduling, capacity provisioning, image pull, container startup, readiness, and minReadySeconds. It is not a perfect statistical model. It is an operational budget for how long “no progress” should be tolerated before the rollout is classified as failed.

The budget is a ceiling, not a fixed wait. Some failures are terminal, and waiting out the rest of the budget buys nothing. If the application container is crashing, no amount of additional scheduling, provisioning, or image-pull time will change the outcome, so we exit early and mark the rollout failed rather than burn the remaining budget.

When the budget is exhausted, or a terminal failure is detected, the verdict should explain where the rollout stopped, not just that a deadline passed. We categorize failures based on pod lifecycle and conditions: things like waiting for scheduling, a platform init container failing, an unpullable image reference, or the application container crashing. That taxonomy and the observability around it is a post of its own. My colleague Vasudev Bongale gave an excellent talk at KubeCon NA 2025 on making rollouts observable, and I’d highly recommend it if you’re building a compute platform.

Success Is a Snapshot Contract

A rollout also needs a success signal, but success is different from progress. Progress tells an upstream system that continued waiting is justified. Success tells it that this rollout has reached a snapshot it can safely branch on.

A rollout is rarely the end of a pipeline. It may be followed by canary analysis, promotion to another stage, or rollout to the next production environment. Each of those steps needs a discrete answer before it can move.

Live status is still useful. It lets users see where the rollout is right now. But a pipeline needs a decision: continue, fail, or promote.

A useful definition of success is that the available replicas on the new version have reached the desired count. But a point-in-time match can be fragile. A pod can pass its readiness probe once and fail it a moment later. Treating that instant as success can promote a rollout that will end poorly.

Use `minReadySeconds`

Kubernetes gives you minReadySeconds for this. When configured, a pod must stay ready for that duration before it counts as available. The default is zero, so the default bar is still “ready once.” Setting it raises the bar to “ready and not flapping,” which is much closer to what a rollout success verdict should mean.

Even then, success is not a permanent guarantee. Pods can become unhealthy later, nodes can fail, or the desired state can change again. That does not mean the rollout never succeeded.

This distinction matters operationally. Rollout success is not the same as ongoing workload health. A rollout answers whether a specific version transition reached its success criteria. Workload health answers whether the running system is healthy now. A pod can become unhealthy after a successful rollout, and that should be handled by health monitoring, remediation, or a new rollout decision. It should not retroactively change the verdict for the rollout that already completed.

The API Lesson

The API design lesson we have learned several times:

Don’t expose a knob just because the system underneath has one.

A good platform API should not be a faithful mirror of Kubernetes. It should expose the outcomes users and higher-level systems need, while the platform owns the machinery required to compute them.

That is the job of a compute platform: turn a continuously reconciling substrate into contracts people can reason about.

Core Kubernetes Deployment conditions do not actually carry an observedGeneration. Only status.observedGeneration at the top level does. That is not ideal, because a client reading a single condition cannot tell which generation produced it and can act on stale data. In our platform we put observedGeneration on every condition and pay close attention to it, since the conditions are a large part of the contract that makes rollout state observable to clients. ↩︎

Keeping GPU Workloads NUMA-Local in Kubernetes

Tue, 26 May 2026 11:00:00 -0400

“NUMA alignment” comes up frequently in GPU infrastructure discussions, but concepts like NUMA nodes, topology policies, and CPU pinning are often assumed rather than well understood. Getting it right is as much the platform engineer’s job as the workload owner’s.

This post isn’t a comprehensive guide to NUMA architecture. It’s a practical account of what happens when you align CPU and GPU resources on Kubernetes nodes: the levels of isolation Kubernetes offers, the gotchas, and what it takes to make it work. My experience is on AMD EPYC hardware. Intel has analogous concepts (Sub-NUMA Clustering instead of NPS, UPI instead of Infinity Fabric), but I haven’t worked with Intel in this context, so I’ll stick to what I know.

Key Terminology
What is NUMA?
Why NUMA Matters for GPU Workloads
Levels of CPU Isolation and NUMA Alignment
How the Kubelet Allocates CPUs
Failure Modes
Topology-Aware Scheduling
Making It Work
DRA and Future Direction

Key Terminology

If you’re already familiar with CPU cache hierarchies, sockets, and PCIe, skip ahead. Otherwise, expand below for the shared vocabulary used throughout the post.

Show terminology

CPU Socket: The physical slot on a motherboard that holds a processor. Multi-socket servers (commonly 2-socket) have multiple processors, each with its own cores and local memory.

Physical Core vs. Logical Core: A physical core is a single processing unit on the CPU die. With SMT (“hyperthreading” on Intel), each physical core presents as 2 logical cores that share the core’s execution resources and caches.

L1/L2 Cache: Small, fast caches private to each physical core (L1 is smaller and faster than L2). Two containers sharing a physical core, one logical core each, compete for the same L1/L2 space.

L3 Cache (Last-Level Cache): A larger cache shared among a group of cores. On AMD EPYC, it’s shared within a Core Complex (CCD) of typically 8 cores. Cores sharing an L3 cache can exchange data quickly through it.

Interconnect: The high-speed link between CPU sockets ( Infinity Fabric on AMD, UPI on Intel). Accessing memory locally is faster than going cross-socket over the interconnect.

PCIe: The bus connecting CPUs to devices like GPUs and NICs. Each PCIe root complex is wired to a specific CPU socket, so a GPU is physically closer to one socket than another.

DMA: Lets devices like GPUs read from and write to system memory directly, without the CPU copying data byte-by-byte. If the data sits in memory attached to a different NUMA node than the GPU, the DMA read crosses the interconnect.

What is NUMA?

NUMA (Non-Uniform Memory Access) describes a memory architecture where the time it takes a CPU core to access memory depends on where that memory physically sits relative to the core.

In a 2-socket server, each socket has its own local memory. A core on socket 0 can access memory attached to socket 0 quickly (local access), but accessing memory attached to socket 1 requires crossing the interconnect, which is slower. On AMD EPYC hardware, cross-socket memory access can incur roughly 3x the latency of local access.

NUMA doesn’t only exist across sockets, though. On AMD EPYC processors, a BIOS setting called NPS (Nodes Per Socket) controls how many NUMA domains a single socket is divided into:

NPS1: Each socket is one NUMA node. A 2-socket machine has 2 NUMA nodes.
NPS2: Each socket is split into 2 NUMA nodes. A 2-socket machine has 4 NUMA nodes.
NPS4: Each socket is split into 4 NUMA nodes. A 2-socket machine has 8 NUMA nodes.

The interactive diagram below shows a simplified view of a 2-socket AMD EPYC machine. Toggle between NPS modes to see how the NUMA boundaries change. In NPS1, each socket is one NUMA node. In NPS2 and NPS4, each socket is further subdivided. The CCD and GPU placement is illustrative, not a promise about every SKU.

The key point: NUMA topology is a function of both the hardware and how it’s configured. You can’t assume a fixed number of NUMA nodes across your fleet unless you control the BIOS settings. Different SKUs have different core counts, different numbers of CCDs, and different NPS configurations, all of which change the NUMA geometry.

Why NUMA Matters for GPU Workloads

GPU inference often follows a CPU-GPU pipeline: the CPU prepares requests, batches them, copies input data into the right format, feeds data to the GPU over PCIe, and then handles postprocessing on the GPU’s output. The GPU does the heavy computation, but it’s the CPU that keeps the pipeline fed.

When a container’s CPUs are on a different NUMA node than its GPU, moving input data from CPU memory to GPU memory may cross a NUMA boundary. The GPU has to read data from memory attached to a farther-away CPU socket instead of memory attached to its local socket. This adds latency on the critical path.

In one inference workload, we observed more than 30% higher p99 tail latency under load for pods whose CPUs spanned both sockets compared with pods whose CPUs stayed on the same socket. For a latency-sensitive service, that is enough to matter, and it happens silently unless someone is explicitly monitoring NUMA alignment. Nothing in Kubernetes surfaces it. The pod is running, serving traffic, and looking healthy, just consistently slower than its peers.

Training workloads are affected too, though the impact profile is different. Data loading workers continuously preprocess batches on CPU and stage them for GPU consumption. Cross-NUMA data loaders contend for inter-socket bandwidth and add latency to every batch transfer. PyTorch’s own performance tuning guide explicitly recommends binding training processes to a single NUMA node.

For GPU workloads where the CPU is on the data path to the GPU, NUMA locality has a direct and measurable impact on performance.

Increasing Levels of CPU Isolation and NUMA Alignment

Kubernetes offers several knobs for CPU isolation, each providing stronger guarantees at the cost of more constraints.

Level 1: Logical Core Pinning with `cpuManagerPolicy: static`

By default, Kubernetes lets the operating system’s CPU scheduler move a container’s processes across any available core. This is efficient for overall CPU utilization, but it means your container’s threads may move across cores, invalidating caches and sharing physical cores with other containers.

Setting cpuManagerPolicy: static in the kubelet config changes this. Containers in Guaranteed QoS pods (where requests == limits) with integer CPU requests get exclusive, pinned logical cores. Kubernetes won’t assign those exclusive CPUs to another container, and your processes stay put. Host daemons and kernel threads can still run there unless the platform also reserves or isolates CPUs for the OS.

The way kubelet pins CPUs is by constraining the container’s cpuset cgroup to the assigned CPU list. The assigned cores can be seen in cpuset.cpus on cgroup v1 and cpuset.cpus.effective on cgroup v2.

On the node, the exact path depends on the cgroup driver, runtime, QoS class, and pod UID formatting. With systemd-style kubepods slices, the files are roughly located here:

KUBEPODS="<kubepods.slice/.../kubepods-pod<uid>.slice/<container>.scope>"
# cgroup v1
cat /sys/fs/cgroup/cpuset/$KUBEPODS/cpuset.cpus
# cgroup v2
cat /sys/fs/cgroup/$KUBEPODS/cpuset.cpus.effective

This alone improves performance consistency. Cache affinity improves because threads aren’t migrating across cores, and container-to-container CPU contention is reduced. But there’s a subtlety: you’re pinning logical cores (hyperthreads), not physical cores. Two containers can still end up sharing a physical core if one gets one hyperthread and the other gets the sibling. They’ll contend for that physical core’s L1 and L2 cache.

What it requires from workload owners: Set requests == limits for CPU and memory on all containers (including init containers and sidecars) to get Guaranteed QoS. CPU requests must be integers for the containers where pinning is desired.

Level 2: Physical Core Pinning with `full-pcpus-only`

The full-pcpus-only CPU manager policy option (cpuManagerPolicyOptions: full-pcpus-only=true) takes isolation further. Instead of allocating individual logical cores, it allocates entire physical cores. Both hyperthreads of each core go to the same container.

This eliminates L1/L2 cache contention between containers that would otherwise share a physical core.

The trade-off: containers that receive exclusive CPUs must request a multiple of the SMT thread count, typically 2. A pinned container requesting 3 CPUs fails with an SMTAlignmentError (covered in Failure Modes below). Any existing pinned containers with odd CPU counts on the node need to be resized before you enable this option.

What it requires from workload owners: Even CPU request values. Audit all containers, including sidecars and init containers. Fractional CPU values on sidecars and init containers are fine as those containers use the shared CPU pool and don’t get pinned.

Level 3: Full NUMA Alignment with `single-numa-node`

CPU pinning ensures your cores are dedicated, and full-pcpus-only ensures the container gets full physical cores. Neither guarantees that all your cores come from the same NUMA node. With the default static policy options, the kubelet’s CPU manager uses a packed allocation strategy that fills one NUMA node before spilling to the next (more on this below), but depending on node fragmentation, your container’s CPUs can still span NUMA boundaries.

The topologyManagerPolicy: single-numa-node setting addresses this. The topology manager sits above the CPU manager, device manager, and memory manager, and coordinates resource allocation by collecting topology hints from each. With single-numa-node, it requires that hinted resources can be satisfied from a single NUMA node. If they can’t, the pod is rejected at admission time with a TopologyAffinityError.

The default scope topologyManagerScope: container computes alignment independently for each container. That’s usually fine when one main container owns the GPU and the exclusive CPUs, while sidecars are unrelated to the latency-critical path and use fractional CPU from the shared pool.

topologyManagerScope: pod is stricter: it asks whether the pod’s effective request fits on one NUMA node. Use it when multiple containers in the same pod are performance-coupled, not just because a logging or metrics sidecar exists.

Caveat: Topology Manager only enforces resources that report topology hints. CPU hints come from CPU Manager, GPU hints come through Device Manager from plugins such as the nvidia-device-plugin, and memory hints come from Memory Manager.

If the GPU device plugin does not report NUMA TopologyInfo, Topology Manager cannot force CPU-GPU locality.

For guaranteed NUMA alignment, set memoryManagerPolicy: Static too. This makes requested memory part of topology admission along with CPU and GPU resources. The workload’s memory request must fit within the target NUMA node. Kubernetes also requires reservedMemory when memoryManagerPolicy: Static is enabled.

This gives you the strongest isolation available. The container stays within one NUMA node and communicates with the GPU on that NUMA node without crossing the socket interconnect. CPU cache locality can still vary within that NUMA node (depending on NPS configuration), but in testing, we have found that a container occupying an entire NUMA node with no overlap from other workloads is materially less affected by cache-thrashing or CPU-intensive noisy neighbors.

What it requires from workload owners: The performance-critical container, or the effective pod request when using topologyManagerScope: pod, must fit within a single NUMA node. That means understanding the machine topology and resizing the workload as hardware or its configuration changes.

A minimal kubelet config for a dedicated NUMA-aligned GPU pool looks like:

cpuManagerPolicy: static
cpuManagerPolicyOptions:
full-pcpus-only: "true"
topologyManagerPolicy: single-numa-node
# Default is container. Use pod only when the whole pod should fit on one NUMA node.
# topologyManagerScope: pod
memoryManagerPolicy: Static
# memoryManagerPolicy: Static requires reservedMemory to be configured.

Roll this out on dedicated, drained nodes. When changing CPU or memory manager policies, clear the kubelet state files before restarting kubelet: <kubelet-root-dir>/cpu_manager_state and <kubelet-root-dir>/memory_manager_state.

How the Kubelet Allocates CPUs

Understanding the allocation algorithm helps explain how and when NUMA spillover happens.

When cpuManagerPolicy: static is enabled with the default policy options, the kubelet uses a packed (bin-pack) allocation strategy: takeByTopologyNUMAPacked. It works top-down through the topology:

First, try to take full NUMA nodes (prefer smaller/more-used ones)
Then, take full physical cores from partially-used NUMA nodes (prefer NUMA nodes with fewer free CPUs to pack them first)
Finally, take individual logical cores if needed

The sort order is key: at every level, it prefers NUMA nodes with fewer remaining free CPUs. This packs nearly-exhausted NUMA nodes before touching less-used ones. The allocator usually keeps CPUs NUMA-local when there is room, but locality is not guaranteed.

But “when possible” is doing a lot of work there. Consider a 2-socket machine with 48 cores per socket (96 vCPUs per socket with SMT). In NPS1 mode, this gives 2 NUMA nodes of 96 vCPUs each. After system and kube-reserved, suppose 90 vCPUs are allocatable per NUMA node, with 4 GPUs per NUMA node. Each pod requests 22 vCPUs.

The first 4 pods land on NUMA 0: 4 x 22 = 88 vCPUs used, leaving only 2 allocatable vCPUs. The 5th pod requests 22 vCPUs, but only 2 remain on NUMA 0. The CPU manager takes those 2 from NUMA 0 and the remaining 20 from NUMA 1. The diagram below (credit to my colleague Roman Lishtaba for identifying this pattern in our GPU inference workloads) shows exactly how this plays out:

Without topology manager enforcement, the kubelet allocates CPUs from multiple NUMA nodes. Pod 5 runs fine, but its performance is degraded. Nothing in Kubernetes will tell you about this.

With topologyManagerPolicy: single-numa-node, the system keeps the allocation bounded within one NUMA node. In this scenario, NUMA 1 still has 90 vCPUs free, so Pod 5 would land there entirely. TopologyAffinityError only fires when no single NUMA node can satisfy the request.

Failure Modes to Be Aware Of

Both cpuManagerPolicyOptions: full-pcpus-only=true and topologyManagerPolicy: single-numa-node introduce hard failure modes that are worth understanding before enabling them.

SMTAlignmentError

When full-pcpus-only is enabled, the kubelet rejects any container that would receive exclusive CPUs but does not request a multiple of the SMT thread count, typically 2. The pod goes into Failed state with an SMTAlignmentError and stays there until someone deletes it. Workload controllers (Deployments, StatefulSets) will recreate the pod, but the replacement hits the same error on any node where full-pcpus-only is in effect.

TopologyAffinityError

When topologyManagerPolicy: single-numa-node is enabled, the kubelet rejects any pod whose containers’ hinted resource requests can’t be satisfied from a single NUMA node. With topologyManagerScope: pod, that check applies to the pod’s effective request. The sequence is:

The scheduler picks a node based on aggregate resource availability
The kubelet receives the pod and runs topology admission
The topology manager collects hints from the CPU manager, device manager, and memory manager
If no single NUMA node can satisfy all resources, the pod is rejected with TopologyAffinityError

Same failure semantics as SMTAlignmentError: the pod is Failed and the scheduler won’t retry. The confusing part is that the node has sufficient aggregate capacity, but the pod still fails because no individual NUMA node has enough room. If you’re not thinking in NUMA terms, this is disorienting.

cpuManagerPolicy: static on its own doesn’t introduce these failure modes. They come from the additional constraints of full-pcpus-only and single-numa-node. Both are node-level kubelet settings that apply to every pod on the node, which means enabling single-numa-node can break existing workloads that don’t fit in a single NUMA node. Dedicated node pools for NUMA-aligned workloads are a practical approach to mitigate this.

Topology-Aware Scheduling

A practical problem with single-numa-node is that the default Kubernetes scheduler sees only aggregate node resources. It doesn’t know that a node’s 60 free vCPUs are split 20/40 across two NUMA nodes. The scheduler can place a pod on a node, only for the kubelet to reject it at admission. The workload controller then creates a replacement, which may fail on the next node too.

The NodeResourceTopologyMatch scheduler plugin reduces this gap. It gives the scheduler per-NUMA-node resource visibility, so it can filter out nodes that can’t satisfy the topology constraints before placing the pod.

Deploying it requires:

A cluster-scoped NodeResourceTopology CRD and one NodeResourceTopology custom resource per node
A topology exporter DaemonSet (such as NFD Topology Updater) on every node, polling the kubelet’s PodResources API and publishing per-NUMA resource availability
The NodeResourceTopologyMatch scheduler plugin configured as a filter and scorer

That’s additional infrastructure for the platform team: a DaemonSet, a cluster-scoped CRD, one custom resource per node refreshed roughly every 60 seconds, and a scheduler plugin with its own cache. Without it, pods repeatedly fail on nodes that look like they have enough capacity.

Making It Work: Collaboration Between Platform and Workload Teams

Getting NUMA alignment right is not something either platform admins or workload owners can do alone. It requires collaboration and shared understanding.

Platform admins: publish topology and sizing guidance

With NUMA alignment, platform admins can’t just hand out node pools and let workload owners request whatever CPU/memory they want. They need to publish clear guidance:

What SKU each node pool uses, and its NUMA geometry (cores per NUMA node, NPS mode)
How many vCPUs are consumed by system-reserved and kube-reserved
Recommended container sizes that align with NUMA boundaries
What constraints are in effect (full-pcpus-only, single-numa-node, and any non-default topology manager scope) and what failure modes they introduce

For example, on a 2-socket machine with 48 cores per socket in NPS1 mode: each NUMA node has 96 vCPUs, about 90 allocatable after reservations, with 4 GPUs per NUMA node. The recommended CPU request per GPU might be 22 vCPUs (22 x 4 = 88, fitting within the 90 available). If the fleet has multiple SKUs with different core counts or NPS configurations, this becomes a matrix of recommendations.

Workload owners: understand the constraints, size accordingly

GPU workloads are inherently more hardware-aware than typical Kubernetes workloads. Unlike a stateless web service where you declare CPU and memory and let the platform figure out placement, GPU inference and training benefit from understanding the machine topology.

This means:

Sizing containers to fit within a NUMA node based on the platform’s published guidance (often it’s better to run multiple smaller pods, each NUMA-local, than one larger pod that spans NUMA nodes)
Using even CPU values when full-pcpus-only is in effect
Ensuring the pod is Guaranteed QoS (requests == limits on all containers)
Updating container sizes when the platform migrates to different SKUs

Verifying alignment in practice

At the node level, start by confirming the hardware topology:

lscpu -e=CPU,CORE,SOCKET,NODE
numactl -H
nvidia-smi topo -m # NVIDIA GPU nodes

Inside a running container, check the workload process’s CPU affinity and compare it with the node’s lscpu output to confirm the allowed CPUs sit within the expected NUMA node:

kubectl exec <pod-name> -c <container-name> -- taskset -cp 1
# If taskset is unavailable:
kubectl exec <pod-name> -c <container-name> -- grep Cpus_allowed_list /proc/1/status

taskset -cp 1 checks PID 1 in the container. If your workload runs as a different PID, check that process instead.

Kubernetes has several CPU manager policy options adjacent to the path described above.

strict-cpu-reservation keeps regular workloads off CPUs reserved for the OS and Kubernetes daemons, which helps reduce system noise on pinned workloads.
prefer-align-cpus-by-uncorecache is a best-effort cache-locality option that tries to keep a container’s CPUs within the same L3 or uncore cache group.
align-by-socket is useful when a container is too large to fit in one NUMA node and must use multiple NUMA nodes. It asks CPU Manager to keep that allocation within one socket when possible.

These can improve isolation or latency, but they do not replace topologyManagerPolicy: single-numa-node for keeping a GPU workload NUMA-local. align-by-socket is also not compatible with single-numa-node.

DRA and Future Direction

The Kubernetes DRA (Dynamic Resource Allocation) CPU driver is interesting because it may allow NUMA-aware CPU placement to happen through the scheduling layer, without some of the post-scheduling admission issues described above. I haven’t explored it deeply enough to recommend it here. I’ll write a follow-up after I spend more time with it.

Wrapping Up

There’s a clear progression of CPU isolation in Kubernetes:

Level	Config	What you get	What it requires
1	`cpuManagerPolicy: static`	Dedicated logical cores, reduced CPU contention	Guaranteed QoS, integer CPU requests
2	+ `full-pcpus-only=true`	Full physical cores, L1/L2 cache isolation	Even CPU request values
3	+ `topologyManagerPolicy: single-numa-node` and `memoryManagerPolicy: Static`	CPU, GPU, and memory admitted only if they fit one NUMA node	Critical container fits in a NUMA node, device plugin topology hints, `reservedMemory`, topology-aware scheduler, sizing guidance from platform team

Each level introduces stricter constraints in exchange for stronger performance isolation. For GPU inference, where the CPU is directly on the data path to the GPU, best-effort alignment is not always good enough. If a pod is misaligned, Kubernetes will not tell you, but the workload may still show worse tail latency. For consistency, a hard failure like TopologyAffinityError is often better than silently serving degraded traffic.

Getting it right takes effort from both sides: platform teams publishing topology guidance and workload owners sizing containers to match. It is more work than treating compute as a black box, but GPU workloads typically need to be more aware of the underlying hardware than ordinary services.

Trading Peace for Pace: A Few Weeks with Claude Code

Tue, 17 Feb 2026 23:00:00 -0500

After a few weeks with Claude Code, it has completely replaced my default workflow. It’s really good, and weirdly addictive! The productivity gains are undeniable. You ship faster, explore more ideas, and iterate constantly. But once the initial magic wears off, you start to feel a trade-off: you are trading peace for pace.

Coding used to have a sense of calmness. Uninterrupted stretches where you’d just be in the zone. Shipping a feature felt like a milestone. Now, because execution is so easy, that emotional reward has moved. The joy shifts from writing code to getting the tool to do what you wanted, cleanly and autonomously. You need more volume to feel productive. There is less deep focus and more context switching. You orchestrate agents, shape prompts, and review output. You get more done, but the work is fragmented, reactive and parallel.

This new pace introduces a subtle pressure: if nothing is running in the background, it feels like wasted potential. Like, agents must always be running! (yes, I realize how absurd that sounds.) You see people on X running multiple agents at once, showcasing what they built and the FOMO is inescapable.

There’s an irony here. AI is supposed to free up time so we can do more with less, but in a culture that glorifies shipping, we just use the freed up time to build even more. It’s also incredibly easy to spend hours optimizing your setup. Refining workflows, writing new skills, installing the new trending plugins to make things just a little better. When the tool does the heavy lifting, you spend a lot of time sharpening the tool itself.

The shift is real though and the leverage is incredibly empowering. We have moved from writing (and reviewing) code to expressing intent. You can take on more ambitious projects and build bespoke solutions for every “I wish this existed” thought. Because the cost of building has dropped so drastically, software is becoming disposable. Permanence starts to matter less. Fulfillment comes less from the effort of execution and more from taste, judgment, and deciding what is actually worth building.

Agents are getting incredibly good at execution. They write clean code and make reasonable choices, often following the exact patterns I would. But as I readily accept their output and offload the thinking, I can’t help but wonder about the risk of my own coding skills atrophying.

Coding agents aren’t a novelty anymore. They are the new baseline expectation for software engineers. And it takes deliberate practice to steer them effectively.

Understanding the Kubernetes Scale subresource

Sat, 13 Jan 2024 11:00:00 -0500

Understanding the `scale` subresource in Kubernetes

Resources like Deployments and Statefulsets in Kubernetes have a scale subresource which captures three things:

spec.replicas: The desired number of replicas.
status.replicas: The actual, current number of replicas.
status.labelSelector: Identifies the pods managed by the resource.

Here’s an example of what a typical query response looks like when you explore the scale subresource:

# Sample output from querying a Kubernetes deployment's scale settings.
➜ curl -s localhost:8001/apis/apps/v1/namespaces/kube-system/deployments/coredns/scale | jq .
{
"kind": "Scale",
"apiVersion": "autoscaling/v1",
"metadata": {
"name": "coredns",
"namespace": "kube-system",
"uid": "0f39b1dd-8cb4-4374-a95d-11d96c0b9d6a",
"resourceVersion": "3260769",
"creationTimestamp": "2023-08-10T15:55:24Z"
},
"spec": {
"replicas": 2
},
"status": {
"replicas": 2,
"selector": "k8s-app=kube-dns"
}
}

Why’s the `scale` subresource Necessary?

When I first learned about the scale subresource, I wondered why it’s needed when the replica information is already available in the spec. As with many things in Kubernetes, it’s needed to support extensibility and flexibility.

Kubernetes doesn’t enforce a uniform schema for representing desired and current replica counts in workload resources. The scale subresource provides a unified interface, enabling different workload resources, both built-in and custom, to consistently expose their scaling settings. This uniform interface allows for seamless integration with the rest of the Kubernetes ecosystem.

This is better understood by looking at how the scale subresource is used.

Uses of the `scale` subresource

The scale subresource is used for the following cases:

By the Horizontal Pod Autoscaler (HPA): HPA uses the scale subresource to dynamically adjust the desired replica count based on some utilization metrics like CPU usage, QPS, etc.
By the Pod Disruption Budget (PDB) controller: When .spec.maxUnavailable or .spec.minAvailable in PDB configurations is specified as a percentage, the PDB controller queries the scale subresource, of the resource managing the pod, to get the desired number of replicas.
Scaling pods through the kubectl scale command: This command allows manually changing the replica count of resources that have the scale subresource enabled.

Scale subresource and Custom Resources

If you’re working with custom resources that manage pods and want them to integrate seamlessly with HPA, PDB, or be compatible with the kubectl scale command, you need to enable the scale subresource in the Custom Resource Definition (CRD).

If you are using kubebuilder to write controllers, you can enable the scale subresource for your custom resource as mentioned here. Also, it seems that the scale subresource can be enabled

PDB, Custom Resources, and the `scale` subresource

For pods managed by a custom resource, PDB can be used without restrictions only if the custom resource supports the scale subresource. This is because when a percentage value is specified for .spec.maxUnavailable or .spec.minAvailable, the PDB controller needs to know the total desired replicas in order to calculate the number of replicas that should be available during a disruption. And PDB controller gets the total desired replicas via the scale subresource of the resource owning the pods.

// if maxUnavailable is set as a percentage
desiredAvailableReplicas := desiredTotalReplicas - (desiredTotalReplicas * maxUnavailable / 100)

// if minAvailable is set a percentage
desiredAvailableReplicas := desiredTotalReplicas * minAvailable / 100

Call paths for reference:

getExpectedPodCount → getExpectedScale → getScaleController

If scale subresource isn’t enabled for your custom resource, you can still use PDB, albeit with certain limitations - you can only use .spec.minAvailable with an integer value, not percentages, as mentioned in the Kubernetes documentation.

What I Wish I Knew About Incident Management

Sat, 07 Nov 2020 18:00:00 -0700

I gave this talk last year at LinkedIn’s internal SRE conference, thought I’d share it here as well.

Why I am writing this post

Like every Software Engineer / SRE, I’ve had my share of troubleshooting software. However, I had never been oncall before I joined Linkedin and the impact of a system outage that affects thousands of engineers made the first week of oncall pretty overwhelming.

But things got better overtime.

In this post, I would like to share the incident management practices I have picked up over the years as an SRE at Linkedin that help me keep calm under pressure and effectively drive incidents to resolution.

What this post is not about

In this post, I am not going to talk about how to debug linux or distributed systems or the various debugging tools. (For stories from the frontlines, check out Software Misadventures Podcast!)

First oncall week

My first few weeks at Linkedin - they were great! I was meeting smart engineers and learning new things. It wasn’t until my first oncall rotation that I started thinking what if there’s an outage and I need to fix it?

Now, don’t get me wrong. LinkedIn has really good systems in place for monitoring/alerting, triaging issues and a very well defined process for incident response. Those are absolutely critical. And to prepare, I had shadowed our oncall the week prior and even had an experienced team member shadow me to guide and help me out, but still, I was anxious.

Here’s what I wish I had known.

Before oncall starts
- Oncall handoff
- Organizing Slack during an oncall week
Signal vs Noise
- Trust, but verify - not all alerts are created equal
- Declaring an incident
Communication during an incident
Incident response is a collaborative process
Towards a resolution
- Looking at changes
- Keep calm and carry on - one step at a time
Learning from the incident

Before Oncall Starts

Oncall handoff

Before an oncall week starts, I talk to the person who is currently oncall to get context on any incidents that happened during the week or any weird bugs that were discovered in our stack. It gives me perspectives on issues that could be getting carried over from the previous week and any critical changes I should be aware of. Although you can’t plan for all that happens during an oncall week, a proper handoff helps you prepare for it.

Organizing Slack during an oncall week

As an SRE on LinkedIn’s container scheduler and deployment infrastructure team, our users are engineers at LinkedIn. We use Slack for internal communication and we have certain channels where users share issues they are experiencing with the tooling or to get our oncall’s attention. During an oncall week, I star these channels and organize my sidebar so that I can easily notice messages from our users and distinguish them from the other messages I receive. As an optional tip - I also like to mute/leave channels that I am not actively participating in to reduce clutter.

As compared to a normal week, I spend more time on Slack when I am oncall - responding to people, answering support requests, helping users with tooling. These Slack notifications can create a little bit of distraction, however, considering all our users are internal, a rise in messages on our channels can also indicate that something might be wrong with our system and our alerting hasn’t caught it yet.

Signal vs Noise

Trust, but verify - not all alerts are created equal

This is something I learned during my interview at LinkedIn and it has been very applicable in my experience since.

In early days of oncall and incident management, it is very common to feel as if things are on fire when any alert gets triggered. Over time, I have realized that gaining context and verifying that the alert is actually indicating an issue helps with the right next steps. It is important to distinguish signal from noise because:

An alert could be non-actionable as it was recently configured with a threshold that’s making it too noisy
The monitoring stack is down and the alert is being triggered because the configuration treats a lack of data points as an issue
Your service is operating perfectly fine, however, the traffic tier routing requests to your service had an issue
Timer on a deliberately silenced alert expired and the alert started triggering

I have experienced all of the above at some point in time. Now when I either receive an alert or a user reports an issue, I check our services (metrics, logs, reproduce the reported problem etc.) to verify that the issue is real.

Declaring an incident

Not all actionable alerts result in an incident. To be effective at identifying the ones that do, it is extremely crucial to think about the bigger picture of mitigating the issue than be overwhelmed by the technical task of resolving the alert.

Some of the qualitative measures that help make this differentiation is to consider whether an issue requires coordinating the fix with other teams or whether the issue is impacting customers or violating an SLO. If any of the conditions are true, declare an incident. It is always better to declare an incident early in the process than waiting too long.

At LinkedIn, we have defined guidelines for all teams about what warrants an incident along with different levels of severity. This takes guesswork out of the picture, and provides a shared understanding to every team member.

Communication during an incident

The incident title and scoping the impact

Every incident gets a title. I didn’t realize it initially, but giving an incident a title forces one to define the problem and communicate it to the stakeholders very succinctly. When communicating to stakeholders, scoping the incident is very important. What I mean by scoping is identifying how big the impact is - which environment is impacted, is the impact limited to a region or is it global, how many customers are impacted, etc. For instance, “the login feature on the site is not working” vs “the login feature on the site is not working for traffic originating from Asia Pacific” say two very different things.

Establish communication channels and an incident lead

We heavily rely on Slack to communicate during an incident. A dedicated slack channel helps focus all the energy and inputs from everyone in one place. It helps the incident lead collect data about symptoms that the users are experiencing and also captures a log of considered/discarded hypotheses and any changes made to the system. Once the Slack channel is created, establish an incident lead and let everyone know who is driving the incident forward.

Eslablishing explicit comms channels for reporting and identifying issues and establishing an incident lead reduce the delay in action and disambiguate any confusion.

Communicate changes to the system

If there’s a fix you’d like to try out, let others know who are and encourage everyone to do the same. This ensures that the potential fix doesn’t make an already bad situation worse and helps catch any blind spots early. If you do end up making a change to the system after getting consensus, let others know and follow up on its effects.

Provide regular updates

While working towards a resolution for the incident, it is very easy to get overwhelmed by the technical details and miss to communicate an update. This leads to angry leadership and annoyed customers who have no insight into what’s happening.

An update provides visibility to the customers that the issue is being worked upon and lets the leadership identify if they can help out with anything. Depending on the severity of the incident, an update every 15-30 mins serves pretty well. The update doesn’t have to be extremely detailed, rather a brief summary describing the current state and immediate next steps is sufficient. An example update:

[UPDATE] Our hypothesis about the memory leak checks out and we have validated the fix in the staging environment. We have pushed out the change and as soon as it goes through the CI pipeline, we’ll canary the new release, monitor metrics and promote the change after verifying the fix.

Incident response is a collaborative process

Get help early

In my earlier days, I used to think that it was solely my responsibility to mitigate the issue, find the root cause, and roll out the fix. If I couldn’t do it, it wouldn’t reflect well on me. In reality, incident management, like much of software development, is a very collaborative process.

One of the big differences in how I approach it now is I focus on actively pulling in other engineers who could help with debugging or resolving the issue early in the process. This change in perspective has relieved me of a lot of unnecessary stress and also made me more effective at resolving incidents.

When requesting help, be specific about the task as well as the urgency. It helps others calibrate their response and manage things they might have at hand.

Working with others

One of the most important things while handling incidents is working with others. Considering the multi-faceted nature of an SRE role, it’s one of the most important while underrated skills.

With multiple people involved, various possibilities get shared and one of the responsibilities of the incident lead is to guide these discussions in a productive direction while filtering out the noise. One should be cautious about going down any rabbit holes and focus on stopping the bleeding first. If there’s a probable cause, share it early - it helps rule out possibilities. And it’s okay to ask questions that seem obvious, but specific questions are more helpful.

Being fearlessly curious, and keeping an open mind enables one to consider possibilities that could be overseen otherwise. Remember, you are working together as a team on sharing and ruling out hypotheses to solve a challenging problem.

Video Conferencing is your friend

We work with engineers in geographically distributed locations (even when it’s not a pandemic) and some discussions are much better conducted synchronously than async. In such scenarios, start a video call sooner in the process. Either a shared conference room or a video call has almost always helped figure out the path forward sooner and coordinate next steps.

Towards a resolution

Looking at changes

Things mostly break when something changes. If we froze our code, configurations and infrastructure, we’d have way less outages than otherwise. Code or config deployments are two of the most common actions that result in change. So, rule those out first. A few things to check for change are:

Code or config deployment targeting the application experiencing the issue
Global configuration changes of the underlying infrastructure (network, OS, etc.)
Anomalous traffic patterns

The intention here is to identify what is unusual about the system that’s resulting in an issue. I must mention that isn’t always a single change, rather a combination of changes that result in the perfect storm.

Keep calm and carry on - one step at a time

While all of this is going on, it’s pretty common to be stressed knowing that users are unable to use your system as promised. If you start to feel you are panicking or overwhelmed, call for support. Panic results in a scrambled path forward which in turn takes longer for an incident to be resolved. And for many people, there’s a linear correlation in length of an incident and the stress they feel.

In addition to calling for support, take a step back to re-evaluate the issue holistically and think about the upstreams/downstreams of your system to rule out possibilities in a structured way. Structuring your thoughts and reminding yourself that you are not alone in this helps keep calm and handle the pressure.

Learning from the incident

I have gone from dreading incidents to treating them as an opportunity to learn something new. I have learnt the most about our software and the underlying infrastructure when something breaks. It makes the task of incident response exciting and fun - like a detective solving a difficult case. And when the incident is finally resolved, it brings a great sense of satisfaction, accomplishment and pride for the team involved in resolving the incident.

When you are not oncall, be available to help your teammates when there’s an incident. They would appreciate it and you’d learn something new in the process. Last but not the least, ensure that not just the postmortem but the incident resolution is blameless. It helps foster a culture of ownership, results in incidents being resolved faster and helps improve the entire team’s performance over time.

Thanks to Guang Yang and Aishanee Shah for reviewing this post and providing very valuable feedback.

If you have any specific incident management practices that have helped you over time, I’d love to know. Feel free to comment on the post or reach out to me via Twitter / Email. I look forward to hearing from you!

How a Kubernetes Pod Gets an IP Address

Fri, 21 Aug 2020 13:04:27 -0700

One of the core requirements of the Kubernetes networking model is that every pod should get its own IP address and that every pod in the cluster should be able to talk to it using this IP address. There are several network providers (flannel, calico, canal, etc.) that implement this networking model.

As I started working on Kubernetes, it wasn’t completely clear to me how every pod is assigned an IP address. I understood how various components worked independently, however, it wasn’t clear how these components fit together. For instance, I understood what CNI plugins were, however, I didn’t know how they were invoked. So, I wanted to write this post to share what I have learned about various networking components and how they are stitched together in a kubernetes cluster for every pod to receive an IP address.

There are various ways of setting up networking in kubernetes and various options for a container runtime. For this post, I will use Flannel as the network provider and Containerd as the container runtime. Also, I am going to assume that you know how container networking works and only share a very brief overview below for context.

Some Background Concepts

Container Networking: A Very Brief Overview

There are some really good posts explaining how container networking works. For context, I will go over a very high level overview here with a single approach that involves linux bridge networking and packet encapsulation. I am skipping details here as container networking deserves a blog post of itself. Some of the posts that I have found to be very educational in this space are linked in the references below.

Containers on the same host

One of the ways containers running on the same host can talk to each other via their IP addresses is through a linux bridge. In the kubernetes (and docker) world, a veth (virtual ethernet) device is created to achieve this. One end of this veth device is inserted into the container network namespace and the other end is connected to a linux bridge on the host network. All containers on the same host have one end of this veth pair connected to the linux bridge and they can talk to each other using their IP addresses via the bridge. The linux bridge is also assigned an IP address and it acts as a gateway for egress traffic from pods destined to different nodes.

Containers on different hosts

One of the ways containers running on different hosts can talk to each other via their IP addresses is by using packet encapsulation. Flannel supports this through vxlan which wraps the original packet inside a UDP packet and sends it to the destination.

In a kubernetes cluster, flannel creates a vxlan device and some route table entries on each of the nodes. Every packet that’s destined for a container on a different host goes through the vxlan device and is encapsulated in a UDP packet. On the destination, the encapsulated packet is retrieved and the packet is routed through to the destined pod.

NOTE: This is just one of the ways how networking between containers can be configured.

What Is CRI?

CRI (Container Runtime Interface) is a plugin interface that allows kubelet to use different container runtimes. Various container runtimes implement the CRI API and this allows users to use the container runtime of their choice in their kubernetes installation.

What is CNI?

CNI project includes a spec to provide a generic plugin-based networking solution for linux containers. It also consists of various plugins which perform different functions in configuring the pod network. A CNI plugin is an executable that follows the CNI spec and we’ll discuss some plugins in the post below.

Assigning Subnets To Nodes For Pod IP Addresses

If all pods are required to have an IP address, it’s important to ensure that all pods across the entire cluster have a unique IP address. This is achieved by assigning each node a unique subnet from which pods are assigned IP addresses on that node.

Node IPAM Controller

When nodeipam is passed as an option to the kube-controller-manager’s --controllers command line flag, it allocates each node a dedicated subnet (podCIDR) from the cluster CIDR (IP range for the cluster network). Since these podCIDRs are disjoint subnets, it allows assigning each pod a unique IP address.

A kubernetes node is assigned a podCIDR when the node first registers with the cluster. To change the podCIDR allocated to nodes in a cluster, nodes need to be de-registered and then re-registered with any configuration changes first applied to the kubernetes control plane. podCIDR for a node can be listed using the following command.

$ kubectl get no <nodeName> -o json | jq '.spec.podCIDR'
10.244.0.0/24

Kubelet, Container Runtime and CNI Plugins - how it’s all stitched together

When a pod is scheduled on a node, a lot of things happen to start up a pod. In this section, I’ll only focus on the interactions that relate to configuring network for the pod.

Once a pod is scheduled on the node, the following interactions result in configuring the network and starting the application container.

Ref: Containerd cri plugin architecture

Interactions between Container Runtime and CNI Plugins

Every network provider has a CNI plugin which is invoked by the container runtime to configure network for a pod as it’s started. With containerd as the container runtime, Containerd CRI plugin invokes the CNI plugin. Every network provider also has an agent that’s installed on each of the kubernetes node to configure pod networking. When the network provider agent is installed, it either ships with the CNI config or it creates one on the node which is then used by the CRI plugin to figure out which CNI plugin to call.

The location for the CNI config file is configurable and the default value is /etc/cni/net.d/<config-file>. CNI plugins need to be shipped on every node by the cluster administrators. The location for CNI plugins is configurable as well and the default value is /opt/cni/bin.

In case of containerd as the container runtime, path for CNI configuration and CNI plugin binaries can be specified under [plugins."io.containerd.grpc.v1.cri".cni] section of the containerd config.

Since we are referring to Flannel as the network provider here, I’ll talk a little about how Flannel is set up. Flanneld is the Flannel daemon and is typically installed on a kubernetes cluster as a daemonset with install-cni as an init container. The install-cni container creates the CNI configuration file - /etc/cni/net.d/10-flannel.conflist - on each node. Flanneld creates a vxlan device, fetches networking metadata from the apiserver and watches for updates on pods. As pods are created, it distributes routes for all pods across the entire cluster and these routes allow pods to connect to each other via their IP addresses. For details on how flannel works, I recommend the linked references below.

The interactions between Containerd CRI Plugin and CNI plugins can be visualized as follows:

As described above, kubelet calls the Containerd CRI plugin in order to create a pod and Containerd CRI plugin calls the CNI plugin to configure network for the pod. The network provider CNI plugin calls other base CNI plugins to configure the network. The interactions between CNI plugins are described below.

Interactions Between CNI Plugins

There are various CNI plugins that help configure networking between containers on a host. For this post, we will refer to 3 plugins.

Flannel CNI Plugin

When using Flannel as the network provider, the Containerd CRI plugin invokes the Flannel CNI plugin using the CNI configuration file - /etc/cni/net.d/10-flannel.conflist.

$ cat /etc/cni/net.d/10-flannel.conflist
{
"name": "cni0",
"plugins": [
{
"type": "flannel",
"delegate": {
"ipMasq": false,
"hairpinMode": true,
"isDefaultGateway": true
}
}
]
}

The Fannel CNI plugin works in conjunction with Flanneld. When Flanneld starts up, it fetches the podCIDR and other network related details from the apiserver and stores them in a file - /run/flannel/subnet.env.

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

The Flannel CNI plugin uses the information in /run/flannel/subnet.env to configure and invoke the bridge CNI plugin.

Bridge CNI Plugin

Flannel CNI plugin calls the Bridge CNI plugin with the following configuration:

{
"name": "cni0",
"type": "bridge",
"mtu": 1450,
"ipMasq": false,
"isGateway": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.0.0/24"
}
}

When Bridge CNI plugin is invoked for the first time, it creates a linux bridge with the "name": "cni0" specified in the config file. For every pod, it then creates a veth pair - one end of the pair is in the container’s network namespace and the other end is connected to the linux bridge on the host network. With Bridge CNI plugin, all containers on a host are connected to the linux bridge on the host network.

After configuring the veth pair, Bridge plugin invokes the host-local IPAM CNI plugin. Which IPAM plugin to use can be configured in the CNI config CRI plugin uses to call the flannel CNI plugin.

Host-local IPAM CNI plugins

The Bridge CNI plugin calls the host-local IPAM CNI plugin with the following configuration:

{
"name": "cni0",
"ipam": {
"type": "host-local",
"subnet": "10.244.0.0/24",
"dataDir": "/var/lib/cni/networks"
}
}

Host-local IPAM (IP Address Management) plugin returns an IP address for the container from the subnet and stores the allocated IP locally on the host under the directory specified under dataDir - /var/lib/cni/networks/<network-name=cni0>/<ip>. /var/lib/cni/networks/<network-name=cni0>/<ip> file contains the container ID to which the IP is assigned.

When invoked, the host-local IPAM plugin returns the following payload

{
"ip4": {
"ip": "10.244.4.2",
"gateway": "10.244.4.3"
},
"dns": {}
}

Summary

Kube-controller-manager assigns a podCIDR to each node. Pods on a node are assigned an IP address from the subnet value in podCIDR. Because podCIDRs across all nodes are disjoint subnets, it allows assigning each pod a unique IP address.

Kubernetes cluster administrator configures and installs kubelet, container runtime, network provider agent and distributes CNI plugins on each node. When network provider agent starts, it generates a CNI config. When a pod is scheduled on a node, kubelet calls the CRI plugin to create the pod. In containerd’s case, Containerd CRI plugin then calls the CNI plugin specified in the CNI config to configure the pod network. And all of this results in a pod getting an IP address.

It took me a while to understand all the interactions and the details involved. I hope this helped you in improving your understanding of how kubernetes works. If you think I got something wrong, please let me know via twitter or email me at hello@ronaknathani.com. If you’d like to discuss something in this post or anything else, feel free to reach out. I’d love to hear from you!

References

Container Networking

How Flannel Works

CRI and CNI

Kubernetes NodePort and iptables rules

Wed, 08 Jul 2020 14:36:33 -0700

I have been working on kubernetes over the last few months and having fun learning about the underlying systems. When I started using kubernetes services, I wanted to learn about the iptables rules that kube-proxy creates to enable them, however, I didn’t exactly know where to start. While there are some really good posts explaining kubernetes networking and how the concept of services works, I couldn’t easily find a starting point to explore these iptables except reading the code or trying to make sense of the output of iptables-save command (which I can’t).

In this post, I share some of what I have learned by digging a little deeper into the iptables rules for NodePort type services and share answers to the following questions I came across while working on kube-proxy:

What happens when a non-kubernetes process starts using a port that’s allocated as a NodePort to a service?
Does the service endpoint continue to route traffic to pods if kube-proxy (configured in iptables mode) process dies on the node?

Some background

Kubernetes allows a few ways to expose applications to the world outside the kubernetes cluster through the concept of services. In our setup at work, we expose a few of our apps through NodePort type services. Also, kube-proxy is the kubernetes component that powers the services concept and we run it in iptables mode (default).

A little about NodePort

NodePort is one of the ways of exposing Kubernetes Services to the world outside the kubernetes cluster. As per Kubernetes docs,

If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767). Each node proxies that port (the same port number on every Node) into your Service.

This means that if we have a service with NodePort 30000, a request to <kubernetes-node-ip>:30000 will get routed to our app. Under normal circumstances kube-proxy binds and listens on all NodePorts to ensure these ports stay reserved and no other processes can use them.

err…a non-kubernetes is using the NodePort!!

A few weeks back at work, one of our kubernetes nodes was rebooted and as it was added back to the cluster, we saw errors in kube-proxy logs - bind: address already in use:

I0707 20:57:38.648179 1 proxier.go:701] Syncing iptables rules
E0707 20:57:38.679876 1 proxier.go:1072] can't open "nodePort for default/azure-vote-front:" (:30450/tcp), skipping this nodePort: listen tcp :30450: bind: address already in use

This error meant that a port allocated as a NodePort to a service was already in use by another process. We found out that there was a non-kubernetes process using this NodePort as a client port to connect to a remote server. This happened due of a race condition post node reboot. This other process started using the NodePort before kube-proxy could bind and listen on the port to reserve it.

Will the NodePort on this node still route traffic to the pods?

This was the immediate question that popped up in our heads as the NodePort allocated to a service was now being used by a non-kubernetes process. One of my colleagues, Alex Dudko pointed out that kube-proxy creates iptables rules in the PREROUTING chain in nat table.

Because of these rules, the answer to the above question is - Yes! Traffic sent to the NodePort on this node will still correctly be routed to the backend pods it targets.

The NodePort would also continue to work if another process was listening on the port as a server and not just using it as a client. Though, would our HTTP GET requests reach this server? More on this later.

Kube-proxy iptables rules

In iptables mode, kube-proxy creates iptables rules for kubernetes services which ensure that the request to the service gets routed (and load balanced) to the appropriate pods.

These iptables rules also help answer the second question mentioned above. As long as these iptables rules exist, requests to services will get routed to the appropriate pods even if kube-proxy process dies on the node. Endpoints for new services won’t work from this node, however, since kube-proxy process won’t create the iptables rules for it.

Rules in the PREROUTING chain

As outlined in this flow chart, there are rules in the PREROUTING chain of the nat table for kubernetes services. As per iptables rules evaluation order, rules in the PREROUTING chain are the first ones to be consulted as a packet enters the linux kernel’s networking stack.

Rules created by kube-proxy in the PREROUTING chain help determine if the packet is meant for a local socket on the node or if it should be forwarded to a pod. It is these rules that ensure that the request to <kubernetes-node-ip>:<NodePort> continue to get routed to pods even if the NodePort is in use by another process.

Below is an over-simplified diagram that demonstrates this:

In rest of this post, I walk through a setup that reproduces the above scenario and examine the iptables rules that make this work.

Setting up a Kubernetes cluster

I created a kubernetes cluster using Azure Kubernetes Service .

Creating pods and a NodePort Service

From the AKS docs, I also created a deployment for an app with a frontend and a backend. I updated the azure-vote-front to type: NodePort service.

azure-vote-front pod in the cluster

$ kubectl get po -l app=azure-vote-front --show-labels -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
azure-vote-front-5bc759676c-x9bwg 1/1 Running 0 24d 10.244.0.11 aks-nodepool1-22391620-vmss000000 <none> <none> app=azure-vote-front,pod-template-hash=5bc759676c

azure-vote-front service targeting the above pod

$ kubectl get svc azure-vote-front -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
azure-vote-front NodePort 10.0.237.3 <none> 80:30450/TCP 24d app=azure-vote-front

The above output shows that the port 30450 is assigned as the NodePort to our service. And we can send HTTP requests to the app from host network:

$ curl -IX GET localhost:30450
HTTP/1.1 200 OK
Server: nginx/1.13.7
Date: Wed, 08 Jul 2020 00:02:08 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 950
Connection: keep-alive
$ curl -sSL localhost:30450 | grep title
<title>Azure Voting App</title>

Examining iptables rules for a NodePort service

Note: AKS doesn’t allow ssh’ing into an AKS worker node directly, instead, a pod needs to be spun up to ssh into the worker node. Instructions here.

Let’s look at the iptables rules for the azure-vote-front service.

After ssh’ing into the kubernetes worker node, let’s look at the PREROUTING chain in the nat table. (PREROUTING chain exists in raw, nat and mangle tables, however, kube-proxy only creates PREROUTING chain rules in nat table)

$ sudo iptables -t nat -L PREROUTING | column -t
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL

There’s a KUBE-SERVICES chain in the target that’s created by kube-proxy. Let’s list the rules in that chain.

$ sudo iptables -t nat -L KUBE-SERVICES -n | column -t
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.202.188 /* kube-system/healthmodel-replicaset-service: cluster IP */ tcp dpt:25227
KUBE-SVC-KA44REDDIFNWY4W2 tcp -- 0.0.0.0/0 10.0.202.188 /* kube-system/healthmodel-replicaset-service: cluster IP */ tcp dpt:25227
KUBE-MARK-MASQ udp -- !10.244.0.0/16 10.0.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- 0.0.0.0/0 10.0.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- 0.0.0.0/0 10.0.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.190.160 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:80
KUBE-SVC-XGLOHA7QRQ3V22RZ tcp -- 0.0.0.0/0 10.0.190.160 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:80
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.195.6 /* kube-system/metrics-server: cluster IP */ tcp dpt:443
KUBE-SVC-LC5QY66VUV2HJ6WZ tcp -- 0.0.0.0/0 10.0.195.6 /* kube-system/metrics-server: cluster IP */ tcp dpt:443
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.129.119 /* default/azure-vote-back: cluster IP */ tcp dpt:6379
KUBE-SVC-JQ4VBJ2YWO22DDZW tcp -- 0.0.0.0/0 10.0.129.119 /* default/azure-vote-back: cluster IP */ tcp dpt:6379
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.237.3 /* default/azure-vote-front: cluster IP */ tcp dpt:80
KUBE-SVC-GHSLGKVXVBRM4GZX tcp -- 0.0.0.0/0 10.0.237.3 /* default/azure-vote-front: cluster IP */ tcp dpt:80
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.0.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- 0.0.0.0/0 10.0.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

The last terget in the KUBE-SERVICES chain is the KUBE-NODEPORTS chain. Since the service we created is of type NodePort, let’s list the rules in KUBE-NODEPORTS chain.

$ sudo iptables -t nat -L KUBE-NODEPORTS -n | column -t
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/azure-vote-front: */ tcp dpt:30450
KUBE-SVC-GHSLGKVXVBRM4GZX tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/azure-vote-front: */ tcp dpt:30450

We can see that the above targets are for packets destined to our NodePort 30450. The comments also show the namespace/pod name - default/azure-vote-front. For requests originating from outside the cluster and destined to our app running as a pod, the KUBE-MARK-MASQ rule marks the packet to be altered later in the POSTROUTING chain to use SNAT (source network address translation) to rewrite the source IP as the node IP (so that other hosts outside the pod network can reply back).

Since KUBE-MARK-MASQ target is to MASQUERADE packets later, let’s follow the KUBE-SVC-GHSLGKVXVBRM4GZX chain to further examine our service.

$ sudo iptables -t nat -L KUBE-SVC-GHSLGKVXVBRM4GZX -n | column -t
Chain KUBE-SVC-GHSLGKVXVBRM4GZX (2 references)
target prot opt source destination
KUBE-SEP-QXDNOBCCLOXLV7LV all -- 0.0.0.0/0 0.0.0.0/0
$ sudo iptables -t nat -L KUBE-SEP-QXDNOBCCLOXLV7LV -n | column -t
Chain KUBE-SEP-QXDNOBCCLOXLV7LV (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.11 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp to:10.244.0.11:80

Here we see the DNAT (Destination Network Address Translation) target is used to rewrite the destination of the packet destined to port 30450 to our pod 10.244.0.11:80. We can verify the podIP and containerPort of our pod as follows:

$ kubectl get po azure-vote-front-5bc759676c-x9bwg -o json | jq -r '[.status.podIP, .spec.containers[0].ports[0].containerPort | tostring] | join(":")'
10.244.0.11:80

Verify kube-proxy is listening on NodePort

Under normal circumstances kube-proxy binds and listens on all NodePorts to ensure these ports stay reserved and no other processes can use them. We can verify this on the above kubernetes node:

$ sudo lsof -i:30450
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
hyperkube 11558 root 9u IPv6 95841832 0t0 TCP *:30450 (LISTEN)
$ ps -aef | grep -v grep | grep 11558
root 11558 11539 0 Jul02 ? 00:06:37 /hyperkube kube-proxy --kubeconfig=/var/lib/kubelet/kubeconfig --cluster-cidr=10.244.0.0/16 --feature-gates=ExperimentalCriticalPodAnnotation=true --v=3

kube-proxy is listening on NodePort 30450.

Create a non-kubernetes process use the NodePort

Now let’s kill kube-proxy process and start a server that listens on this NodePort instead.

$ kill -9 11558
$ python -m SimpleHTTPServer 30450 &
[1] 123578
$ sudo lsof -i:30450
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 123578 azureuser 3u IPv4 95854834 0t0 TCP *:30450 (LISTEN)
$ ps -aef | grep -v grep | grep 123578
azureus+ 123578 85107 0 00:49 pts/5 00:00:00 python -m SimpleHTTPServer 30450

Where will the requests to NodePort be routed - pod or the non-kubernetes process?

While the python process is listening on port 30450, also allocated as a NodePort to our kubernetes service, the iptables rules in the PREROUTING chain will route all requests to port 30450 to our pod. We can verify this as below:

$ curl -I -XGET localhost:30450
HTTP/1.1 200 OK
Server: nginx/1.13.7
Date: Wed, 08 Jul 2020 00:53:09 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 950
Connection: keep-alive

As we send a GET request to port 30450, we receive a response from an nginx server hosting Azure Voting App, like we saw when kube-proxy was listening on port 30450.

$ curl -sSL localhost:30450 | grep title
<title>Azure Voting App</title>

Summary

When kube-proxy is used in iptables mode, routing requests to services continues to work for existing services even when the kube-proxy process dies on the node
Kube-proxy binds and listens (on all k8s nodes) to all ports allocated as NodePorts to ensure these ports stay reserved and no other processes can use them
Even if a process starts using NodePort, iptables rules (because they are in PRESOUTING chain) ensure that the traffic sent to the NodePort gets routed to the pods

References

@thockin’s tweet, describing kube-proxy iptables rules in a flow chart, is an excellent way to holistically follow a packet’s path through various iptables rules before it reaches the destined pod.
Kubernetes Networking Demystified: A Brief Guide is a great post explaining how kubernetes services and kubernetes networking work.

Ronak Nathani

Kubernetes Has No "Done": Modeling Rollout Progress and Success

What Is Progress, Really?

The Progress Deadline Defines Failure, but It Is Not the Right User Knob

Success Is a Snapshot Contract

Use minReadySeconds

The API Lesson

Keeping GPU Workloads NUMA-Local in Kubernetes

Table of Contents

Key Terminology

What is NUMA?

Why NUMA Matters for GPU Workloads

Increasing Levels of CPU Isolation and NUMA Alignment

Level 1: Logical Core Pinning with cpuManagerPolicy: static

Level 2: Physical Core Pinning with full-pcpus-only

Level 3: Full NUMA Alignment with single-numa-node

How the Kubelet Allocates CPUs

Failure Modes to Be Aware Of

SMTAlignmentError

TopologyAffinityError

Topology-Aware Scheduling

Making It Work: Collaboration Between Platform and Workload Teams

Platform admins: publish topology and sizing guidance

Workload owners: understand the constraints, size accordingly

Verifying alignment in practice

Appendix: Related CPU Manager Options

DRA and Future Direction

Wrapping Up

Trading Peace for Pace: A Few Weeks with Claude Code

Understanding the Kubernetes Scale subresource

Understanding the scale subresource in Kubernetes

Why’s the scale subresource Necessary?

Uses of the scale subresource

Scale subresource and Custom Resources

PDB, Custom Resources, and the scale subresource

What I Wish I Knew About Incident Management

Why I am writing this post

What this post is not about

First oncall week

Before Oncall Starts

Oncall handoff

Organizing Slack during an oncall week

Signal vs Noise

Trust, but verify - not all alerts are created equal

Declaring an incident

Communication during an incident

The incident title and scoping the impact

Establish communication channels and an incident lead

Communicate changes to the system

Provide regular updates

Incident response is a collaborative process

Get help early

Working with others

Video Conferencing is your friend

Towards a resolution

Looking at changes

Keep calm and carry on - one step at a time

Learning from the incident

How a Kubernetes Pod Gets an IP Address

Some Background Concepts

Container Networking: A Very Brief Overview

Containers on the same host

Containers on different hosts

What Is CRI?

What is CNI?

Assigning Subnets To Nodes For Pod IP Addresses

Node IPAM Controller

Kubelet, Container Runtime and CNI Plugins - how it’s all stitched together

Interactions between Container Runtime and CNI Plugins

Interactions Between CNI Plugins

Flannel CNI Plugin

Bridge CNI Plugin

Host-local IPAM CNI plugins

Summary

References

Container Networking

How Flannel Works

CRI and CNI

Kubernetes NodePort and iptables rules

Some background

Use `minReadySeconds`

Level 1: Logical Core Pinning with `cpuManagerPolicy: static`

Level 2: Physical Core Pinning with `full-pcpus-only`

Level 3: Full NUMA Alignment with `single-numa-node`

Understanding the `scale` subresource in Kubernetes

Why’s the `scale` subresource Necessary?

Uses of the `scale` subresource

PDB, Custom Resources, and the `scale` subresource