Kubernetes Interview Questions & Answers - Fundamentals

Ans: Kubernetes (K8s for short) is a tool that runs your apps inside containers and takes care of them for you, so you’re not stuck babysitting servers by hand. Here’s the easiest way to picture what it actually does, step by step:

1. You package your app into a container - a small, self-contained box holding your app and everything it needs to run.
2. You tell Kubernetes what you want - something like “run 3 copies of this app, and always keep 3 running.”
3. Kubernetes finds room and starts it - it looks across its machines (called nodes), picks ones with free capacity, and starts your containers there.
4. If a container crashes, Kubernetes notices and fixes it - a replacement gets started automatically, with nobody needing to log in and restart anything.
5. If a whole machine dies, Kubernetes moves your app - it starts fresh copies on a healthy machine instead.
6. If traffic grows, Kubernetes can scale up - run more copies, and spread incoming traffic across all of them evenly.
7. When you ship a new version, Kubernetes rolls it out gradually - swapping old containers for new ones a few at a time, so users never see downtime, and it can undo the change instantly if something breaks.

That’s really the whole idea: you describe what you want, and Kubernetes keeps making it true, automatically, all the time - instead of you doing steps 3 through 7 by hand.

Why this matters: doing that reliably across dozens or hundreds of machines, all with different capacity, that can fail at any time, just isn’t realistic to manage by hand. Google built Kubernetes based on lessons from their own internal system (called Borg) specifically to automate this, and later donated it to the Cloud Native Computing Foundation (CNCF), which maintains it today. That’s exactly why it became the standard way to run containers in production.

Ans: A Kubernetes cluster is a set of machines, called nodes, working together under Kubernetes’ management. Every cluster splits into two halves with very different jobs: a control plane that makes decisions, and one or more worker nodes where your actual applications run.

The control plane doesn’t run your application containers at all - its entire job is to watch the cluster’s state, make decisions, and keep things consistent. It’s made up of a handful of components:

• kube-apiserver - the front door; every request from any source goes through here.
• etcd - the cluster’s single source of truth, storing every object’s current state.
• kube-scheduler - decides which node a new Pod should run on.
• kube-controller-manager - runs the reconciliation loops that keep reality matching what you asked for.

Worker nodes are where Pods actually live and run, and each one runs three things to make that possible:

• kubelet - the agent that talks to the control plane and makes sure its assigned Pods are actually running.
• container runtime - pulls images and starts/stops containers (containerd is the common one today).
• kube-proxy - handles the networking so traffic can reach Pods on that node.

On a managed service like EKS, AWS runs the entire control plane for you behind the scenes - you only ever see and manage the worker nodes (and even those can be serverless with Fargate). But conceptually, every Kubernetes cluster anywhere follows this same two-part shape.

Ans: Kubernetes architecture is built around a clean separation: the control plane decides what should happen, and worker nodes carry it out. Nothing talks directly to anything else - every component, on both sides, communicates exclusively through kube-apiserver.

Control-plane components and their roles:

• kube-apiserver - exposes the REST API and is the only component that ever talks to etcd directly. Every read and write in the whole cluster passes through it, and it also runs authentication, authorization, and admission control on every request.
• etcd - a distributed, consistent key-value store that holds the entire state of the cluster: every object’s spec and status. If etcd is lost, the cluster forgets everything it was supposed to be running.
• kube-scheduler - watches for Pods that don’t have a node assigned yet, and picks the best available node for each one, based on resource availability, taints/tolerations, and affinity rules. It only decides; it never starts anything itself.
• kube-controller-manager - runs dozens of independent reconciliation loops (the Node controller, ReplicaSet controller, Endpoints controller, and more) bundled into a single process, each one constantly nudging reality toward whatever was declared.

Worker-node components and their roles:

• kubelet - the node’s agent; watches the API server for Pods assigned to its node, and makes sure they’re actually running and healthy, restarting anything that fails.
• container runtime - does the real work of pulling images and starting/stopping containers, communicating with kubelet through the Container Runtime Interface (CRI).
• kube-proxy - programs the node’s networking rules so traffic aimed at a Service reaches a healthy backing Pod, wherever it happens to be running.

How they work together: when you submit a manifest, it lands on kube-apiserver, gets validated and written to etcd, and from there it’s all reactive. Controllers watching the API server notice the new object and create whatever’s needed underneath it (a Deployment creates a ReplicaSet, which creates Pods). The scheduler notices unscheduled Pods and assigns them to nodes. kubelet on the chosen node notices a Pod assigned to it and starts the containers via the runtime. Nothing is pushed by the control plane directly to a node - every component pulls its own work by watching the API server, which is exactly what keeps the architecture loosely coupled and resilient to individual pieces restarting or falling behind temporarily.

Ans: These three terms get used almost interchangeably in conversation, but they mean slightly different things, and the distinction actually matters once you’re reading Kubernetes documentation closely.

An object is a specific, named record of something you want to exist - “a Deployment called web-app with 3 replicas running nginx:1.25” is an object. It’s a concrete instance, persisted in etcd, with its own spec (what you asked for) and status (what’s actually happening).

A resource is the general term for a type of object and the API endpoint used to manage it. deployments, pods, and services are all resources - and kubectl get pods is really just a GET request to the pods resource’s API endpoint. Every specific Pod you create is an object of the pods resource.

An API resource is the formal name for a resource as exposed through the Kubernetes API, tied to a specific API group and version - for example, deployments lives under the apps/v1 API group and version, while pods lives in the core (unnamed) v1 group. kubectl api-resources lists every API resource a cluster currently supports, along with which group it belongs to and what operations (get, list, watch, delete, and so on) are available on it.

So in short: resource = the type/endpoint (pods), object = a specific instance of that type (my-app-pod-abc123), and API resource = the resource as registered with a specific group and version in the API server’s discovery system. In everyday conversation people blur “object” and “resource” together constantly, and that’s usually fine - but when someone says “API resource” they specifically mean the group/version-scoped API endpoint.

Ans: Desired state is what you’ve told Kubernetes you want to be true - “always keep 3 replicas of this Deployment running,” for example. You express it declaratively in a manifest and hand it to the API server, and it gets stored in etcd as the authoritative record of your intent.

Current state is what’s actually happening in the cluster right now - maybe only 2 of those 3 replicas are healthy because one crashed, or a node just went offline taking a Pod with it. Kubernetes constantly observes this real, live state through its components reporting status back to the API server.

The whole system is built around continuously comparing these two things and closing any gap. Controllers watch both the desired state (from etcd) and the current state (from what’s actually running), and the instant they diverge - a Pod crashes, a node dies, someone deletes something by accident - a controller notices and takes corrective action to bring current state back in line with desired state. This is what people mean when they say Kubernetes is “self-healing”: it’s not magic, it’s just this comparison loop running constantly, for every object, all the time, regardless of what caused the drift in the first place.

Ans: Reconciliation is the mechanism that actually makes “desired state vs current state” mean something in practice. Every controller in Kubernetes runs its own reconciliation loop, and despite managing wildly different kinds of objects, they all follow the same basic shape:

1. Watch - subscribe to changes on the objects the controller cares about, via the API server’s watch mechanism.
2. Compare - check the current, real state of those objects against what was declared as desired.
3. Act - if there’s a gap, do whatever’s needed to close it (create a missing Pod, delete an extra one, update a field).
4. Go back to step 1, forever.

This is often described as “level-triggered” rather than “edge-triggered” - a controller doesn’t just react to a single event and assume it worked; it keeps re-checking the entire desired state against reality, every time. That matters a lot in practice: if a controller misses an update (a network blip, a restart), it doesn’t matter, because the next reconciliation pass re-derives the correct action from scratch by comparing full states rather than relying on having seen every individual event. That’s exactly what makes Kubernetes resilient to controllers restarting, missing updates, or racing with each other - the loop is self-correcting by design, not by careful event bookkeeping.

Ans: Declarative configuration means you describe what you want to exist, and let Kubernetes figure out how to get there - as opposed to an imperative approach, where you’d issue a sequence of commands telling it exactly what actions to take (“create this Pod,” “now delete that one,” “now update this field”).

In practice, this means writing a manifest - a YAML (or JSON) file describing an object’s desired spec - and handing it to Kubernetes with kubectl apply -f manifest.yaml. Kubernetes compares what you submitted against what currently exists and figures out the difference itself: creating the object if it’s new, updating just the fields that changed if it already exists, and leaving everything else alone.

This is the foundation that reconciliation and self-healing are built on. Because you’ve declared an end state rather than a sequence of steps, Kubernetes can keep re-applying that same declaration indefinitely - after a crash, after a node failure, after literally any kind of disruption - and always arrive back at what you asked for. An imperative model can’t do this nearly as cleanly, since it has no persistent record of your original intent, only the individual commands you already ran.

Ans: A manifest is the YAML (or JSON) file you write to declare a Kubernetes object’s desired state. Every manifest shares the same basic structure, no matter what kind of object it describes:

apiVersion: apps/v1        # Which API group/version this object belongs to
kind: Deployment            # What type of object this is
metadata:                   # Identifying info: name, namespace, labels, annotations
  name: nginx-deployment
  labels:
    app: nginx
spec:                       # What you want: the desired state
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.25

apiVersion and kind together tell the API server which schema to validate the object against. metadata identifies the object (its name has to be unique within its namespace) and can carry labels and annotations for organizing and describing it. spec is where the actual desired state lives, and its shape is completely different depending on kind - a Deployment’s spec looks nothing like a Service’s.

How it gets processed:

1. kubectl apply -f manifest.yaml reads the file, converts it to JSON, and sends it to kube-apiserver.
2. The API server authenticates the request, checks authorization (RBAC), and runs it through admission control.
3. The object gets validated against its schema and written to etcd.
4. Any relevant controllers, watching for changes, notice the new or updated object and start reconciling toward it.

Once an object exists, status also appears in it automatically - added and maintained by Kubernetes itself to reflect what’s actually happening, distinct from the spec you wrote, which only ever reflects what you asked for.

Ans: kubectl is the command-line tool for interacting with a Kubernetes cluster - creating and inspecting objects, checking logs, exec’ing into containers, debugging problems. If you work with Kubernetes regularly, this is the tool you live in day to day.

Under the hood, kubectl isn’t doing anything magical - every command it runs boils down to a plain HTTPS REST call to kube-apiserver. kubectl get pods is a GET to /api/v1/namespaces/<namespace>/pods; kubectl apply -f file.yaml is a POST or PATCH with the manifest’s content as the body. kubectl just formats the JSON response nicely for your terminal and adds a lot of convenience on top (YAML parsing, diffing on apply, tab completion, and so on).

A few concepts matter for understanding how it actually operates:

• kubeconfig - kubectl reads this file to figure out which cluster to talk to and what credentials to authenticate with.
• context - a named cluster + user + namespace combination inside kubeconfig; whichever context is “current” decides where your commands actually go.
• verbs - get, create, apply, delete, describe, logs, exec, and so on, each mapping to a different kind of API call or client-side behavior.
• output formatting - -o yaml, -o json, -o wide, -o jsonpath=... all control how the API server’s response gets rendered back to you.

Because everything goes through the same API server as every other client, kubectl has no special privileges of its own - whatever it can and can’t do is governed entirely by the same RBAC rules that apply to any other identity making the same request.

Ans: kubeconfig is the file (usually ~/.kube/config, though you can point elsewhere with $KUBECONFIG or --kubeconfig) that tells kubectl how to find and authenticate to a cluster. It’s a plain YAML file built around three sections that get combined into a usable connection:

apiVersion: v1
kind: Config
clusters:
- name: prod-cluster
  cluster:
    server: https://api.prod.example.com
    certificate-authority-data: <base64 CA cert>
users:
- name: my-user
  user:
    client-certificate-data: <base64 cert>
    client-key-data: <base64 key>
    # or: token, or an exec plugin for things like AWS IAM auth
contexts:
- name: prod
  context:
    cluster: prod-cluster
    user: my-user
    namespace: default
current-context: prod

• clusters - the server address and CA certificate needed to trust and reach a given cluster.
• users - the credentials to authenticate as (a client cert, a bearer token, or an exec plugin that generates a token dynamically, which is how EKS auth works).
• contexts - a named combination of a cluster, a user, and a default namespace, bundled together under one name.

You can merge multiple kubeconfig files (by pointing $KUBECONFIG at several paths separated by a colon), which lets you accumulate access to many clusters in one place and just switch between them by changing which context is active, instead of juggling separate flags for cluster address and credentials every time.

Ans: A context is a named shortcut inside kubeconfig that bundles together a cluster, a user, and a default namespace, so you don’t have to specify all three separately on every single command. Instead of remembering to pass --server=..., --user=..., and --namespace=... every time, you just set one context as current and every subsequent command automatically targets the right combination.

# See every context kubectl knows about
kubectl config get-contexts

# See which one is currently active
kubectl config current-context

# Switch to a different one
kubectl config use-context staging-cluster

This is exactly what makes it safe and convenient to work across dev, staging, and production from the same terminal - you switch contexts explicitly rather than having to reconfigure connection details each time, which also reduces the risk of accidentally running a command against the wrong cluster. kubectl resolves which context to use in a clear order: an explicit --context flag on the command overrides everything, otherwise it falls back to whatever current-context is set to inside the active kubeconfig file.

Ans: A namespace lets you divide a single physical cluster into multiple virtual ones. It’s a way of scoping objects so that names only need to be unique within a namespace, not across the whole cluster - two teams can both have a Service called api as long as each lives in its own namespace.

Why they’re used: namespaces are the standard way to separate concerns on shared infrastructure - splitting dev, staging, and prod environments, or isolating different teams’ or projects’ resources from each other, all on the same physical cluster instead of needing entirely separate clusters for each.

How the isolation works: most namespaced objects (Pods, Services, Deployments, ConfigMaps, Secrets) get stored with their namespace baked directly into their key in etcd, so lookups and name uniqueness are naturally scoped. On top of that, several Kubernetes features are namespace-aware and let you apply real boundaries: RBAC Roles and RoleBindings can grant permissions scoped to just one namespace, ResourceQuotas can cap how much CPU/memory/object-count a namespace is allowed to consume, and NetworkPolicies can restrict traffic between namespaces.

When to use them: as soon as more than one team or more than one environment is sharing a cluster, namespaces are worth using. A few things are deliberately cluster-scoped and sit outside any namespace entirely - Nodes and PersistentVolumes being the classic examples - because they represent physical or shared infrastructure rather than something that belongs to any one team or environment.

Ans: Both labels and annotations are key-value pairs you attach to an object’s metadata, but they exist for very different reasons.

Labels are meant to be queried against - they’re how Kubernetes finds and groups related objects. A Service finds the Pods it should route to by matching labels, not by any hardcoded list of Pod names; a Deployment’s ReplicaSet knows which Pods belong to it the same way. Labels are meant to be short, structured, and meaningful for selection: app: frontend, env: production, tier: backend.

metadata:
  labels:
    app: frontend
    env: production

Annotations are purely descriptive - metadata that’s useful to a human or a tool, but that nothing should ever select by. Build numbers, a git commit hash, a contact email, configuration for some third-party controller - these all belong in annotations, and they can hold much larger or less structured values than labels are meant for.

The rule of thumb: if something needs to be found or grouped by it, use a label. If it’s just informational, use an annotation. Labels power the entire selection mechanism in Kubernetes - Services, Deployments, NetworkPolicies, and more all rely on label selectors to find the objects they care about, which is exactly why keeping labels clean and consistent across a cluster matters so much in practice.

Ans: A label selector is a query that finds objects by matching against their labels, rather than by name. It’s the mechanism that lets one object dynamically “point at” a changing set of others.

There are two forms: equality-based selectors (app=frontend, or the negated env!=prod) and set-based selectors (environment in (prod, staging), tier notin (cache), or just app to match anything that has the key at all, regardless of value).

apiVersion: v1
kind: Service
metadata:
  name: frontend-svc
spec:
  selector:
    app: frontend
  ports:
    - port: 80

Here, the Service continuously watches for any Pod carrying the label app: frontend and adds it as a routable endpoint automatically - including Pods that show up after the Service already exists. Delete a Pod, and it drops out of the endpoint list just as automatically. Deployments and ReplicaSets work the same way: a ReplicaSet’s spec.selector defines which Pods it considers “mine,” and it continuously reconciles the count of matching Pods against replicas, regardless of whether those Pods were created by this exact ReplicaSet or just happen to carry matching labels (which is also why label collisions between unrelated ReplicaSets are a real and pretty common source of bugs).

Ans: An owner reference is a pointer stored on an object’s metadata, pointing back to whichever object created it. A Pod created by a ReplicaSet carries an owner reference back to that ReplicaSet; the ReplicaSet, in turn, carries one back to the Deployment that created it - forming a chain all the way up.

Two fields on an owner reference matter beyond just “who created this”: controller: true marks the managing owner (only one owner reference on an object can set this, and it’s how a ReplicaSet asserts “this Pod is mine” so two controllers don’t fight over the same Pod), and blockOwnerDeletion: true tells the garbage collector to hold off deleting the owner until this specific dependent is gone.

This is what powers garbage collection. When you delete a Deployment, Kubernetes doesn’t need any special-cased “also delete its ReplicaSets and Pods” logic - the garbage collector controller simply watches for objects whose owners have disappeared and cleans up anything left orphaned, following the reference chain down automatically. You can control exactly how that cleanup happens with a deletion propagation policy: Background (the default - the owner is deleted immediately and dependents get cleaned up shortly after), Foreground (the owner is marked for deletion but stays visible until every dependent is actually gone first), or Orphan (the owner is deleted but dependents are deliberately left behind, with their owner reference just stripped).

Ans: Admission control is the stage of the API request lifecycle that runs after a request has been authenticated and authorized, but before it’s ever written to etcd. It’s the API server’s last checkpoint - a chance to inspect, modify, or outright reject an object before it becomes real.

Admission controllers are built-in pieces of logic compiled into kube-apiserver - NamespaceLifecycle (blocks creating new objects in a namespace that’s being deleted) and ResourceQuota (enforces per-namespace resource caps) are two common examples. They run in two ordered groups: every mutating controller runs first, and each one can modify the incoming object (setting a default value, for instance); then every validating controller runs against the final, fully-mutated object, and any single rejection fails the whole request.

Admission webhooks let you plug your own custom logic into that exact same pipeline, over HTTP, instead of relying only on what’s built into Kubernetes itself. There are two kinds, matching the same mutating/validating split: a MutatingAdmissionWebhook can actually change the object (auto-injecting a sidecar container is a classic example - this is how Istio adds its Envoy proxy to every Pod), while a ValidatingAdmissionWebhook can only approve or reject a request, never alter it. Tools like OPA Gatekeeper (policy enforcement) and cert-manager (webhook-based cert validation) are both built entirely on this mechanism, which is exactly what lets Kubernetes stay extensible without every possible policy needing to be baked into the core API server.

Ans: The container runtime is the component that does the actual, low-level work of running a container - pulling the image, setting up Linux namespaces and cgroups for isolation and resource limits, starting and stopping the process, and managing its filesystem. Kubernetes itself never runs a container directly; it always delegates that job to a runtime.

The interface between kubelet and the runtime is the CRI (Container Runtime Interface) - a gRPC API that covers the basics any runtime needs to support: pull an image, create a container, start it, stop it, remove it. Because this interface is standardized, Kubernetes doesn’t need to know anything runtime-specific, and you can swap which runtime a node uses without changing anything about Kubernetes itself.

containerd is the most common runtime today - a lightweight, CNCF graduated project, and the default on most managed Kubernetes services including EKS. Interestingly, Docker itself is built on top of containerd; when people ran Docker as the container runtime in older Kubernetes versions, Kubernetes was really just talking to containerd underneath it anyway (with an extra shim layer that’s since been removed as Docker-as-runtime support was dropped from Kubernetes). Under containerd, the actual container creation happens via runc, a low-level OCI-compliant runtime that does the final work of setting up namespaces and cgroups and launching the process.

Ans: kubelet is the agent that runs on every worker node and acts as the bridge between that node and the rest of the cluster. Its entire job boils down to one thing: make sure whatever Pods have been assigned to this node are actually running and healthy, and keep the control plane informed about their status.

How it communicates with the control plane: kubelet continuously watches kube-apiserver for Pods scheduled to its node - it’s a pull model, not a push model, which means nodes only ever need outbound access to the API server, never the other way around. That’s a deliberate design choice that makes life much easier in locked-down network environments, like nodes sitting in private subnets. It authenticates to the API server with a client certificate (often provisioned via TLS bootstrapping when the node first joins), and it also runs its own small HTTPS server so the API server can reach into it for things like kubectl exec, logs, and port-forward.

How it manages Pods: for every Pod assigned to it, kubelet calls the container runtime through CRI to pull the required images and start the containers, mounts any volumes and injects config from ConfigMaps and Secrets, and then keeps running liveness, readiness, and startup probes for the Pod’s entire lifetime - restarting containers that fail liveness checks, and pulling Pods with failing readiness checks out of Service traffic without killing them. Throughout all of this, it keeps reporting node and Pod status back to the API server, which is exactly what other components (like the scheduler and any watching controllers) rely on to know the real, current state of the cluster.

Ans: The Kubernetes API is the REST interface exposed by kube-apiserver, and it is genuinely the only way to change anything in a cluster - kubectl, controllers, kubelets, Helm, Terraform, any third-party tool, they all go through this same API with no back door or shortcut around it.

• API resources are the named types you interact with - pods, deployments, services - each one exposed as a REST endpoint supporting a consistent set of operations (list, get, create, update, delete, watch) over standard HTTP verbs.
• API groups organize resources so different parts of the API can evolve independently instead of one giant flat namespace. Core, foundational types like Pods and Services live in the legacy, unnamed “core” group (referenced as just v1); Deployments and StatefulSets live under apps; Jobs and CronJobs live under batch.
• API versions (v1alpha1, v1beta1, v1) indicate how stable a given API is. alpha can change or vanish without warning, beta is more solid but might still shift slightly, and v1/GA comes with real backward-compatibility guarantees. A resource can even be served at multiple versions simultaneously while it matures from alpha through to GA.

How clients interact with it: every client, kubectl included, first uses the API server’s discovery endpoints (/api, /apis) to figure out what groups, versions, and resources a given cluster actually supports, then issues plain HTTPS requests against the relevant endpoint - GET /apis/apps/v1/namespaces/default/deployments/my-app, for instance. Every one of those requests goes through the same authentication, authorization, and admission pipeline regardless of which client sent it, which is exactly what makes the API the single, consistent enforcement point for everything that happens in a cluster.

Ans: These are all apiVersion values you’ll write at the top of a manifest, and each one identifies a specific API group plus a specific version within that group.

• v1 refers to the stable, legacy “core” API group, which has no name of its own (it predates the group system). This is where the foundational object types live: Pod, Service, Namespace, ConfigMap, Secret, Node, PersistentVolume. When a manifest just says apiVersion: v1, it’s this group being referenced.
• apps/v1 is the apps API group at version v1, and it’s where Kubernetes’ workload controllers live: Deployment, StatefulSet, DaemonSet, ReplicaSet. This is the version to use today - earlier versions like apps/v1beta1 and apps/v1beta2 existed during Kubernetes’ early days and have since been removed from modern clusters entirely.
• batch/v1 is the batch group, covering run-to-completion workloads: Job and CronJob. A Job runs Pods until they finish successfully, in contrast to a Deployment’s Pods, which are expected to run indefinitely; a CronJob just creates Jobs on a schedule.

Why the split exists: grouping resources this way lets different parts of the Kubernetes API evolve, version, and stabilize independently of each other. The apps group could go through alpha/beta iterations for new workload types without having to bump the version number of the entire core API, and vice versa - each group moves through its own stabilization lifecycle on its own schedule, which keeps Kubernetes’ overall API surface manageable as it grows.

Ans: Amazon EKS (Elastic Kubernetes Service) is AWS’s managed Kubernetes offering - instead of standing up and operating your own control plane (API server, etcd, scheduler, controller-manager) on EC2 instances you patch and babysit yourself, AWS runs and maintains all of that for you, and hands you a standard Kubernetes API endpoint that behaves exactly like any other Kubernetes cluster would.

Why EKS is used: running a highly available, correctly patched, properly secured Kubernetes control plane by hand is genuinely hard work - it means managing etcd backups, coordinating version upgrades, securing access to the API server, and handling failover, all before you’ve even deployed a single application. EKS takes that entire burden off your plate, letting your team focus on the applications running on top instead of the plumbing underneath.

What AWS manages: the entire control plane - the API server, etcd, the scheduler, the controller-manager - spread across multiple Availability Zones for high availability, patched and upgraded by AWS, with the underlying infrastructure kept completely invisible to you. You never SSH into a control-plane instance; the Kubernetes API is the only interface you ever get.

What remains the customer’s responsibility: everything on the data-plane side and above. That means the worker nodes themselves (unless you’re using Fargate, where AWS takes on even more of this), the workloads you deploy, your VPC and subnet design, IAM roles and RBAC configuration, and keeping your applications and node-level software patched and secure. EKS makes running Kubernetes dramatically easier, but it doesn’t make you a passive bystander - you’re still very much responsible for what you build on top of it.

Ans: An EKS cluster has two distinct halves that work together but are managed very differently: a control plane AWS runs for you, and a data plane you’re responsible for.

The control plane runs inside an AWS-owned and AWS-managed VPC, not your own account, and consists of multiple API server replicas, an etcd cluster, the scheduler, and the controller-manager, all spread across at least three Availability Zones. AWS reaches into your VPC by provisioning elastic network interfaces (ENIs) directly into the subnets you specify at cluster creation, which is how the control plane can talk to your worker nodes even though its own infrastructure stays completely hidden from you.

Your VPC and subnets house everything you’re responsible for: worker nodes, the ENIs the control plane places into your subnets, and typically a mix of public subnets (for internet-facing load balancers) and private subnets (for worker nodes, following AWS’s recommended practice).

Nodes - EC2 instances (self-managed or in a managed node group) or serverless Fargate - run your actual Pods, each one running kubelet, kube-proxy, and a container runtime, exactly like any other Kubernetes worker node.

Networking relies on the Amazon VPC CNI plugin by default, which assigns each Pod a real IP address from your VPC’s own address space, rather than an overlay network - meaning Pods are directly routable within your VPC just like EC2 instances are.

Authentication swaps Kubernetes’ usual certificate or static-token model for IAM: callers authenticate using signed AWS credentials, and that identity gets mapped to a Kubernetes user or group through access entries, after which normal Kubernetes RBAC takes over for authorization.

AWS integrations run throughout: the cloud-controller-manager equivalent logic provisions load balancers for LoadBalancer Services, EBS/EFS CSI drivers (installable as EKS add-ons) handle persistent storage, and IAM Roles for Service Accounts (IRSA) let individual Pods assume fine-grained IAM permissions without sharing a node-wide role.

Ans: The EKS managed control plane is the set of Kubernetes control-plane components - API server, etcd, scheduler, controller-manager - that AWS provisions, operates, and keeps healthy on your behalf, so you never have to run this infrastructure yourself.

Where it runs: inside a VPC that AWS owns and manages, completely separate from your own AWS account’s infrastructure. AWS reaches into your VPC only by placing elastic network interfaces into the subnets you designate at cluster creation - that’s the sole connection point between AWS’s control-plane infrastructure and your network.

How it’s made highly available: control-plane components run as multiple replicas spread across at least three Availability Zones, sitting behind a load-balanced cluster endpoint. The etcd cluster is similarly replicated across AZs and kept consistent via the Raft protocol, so losing an entire AZ still leaves a working quorum. AWS continuously health-checks every piece and automatically replaces anything unhealthy, without any action (or even visibility) required from you.

How customers interact with it: exclusively through the standard Kubernetes API - kubectl, eksctl, Terraform, Helm, or any other tool all just talk to the cluster’s API endpoint the same way they would with any Kubernetes cluster anywhere. There’s no host-level access, no SSH, no way to log into a control-plane instance even if you wanted to - the API is intentionally the only door in, which is also exactly what keeps the underlying infrastructure fully abstracted away from you.

Ans: The cluster endpoint is the URL that kubectl and every other client use to reach your cluster’s API server - a managed, highly available address sitting in front of the API server replicas, so you never connect to an individual instance directly. At cluster creation (and any time after), you choose how it’s exposed: public, private, or both together.

Public endpoint: the API server gets a public DNS name reachable over the internet, secured behind AWS-managed infrastructure. By default it’s open to 0.0.0.0/0, but you can restrict it to specific IP ranges with --public-access-cidrs. It’s the easiest way to reach your cluster from anywhere without extra networking setup, but leaving it wide open does widen your attack surface, so locking down the allowed CIDRs is a common and sensible hardening step.

Private endpoint: when enabled, EKS sets up VPC endpoints via AWS PrivateLink inside your subnets, and the cluster’s DNS name resolves to private IPs reachable only from inside the VPC (or anything connected to it - VPN, Direct Connect, peered VPCs). There’s no path in from the public internet at all in this mode.

When to use each: public access (ideally CIDR-restricted) is simplest for small teams or when CI/CD runners live outside your VPC. Private-only is the stronger security posture for production, security-sensitive environments, at the cost of needing VPN, Direct Connect, or a bastion host for anyone who needs access from outside the VPC. Many teams run both together - private access for normal traffic, plus a tightly CIDR-restricted public endpoint as a convenient fallback path.

Ans: Every EKS cluster is anchored to a VPC in your own AWS account - it’s where your subnets live, where worker nodes and load balancers get placed, and its routing and security configuration decides how your cluster reaches the internet, other AWS services, and anywhere on-prem you’re connected to.

Subnets are how EKS places both worker nodes and the control plane’s own network interfaces within your VPC, and each subnet is tied to a single Availability Zone. AWS’s standard guidance is to spread subnets across at least two (ideally three) AZs for resilience, use public subnets for internet-facing resources like load balancers, and keep worker nodes in private subnets with outbound access via a NAT gateway.

Networking requirements for control-plane-to-node communication: when you create the cluster, you tell EKS which subnets to use, and AWS provisions elastic network interfaces directly into those subnets so the control plane can reach kubelet on every node (for things like kubectl exec and logs) and nodes can reach the API server. This means your subnet’s route tables, NACLs, and the EKS-managed cluster security group all have to actually allow this traffic to flow - a subnet with broken routing or an overly restrictive NACL can silently break control-plane-to-node communication even though everything looks fine from the EKS console.

Because subnets are AZ-specific, they also matter for storage: an EBS volume is tied to the AZ it was created in, so a Pod’s PVC needs to land in the same AZ as the node it’s scheduled to, which is exactly why volumeBindingMode: WaitForFirstConsumer on your StorageClass matters so much in a multi-AZ EKS setup.

Ans: The cluster security group is a security group that EKS creates and manages automatically, and it’s specifically what allows traffic to flow correctly between the control plane and your worker nodes and Pods. AWS attaches it to the ENIs it places in your subnets, and by default to your worker nodes as well.

Its whole purpose is to guarantee that the specific ports and protocols Kubernetes actually needs - the API server reaching kubelet on each node, nodes reaching back to the API server, health checks, and so on - stay open, without you needing to hand-craft those exact rules yourself. Since EKS manages it, it’s intentionally permissive between the control plane and nodes: you’re not expected to lock this one down further, since doing so risks breaking core cluster functionality in ways that can be genuinely confusing to debug.

It’s important to understand what this security group doesn’t cover, though - it’s not a substitute for your own security groups controlling node-to-node traffic, node-to-internet egress, or traffic between your Pods and other AWS services. Those you still design, attach, and lock down yourself, same as you would for any other EC2-based infrastructure in your account.

Ans: An EKS node is a worker machine, typically an EC2 instance, that’s part of your data plane and actually runs your Pods. Aside from being provisioned and managed within AWS’s tooling, it’s functionally identical to a worker node in any other Kubernetes cluster.

Its role: run the Pods that get scheduled to it, report its health and capacity back to the control plane, and handle the actual pulling of images, starting of containers, and routing of Service traffic locally.

How it connects to the cluster: on boot, a bootstrap script points kubelet at the cluster’s API endpoint and CA certificate, and kubelet authenticates using the node’s IAM role credentials via the same webhook token authenticator mechanism EKS uses for human users. Once that identity is validated and mapped (through an access entry or the older aws-auth ConfigMap) to a Kubernetes identity in the system:nodes group, kubelet registers a Node object with the API server and the node becomes schedulable.

Major components running on it:

• kubelet - the node agent, same role as anywhere else.
• kube-proxy - programs the networking rules for Service traffic.
• container runtime - containerd by default on EKS-optimized AMIs.
• VPC CNI plugin - assigns each Pod a real, routable IP address from the VPC’s own address space, rather than using an overlay network.

None of this is EKS-specific behavior at the Kubernetes layer - the only genuinely EKS-specific part is how the node authenticates (via IAM) and how it’s provisioned (via a managed node group, self-managed ASG, or Fargate).

Ans: A managed node group is EKS creating and operating an Auto Scaling Group of EC2 worker nodes on your behalf, using either an EKS-optimized AMI or a custom launch template you supply.

Architecture: under the hood it really is just an ASG - EKS just owns and orchestrates its lifecycle for you, rather than you managing the ASG directly through EC2’s own console or API.

Lifecycle management: EKS bootstraps each new instance into the cluster automatically (no manual bootstrap scripting required), and it hooks into the ASG’s lifecycle events so it can intercept a scale-in and drain a node properly before terminating it, instead of yanking it out from under running Pods.

Scaling: you can scale a managed node group directly through the EKS API or eksctl, or pair it with the Cluster Autoscaler or Karpenter for automatic scaling based on pending Pod demand.

Upgrades: when you update the node group’s AMI or Kubernetes version, EKS performs a rolling replacement rather than an in-place patch:

1. Launches new nodes on the updated AMI/launch template.
2. Waits for them to become Ready.
3. Cordons the old nodes so no new Pods land there.
4. Drains each old node, respecting any PodDisruptionBudgets.
5. Terminates the drained nodes and shrinks the ASG back down.

AWS responsibilities: provisioning and terminating instances, handling the bootstrap process, and orchestrating the rolling upgrade safely. Your responsibilities: choosing instance types and sizing, setting updateConfig (max unavailable during upgrades), and making sure PodDisruptionBudgets exist so upgrades don’t cause an outage. Managed node groups are the right default for most teams - they remove almost all the manual node lifecycle work while still leaving you full control over sizing and scaling behavior.

Ans: A self-managed node is a worker node you provision and operate entirely yourself - your own Auto Scaling Group, your own AMI and launch template, joined to the cluster manually rather than through EKS’s managed node group tooling.

How it differs from a managed node group: with a managed node group, EKS handles bootstrapping, rolling upgrades, and lifecycle-aware draining automatically. With a self-managed node, none of that is automatic - you write and maintain your own bootstrap logic (or rely on the EKS-optimized AMI’s built-in script), configure the ASG yourself, and orchestrate upgrades and drains on your own, typically using a tool like eksctl, Terraform, or your own scripts and hooks.

What falls on the customer: everything about the instance’s lifecycle - choosing and maintaining the AMI, writing correct bootstrap configuration pointing at the right cluster endpoint and CA data, setting up the node IAM role and its access entry mapping, handling AMI/Kubernetes version upgrades through your own rolling-replacement process, and monitoring node health yourself instead of relying on EKS to do it.

When it’s still worth it: self-managed nodes matter when you need something a managed node group doesn’t support - custom AMIs with a nonstandard kernel or security hardening that AWS’s managed AMIs don’t offer, specific driver requirements for specialized hardware like GPUs, or infrastructure patterns your organization’s existing automation is already built around. For the majority of workloads, though, a managed node group removes real operational burden for very little loss of control, which is why it’s the recommended default.

Ans: EKS Fargate is a serverless compute option for EKS - Pods run without you ever provisioning, patching, or managing an EC2 instance underneath them. AWS handles the compute entirely.

How it works without EC2 worker nodes: when a Pod is scheduled to run on Fargate, AWS provisions a right-sized, isolated micro-VM just for that one Pod, runs it there, and tears the micro-VM down once the Pod is gone. It still shows up as a Node object if you run kubectl get nodes, so the rest of Kubernetes (Services, DNS, kube-proxy routing) works normally - but you’ll never patch, scale, or SSH into that “node,” because there’s no persistent underlying instance for you to manage at all.

How Fargate profiles decide placement: a Fargate profile is a rule you define saying “any Pod whose namespace (and optionally whose labels) match this profile should run on Fargate instead of EC2.” When a new Pod is created, EKS checks it against your configured Fargate profiles; a match routes it to Fargate, no match sends it through normal EC2-based scheduling instead.

Real constraints worth knowing: because each Pod is fully isolated with no shared underlying node, Fargate doesn’t support DaemonSets, privileged Pods, hostNetwork or hostPort, and Fargate profile subnets must be private. It’s a great fit for batch jobs, infrequent workloads, or teams who want zero node management, but it’s not a universal replacement for EC2-backed nodes.

Ans: EKS add-ons are AWS-curated, versioned packages for common pieces of cluster infrastructure - the VPC CNI, CoreDNS, kube-proxy, and the EBS/EFS CSI drivers are the most common examples - that you install and keep updated through the EKS API instead of applying and maintaining raw Kubernetes manifests yourself.

How they integrate: under the hood, an add-on is still just Kubernetes objects (DaemonSets, Deployments, ConfigMaps, RBAC) applied to your cluster - the difference is that EKS tracks it as a managed resource with its own version and health status, similar to how CloudFormation tracks a stack resource. Many add-ons also integrate with IAM through IRSA, so their Pods can assume specific AWS permissions (the EBS CSI driver needing permission to create and attach volumes, for instance) without those permissions living on the node’s own IAM role.

Why they’re used: it removes the manual work of tracking compatible versions, applying manifests correctly, and remembering to keep core cluster infrastructure patched - AWS does the version compatibility checking and can apply updates for you on a schedule you control.

How AWS manages the lifecycle: when you install or upgrade an add-on, EKS applies the underlying resources and then continuously watches them for drift - if something modifies those resources outside of EKS’s control, you get a conflict, and you choose upfront how that’s resolved via a conflict resolution strategy: OVERWRITE (EKS’s version always wins), PRESERVE (your manual changes are left alone), or NONE (fail loudly rather than silently pick a side).

Ans: An access entry is the modern, API-managed way to grant an IAM user or role access to an EKS cluster, replacing the older approach of hand-editing the aws-auth ConfigMap directly.

How they provide access: each access entry associates a specific IAM principal (a user or role’s ARN) with the cluster, and then attaches either a built-in EKS access policy (a managed permission set like AmazonEKSClusterAdminPolicy or AmazonEKSViewPolicy, optionally scoped to just one namespace via an access scope), or a mapping to a plain Kubernetes username and group, which you then wire up to your own RBAC RoleBindings yourself, similar to how aws-auth worked.

How they relate to authentication: authentication in EKS is entirely IAM-based - a caller proves who they are with signed AWS credentials, validated through a webhook token authenticator, which is a completely separate step from whether they’re actually granted anything on the cluster.

How they relate to authorization: an access entry is exactly the missing link between “this is a valid, authenticated IAM identity” and “here’s what that identity can do in Kubernetes.” Without an access entry (or the equivalent aws-auth mapping) for a given IAM principal, that principal can authenticate successfully but has no Kubernetes identity at all - every request comes back Forbidden, because RBAC has nothing to evaluate against. Access entries are managed directly through the EKS API, console, or CLI, and changes take effect immediately - no ConfigMap editing, no risk of a YAML typo locking you out of your own cluster, which was a real and fairly common failure mode with the old aws-auth approach.

Ans: EKS layers IAM-based authentication on top of standard Kubernetes RBAC authorization, connected by access entries acting as the translation step between the two systems.

1. A client generates a signed request. Tools like aws eks get-token or the aws-iam-authenticator exec plugin create a short-lived, cryptographically signed token derived from the caller’s actual AWS credentials - this is really just a signed AWS STS GetCallerIdentity request, packaged as a bearer token.
2. The API server authenticates it. EKS registers a webhook token authenticator with kube-apiserver; when a request arrives with this token, the API server calls out to validate it against IAM and confirm exactly which IAM principal it belongs to. This step only establishes who the caller is.
3. The access entry maps that identity. The validated IAM principal gets mapped to a Kubernetes username and group set, based on whatever access entry (or, on older clusters, aws-auth ConfigMap mapping) exists for that principal. No mapping means no Kubernetes identity, full stop.
4. RBAC decides what’s allowed. From here it’s completely standard Kubernetes - RoleBindings and ClusterRoleBindings tied to that mapped username or group determine exactly what the caller can do. This step has no awareness of AWS or IAM at all; as far as RBAC is concerned, it’s evaluating a normal Kubernetes identity.

The reason this two-layer design matters is that it cleanly separates “prove who you are” (IAM’s job) from “decide what you’re allowed to do” (RBAC’s job) - a valid, authenticated IAM caller with no RBAC grant behind their mapped identity still gets nothing, and conversely, RBAC rules stay portable and identical to what you’d write on any non-EKS Kubernetes cluster.

Ans: eksctl is the official CLI purpose-built for creating and managing EKS clusters - a much faster, simpler path than hand-assembling the equivalent CloudFormation or Terraform yourself, especially for getting a working cluster up quickly.

How it creates and manages clusters: a single command, or a declarative YAML config file, describes what you want - cluster name, region, node groups, Fargate profiles, add-ons, IAM roles - and eksctl translates that into a series of AWS API calls and CloudFormation stacks under the hood. It provisions the VPC and subnets (unless you point it at existing ones), the IAM roles the cluster and node groups need, the cluster itself, and any node groups or add-ons you asked for, waiting for each step to complete before moving to the next. Once it’s done, it automatically updates your local kubeconfig, so you can start running kubectl immediately without a separate step.

# Minimal cluster creation
eksctl create cluster --name my-cluster --region us-east-1 --nodes 3

# Or declaratively, from a config file describing the whole cluster
eksctl create cluster -f cluster-config.yaml

How it differs from the AWS CLI: the aws eks commands are lower-level - they let you create or describe a cluster object itself, but they don’t provision the surrounding infrastructure (VPC, IAM roles, node groups) for you; you’d need to script all of that yourself alongside aws eks create-cluster.

How it differs from Terraform: Terraform gives you full infrastructure-as-code control, state management, and integration with the rest of your AWS infrastructure as code, but it takes more upfront work to write and maintain, and doesn’t come with EKS-specific conveniences (like automatic kubeconfig updates or simplified node group YAML) built in the way eksctl does. In practice, eksctl is great for quickly standing up or tearing down clusters and for teams who want an EKS-native workflow, while Terraform is the better choice when EKS needs to be one piece of a larger, unified infrastructure-as-code setup managed alongside everything else in your AWS account.

Ans: Every one of these components only ever talks to kube-apiserver - nothing in Kubernetes talks directly to anything else, and nothing is ever pushed down to a node. It’s a pull-based model built entirely around watching the API server.

1. kube-apiserver receives a request (from kubectl, another controller, whatever), authenticates and authorizes it, runs it through admission control, and writes the result to etcd.
2. kube-controller-manager’s relevant controller is watching the API server for objects it cares about, notices the change, and creates whatever’s needed underneath (a Deployment change triggers ReplicaSet and Pod creation, for instance).
3. kube-scheduler is separately watching for Pods with no node assigned. It filters out nodes that can’t run the Pod, scores what’s left, and writes the winning node’s name back to the Pod object through the API server - it never talks to the node directly, only to the API server.
4. kubelet, on every node, is watching the API server for Pods assigned to its node specifically. The instant it sees one, it calls the container runtime over CRI to pull the image and start the container, then keeps running probes and reporting status back up through the API server.
5. kube-proxy, also on every node, watches Services and Endpoints (not Pods directly), and programs local networking rules (iptables or IPVS) so traffic aimed at a Service reaches whichever Pod is currently healthy and backing it.

The reason this “everyone watches the API server, nobody talks to each other directly” design matters so much is that it keeps components loosely coupled - any one of them can restart, fall behind, or be temporarily unreachable without breaking the others, since they all just resume watching from wherever they left off.

Ans: Maintaining desired state isn’t one mechanism, it’s a chain of independent reconciliation loops, each one only responsible for translating its layer of the desired state into the next layer down.

1. You submit a manifest (say, a Deployment asking for 3 replicas of nginx:1.25) via kubectl apply. It’s validated and stored in etcd as the Deployment’s desired spec.
2. The Deployment controller watches for Deployment changes, compares the desired replica count and Pod template against the ReplicaSet(s) it currently owns, and creates (or updates) a ReplicaSet to match.
3. The ReplicaSet controller watches for ReplicaSet changes, compares its desired replica count against how many matching Pods actually exist right now, and creates or deletes Pod objects to close the gap.
4. kube-scheduler watches for Pods with no node assigned and picks one for each.
5. kubelet on the assigned node watches for Pods scheduled to it, and actually starts the containers through the runtime.
6. From here on, it’s continuous: if a Pod crashes, kubelet restarts the container; if a node dies, the Node controller notices and the Pod gets rescheduled elsewhere by the ReplicaSet controller creating a replacement; if you edit the Deployment’s replica count, the whole chain re-triggers from step 2 downward.

No single component “owns” the whole process end to end - it’s a sequence of independent, narrowly-scoped reconciliation loops, each one only responsible for translating one layer of desired state into the next, which is exactly what makes the overall system resilient: any individual loop can be restarted or momentarily behind without the others needing to know or care.

Ans: kube-apiserver is a stateless REST server, and every single request it handles goes through the same fixed pipeline, regardless of who sent it or what it’s asking for.

1. Receive - the request arrives as an HTTPS call (kubectl, a controller, another cluster component, anything) against a specific API resource endpoint, like POST /apis/apps/v1/namespaces/default/deployments.
2. Authenticate - the server runs through its configured authenticators (client certs, bearer tokens, ServiceAccount tokens, an OIDC provider, or a webhook authenticator like EKS uses) in order, until one successfully identifies the caller. This step only answers “who is this,” nothing about permissions.
3. Authorize - the identified caller gets checked against whatever authorization modes are configured, almost always RBAC in practice: does any RoleBinding or ClusterRoleBinding grant this identity the right verb on this resource? No explicit allow means the request is denied.
4. Admission control - mutating admission controllers and webhooks run first as a group (each one can modify the object), then validating ones run against the final object (any single rejection fails the whole request).
5. Schema validation - the object is checked against its type’s schema to make sure it’s actually well-formed.
6. Write to etcd - for a write request, the validated object gets persisted, and its resourceVersion is bumped.
7. Notify watchers - anything with an open watch on this resource (controllers, kubelets, kubectl get -w) gets pushed the change immediately, which is what kicks reconciliation into motion.
8. Respond - the API server returns the result (the created/updated object, or an error) back to the caller.

For a read request, steps 4-6 are skipped and the server either serves the response from its in-memory watch cache or reads through to etcd directly, depending on the request. The API server never talks to anything else in the cluster on its own initiative - it’s purely reactive, responding to whatever request just arrived.

Ans: kubectl apply is meaningfully different from kubectl create - it’s designed to be run repeatedly against the same file, computing a diff each time rather than blindly creating or overwriting.

1. kubectl reads the local YAML file and converts it to JSON.
2. It checks whether an object with that name already exists, by querying the API server.
3. If it doesn’t exist, kubectl sends a plain POST to create it, and stores the full applied configuration in an annotation (kubectl.kubernetes.io/last-applied-configuration) on the object, for future diffing.
4. If it already exists, kubectl computes a three-way merge: comparing the last-applied-configuration (what you applied last time), the current live object (what might have been changed by something else since), and the newly applied configuration (what you’re applying now). This three-way diff is what lets apply correctly figure out which fields you’re actively changing versus fields something else modified that you never mentioned - and it sends that as a PATCH, not a full overwrite.
5. The API server runs the patch through its normal pipeline: authentication, authorization, admission control, schema validation, and finally a write to etcd.
6. Controllers watching that resource notice the change and start reconciling toward the new desired state.

# What actually happens under the hood, roughly:
kubectl apply -f deployment.yaml
# 1. GET the existing object (if any)
# 2. Compute 3-way merge patch (last-applied vs live vs new)
# 3. PATCH (or POST if it doesn't exist yet)
# 4. Update the last-applied-configuration annotation

This three-way merge is exactly why apply is safe to run repeatedly and why it’s the recommended default over create or replace - it only touches the fields you’re actually managing through this file, leaving anything else (like a field a controller or another tool set independently) untouched.

Ans: etcd is a distributed, strongly consistent key-value store, and it’s the single source of truth for absolutely everything in a Kubernetes cluster - every object’s spec, every object’s status, all of it lives here and nowhere else.

What it stores: every Kubernetes object gets serialized (as protobuf, by default) and stored under a key that mirrors its REST API path - something like /registry/pods/production/web-app-abc123. That covers Pods, Deployments, Secrets, ConfigMaps, Nodes, RBAC bindings, literally every object type in the cluster.

How kube-apiserver interacts with it: kube-apiserver is etcd’s only client - nothing else in the cluster, not even other control-plane components, ever talks to etcd directly. It connects over gRPC secured with mutual TLS, uses etcd’s compare-and-swap semantics to handle concurrent writes safely (so two conflicting updates to the same object can’t silently clobber each other), and layers a watch cache on top so most reads and watch subscriptions are served from memory rather than hitting etcd on every single request.

Why it’s critical: if etcd is lost or corrupted, the cluster loses its memory of what it’s supposed to be running entirely - not just current status, but the desired state itself. Already-running Pods keep running for a while since kubelet and kube-proxy operate on their last-known state, but nothing can be created, updated, or properly healed until etcd (or a restored backup of it) is available again. This is exactly why etcd runs as a Raft-replicated cluster (3 or 5 members) rather than a single instance, and why regular etcd backups are considered non-negotiable in any serious production Kubernetes operation.

Ans: kube-scheduler runs its own watch loop, separate from every other controller, specifically looking for Pods whose spec.nodeName field is empty.

1. Identify unscheduled Pods - the scheduler watches the API server for Pods with no node assigned yet, and queues each one for scheduling.
2. Filtering - for each Pod, it evaluates every node in the cluster and eliminates any that can’t possibly run it: not enough free CPU/memory, a taint the Pod doesn’t tolerate, a node selector or required affinity rule that doesn’t match, a port conflict, and so on. What’s left is the set of feasible nodes.
3. Scoring - each feasible node gets ranked by a set of scoring functions - things like how much spare capacity it has left, whether it satisfies soft (preferred) affinity rules, and how evenly spreading Pods across nodes or zones would balance out. Every function contributes a score, and they’re combined into a final ranking.
4. Selection and binding - the highest-scoring node wins, and the scheduler writes that decision back by creating a Binding object through the API server, which sets the Pod’s nodeName. The scheduler itself never talks to the node - it only ever writes its decision through the API server, same as everything else.

From there, kubelet on the chosen node picks up the now-assigned Pod through its own watch and actually starts it - the scheduler’s job ends the moment it makes and records the decision.

Ans: kube-controller-manager is a single running process, but it’s really just a container for dozens of independent controllers bundled together for convenience of deployment - logically, each one is completely separate and has no awareness of the others.

What a controller is: a controller is a piece of logic responsible for one specific type of object (or a small related set), whose entire job is to watch that object’s desired state and continuously reconcile the real world to match it. The Node controller monitors node health and marks unresponsive nodes; the ReplicaSet controller keeps the right number of Pods running; the Endpoints controller keeps a Service’s backing Pod list current; and there are many more, each with its own narrow scope.

How they watch resources: every controller uses the same list-watch pattern against the API server - it does an initial LIST to get current state and a starting resourceVersion, then opens a WATCH to stream subsequent changes. In practice, most controllers use an informer underneath, which maintains a local, continuously-updated cache of the objects it cares about, so the controller can read current state instantly without hitting the API server on every single reconcile.

How they reconcile: each controller runs its own loop - watch for a change, compare the object’s current state against what’s declared, and take whatever action closes the gap (create a missing Pod, delete an extra one, update a status field). Critically, these loops are “level-triggered,” not “edge-triggered” - a controller re-derives what needs to happen by comparing the entire current state against desired state every time, rather than trusting that it correctly processed every individual event it’s ever seen. That’s what makes the whole system tolerant of controllers restarting, missing an update, or racing against each other.

Ans: kubelet is the single component on every node responsible for actually making Pods real, and it operates through a continuous sync loop.

Watching Pod specs: kubelet watches the API server, filtered specifically to Pods whose spec.nodeName matches its own node - it never sees or cares about Pods assigned elsewhere.

Communicating with the API server: all communication is authenticated with a client certificate (obtained via TLS bootstrapping when the node first joins, and auto-rotated well before expiry). It’s a two-way relationship: kubelet pulls its assigned Pod specs and pushes status updates outward, but the API server also needs to reach into kubelet for things like kubectl exec, logs, and port-forward - for that, kubelet runs its own small HTTPS server that the API server calls into directly.

Interacting with the container runtime: kubelet never touches containers, namespaces, or cgroups directly - it issues gRPC calls to the container runtime over CRI (the RuntimeService for container lifecycle, the ImageService for pulling/managing images), and the runtime does the actual low-level work.

Maintaining Pod health: kubelet continuously runs whatever probes are configured for the lifetime of the Pod - restarting a container if its liveness probe fails, and pulling a Pod out of Service traffic (without killing it) if its readiness probe fails. It uses the PLEG (Pod Lifecycle Event Generator) to efficiently detect container-level changes rather than doing a full inspection of every container on every loop iteration, and it keeps reporting Pod phase (Pending, Running, Succeeded, Failed, Unknown) and node conditions back to the API server the entire time.

Ans: A Service gets a stable virtual IP (ClusterIP) that never changes, even as the actual Pods behind it come and go constantly. kube-proxy is what makes that virtual IP actually work at the network level.

1. Watch - kube-proxy, running on every node, watches the API server for Services and their EndpointSlices (the list of actual, currently-healthy Pod IPs backing each Service).
2. Program rules - whenever that list changes, kube-proxy rewrites the node’s local networking rules to match, using one of a few backends: iptables (the long-time default, using DNAT rules), IPVS (a kernel-level virtual server, better suited to clusters with huge numbers of Services), or the newer nftables mode.
3. Route traffic - when any process on that node sends a packet to a Service’s ClusterIP, the kernel applies those rules and rewrites the destination to one of the real, currently-healthy backing Pod IPs, essentially transparently.

This all happens at the OS networking layer - the client Pod has no idea which actual backend it ended up talking to, and it never needs to. It’s also worth knowing this is why kube-proxy going down doesn’t break existing connections immediately: the rules it already programmed keep working since they’re just static kernel state, but the node’s view of Service membership stops updating until kube-proxy comes back.

Ans: Taking kubectl apply -f pod.yaml as the starting point, here’s the full chain from file to running container:

1. kubectl converts the YAML to JSON and sends it to kube-apiserver.
2. The API server authenticates the caller, checks RBAC authorization, and runs the object through admission control (mutating, then validating).
3. The object passes schema validation and gets written to etcd with spec.nodeName still empty.
4. kube-scheduler’s watch picks up the new, unscheduled Pod - it filters out infeasible nodes, scores what’s left, and writes the winning node back through a Binding, setting spec.nodeName.
5. kubelet on that specific node, watching for Pods assigned to it, sees the now-scheduled Pod.
6. kubelet calls the container runtime over CRI to pull the container image (if it isn’t already cached locally).
7. The runtime creates the container using Linux namespaces (isolation) and cgroups (resource limits), and starts the process.
8. kubelet begins running any configured probes, and continuously reports the Pod’s status (Pending → Running) back to the API server.

Every arrow in that diagram is a watch-driven reaction, not a direct call - no component ever reaches out and tells another “go do this now.” Each one is independently watching the API server and reacting the instant it sees something relevant change.

Ans: These three stages run in a strict, fixed order for every single request that reaches kube-apiserver, and each one answers a genuinely different question.

1. Authentication answers “who is making this request.” The API server tries each configured authenticator in turn (client certs, bearer tokens, ServiceAccount tokens, OIDC, a webhook authenticator) until one succeeds, producing a username and group list. Fail every authenticator (with anonymous access disabled) and the request is rejected with a 401 before authorization ever runs.
2. Authorization answers “is this identity allowed to do this specific thing.” With RBAC as the authorizer, it checks whether any RoleBinding or ClusterRoleBinding grants the identified user, group, or ServiceAccount the requested verb on the requested resource. No explicit allow means denial (403), by default - there’s no implicit yes anywhere in this model.
3. Admission control runs only once a request is both authenticated and authorized, and it’s the only stage that can actually change the object (via mutating webhooks/controllers) rather than just accept or reject it. Mutating admission runs first as a full pass, then validating admission runs against the final, fully-mutated object - any single validating rejection fails the entire request, and nothing partially applies.

Only after all three stages succeed does the object get schema-validated and written to etcd. Knowing this order matters practically too - a 401 means fix your credentials, a 403 means fix RBAC, and an admission-related error or timeout means look at your webhooks, since each failure mode points at a completely different stage of this exact same pipeline.

Ans: These five mechanisms solve different problems, but they combine constantly in everyday cluster operation:

• Namespaces provide the outer boundary - a scope within which object names must be unique, and which RBAC, ResourceQuotas, and NetworkPolicies can all target.
• Labels are the key-value tags that make objects findable and groupable within (or across) that boundary - app: frontend, env: production.
• Selectors are the queries built against those labels - a Service’s spec.selector continuously finds every Pod carrying matching labels, automatically picking up new ones and dropping deleted ones, with zero manual wiring.
• Annotations carry information that’s useful to humans or tooling but should never be selected against - a build number, a git commit hash, config for a third-party controller.
• Owner references track parent-child relationships between objects (a Pod owned by a ReplicaSet, owned by a Deployment), independent of labels entirely, and they’re what drives automatic cascading garbage collection when a parent is deleted.

How they combine in practice: picture a Deployment named web-app in the production namespace. It carries the label app: web-app on its Pod template, so its owned ReplicaSet, and in turn every Pod that ReplicaSet creates, inherit that label too. A Service’s selector (app: web-app) picks up exactly those Pods, entirely by label match, with no reference to the ownership chain at all. Meanwhile, each Pod also carries an owner reference back to its ReplicaSet (unrelated to any label) - so if you delete the Deployment, garbage collection cascades down through the owner reference chain and cleans up the ReplicaSet and Pods, while the Service, watching by label selector rather than ownership, simply ends up with zero matching endpoints. An annotation on the Deployment, like kubectl.kubernetes.io/last-applied-configuration, sits alongside all of this purely as bookkeeping - kubectl apply reads it for diffing, but nothing in the ownership or selection logic ever looks at it.

Ans: Kubernetes avoids polling almost entirely - detection is push-based, built around a long-lived watch mechanism rather than clients repeatedly asking “has anything changed yet.”

1. etcd’s own watch feature is the root of this - it can notify a subscriber the instant a key changes, rather than making them poll.
2. kube-apiserver subscribes to etcd’s watches and re-exposes that same capability to its own clients: any client can open a WATCH request against a resource and receive a stream of ADDED, MODIFIED, and DELETED events as they happen, starting from a given resourceVersion.
3. Controllers consume this through an informer, which runs the list-watch loop, maintains a local cache of the objects it cares about, and fires event handlers (add/update/delete) as changes come in. Rather than doing real work directly inside those handlers, informers typically just enqueue the object’s key onto a rate-limited work queue.
4. Reconciliation happens as a separate step - worker goroutines pull keys off that queue and re-read the object’s current state fresh (not the possibly-stale object from the original event), compare it against desired state, and act accordingly. If that fails, the key gets re-queued with exponential backoff rather than dropped or retried immediately.

The reason step 4 deliberately re-reads current state instead of trusting the event’s payload is what makes this whole system “level-triggered” rather than “edge-triggered” - a controller doesn’t need to have correctly processed every single event it ever received; it just needs to eventually notice that current state and desired state don’t match, from whatever trigger got it to look. Missed or duplicate events, connection drops, and restarts all become non-issues, because the next reconciliation pass just re-derives the right action from scratch.

Ans: EKS’s entire design is about making the boundary between “what AWS manages” and “what you manage” as clean as possible, while still behaving like plain, standard Kubernetes at the API layer.

The control plane runs in an AWS-owned VPC, completely separate from your account’s infrastructure. AWS reaches into your VPC only through elastic network interfaces it places into subnets you specify at cluster creation - that’s the entire connection surface between AWS’s infrastructure and yours.

Your VPC and nodes are where you’re responsible: worker nodes (EC2 or Fargate) live in your subnets, run standard kubelet/kube-proxy/container runtime, and get scheduled Pods exactly like any Kubernetes worker node anywhere.

The Kubernetes API is the only interface between the two sides - kubectl, eksctl, Terraform, Helm, anything, all talk to the same cluster endpoint the control plane exposes, and it behaves identically to a self-managed cluster’s API from the client’s point of view.

IAM replaces the usual certificate/static-token authentication model - callers authenticate with signed AWS credentials rather than a Kubernetes-native mechanism, get validated through a webhook authenticator, and get mapped to a Kubernetes identity via an access entry, at which point ordinary RBAC takes over.

Networking is handled by the Amazon VPC CNI by default, assigning each Pod a real, routable IP directly from your VPC’s address space rather than an overlay network, which is what lets Pods talk to other AWS resources (RDS, other EC2 instances) as naturally as any other VPC-native resource would.

AWS service integrations run throughout: LoadBalancer Services provision real ALBs/NLBs automatically, EBS/EFS CSI drivers (as EKS add-ons) handle persistent storage, and IAM Roles for Service Accounts (IRSA) let individual Pods assume scoped IAM permissions instead of sharing a broad node-wide role.

The net effect: to Kubernetes itself, it looks like any other cluster - the “EKS-ness” is entirely in how the control plane is operated and how auth/networking bridge into AWS, not in the Kubernetes API surface itself.

Ans: Communication flows both directions, and each direction has its own path.

Nodes reaching the control plane: every worker node’s kubelet needs outbound access to the cluster’s API endpoint (public, private, or both, depending on your configuration). It authenticates using the node’s IAM role credentials, validated through the same webhook token authenticator EKS uses for human callers, and from there behaves exactly like any Kubernetes kubelet - watching for assigned Pods and reporting status back.

The control plane reaching nodes: this is where EKS’s networking model matters. AWS provisions elastic network interfaces directly into the subnets you specified at cluster creation, and it’s through these ENIs that the control plane reaches kubelet’s own HTTPS server on each node - which is what’s needed for things like kubectl exec, logs, and port-forward, none of which flow through the normal Pod-watching path.

What controls this traffic: the EKS-managed cluster security group is what allows this bidirectional traffic to actually flow - it’s attached to both the control plane’s ENIs and, by default, your worker nodes, and it’s specifically configured to permit exactly the ports Kubernetes needs. Modifying it without understanding what it protects is one of the most common ways to accidentally break control-plane-to-node communication in an EKS cluster, since the breakage often doesn’t show up as an obvious error, just nodes slowly going NotReady or kubectl exec mysteriously failing.

The important thing to internalize is that this is architecturally identical to any self-managed cluster’s control-plane-to-node relationship - EKS hasn’t changed the shape of the communication, only where the control plane’s infrastructure physically lives and how it gets network access into your VPC.

Ans: The cluster endpoint is the single URL every client uses to reach the API server, and EKS lets you choose exactly how it’s exposed.

Public-only means the API server gets a public DNS name behind an AWS-managed load balancer. Anyone with network access to the allowed CIDR range can attempt to connect (they still need valid IAM credentials and RBAC permissions to do anything, of course, but the network path itself is open). This is the easiest to work with for small teams or when CI/CD runners live outside the VPC, and it’s the right call as long as the CIDR list is actually kept tight.

Private-only provisions VPC endpoints via AWS PrivateLink, and DNS resolves to private IPs only reachable from inside the VPC. There is no path in from the public internet at all - not “restricted,” genuinely absent. This is the standard choice for security-sensitive production environments, at the cost of needing a VPN, Direct Connect, or bastion host for anyone who needs to reach it from outside.

Public and private together is the common middle ground - nodes and in-VPC tooling use the private path automatically, while a tightly CIDR-restricted public endpoint stays available as a convenient fallback for external admins or CI/CD, without needing a full VPN setup just for occasional access.

Ans:

1. Provisioning - an EC2 instance launches, either through a managed node group’s Auto Scaling Group or your own self-managed ASG, using an EKS-optimized AMI or a custom one.
2. Bootstrap configuration - the instance’s user data runs a bootstrap script that writes kubelet’s config and kubeconfig, pointing it at the cluster’s API endpoint and CA certificate, and wires up the AWS IAM authenticator as its auth mechanism.
3. Authentication - kubelet signs a request using the node’s IAM role credentials (via its instance profile), which gets validated through the same webhook token authenticator flow used for any EKS caller, and maps to a Kubernetes identity (typically system:node:<name> in the system:nodes group) via an access entry or aws-auth mapping.
4. Networking - the VPC CNI plugin initializes on the node, ready to assign real VPC IP addresses to Pods as they’re scheduled there.
5. kubelet registration - once authenticated, kubelet registers a new Node object with the API server, representing this instance as a schedulable part of the cluster.
6. Readiness - the node reports its capacity and status; once it passes internal readiness checks (network plugin ready, no blocking conditions), it becomes schedulable and the scheduler can start placing Pods on it.

A failure at any of these steps produces a very different symptom: a bad bootstrap script means the instance never even registers a Node object; a missing IAM mapping means it registers but authentication keeps failing; a broken security group means it can boot and authenticate but never actually reach the API endpoint in the first place - which is exactly why troubleshooting “node not joining” always starts with figuring out which of these stages it’s actually stuck at.

Ans:

Creation: you specify instance type, sizing, and either an EKS-optimized AMI or a custom launch template. EKS translates this into an Auto Scaling Group it creates and owns on your behalf - so under the hood it genuinely is just an ASG, but with EKS as the layer orchestrating it instead of you touching the ASG API directly.

EC2 instances and registration: as the ASG launches instances, each one runs the standard EKS bootstrap process (covered in node-joining) automatically - no manual scripting required, since EKS wires the correct bootstrap configuration into the launch template for you.

Scaling: you can scale a managed node group directly through the EKS API, eksctl, or the console, or hand scaling decisions to the Cluster Autoscaler or Karpenter, which adjust desired capacity based on pending Pod demand.

Lifecycle management: EKS hooks into the ASG’s lifecycle events specifically so it can intercept a scale-in or termination and drain the node properly - cordoning it, evicting Pods respecting PodDisruptionBudgets - before the instance actually goes away, rather than yanking it out from under running workloads.

Upgrades follow a rolling replacement pattern rather than an in-place patch:

1. New nodes launch on the updated AMI or launch template version.
2. EKS waits for them to become Ready.
3. Old nodes are cordoned, then drained (respecting PDBs).
4. Drained nodes are terminated and the ASG shrinks back to the desired count.

Throughout all of this, AWS owns the mechanics of provisioning, bootstrapping, and safe rolling replacement, while you retain control over sizing, instance types, updateConfig pacing (max unavailable during upgrades), and whatever PodDisruptionBudgets need to exist for those upgrades to actually be safe for your workloads.

Ans:

1. You define one or more Fargate profiles on the cluster, each specifying a namespace (and optionally labels) that should run on Fargate instead of EC2.
2. When a new Pod is created, its namespace and labels are checked against every configured Fargate profile.
3. On a match, the Pod is picked up by Fargate’s scheduling path rather than being handed to a Node object backed by an EC2 instance - AWS provisions a right-sized, isolated micro-VM specifically for that Pod.
4. The Pod runs inside that micro-VM, with its own kernel-level isolation from every other Pod, including ones from the same Deployment.
5. It still registers as a Node object if you run kubectl get nodes, so the rest of Kubernetes (Service routing, DNS, kube-proxy) treats it like any other node, but there’s no persistent instance behind it for you to patch, scale, or access at the OS level.
6. Once the Pod terminates, the micro-VM is torn down entirely - there’s nothing left running or billed once the Pod is gone.

On no match, the Pod just goes through completely normal EC2-based scheduling instead - Fargate profiles only ever pull matching Pods out of the default path, they don’t change how anything else in the cluster is scheduled. And because there’s no shared node underneath any given Pod, Fargate can’t support things that assume node-level presence - DaemonSets, privileged containers, or hostNetwork/hostPort - since there’s no persistent host for those concepts to attach to.

Ans:

Deployment: when you install an add-on (the VPC CNI, CoreDNS, kube-proxy, the EBS/EFS CSI drivers), EKS applies the underlying Kubernetes resources it needs - typically a DaemonSet or Deployment, plus supporting ConfigMaps and RBAC - directly to your cluster. From the cluster’s point of view, these are ordinary Kubernetes objects; the difference is that EKS tracks the add-on as a managed resource with its own version and health status, the way CloudFormation tracks a stack resource.

Configuration: most add-ons accept configuration through the EKS API at install or update time (things like the CNI’s IP allocation behavior, or which IAM role the CSI driver should assume), which EKS then applies as part of the underlying resources.

Upgrades: you choose a target version, and EKS updates the add-on’s resources to match - it also checks version compatibility against your cluster’s Kubernetes version to avoid installing something that isn’t actually supported.

Integration with IAM: many add-ons need real AWS permissions to function (the EBS CSI driver needs to create and attach volumes, for instance), and this is handled through IRSA - IAM Roles for Service Accounts - so the add-on’s Pods can assume a scoped IAM role rather than needing broad permissions baked into every node’s own role.

Drift handling: EKS continuously watches the add-on’s resources for drift - if something modifies them outside of EKS’s control, you hit a conflict, and how that’s resolved depends on the conflict resolution strategy you chose: OVERWRITE (EKS’s version wins), PRESERVE (manual changes are left alone), or NONE (fail rather than silently pick a side). This is the mechanism that keeps critical cluster infrastructure from silently drifting out of a known-good, tracked state over time.

Ans: This is a two-system handoff - IAM proves identity, and standard Kubernetes RBAC decides permissions, with access entries as the bridge connecting them.

1. A client generates a signed, short-lived token derived from real AWS credentials (aws eks get-token or the aws-iam-authenticator exec plugin do this) - functionally, it’s a signed STS GetCallerIdentity request packaged as a bearer token, typically valid for about 15 minutes.
2. kube-apiserver receives the request with this token and calls out to a registered webhook token authenticator, which validates the token against IAM and confirms exactly which IAM principal (user or role) it belongs to. This step establishes identity only - it has no concept of Kubernetes permissions.
3. The API server looks up whether an access entry (or, on older clusters, an aws-auth ConfigMap mapping) exists for that specific IAM principal. If one does, it maps the caller to a Kubernetes username and group set - if not, the caller is authenticated but has no Kubernetes identity at all, and every subsequent request fails authorization.
4. From here, it’s entirely standard RBAC - the mapped username/group gets evaluated against RoleBindings and ClusterRoleBindings exactly as it would on any non-EKS cluster, with zero further involvement from IAM.

The key thing to hold onto is that these are genuinely two separate systems stitched together at exactly one point - the access entry mapping. IAM never makes an authorization decision, and RBAC never makes an authentication decision; a valid IAM caller with no matching RBAC grant behind their mapped identity still gets nothing, and conversely, RBAC rules written for EKS look and behave identically to rules written for a plain self-managed cluster.

Ans: Before access entries existed, mapping an IAM identity to Kubernetes access meant directly editing the aws-auth ConfigMap - a single YAML object living inside the cluster that every IAM-to-RBAC mapping had to be hand-added to.

How access entries work: each one directly associates an IAM principal’s ARN with the cluster through the EKS API itself, and attaches either a built-in EKS access policy (a managed permission set like AmazonEKSClusterAdminPolicy or AmazonEKSViewPolicy, optionally scoped to a single namespace via an access scope) or a mapping to a plain Kubernetes username/group that you then wire to your own RBAC RoleBindings, same as before.

What was painful about aws-auth:

• It required editing raw YAML inside a ConfigMap, with no built-in validation - a single indentation mistake could silently break the mapping, or worse, lock out every IAM identity relying on it.
• Changes required kubectl access to the cluster in the first place, creating an awkward bootstrapping problem for automation.
• There was no way to see access grants without reading and parsing that ConfigMap by hand.

What access entries fix:

• They’re managed directly through the EKS API, CLI, or console - no ConfigMap editing, and no risk of a malformed YAML edit taking down access for everyone.
• Changes apply immediately, with proper validation at the API layer before anything is accepted.
• Built-in access policies cover common cases (admin, view-only) without you needing to write and maintain the RBAC objects yourself.
• Access can be granted or audited through standard AWS tooling (IAM policies, CloudTrail) rather than needing cluster access just to inspect who has access.

Existing aws-auth mappings still work on clusters that haven’t migrated, and both mechanisms can coexist during a transition, but access entries are the current, safer default for any new cluster.

Ans: A single eksctl create cluster command (or a declarative config file) triggers a whole sequence of AWS provisioning steps under the hood, using CloudFormation as its actual execution engine.

1. VPC and subnets - unless you point eksctl at an existing VPC, it creates a new one with public and private subnets spread across multiple Availability Zones, along with the NAT gateways and route tables needed for private subnet internet access.
2. IAM roles - it creates the cluster’s own IAM role (giving EKS permission to manage AWS resources on your behalf) and, separately, an IAM role for each node group.
3. The EKS cluster itself - eksctl calls the EKS API to create the managed control plane, using the VPC/subnets and cluster IAM role from the previous steps.
4. Node groups (if requested) - it provisions Auto Scaling Groups (for managed node groups) or launches EC2 instances directly with a bootstrap script (for self-managed), attaching the appropriate node IAM role.
5. Add-ons (if requested) - it can install core add-ons like the VPC CNI, CoreDNS, and kube-proxy as part of the same process.
6. Access configuration - it sets up the initial access entry (or aws-auth mapping, on older versions) so the creating identity actually has cluster access once everything’s up.
7. kubeconfig update - once the cluster is active, eksctl writes an entry into your local kubeconfig automatically, so kubectl works immediately without a separate manual step.

Each of these steps is backed by one or more CloudFormation stacks that eksctl creates and manages for you - which is also why eksctl delete cluster is reliable at tearing everything back down cleanly, since it’s really just deleting the same CloudFormation stacks in reverse order.

Ans: The aws eks command group is a thin, direct wrapper around the EKS API - it doesn’t do the orchestration work eksctl does; it just exposes individual API operations.

aws eks describe-cluster sends a request to the EKS API and returns the cluster’s metadata directly - its status (ACTIVE, CREATING, UPDATING, FAILED), its endpoint URL, its CA certificate data, its VPC configuration, and the Kubernetes version it’s running. This is genuinely just a read operation against AWS’s own records about the cluster; it doesn’t talk to the Kubernetes API at all.

aws eks describe-cluster --name my-cluster --query "cluster.status"

aws eks update-kubeconfig does something more interesting: it calls describe-cluster internally to get the endpoint and CA data, then writes a new entry into your local kubeconfig file with:

1. A cluster entry containing the endpoint URL and CA certificate.
2. A user entry configured with an exec block - not a static credential, but a command (aws eks get-token or the older aws-iam-authenticator) that kubeconfig runs every time kubectl needs to authenticate, generating a fresh, short-lived token from your current AWS credentials on the fly.
3. A context tying the two together, which it can also set as your current context.

aws eks update-kubeconfig --name my-cluster --region us-east-1

This exec-based design is exactly why EKS credentials never sit around as long-lived static tokens in your kubeconfig - every single kubectl command transparently triggers a brand-new signed token behind the scenes, scoped to whatever your current AWS credentials (an assumed role, SSO session, or access keys) allow at that exact moment.

Ans: Tracing a single kubectl apply end to end shows every stage a request actually passes through, in a strict order that never varies.

1. Client-side (kubectl): kubectl uses client-go under the hood, which first consults the API server’s discovery documents (or a cached version of them) to figure out which group/version/resource your object maps to, and builds a REST client scoped to that endpoint.
2. Transport: the request goes out as an HTTPS call, authenticated with whatever credential your kubeconfig specifies (client cert, bearer token, or an exec-plugin-generated token).
3. Authentication: kube-apiserver runs the request through its configured authenticator chain until one succeeds, establishing a username and group list. No successful authenticator (with anonymous access off) means an immediate 401, before anything else runs.
4. Authorization: the identified caller is checked against the configured authorizers, almost always RBAC - does any RoleBinding/ClusterRoleBinding grant this verb on this resource. No explicit allow means a 403.
5. Mutating admission: every registered mutating admission controller and webhook runs, in order, each one able to modify the object (defaulting fields, injecting a sidecar, and so on).
6. Schema validation: the object, now fully mutated, is validated against its OpenAPI schema.
7. Validating admission: every validating controller and webhook runs against the final object; a single rejection fails the entire request with nothing partially applied.
8. Persistence: the object is serialized (protobuf, by default) and written to etcd, with a new resourceVersion assigned as part of that write.
9. Watch notification: etcd’s own watch mechanism notifies kube-apiserver, which immediately pushes the change to every client with an open watch on that resource - this is what kicks reconciliation into gear on the controller side.
10. Response: the API server serializes the final object (or an error) back to the client in the format it requested (JSON by default, protobuf for some internal clients).

Every one of these stages is independently observable in practice - a 401 means look at credentials, a 403 means look at RBAC, an admission-related failure or timeout means look at webhooks, and a request that succeeds but nothing seems to happen afterward means look at whether any controller is actually watching that resource at all.

Ans: kube-apiserver is etcd’s sole client in the entire cluster, and every interaction goes over gRPC, secured with mutual TLS between the two.

Storage: every object is serialized (protobuf by default, though JSON is also supported) and stored under a key that mirrors its REST path - /registry/pods/production/web-app-abc123, for example. This flat, path-based keying is what makes range queries (like “list every Pod in this namespace”) efficient - it’s just a prefix scan over etcd’s sorted keyspace.

Retrieval: a plain GET/LIST can be served two ways - either as a direct quorum read straight from etcd (guaranteeing you see the absolute latest committed state), or, more commonly for LIST/WATCH heavy traffic, served out of kube-apiserver’s in-memory watch cache, which stays synchronized with etcd via its own persistent watch and is far cheaper for the API server to serve at scale.

Watching: etcd has native support for watching a key or key-range and streaming every subsequent change (with its own internal revision number attached). kube-apiserver maintains one watch connection to etcd per resource type it cares about, and re-exposes that same capability to its own clients - so a client watching Pods through the Kubernetes API is really just riding on top of kube-apiserver’s own etcd watch, translated into Kubernetes’ ADDED/MODIFIED/DELETED event format.

Updating: writes use etcd’s compare-and-swap semantics rather than blind overwrites - kube-apiserver includes the object’s expected resourceVersion (etcd’s own revision number, effectively) in the write, and etcd only commits the change if that still matches current state. If it doesn’t (because someone else updated the object in between), the write is rejected with a conflict error, and the client is expected to re-fetch and retry - which is exactly what prevents two concurrent updates to the same object from silently overwriting each other.

Ans: Internally, a controller’s reconciliation loop is built from a handful of distinct, cooperating pieces, not one monolithic function.

1. Reflector - runs the actual list-watch loop against the API server for a given resource type, and feeds every change into a DeltaFIFO queue as it arrives.
2. Informer / local cache (Indexer) - consumes that DeltaFIFO, keeps an in-memory cache of the resource’s current state continuously in sync, and fires registered event handlers (OnAdd, OnUpdate, OnDelete) as changes are processed.
3. Work queue - rather than doing real reconciliation work directly inside those event handlers, the handlers typically just extract the object’s key (namespace/name) and enqueue it onto a rate-limited work queue. This decouples “noticing a change happened” from “actually processing it.”
4. Worker goroutines - a pool of workers continuously pull keys off the queue, one at a time, and invoke the controller’s syncHandler function for each.
5. syncHandler - critically, this re-reads the object’s current state fresh from the local cache (not the possibly-stale object captured at event time), compares it against desired state, and performs whatever create/update/delete actions are needed to close the gap.
6. Requeue on failure - if a sync fails, the key gets requeued with exponential backoff rather than retried immediately or dropped, so a controller can ride out a transient failure (an API server hiccup, a temporary permissions issue) without hammering the system or silently giving up.

Because syncHandler always re-derives the right action from current state rather than trusting the event payload, this design is naturally “level-triggered” - a controller doesn’t need to have correctly processed every single event it’s ever seen; it just needs to eventually get triggered to look, and it’ll figure out the correct action from scratch every time.

Ans: Without informers, every controller that needed to know about, say, Pods would have to run its own independent list-watch against the API server - and in a cluster with dozens of controllers all caring about overlapping sets of resources, that’s a lot of redundant watch connections and redundant in-memory copies of the same data.

Informers solve this with sharing. A SharedInformerFactory lets multiple controllers register interest in the same resource type while only maintaining one underlying watch connection and one local cache, which all of them read from. Each controller just registers its own event handlers against the shared informer - the expensive part (the actual watch against the API server) happens exactly once no matter how many controllers care about the result.

Why this matters for efficiency:

• No polling - informers rely entirely on etcd’s push-based watch mechanism (relayed through kube-apiserver), so there’s no wasted request volume from repeatedly asking “has anything changed yet.”
• Local reads - once an informer’s cache is populated, a controller reading current state for any object is a fast, in-memory lookup, not a network round-trip to the API server.
• Shared cost - one watch connection and one cache serve every controller interested in that resource type, instead of N independent ones.

Work queues solve a different problem: decoupling “I noticed a change” from “I need to act on it.” Rather than doing real work inside an event handler (which would run on the informer’s own goroutine and could block event processing for everyone), handlers just enqueue an object’s key. A pool of workers then processes that queue independently, which gives you three things for free: deduplication (the same key showing up multiple times before it’s processed just collapses into one entry), rate limiting (workers process at a controlled pace instead of a burst), and exponential backoff on retry (a failing key doesn’t get hammered immediately, it backs off progressively).

Together, informers plus work queues are what let a cluster run dozens of controllers, each reconciling potentially thousands of objects, without turning kube-apiserver into a bottleneck under constant redundant polling.

Ans: Modern kube-scheduler is built on the scheduling framework - a pipeline of well-defined extension points, each one implemented by pluggable plugins, rather than one hardcoded filter-then-score routine.

1. PreFilter - plugins precompute anything the Filter stage will need, and can reject the Pod outright here if scheduling is already impossible.
2. Filter - each plugin eliminates nodes that flat-out can’t run the Pod (insufficient resources, an untolerated taint, a failed required affinity rule, a port conflict). What survives is the feasible set.
3. PostFilter - only runs if the feasible set is empty. This is where preemption logic lives: the scheduler looks for a node where evicting one or more lower-priority Pods would make room, and if it finds one, it evicts those victims (respecting their own graceful termination) rather than leaving the new Pod unschedulable. The preempted Pod itself isn’t bound immediately - it gets nominated for that node, and the actual binding happens on a subsequent scheduling cycle once the victims have actually terminated.
4. PreScore / Score - every feasible node gets ranked by scoring plugins (balancing resource usage, honoring soft affinity preferences, spreading across topology domains), and the scores get normalized and combined into one final ranking.
5. Reserve - the scheduler tentatively reserves the winning node’s resources for this Pod, so a second Pod being scheduled concurrently can’t race in and claim the same capacity.
6. Permit - a final gate, primarily used by plugins implementing things like gang-scheduling, which might hold a Pod here until a whole group of related Pods is ready to be admitted together.
7. PreBind / Bind - the scheduler actually writes the decision back through the API server as a Binding object, setting spec.nodeName.
8. PostBind - runs after a successful bind, mainly for cleanup or plugin-specific bookkeeping.

This plugin-based architecture is exactly what lets Kubernetes support custom scheduling behavior (gang scheduling, custom scoring for specialized hardware) without forking the scheduler itself - you write a plugin that hooks into whichever extension points it needs, and it runs inside the same pipeline as every built-in plugin.

Ans: kube-scheduler and kube-controller-manager are both singleton-by-design components - only one instance should ever be actively making decisions at a time, because two active schedulers could race and double-bind a Pod, or two active controller-managers could both try to reconcile the same object in conflicting ways. But you still want to run multiple replicas for high availability. Leader election is what reconciles those two requirements.

How it works: every replica tries to acquire (or renew) a Lease object (in the kube-system namespace) associated with that component. Whichever replica successfully holds the lease is the active leader and does the actual work; every other replica sits idle in standby, continuously watching the lease and periodically attempting to acquire it themselves.

1. The current leader renews its lease well before it expires, on a configurable interval (renewDeadline).
2. If the leader dies, stalls, or loses network connectivity, it stops renewing.
3. Once the lease’s expiry passes, standby replicas (checking on their own retryPeriod) race to acquire it.
4. Whichever one wins becomes the new active leader and picks up work immediately.

Why this matters for HA: without leader election, you’d either have to run these components as a true single point of failure (no redundancy at all), or accept the real risk of split-brain behavior from multiple active instances fighting over the same decisions. Leader election gives you hot standby replicas that add zero coordination overhead during normal operation (only the active leader does real work) while still providing fast, automatic failover - typically within seconds of the lease expiring - the moment the active instance goes away.

Ans: kubelet’s job spans from the moment a Pod is assigned to its node all the way to the moment it’s fully torn down, and it runs a continuous sync loop the entire time.

1. Pod admission (local): before accepting a newly assigned Pod, kubelet runs its own local admission checks, independent of the API server’s - re-verifying the Pod still fits available node resources, checking node-level admission plugins, and rejecting the Pod locally (marking it Failed) if something no longer checks out, rather than assuming the scheduler’s decision is still perfectly valid by the time it arrives.
2. Container creation: kubelet calls the container runtime over CRI - first creating a Pod sandbox (which sets up the shared network namespace all containers in the Pod will use), then pulling images and creating/starting each container inside that sandbox, running init containers to completion first, in order, before any regular containers start.
3. Probes: once containers are running, kubelet begins executing configured startup, liveness, and readiness probes on their own schedules. A failed liveness probe triggers a container restart; a failed readiness probe removes the Pod from Service endpoints without touching the container itself; startup probes gate the other two until the app reports it’s actually finished initializing.
4. Restarts: governed by the Pod’s restartPolicy (Always, OnFailure, Never). When a restart is warranted, kubelet applies exponential backoff between attempts (capping out at 5 minutes) to avoid hammering a container that’s crash-looping.
5. Status reporting: kubelet uses the PLEG (Pod Lifecycle Event Generator) to efficiently detect container-level state changes without doing a full, expensive inspection of every container on every loop tick, and continuously reports Pod phase and container statuses back to the API server.
6. Termination: on deletion, kubelet removes the Pod from Service endpoints (via the endpoint controller reacting to the deletion timestamp), runs any preStop hook, sends SIGTERM to each container, waits up to terminationGracePeriodSeconds, and sends SIGKILL to anything still running after that window closes. Once every container has actually stopped, kubelet reports the Pod as terminated and it’s removed from the API server.

Throughout all of this, kubelet is also running periodic garbage collection on dead containers and unused images on the node, independent of any specific Pod’s lifecycle, to keep the node from slowly filling up with stopped container remnants.

Ans: kubelet’s relationship with the API server is secured with mutual TLS in both directions, and getting there involves a proper bootstrapping process rather than a static, pre-baked credential.

1. Initial bootstrap: a brand-new node doesn’t yet have a client certificate, so it authenticates its very first request using a short-lived bootstrap token, which only has permission to submit a CertificateSigningRequest (CSR).
2. Certificate issuance: kubelet generates a keypair locally and submits a CSR asking for a client certificate identifying it as system:node:<nodename> in the system:nodes group. In most clusters this gets auto-approved by a controller watching for CSRs matching the expected bootstrap pattern.
3. Ongoing authentication: from then on, kubelet authenticates every request to kube-apiserver using that client certificate over mutual TLS - the API server verifies it against the cluster’s CA, exactly like any other certificate-based authenticator.
4. Certificate rotation: kubelet automatically requests a new certificate well before the current one expires (again via the CSR API), so long-lived nodes never need manual re-bootstrapping.
5. Authorization - the Node authorizer: once authenticated as system:node:<nodename>, a special-purpose authorizer (distinct from general RBAC) restricts that identity to only the objects related to its own node - it can read Secrets and ConfigMaps mounted by its own Pods, but not another node’s; it can update its own Node object’s status, but not another node’s. This is a deliberate, narrow scope specifically because a compromised node shouldn’t be able to read every Secret in the cluster just by virtue of being a valid, authenticated node.
6. The reverse direction: kubelet also runs its own small HTTPS server, secured the same way, which the API server calls into for kubectl exec, logs, and port-forward - this direction is also authenticated and subject to RBAC on the caller’s side, not just trusted blindly because the request came from the control plane.

This combination (mutual TLS plus a purpose-built, narrowly-scoped Node authorizer) is what keeps a single compromised node’s blast radius contained to roughly “its own Pods and their directly-mounted secrets,” rather than the whole cluster.

Ans: kubelet never touches Linux namespaces, cgroups, or the container filesystem directly - every interaction with the runtime goes through CRI, a gRPC interface split into two services, connected over a local Unix socket.

1. RunPodSandbox - the very first CRI call for a new Pod. This creates the Pod sandbox - a paused, minimal container (often literally called the “pause container”) that owns the shared network namespace every container in the Pod will use. This is also the point where the network is actually attached, via the CNI plugin, giving the Pod its IP address.
2. PullImage (ImageService) - for each container the Pod needs, kubelet checks if the image is already cached locally, and if not, calls the ImageService to pull it, using whatever imagePullSecret and imagePullPolicy apply.
3. CreateContainer (RuntimeService) - for each container, kubelet calls CreateContainer, passing in the container’s spec (command, env vars, resource limits, mounted volumes), attaching it to the Pod’s already-created sandbox and its shared network namespace.
4. StartContainer - actually starts the process inside the container. Init containers are run through this same sequence one at a time, sequentially, with each one having to complete successfully before the next init container (or the first regular container) starts.
5. Ongoing lifecycle calls - StopContainer and RemoveContainer handle graceful shutdown and cleanup; ListContainers/ContainerStatus are what kubelet polls (efficiently, via PLEG) to keep its own view of container state accurate.

Because every one of these is a standardized CRI call rather than something runtime-specific, kubelet’s logic is identical whether the underlying runtime is containerd or CRI-O - all the runtime-specific work (talking to the kernel, managing namespaces and cgroups, actually invoking runc) happens entirely on the other side of that gRPC boundary.

Ans: Node-level networking in Kubernetes is really the combination of two mostly-independent systems: CNI plugins handling Pod-to-Pod connectivity, and kube-proxy handling Service virtual IPs.

Pod networking (CNI): when kubelet creates a Pod’s sandbox, it calls the configured CNI plugin’s ADD command, passing in the Pod’s network namespace. The plugin is responsible for assigning the Pod a real IP address, wiring up its network interface, and making sure that IP is actually routable to and from other nodes. Exactly how it achieves that routability varies by plugin - the AWS VPC CNI assigns IPs directly out of the VPC’s own address space (so Pods are natively routable within the VPC, no overlay needed), while plugins like Calico or Flannel commonly use an overlay network (VXLAN or similar) or BGP-advertised routes to stitch Pod IPs across nodes that aren’t natively on the same L2/L3 segment.

Service networking (kube-proxy): a Service’s ClusterIP isn’t attached to any real network interface - it only exists as a set of forwarding rules. kube-proxy watches Services and EndpointSlices, and programs the node’s iptables or IPVS rules so that any packet sent to a Service’s ClusterIP gets rewritten (DNAT) to one of the currently-healthy backing Pod IPs.

How routes tie it together: for direct Pod-to-Pod traffic (not through a Service), the node’s own routing table (populated by the CNI plugin, or handled natively if using VPC-native IPs) is what actually gets a packet from one node to another node hosting the destination Pod. For Service traffic, kube-proxy’s rules do the DNAT locally on the sending node before the packet ever leaves it, rewriting the destination to a real Pod IP, at which point normal Pod-to-Pod routing takes over for the rest of the trip. This split is exactly why a broken CNI plugin and a broken kube-proxy produce very different symptoms - one breaks all Pod connectivity outright, the other leaves direct Pod-to-Pod traffic working fine while only Service-based traffic fails.

Ans: etcd’s consistency guarantees come entirely from the Raft consensus algorithm, and understanding Raft is really the whole answer here.

Replication and quorum: an etcd cluster runs as an odd number of members (3 or 5, typically). Every write has to go through the current Raft leader, which replicates the entry to followers and only considers it committed once a majority (quorum) of members have persisted it - 2 out of 3, or 3 out of 5. This is why odd numbers matter: with 3 members you can lose 1 and still have a quorum of 2; with 5 you can lose 2.

Leader election: if the current leader stops responding (crash, network partition), the remaining members detect the missing heartbeats and hold a new election among themselves, using randomized election timeouts specifically designed to avoid repeated split votes. Only once a new leader is elected does the cluster resume accepting writes - which is why etcd briefly (typically well under a second) rejects writes during a failover.

Durability: every write is appended to a write-ahead log (WAL) on disk before being applied, so a member that crashes and restarts can replay its WAL to recover exactly where it left off. Periodic snapshots compact that history so a restarting or newly-joining member doesn’t need to replay the cluster’s entire write history from the beginning.

Failure scenarios:

• Losing a minority of members - the cluster keeps operating normally; quorum is unaffected.
• Losing the leader specifically - a brief write pause during re-election, then normal operation resumes with a new leader.
• Losing a majority (quorum loss) - the cluster stops accepting writes entirely, since no majority can be reached to commit anything. Recovery at this point typically requires restoring from a snapshot backup or manually reconstructing quorum from surviving members.
• Disk latency issues - since every write requires an fsync before being acknowledged, slow disks are one of the most common real-world causes of leader elections flapping repeatedly, even without an actual member failure.

Ans: Every control-plane component achieves HA differently, based on whether it holds state or not, and understanding that distinction is the key to the whole answer.

kube-apiserver - stateless, horizontally scaled: because it holds no state of its own (everything lives in etcd), you can run any number of API server replicas behind a load balancer, and any of them can serve any request equally well. Losing one is a non-event from the cluster’s point of view - the load balancer simply stops routing to it, and clients with open connections reconnect to a healthy replica.

etcd - stateful, quorum-replicated: etcd can’t just be “horizontally scaled” the same way, since every member needs to agree on the same sequence of writes. Instead it runs as a Raft cluster (3 or 5 members) spread across failure domains, tolerating the loss of a minority without losing the ability to accept writes.

kube-scheduler and kube-controller-manager - stateless, but singleton-by-design: these can’t simply run as N independent active replicas, because having two schedulers or two controller-managers both actively making decisions risks conflicting actions. Instead, you run multiple replicas but only one is ever active, decided through leader election via a Lease object - the rest sit as hot standbys ready to take over within seconds if the active one disappears.

Put together: you can lose an entire Availability Zone’s worth of control-plane instances (an API server replica, an etcd member, and a scheduler/controller-manager replica all at once) and the cluster keeps functioning - API traffic reroutes to healthy API server replicas, etcd retains quorum through its remaining members, and whichever scheduler/controller-manager replica was standing by simply becomes the new active leader. This exact setup is what managed offerings like EKS give you automatically, spread across at least three AZs by default.

Ans: Kubernetes’ API surface has to keep evolving without constantly breaking every existing client, and it manages that through a fairly disciplined versioning and deprecation system.

Groups and versions: resources are organized into API groups (apps, batch, the unnamed core group) so different parts of the API can evolve independently, and each group exposes one or more versions (v1alpha1, v1beta1, v1) simultaneously while a resource matures.

Discovery: every client can query /api and /apis to get a live list of every group, version, and resource the current cluster actually supports, along with which operations are valid on each. This is how tools like kubectl avoid hardcoding assumptions about what a given cluster supports - they discover it at runtime.

Compatibility across versions: when a resource is served at multiple versions simultaneously, the API server internally converts between them using a designated storage version - whatever version the object is actually persisted as in etcd - and converts to/from whatever version a given request asked for. For built-in types, Kubernetes ships this conversion logic itself; for CRDs, you provide a conversion webhook.

Deprecation policy: Kubernetes follows a formal, published deprecation policy with real guarantees attached to each maturity level:

A GA API being removed is always preceded by a formal deprecation announcement well ahead of time, precisely so cluster operators have a real, guaranteed window to migrate manifests and tooling before anything actually breaks - kubectl convert and tools like pluto exist specifically to help find and update manifests still referencing an API version that’s scheduled for removal.

Ans: The aggregation layer solves a specific problem: how do you extend the Kubernetes API surface with functionality that needs its own storage backend, its own scaling characteristics, or logic too complex to reasonably express through Kubernetes’ built-in generic storage - without forking or patching kube-apiserver itself.

How it works: you register an APIService object telling kube-apiserver “requests for this specific group/version should be proxied to this Service instead of handled locally.” From that point on, matching requests get authenticated and authorized by kube-apiserver as normal, then transparently forwarded to your own, completely separate extension API server process, which handles the request however it wants - its own storage, its own business logic, anything.

How this differs from CRDs: a Custom Resource Definition extends the API surface without leaving kube-apiserver’s own process at all - it’s stored in the same etcd, using the same generic storage and validation machinery every built-in type uses. That makes CRDs dramatically simpler to build and operate (no separate server to run, deploy, or scale), but it also means you’re constrained to whatever the generic apiserver framework supports - you can’t bring your own storage backend or truly custom request-handling logic.

How this differs from normal built-in resources: built-in resources (Pods, Deployments) are compiled directly into kube-apiserver itself; CRDs are data-driven extensions handled by the same generic machinery; aggregated APIs are genuinely separate server processes, only stitched into the same API surface at the routing layer.

The real-world example almost everyone runs into is the metrics API (metrics.k8s.io) backing kubectl top - it looks and feels like a completely normal part of the Kubernetes API, but it’s actually served entirely by the separate metrics-server process, proxied through via exactly this aggregation mechanism.

Ans: Admission webhooks let you plug custom logic directly into the request pipeline, and the mechanics of exactly how the API server talks to them matter a lot for understanding failure behavior.

When they’re called: mutating webhooks run as a group first, after built-in mutating admission controllers, and before schema validation; validating webhooks run afterward, against the final, fully-mutated object, right before the write to etcd. Within each group, multiple webhooks can be registered and all of them run, though the order between webhooks in the same phase isn’t something you should depend on.

How the interaction works: for each matching request, kube-apiserver sends the webhook server an AdmissionReview object over HTTPS, containing the full request context - the object itself, the operation (CREATE/UPDATE/DELETE), and who sent it. The webhook does its logic and returns another AdmissionReview with allowed: true/false; a mutating webhook can additionally include a JSONPatch describing exactly what it wants changed.

What happens on webhook failure: this is where two settings matter enormously. timeoutSeconds (30s by default) caps how long the API server waits for a response before treating the call as failed. failurePolicy decides what happens when that failure occurs - Ignore lets the request through as if the webhook approved it, while Fail blocks the request entirely. A failurePolicy: Fail webhook that’s unreachable can genuinely lock you out of the cluster (which is exactly the scenario cert-manager and similar tools go to real lengths to avoid, by making their own webhook extremely resilient). There’s also a reinvocationPolicy setting for mutating webhooks specifically - if a later mutating webhook changes the object again after an earlier one already ran, IfNeeded will re-invoke the earlier webhook so it sees the final, fully-mutated object rather than an intermediate one it never got to react to.

Ans: Garbage collection in Kubernetes is entirely driven by the owner reference graph, watched continuously by a dedicated garbage collector controller running inside kube-controller-manager.

The graph: every dependent object (a Pod created by a ReplicaSet, a ReplicaSet created by a Deployment) carries an owner reference pointing back to its creator. The GC controller builds and maintains an in-memory graph of these relationships across the whole cluster.

What triggers cleanup: when an owner object is deleted, the GC controller detects that its dependents now point at a nonexistent (or being-deleted) owner, and acts according to whichever deletion propagation policy was specified:

• Background (the default) - the owner is deleted immediately, and the GC controller cleans up dependents shortly afterward, asynchronously.
• Foreground - the owner isn’t actually removed yet; instead, Kubernetes adds a special built-in foregroundDeletion finalizer to it, sets its deletionTimestamp, and only removes it once every dependent with blockOwnerDeletion: true has actually been deleted first. This is how you get a guarantee that nothing is left dangling, at the cost of the owner staying visible (in Terminating state) longer.
• Orphan - the owner is deleted immediately, but its dependents are deliberately left behind, with their owner reference simply stripped rather than triggering any cleanup.

How finalizers fit in: finalizers are the general mechanism that lets any controller (not just the built-in GC one) block deletion until it’s done its own cleanup - and foreground propagation is really just the GC controller using that same generic finalizer mechanism for its own purposes. This is exactly why an object can appear stuck in Terminating for reasons that have nothing to do with owner references at all - any controller-owned finalizer that never gets removed (because its owning controller crashed, or hit a permissions error trying to clean up an external resource) will block deletion indefinitely, regardless of what’s happening with the owner reference graph.

Ans: A finalizer is a plain string stored in an object’s metadata, and its presence is what turns kubectl delete from an immediate action into a two-phase process.

1. When you delete an object that has one or more finalizers set, Kubernetes doesn’t remove it right away. Instead, it sets metadata.deletionTimestamp to the current time and leaves the object exactly as it was otherwise - this is what you see rendered as Terminating in kubectl get.
2. It’s on whichever controller registered a given finalizer to notice the deletionTimestamp (via its normal watch), perform whatever cleanup it’s responsible for (deprovisioning a cloud load balancer, releasing an external volume, running any last logic that needs to happen before this object truly disappears), and then remove its own finalizer from the list once that’s done.
3. Only once the finalizer list is completely empty does Kubernetes actually remove the object from etcd for good.

Why an object can get stuck: the object stays in Terminating for exactly as long as any finalizer remains on it. Real-world causes include: the controller responsible for a finalizer has crashed or is crash-looping and simply never gets to run its cleanup logic; the cleanup logic itself is failing (an external cloud resource it’s trying to deprovision is already gone, or it’s hitting a permissions error); or the controller was uninstalled from the cluster entirely, leaving its finalizer with nothing left to ever remove it.

# See what's actually blocking deletion
kubectl get pod my-pod -o jsonpath='{.metadata.finalizers}'

# Last-resort escape hatch - understand what you're skipping first
kubectl patch pod my-pod -p '{"metadata":{"finalizers":[]}}' --type=merge

Why this design exists at all: finalizers give controllers a real guarantee that they get a chance to clean up before an object truly disappears, rather than racing to notice a deletion event and possibly missing it. The tradeoff is exactly the stuck-Terminating failure mode - which is why force-removing a finalizer should always come with understanding exactly what cleanup step you’re bypassing, since it’s very possible you’re about to leave a real external resource (a load balancer, a volume, a DNS record) orphaned and never cleaned up.

Ans: An EKS control plane runs entirely within infrastructure AWS owns and operates - it is not a set of EC2 instances sitting inside your account that you simply don’t have credentials for; it’s genuinely isolated at the account and network level.

What AWS manages: multiple API server replicas, an etcd cluster, the scheduler, and the controller-manager, all deployed and kept healthy inside an AWS-managed VPC dedicated to running EKS control planes, spread across multiple Availability Zones for resilience.

Isolation: your cluster’s control plane is logically isolated from every other customer’s - there’s no shared etcd, no shared API server process, and no network path between your control plane and anyone else’s. From your own AWS account’s perspective, you have zero visibility into or access to this infrastructure at all - no EC2 instances to describe, no way to SSH in, nothing to patch yourself.

How it communicates with your VPC: the only connection point is a set of elastic network interfaces (ENIs) that AWS provisions directly into the subnets you specify at cluster creation. These ENIs are what let the control plane reach kubelet on your worker nodes (for exec/logs/port-forward), and they’re governed by the EKS-managed cluster security group, which permits exactly the traffic Kubernetes needs between the control plane and your nodes. Your cluster’s public and/or private endpoint (which clients like kubectl actually connect to) sits in front of the API server replicas behind an AWS-managed load balancer, itself also determined by the endpoint access configuration you chose.

The net effect is that AWS gets to patch, scale, and replace control-plane infrastructure completely transparently to you - there’s genuinely nothing on your side that could be affected by that work, because there’s no shared infrastructure between what AWS operates and what you operate at all.

Ans: EKS applies the exact same HA principles any well-run Kubernetes control plane needs, just fully managed and automated on your behalf.

API server: multiple replicas run across at least three Availability Zones, sitting behind an AWS-managed Network Load Balancer that your cluster’s endpoint DNS name actually resolves to. Since API server replicas are stateless, losing any individual one is a non-event - the load balancer simply routes around it.

etcd: the etcd cluster backing your control plane is also spread across multiple AZs and kept consistent through the same Raft replication every etcd cluster relies on, tolerating the loss of a minority of members without losing the ability to accept writes.

Automated health management: AWS continuously monitors every control-plane component’s health and automatically replaces anything unhealthy - a failed API server instance, a struggling etcd member - without any action, or even visibility, on your part. Version patches and minor upgrades happen the same way, rolled out by AWS during the window you configure, with the load-balanced endpoint absorbing the transition without downtime to API access.

What’s outside AWS’s HA guarantee: the data plane. EKS gives you a highly available control plane by default, but your worker nodes still need to be deliberately spread across multiple AZs yourself (typically via your ASG’s subnet configuration), and your own workloads need PodDisruptionBudgets and topology spread constraints if you want them to actually survive a node or AZ failure gracefully - the managed control plane’s HA doesn’t automatically extend to protect your applications’ availability.

Ans: This communication is genuinely bidirectional, with a different path and purpose in each direction.

Nodes reaching the API server: every worker node needs outbound network access to the cluster’s endpoint - public, private, or both, per your configuration. kubelet authenticates using the node’s IAM role credentials, validated through EKS’s webhook token authenticator, mapped to a Kubernetes identity, and from there it watches for Pods assigned to it and reports status back, exactly like any Kubernetes cluster.

The control plane reaching kubelet: this direction uses the elastic network interfaces AWS placed directly into your subnets at cluster creation. Through those ENIs, the API server can reach kubelet’s own small HTTPS server on each node, which is specifically what’s needed for kubectl exec, logs, and port-forward - none of which are covered by the normal Pod-watching path, since those are the control plane initiating contact with a node rather than the other way around.

Security groups: the EKS-managed cluster security group governs this entire path - it’s attached both to the control plane’s ENIs and (by default) to your worker nodes, and it’s specifically configured to permit exactly the ports this bidirectional traffic needs. This is deliberately permissive by EKS’s design; you’re not meant to lock it down further, since doing so is one of the most common (and confusing to diagnose) ways to accidentally break the cluster.

Endpoints: whichever endpoint mode you’ve configured (public, private, or both) governs the client-to-API-server leg of this whole picture - it doesn’t change how the control plane reaches into your VPC via its ENIs, which happens regardless of your endpoint configuration, since that path is what nodes fundamentally depend on to function at all.

Authentication summary: nodes authenticate to the control plane via IAM (their instance role); the control plane authenticates to kubelet using its own TLS setup, and kubelet’s RBAC-equivalent authorization on that inbound side comes from whatever’s configured to call it (kubectl exec still goes through the caller’s own Kubernetes RBAC check before the API server even reaches out to kubelet on their behalf).

Ans:

1. Instance launch: the EC2 instance boots with an attached IAM instance profile (the node IAM role) and user data containing the bootstrap configuration - either supplied directly, or, for managed node groups, wired in automatically by EKS’s own launch template.
2. Bootstrap script execution: the standard bootstrap.sh (on EKS-optimized AMIs) runs, pulling the cluster’s endpoint URL and CA certificate - either passed directly in user data, or fetched live from the EKS API using the instance’s IAM credentials.
3. kubeconfig generation: the script writes kubelet’s own kubeconfig, configured with an exec-based auth plugin (aws-iam-authenticator or equivalent) that generates a signed token from the instance’s IAM role credentials, obtained transparently through the instance metadata service (IMDS), every time kubelet needs to authenticate.
4. First authenticated request: kubelet signs its very first request to the API server using that IAM-derived token. The API server’s webhook token authenticator validates it against IAM’s STS GetCallerIdentity, confirming exactly which IAM role signed it.
5. Identity mapping: the validated IAM role gets mapped to a Kubernetes identity, system:node:<nodename> in the system:nodes group, via whatever access entry (or aws-auth mapping) exists for that role. No mapping here means the node authenticates successfully but can never actually register.
6. Node registration: with a valid, mapped identity, kubelet registers a new Node object with the API server, reporting its capacity and initial conditions.
7. Readiness: the node reports itself Ready once its network plugin (the VPC CNI) initializes successfully and no blocking conditions remain, at which point the scheduler can start placing Pods on it.

A failure at any step produces a distinctly different, diagnosable symptom - a broken bootstrap script means no Node object ever appears at all; a missing access entry mapping means the node authenticates but never successfully registers; a security group or subnet routing problem means the instance never even reaches the API endpoint to attempt any of this in the first place.

Ans: A managed node group upgrade is a rolling replacement, never an in-place patch - old instances are never modified, only replaced by new ones built from the updated AMI or launch template.

1. New instances launch first. EKS increases the underlying ASG’s capacity and launches new nodes on the updated AMI/launch template version, bootstrapping and registering them exactly like any new node join.
2. Readiness wait. EKS waits for these new nodes to report Ready before touching anything on the old side - workloads should never be asked to move to capacity that isn’t actually proven healthy yet.
3. Cordon old nodes. Once new capacity is confirmed ready, EKS cordons the old nodes (marking them unschedulable) so no new Pods land there, without disturbing anything already running.
4. Drain, respecting PodDisruptionBudgets. Each old node is drained - Pods are evicted and rescheduled onto the new nodes, but the drain actively respects any PDBs in place, meaning it will pause rather than force an eviction that would violate one.
5. Terminate. Once a node is fully drained, it’s terminated and removed from the ASG.
6. Pace control. You control how aggressively this rolls out via the node group’s updateConfig - a maxUnavailable count or percentage that caps how many nodes are being replaced simultaneously.

What minimizes disruption: new capacity always arrives before old capacity leaves (never the reverse), PDBs are a hard constraint the drain won’t violate, and the pace is entirely under your control. What can stall an upgrade: a PDB that makes eviction mathematically impossible (no slack left to remove even one Pod), a Pod that simply won’t terminate within its grace period, or insufficient capacity for new nodes to launch in the first place (a quota limit, or a subnet that’s run out of available IPs) - any of these leaves the upgrade paused rather than forcing something unsafe through.

Ans: EKS bridges two genuinely separate systems - IAM (identity) and Kubernetes RBAC (permissions) - and the whole integration comes down to exactly one translation point: the access entry mapping.

1. IAM identity and token generation: a caller (a person, a CI pipeline, a node) has an IAM user or role. A tool like aws eks get-token generates a short-lived, signed token from that identity’s actual AWS credentials - functionally a signed STS GetCallerIdentity request, valid for roughly 15 minutes.
2. Authentication via webhook: kube-apiserver, configured with EKS’s webhook token authenticator, receives this token on incoming requests and calls out to validate it against IAM, confirming exactly which IAM principal (by ARN) it belongs to. This step establishes identity only.
3. Mapping to a Kubernetes identity: the validated IAM principal is looked up against configured access entries (or, on older clusters, the aws-auth ConfigMap). A match produces a Kubernetes username and a set of Kubernetes groups for that request; no match means the caller is authenticated but has no Kubernetes identity whatsoever.
4. Kubernetes groups and RBAC: once mapped, everything downstream is completely standard Kubernetes RBAC - the mapped username/groups get checked against RoleBindings and ClusterRoleBindings, identically to how a certificate-based or OIDC-based identity would be evaluated on any non-EKS cluster.
5. API authorization decision: the request either proceeds (if some binding grants the needed verb on the needed resource) or gets a 403, exactly like standard Kubernetes - IAM plays no further role at this stage at all.

The clean separation matters practically: RBAC rules you write for an EKS cluster are portable, identical to rules you’d write anywhere else - there’s genuinely no EKS-specific RBAC syntax. And a caller with valid, fully authenticated IAM credentials but no matching access entry gets exactly nothing, since authentication succeeding never implies any Kubernetes permissions on its own.

Ans: EKS access management has a few interacting layers, and understanding each one’s job separately is what makes the whole system make sense.

Authentication modes (a cluster-level setting): each EKS cluster is configured with an authenticationMode deciding which identity-mapping mechanisms are active:

Access entries: each one ties a specific IAM principal ARN to the cluster, attaching either a built-in EKS access policy (like AmazonEKSClusterAdminPolicy, optionally scoped to a namespace via an access scope) or a mapping to a Kubernetes username/group you wire to your own RBAC objects.

IAM’s role: purely authentication - proving which principal is making a given request, with zero awareness of what that principal is actually allowed to do inside Kubernetes.

RBAC’s role: the only thing that actually grants permissions on Kubernetes objects, regardless of whether the caller’s identity came from an IAM-mapped access entry, a Kubernetes ServiceAccount token, or an OIDC identity - RBAC treats all of them identically once they’re mapped to a username/group.

Permissions actually required to access a cluster: two genuinely separate grants have to both be present -

1. An IAM permission to call the EKS API at all (eks:DescribeCluster, eks:AccessKubernetesApi) - without this, a caller can’t even fetch cluster connection info or successfully authenticate a request.
2. An access entry mapping plus RBAC grant - without this, the caller can authenticate fine but has no Kubernetes permissions whatsoever, since IAM alone was never enough on its own.

Missing either one produces a different, distinguishable failure - missing the IAM-level permission means you can’t even generate a usable token or reach the cluster; missing the access entry/RBAC grant means you authenticate fine but every Kubernetes-level action comes back Forbidden.

Ans: Fargate plugs into Kubernetes at the scheduling layer specifically, intercepting matched Pods before kube-scheduler ever gets a chance to bind them to an EC2 node.

Scheduling determination: when a new Pod is created, EKS checks its namespace (and optionally labels) against every configured Fargate profile. A match routes the Pod through Fargate’s own scheduling path instead of normal EC2-based scheduling - conceptually, a Fargate-aware component claims the Pod the way a virtual kubelet would, rather than kube-scheduler filtering and scoring real EC2 nodes for it.

Where AWS runs it: for a matched Pod, AWS provisions a right-sized, fully isolated micro-VM specifically for that one Pod, using the resource requests defined in its spec to determine sizing. There’s no shared node underneath - each Pod effectively gets its own dedicated, minimal “node,” which still registers as a real Node object so the rest of the cluster can interact with it normally.

Networking: Fargate Pods use the same VPC CNI model as EC2-backed Pods - each Pod gets a real, routable IP address directly from your VPC’s address space, via an ENI attached to its micro-VM. This is exactly why Fargate profile subnets must be private subnets, and it’s what lets Fargate Pods communicate with other Pods, Services, and AWS resources exactly like any other Pod in the cluster, with no special-cased networking path required.

What this rules out: because there’s no persistent, shared node, Fargate can’t support DaemonSets (nothing to run “on every node”), privileged Pods, or hostNetwork/hostPort - all of which assume node-level presence that simply doesn’t exist in this model.

Ans: An EKS add-on is a Kubernetes-native workload underneath (a DaemonSet, a Deployment, RBAC, ConfigMaps) - what makes it an “add-on” specifically is that EKS itself tracks and reconciles it as a managed resource, similar to how CloudFormation tracks a stack’s resources.

Integration with the control plane: installing an add-on causes EKS to apply the underlying Kubernetes objects through the same API path any client would use, and it registers a corresponding EKS-side resource with its own status and version, independent of but reflecting the state of those underlying Kubernetes objects.

Integration with worker nodes: most core add-ons (VPC CNI, kube-proxy) run as DaemonSets specifically so they have a Pod on every node - which is exactly the kind of workload Fargate can’t host, and part of why Fargate-only clusters have a different add-on story than EC2-backed ones.

Version management: each add-on has its own versioning independent of the cluster’s Kubernetes version, but EKS validates compatibility between the two before allowing an install or upgrade, preventing you from installing something the current control-plane version doesn’t actually support.

Configuration: add-ons accept configuration parameters through the EKS API at install or update time, which EKS then applies to the underlying resources - and for add-ons needing real AWS permissions (the EBS/EFS CSI drivers, for instance), configuration typically includes wiring up an IRSA role so the add-on’s Pods can assume scoped IAM permissions rather than relying on broad node-level access.

Lifecycle and drift handling: EKS continuously reconciles the add-on’s actual resources against what it expects, using a health status (CREATING, ACTIVE, DEGRADED, DELETING) to reflect current state. If something external modifies the add-on’s resources, EKS detects the conflict and resolves it according to the strategy you chose: OVERWRITE (EKS’s expected state always wins), PRESERVE (external changes are respected and left alone), or NONE (fail rather than silently choose a side) - this is the mechanism that keeps critical, cluster-wide infrastructure from silently drifting into an unknown or broken state over time.

Ans: An EKS upgrade is a deliberately sequenced process, and doing the steps out of order is the most common way to cause real problems.

1. Pre-upgrade checks. Before upgrading, you should confirm your workloads and tooling don’t depend on any Kubernetes API that’s being removed in the target version - kubectl and third-party tools like pluto can scan for deprecated API usage across your manifests.
2. Control-plane upgrade (in place). AWS upgrades the control plane’s API server, etcd, scheduler, and controller-manager to the target version behind the scenes, with no need to recreate the cluster. Because it’s stateless and load-balanced, this happens with no meaningful API-server downtime from the client’s perspective.
3. Version skew window. Once the control plane is upgraded, your existing nodes (still on the old kubelet version) keep working immediately - Kubernetes’ version skew policy explicitly allows kubelets to run a few minor versions behind the control plane. This window exists to give you room to upgrade nodes safely, not as a permanent state to leave things in.
4. Add-on compatibility and upgrade. Core add-ons (VPC CNI, CoreDNS, kube-proxy) need versions compatible with the new control-plane version - EKS will flag or block incompatible combinations, and you typically upgrade these shortly after the control plane, before touching node groups.
5. Managed node-group upgrades. Nodes are upgraded via the standard rolling-replacement process (new nodes launch on the updated AMI, old ones are cordoned, drained respecting PDBs, then terminated) - this is a separate, explicit step you trigger yourself.
6. Workload considerations. Anything relying on a now-removed API version needs to have been migrated before the control-plane upgrade actually happens - the upgrade itself won’t rewrite your manifests for you, and workloads still referencing a removed API will simply start failing once the old version is gone.

The version skew window is genuinely useful for staging the process safely, but it’s not meant to be a long-term state - leaving nodes several minor versions behind the control plane for an extended period eventually runs into real compatibility issues that the skew policy simply doesn’t cover.

Ans: By default, EKS control-plane logs aren’t sent anywhere - since AWS operates that infrastructure, you’d otherwise have zero visibility into it at all, so enabling this logging is usually one of the first things worth turning on for any production cluster.

Available log types:

• api - API server request logs; useful for seeing exactly what requests are being made and how the server is responding.
• audit - a detailed, security-focused record of who did what; the log type you reach for when investigating unexpected or unauthorized activity.
• authenticator - logs from the IAM webhook authenticator specifically; invaluable for debugging “why can’t this IAM role authenticate or get mapped correctly” issues.
• controllerManager - reconciliation activity from the built-in controllers.
• scheduler - scheduling decisions and any scheduling-related errors.

How they’re delivered: you enable whichever log types you want per cluster (via aws eks update-cluster-config or the console), and enabled types stream to CloudWatch Logs, into a dedicated log group following the pattern /aws/eks/<cluster-name>/cluster, with each log type getting its own log stream within that group.

aws eks update-cluster-config \
  --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator"],"enabled":true}]}'

Using them for troubleshooting: CloudWatch Logs Insights is the practical tool here - you can write queries filtering by error patterns, specific usernames, or time windows across whichever log type is relevant. audit combined with authenticator is the standard pair for chasing down “why does this IAM role get Forbidden” issues, since together they show both the raw authentication attempt and the resulting authorized (or denied) action. api is generally the first place to look for API server-level errors or unexpected latency, while scheduler and controllerManager are more useful when Pods aren’t landing where expected or reconciliation seems stuck.

One practical note worth keeping in mind: enabling every log type on a busy cluster can generate real CloudWatch Logs volume (and cost) fast, so many teams start with audit and authenticator (the highest-value, lowest-volume pair for security and access troubleshooting) and add api, scheduler, or controllerManager selectively when actively debugging something specific, rather than leaving everything on indefinitely by default.