Kubernetes Interview Questions & Answers — Advanced Kubernetes
Advanced Kubernetes interview questions on Helm, Operators and CRDs, multi-cluster and multi-tenant architectures, service mesh, cluster upgrades, etcd internals, custom schedulers, GitOps, and advanced deployment strategies like canary and blue-green.
Helm is the package manager for Kubernetes. It lets you define, install, and upgrade complex Kubernetes applications using charts (packaged YAML templates).
Without Helm — manually apply 10+ YAML files:
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml
kubectl apply -f configmap.yaml
kubectl apply -f secret.yaml
# ...and more
With Helm — one command:
helm install my-app ./my-chart --namespace production
Chart structure:
my-chart/
├── Chart.yaml ← Chart metadata (name, version, description)
├── values.yaml ← Default configuration values
├── templates/
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── configmap.yaml
│ └── _helpers.tpl ← Reusable template functions
└── charts/ ← Dependency charts
Key Helm commands:
# Add a chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
# Search for charts
helm search repo nginx
# Install a chart
helm install my-nginx bitnami/nginx \
--namespace web \
--create-namespace \
--set replicaCount=3 \
--set service.type=LoadBalancer
# Install with custom values file
helm install my-nginx bitnami/nginx -f custom-values.yaml
# Upgrade a release
helm upgrade my-nginx bitnami/nginx --set image.tag=1.25
# Rollback to previous version
helm rollback my-nginx 1
# List all releases
helm list --all-namespaces
# Uninstall a release
helm uninstall my-nginx -n web
# Render templates without installing (dry-run)
helm template my-nginx bitnami/nginx
helm install my-nginx bitnami/nginx --dry-run --debug
Ans:
# Method 1: Rolling Update (default)
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0 # Never have 0 available pods
maxSurge: 1 # Allow 1 extra pod during update
# Method 2: Add readiness probes (critical!)
containers:
- name: app
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
# K8s only routes traffic to pods that pass readiness probe
# Perform rolling update
kubectl set image deployment/my-deploy app=myimage:2.0
# Monitor rollout
kubectl rollout status deployment/my-deploy
# Rollback if issues
kubectl rollout undo deployment/my-deploy
# Method 3: Blue/Green deployment
# Deploy new version as separate deployment
# Switch service selector from app: blue → app: green
# Method 4: Canary with Ingress annotations
# Route 10% traffic to canary, 90% to stable
# nginx.ingress.kubernetes.io/canary: "true"
# nginx.ingress.kubernetes.io/canary-weight: "10"
Also use PodDisruptionBudget to prevent too many pods being unavailable:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: my-pdb
spec:
minAvailable: 2 # Always keep at least 2 pods
selector:
matchLabels:
app: myapp
🎯 Scenario: Your base application YAML works for dev, but production needs 5 replicas, different resource limits, a different image tag, and production database URLs.
Answer:
Directory structure:
k8s/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
└── overlays/
├── dev/
│ ├── kustomization.yaml
│ └── dev-patch.yaml
├── staging/
│ └── kustomization.yaml
└── production/
├── kustomization.yaml
├── prod-patch.yaml
└── prod-configmap.yaml
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
commonLabels:
app: web-api
managed-by: kustomize
# base/deployment.yaml (minimal baseline)
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-api
spec:
replicas: 1
template:
spec:
containers:
- name: api
image: myapp:latest
resources:
requests:
cpu: 100m
memory: 128Mi
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
namespace: production
images:
- name: myapp
newTag: "v2.1.0-prod" # Override image tag
patches:
- path: prod-patch.yaml
configMapGenerator:
- name: app-config
literals:
- DB_URL=postgresql://prod-db.internal:5432/myapp
- LOG_LEVEL=warn
- REPLICAS=5
# overlays/production/prod-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-api
spec:
replicas: 5
template:
spec:
containers:
- name: api
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "2"
memory: 1Gi
# Preview production output
kubectl kustomize overlays/production/
# Apply production config
kubectl apply -k overlays/production/
# Apply dev config
kubectl apply -k overlays/dev/
🎯 Scenario: Your cluster has no monitoring. Set up CPU, memory, pod health metrics with dashboards and alerts.
Answer:
# Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager + node-exporter)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--set grafana.adminPassword=changeme \
--set prometheus.prometheusSpec.retention=30d \
--set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=fast-ssd \
--set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi
# ServiceMonitor — tell Prometheus to scrape your app
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: api-server-monitor
namespace: production
labels:
release: monitoring # Must match Prometheus serviceMonitorSelector
spec:
selector:
matchLabels:
app: api-server
endpoints:
- port: metrics
path: /metrics
interval: 15s
scrapeTimeout: 10s
# PrometheusRule — define alerting rules
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: api-server-alerts
namespace: production
labels:
release: monitoring
spec:
groups:
- name: api-server
rules:
- alert: HighErrorRate
expr: |
(rate(http_requests_total{status=~"5..",job="api-server"}[5m]) /
rate(http_requests_total{job="api-server"}[5m])) > 0.05
for: 3m
labels:
severity: critical
team: backend
annotations:
summary: "High error rate on {{ $labels.pod }}"
description: "Error rate {{ $value | humanizePercentage }} for 3+ minutes"
runbook_url: "https://wiki/runbook/high-error-rate"
- alert: PodCrashLooping
expr: increase(kube_pod_container_status_restarts_total[1h]) > 3
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
- alert: PodMemoryNearLimit
expr: |
container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.90
for: 5m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} memory at {{ $value | humanizePercentage }} of limit"
- alert: NodeDiskSpaceLow
expr: |
(node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.10
for: 5m
labels:
severity: critical
annotations:
summary: "Node {{ $labels.instance }} disk is {{ $value | humanizePercentage }} free"
🎯 Scenario: You need an instant traffic switch to a new version with instant rollback capability — rolling update is too slow.
Answer:
# Blue deployment — current production (receives all traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app-blue
spec:
replicas: 3
selector:
matchLabels:
app: web-app
slot: blue
template:
metadata:
labels:
app: web-app
slot: blue
version: v1.0
spec:
containers:
- name: web
image: myapp:v1.0
---
# Green deployment — new version (deployed, but receives no traffic yet)
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app-green
spec:
replicas: 3
selector:
matchLabels:
app: web-app
slot: green
template:
metadata:
labels:
app: web-app
slot: green
version: v2.0
spec:
containers:
- name: web
image: myapp:v2.0
---
# Service — currently pointing to blue
apiVersion: v1
kind: Service
metadata:
name: web-app
spec:
selector:
app: web-app
slot: blue # ← Change to "green" to switch all traffic instantly
ports:
- port: 80
targetPort: 8080
# Deploy green alongside blue (no traffic yet)
kubectl apply -f green-deployment.yaml
# Test green directly before switching
kubectl port-forward deployment/web-app-green 8081:8080
curl http://localhost:8081/health
curl http://localhost:8081/api/test
# Run smoke tests against green
# ...all good?
# SWITCH TRAFFIC — instant, takes effect in <1 second
kubectl patch service web-app \
-p '{"spec":{"selector":{"slot":"green"}}}'
# Verify switch took effect
kubectl get endpoints web-app # Should show green pod IPs
# Monitor error rate for 15 minutes...
# If issues: INSTANT ROLLBACK (1 command)
kubectl patch service web-app \
-p '{"spec":{"selector":{"slot":"blue"}}}'
# After successful validation: clean up blue
kubectl delete deployment web-app-blue
🎯 Scenario: Broken YAML or misconfigured manifests reach production, causing deployment failures. How do you catch these in CI?
Answer:
# GitHub Actions — comprehensive manifest validation
name: Validate K8s Manifests
on: [pull_request]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# 1. YAML syntax validation
- name: Validate YAML
run: |
pip install yamllint
yamllint -d relaxed k8s/
# 2. Kubernetes schema validation with kubeval
- name: Kubeval
run: |
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
./kubeval --kubernetes-version=1.28.0 k8s/**/*.yaml
# 3. Advanced validation with kubeconform
- name: Kubeconform
uses: docker://ghcr.io/yannh/kubeconform:latest
with:
args: "-strict -summary -kubernetes-version 1.28.0 k8s/"
# 4. Security scanning with Trivy
- name: Trivy K8s scan
uses: aquasecurity/trivy-action@master
with:
scan-type: config
scan-ref: k8s/
severity: CRITICAL,HIGH
exit-code: 1
# 5. Policy compliance with Checkov
- name: Checkov K8s policies
uses: bridgecrewio/checkov-action@master
with:
directory: k8s/
framework: kubernetes
soft_fail: false
# 6. Kustomize build verification
- name: Kustomize build
run: |
for env in dev staging production; do
echo "Building $env..."
kubectl kustomize k8s/overlays/$env > /dev/null
echo "$env: OK"
done
# 7. Helm chart linting
- name: Helm lint
run: |
helm lint charts/web-app/ \
--values charts/web-app/values-prod.yaml
Answer:
etcd is a distributed, consistent key-value store used as Kubernetes’ backing store for all cluster data. Every API object (Pods, Services, ConfigMaps, Secrets, etc.) is stored in etcd.
Key properties:
- Consistency: Uses Raft consensus algorithm for leader election and data replication
- High availability: Typically run as a 3 or 5-node cluster (odd number for quorum)
- Watch API: Enables Kubernetes controllers to watch for changes
Important facts:
- All communication with etcd goes through the API server
- Backing up etcd is critical for disaster recovery
- In EKS, etcd is fully managed by AWS
# In a self-managed cluster — backup etcd
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
Answer:
A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application using custom controllers and CRDs. Operators encode operational knowledge (how to deploy, scale, upgrade, backup) into software.
Operator pattern:
- Define a CRD (e.g.,
PostgreSQLCluster) - Implement a Controller that watches the CRD
- Controller reconciles the actual state with the desired state
Popular Operators:
- Prometheus Operator
- PostgreSQL Operator (Zalando or CrunchyData)
- Cert-Manager
- ArgoCD
# Example: Using the Prometheus Operator CRD
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
replicas: 2
retention: 30d
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 50Gi
Answer:
EKS Anywhere allows you to create and manage Kubernetes clusters on your own infrastructure (on-premises, VMware vSphere, bare metal, or other cloud providers) using the same EKS tools and configurations used in AWS.
Use cases:
- Regulatory requirements that prevent cloud usage
- Data sovereignty requirements
- Hybrid cloud architectures
- Air-gapped environments
Key features:
- Uses same EKS configuration API
- Supports curated packages (CoreDNS, Cilium, etc.)
- Optionally connect to AWS via EKS Connector for management in the AWS Console
Answer: This is an experience question — a strong answer names a specific strategy, ties it to a concrete reason, and shows you considered the trade-off, not just recited the options.
Example of how to structure it: “For our main API service, I used rolling updates with maxSurge: 1, maxUnavailable: 0 — it’s the default, requires no extra infrastructure, and zero-downtime was the main requirement, not instant rollback. Readiness probes were critical here — without a well-tuned readiness probe, Kubernetes will happily route traffic to a Pod that’s technically running but not yet ready to serve, which caused a handful of 502s before we tightened it.”
spec:
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
template:
spec:
containers:
- name: api
readinessProbe:
httpGet: { path: /ready, port: 8080 }
initialDelaySeconds: 10
periodSeconds: 5
“For a higher-risk service — our payments API — we used canary via Argo Rollouts instead: 10% traffic to the new version, automated analysis against error-rate and latency metrics from Prometheus for 10 minutes, auto-promote on pass or auto-rollback on fail. The extra tooling was worth it there specifically because a bad payments deploy is much more expensive than a bad deploy anywhere else in the system.”
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 10m }
- analysis:
templates: [{ templateName: success-rate }]
What makes this a strong answer: naming different strategies for different services based on their actual risk profile — not applying the same strategy everywhere out of habit — is what separates real production experience from a memorized definition of blue-green/canary/rolling.
Zero-downtime deployments require a combination of correct deployment strategy, pod lifecycle hooks, and health probes.
Complete zero-downtime deployment configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2 # Can temporarily have 8 pods (6+2)
maxUnavailable: 0 # Never drop below 6 healthy pods
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
# 1. Give pods time to finish in-flight requests before shutdown
terminationGracePeriodSeconds: 60
containers:
- name: web
image: web-app:v2.0
ports:
- containerPort: 8080
# 2. Readiness probe — pod only gets traffic when truly ready
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
# 3. Liveness probe — restart if pod is dead
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 15
# 4. preStop hook — wait for traffic to drain before shutdown
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 15 # Wait 15s for load balancer to remove pod from rotation
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
Canary deployment pattern:
# Deploy v2 to 10% of traffic first
kubectl scale deployment web-app-v1 --replicas=9 # 9 old pods
kubectl scale deployment web-app-v2 --replicas=1 # 1 new pod (10%)
# If v2 is healthy, gradually increase
kubectl scale deployment web-app-v2 --replicas=5 # 50%
kubectl scale deployment web-app-v2 --replicas=10 # 100%
kubectl scale deployment web-app-v1 --replicas=0 # Remove old
Blue-Green deployment:
# Switch Service selector from blue to green instantly
kubectl patch service web-svc \
-p '{"spec":{"selector":{"version":"v2"}}}'
# Rollback instantly by switching back
kubectl patch service web-svc \
-p '{"spec":{"selector":{"version":"v1"}}}'
etcd is a distributed, consistent key-value store that serves as Kubernetes’ source of truth. Every object (pods, services, configmaps, secrets) is stored in etcd.
Architecture:
All cluster state stored in etcd:
/registry/pods/default/my-pod
/registry/services/default/my-svc
/registry/deployments/production/web-app
/registry/secrets/default/db-secret
etcd uses the Raft consensus algorithm:
- Requires a quorum (majority) to function:
(n/2) + 1 - 3 members → can tolerate 1 failure
- 5 members → can tolerate 2 failures
- Always use odd numbers of etcd members
| Cluster Size | Quorum | Tolerable Failures |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
What happens when etcd goes down:
etcd down → API server cannot read/write state
→ No new pods can be scheduled
→ Existing pods keep running (kubelet works independently)
→ kubectl commands fail
→ New deployments fail
Backup etcd (critical for disaster recovery):
# Take an etcd snapshot
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Verify the snapshot
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-$(date +%Y%m%d).db
# Restore from snapshot
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db \
--data-dir=/var/lib/etcd-restored
# Automate daily backups via CronJob
kubectl apply -f etcd-backup-cronjob.yaml
Best practices:
- Always run etcd on separate dedicated nodes from worker nodes
- Use SSDs — etcd is I/O intensive
- Monitor etcd latency (should be < 10ms)
- Take snapshots before every cluster upgrade
A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application using Custom Resource Definitions (CRDs) and custom controllers that encode operational knowledge.
The Operator pattern:
Human Operator knowledge → encoded in → Custom Controller
↓
CRD (custom resource) → Controller reconciles → Desired state
When to build a Kubernetes Operator:
- Managing stateful applications (databases, message queues)
- Automating complex operational tasks (backups, upgrades, failover)
- When your app needs more than Deployment/StatefulSet
- Encoding domain-specific knowledge (e.g., how to scale a database cluster)
Example CRD — custom database resource:
# 1. Define the Custom Resource Definition
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: postgresclusters.db.example.com
spec:
group: db.example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
replicas:
type: integer
version:
type: string
backupSchedule:
type: string
scope: Namespaced
names:
plural: postgresclusters
singular: postgrescluster
kind: PostgresCluster
---
# 2. Use the custom resource (like any K8s object now)
apiVersion: db.example.com/v1
kind: PostgresCluster
metadata:
name: my-database
namespace: production
spec:
replicas: 3
version: "15.2"
backupSchedule: "0 2 * * *" # Operator handles backups automatically
Popular real-world operators:
# Install cert-manager operator (manages TLS certificates)
helm install cert-manager jetstack/cert-manager --set installCRDs=true
# Install Prometheus operator (manages monitoring stack)
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack
# Install Strimzi operator (manages Kafka clusters)
helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator
Build your own operator:
# Use Operator SDK (most popular framework)
operator-sdk init --domain example.com --repo github.com/example/my-operator
operator-sdk create api --group apps --version v1 --kind MyApp --resource --controller
# Or use Kubebuilder
kubebuilder init --domain example.com
kubebuilder create api --group apps --version v1 --kind MyApp
GitOps is a deployment methodology where Git is the single source of truth for cluster state. ArgoCD continuously syncs the cluster to match what’s in Git.
GitOps principles:
- Entire system described declaratively in Git
- Desired state versioned in Git
- Approved changes automatically applied to the cluster
- Software agents ensure correctness and alert on divergence
Install ArgoCD:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
# Port-forward to access UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
Create an ArgoCD Application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-app-production
namespace: argocd
spec:
project: default
# Source — where your manifests live in Git
source:
repoURL: https://github.com/myorg/k8s-configs
targetRevision: main
path: apps/web-app/production
# Destination — where to deploy in the cluster
destination:
server: https://kubernetes.default.svc
namespace: production
# Sync policy — auto-sync when Git changes
syncPolicy:
automated:
prune: true # Delete resources removed from Git
selfHeal: true # Revert manual changes to cluster
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
retry:
limit: 5
backoff:
duration: 5s
maxDuration: 3m
factor: 2
GitOps workflow:
# Developer makes a change
git checkout -b feature/update-image
# Edit k8s/deployment.yaml — change image tag
git commit -m "deploy: bump web-app to v2.5"
git push origin feature/update-image
# Create PR → review → merge to main
# ArgoCD detects the change within 3 minutes
# ArgoCD applies the change to cluster automatically
# Check sync status
argocd app get web-app-production
argocd app sync web-app-production # Manual sync if needed
argocd app history web-app-production # Deployment history
A production-grade Kubernetes architecture for high-traffic needs to address availability, security, scalability, and observability.
Cluster architecture:
┌─────────────────────────────────┐
│ CONTROL PLANE (HA) │
│ 3x master nodes (multi-AZ) │
│ etcd cluster (separate nodes) │
└──────────────┬──────────────────┘
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ AZ-1 Nodes │ │ AZ-2 Nodes │ │ AZ-3 Nodes │
│ App workers │ │ App workers │ │ App workers │
│ GPU nodes │ │ GPU nodes │ │ Spot nodes │
└─────────────┘ └─────────────┘ └─────────────┘
Node pool strategy:
# System node pool — control plane components
nodePool: system
instanceType: m5.xlarge
count: 3
taints: [CriticalAddonsOnly=true:NoSchedule]
# Application node pool — production workloads (on-demand)
nodePool: app-ondemand
instanceType: m5.2xlarge
minCount: 6
maxCount: 50
availabilityZones: [us-east-1a, us-east-1b, us-east-1c]
# Spot node pool — batch/non-critical workloads (80% cheaper)
nodePool: app-spot
instanceTypes: [m5.2xlarge, m5.4xlarge, m5a.2xlarge]
spot: true
minCount: 0
maxCount: 100
Production deployment configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
spec:
replicas: 9 # 3 per AZ
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 3
maxUnavailable: 0
template:
spec:
# Spread across zones and nodes
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: web-app
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: web-app
# Don't schedule on spot nodes (critical app)
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node.kubernetes.io/lifecycle
operator: NotIn
values: [spot]
containers:
- name: web-app
image: web-app:v3.0
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"
readinessProbe:
httpGet: {path: /ready, port: 8080}
periodSeconds: 5
livenessProbe:
httpGet: {path: /health, port: 8080}
periodSeconds: 15
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
terminationGracePeriodSeconds: 60
Observability stack:
# Metrics — Prometheus + Grafana
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack
# Logs — EFK or Loki stack
helm install loki grafana/loki-stack --set grafana.enabled=false
# Tracing — Jaeger or Tempo
helm install jaeger jaegertracing/jaeger
# Alerts — configure PagerDuty/Slack in Alertmanager
Key SLOs to monitor:
# Error rate < 0.1%
# P99 latency < 200ms
# Availability > 99.9%
# Pod restart rate < 1/hour
# Node CPU < 70%
# Node Memory < 80%
Answer:
The problem with self-managed Kubernetes on EC2:
When you run Kubernetes yourself on EC2 you are responsible for everything: installing and upgrading the control plane (API server, etcd, scheduler, controller manager), patching etcd for security vulnerabilities, ensuring etcd backups, managing control plane HA across multiple EC2 instances, and debugging issues when the API server goes down at 2 AM. A typical self-managed control plane requires 3–5 dedicated EC2 instances just for the control plane, a dedicated team to maintain it, and on-call engineers who deeply understand Kubernetes internals.
What EKS gives you:
EKS is AWS’s managed Kubernetes service. AWS runs and manages the control plane for you. The Kubernetes API server, etcd, and control plane components run in an AWS-managed account — you never see them, never patch them, never back them up. AWS guarantees 99.95% SLA on the control plane.
Real-world comparison:
| Responsibility | Self-Managed on EC2 | EKS |
|---|---|---|
| Kubernetes version upgrades | Your team | One-click in console |
| etcd backup and restore | Your team | AWS handles it |
| Control plane HA | Your team (3+ EC2 instances) | AWS handles it |
| Control plane security patches | Your team | AWS handles it |
| Cost of control plane EC2 | You pay for 3–5 instances | $0.10/hour flat fee |
| Worker nodes | You manage | You manage (but with managed node groups available) |
Real-world example:
A fintech startup was spending 40% of their DevOps team’s time maintaining a self-managed Kubernetes cluster — patching, backing up etcd, debugging control plane issues. After migrating to EKS, that 40% of time was redirected to building product features. The control plane costs $0.10/hour (~$72/month) which was less than the EC2 cost of their self-managed control plane.
Answer:
Architecture: Fluent Bit → CloudWatch Logs
Fluent Bit is a lightweight log processor that runs as a DaemonSet — one pod on every node. It reads logs from all containers on its node and ships them to CloudWatch Logs.
Step 1 — Create IAM permissions for log shipping:
eksctl create iamserviceaccount \
--name fluent-bit \
--namespace amazon-cloudwatch \
--cluster production-cluster \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--approve
Step 2 — Deploy Fluent Bit using the AWS-provided configuration:
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
Step 3 — Configure CloudWatch log retention:
# Set 90-day retention on all EKS log groups
aws logs put-retention-policy \
--log-group-name /aws/containerinsights/production-cluster/application \
--retention-in-days 90
Log structure in CloudWatch:
/aws/containerinsights/production-cluster/
/application → all container stdout/stderr logs
/dataplane → kubelet, kube-proxy system logs
/host → EC2 instance system logs
/performance → CPU, memory, network metrics per pod
Searching logs:
# Use AWS CLI for quick searches
aws logs filter-log-events \
--log-group-name /aws/containerinsights/production-cluster/application \
--filter-pattern "ERROR" \
--start-time $(date -d '1 hour ago' +%s)000
Or use CloudWatch Log Insights:
fields @timestamp, kubernetes.pod_name, log
| filter kubernetes.namespace_name = "production"
| filter log like /ERROR/
| sort @timestamp desc
| limit 100
Answer:
What is GitOps:
GitOps is a practice where the entire desired state of your Kubernetes cluster is stored in a Git repository. ArgoCD continuously reconciles what is in Git with what is actually running in the cluster. If someone manually changes something in the cluster, ArgoCD detects the drift and reverts it. If you change something in Git, ArgoCD automatically applies it to the cluster.
Architecture:
Developer pushes code
→ GitHub Actions CI pipeline runs:
- Builds Docker image
- Runs tests
- Pushes image to ECR
- Updates image tag in Git (Kubernetes manifests repo)
→ ArgoCD detects change in Git repo
→ ArgoCD applies new manifests to EKS cluster
→ ArgoCD reports sync status (Synced / OutOfSync / Degraded)
Step 1 — Install ArgoCD:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
Step 2 — Create an Application pointing to your Git repo:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-api-production
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/mycompany/k8s-manifests
targetRevision: main
path: apps/web-api/production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true # Delete resources removed from Git
selfHeal: true # Revert any manual changes to the cluster
syncOptions:
- CreateNamespace=true
Step 3 — Deployment workflow:
A developer does not run kubectl apply to deploy. They submit a Pull Request changing the image tag in the manifests repository. After review and merge, ArgoCD detects the change and automatically deploys within 3 minutes.
Benefits realized in production:
- Complete audit trail: every deployment is a git commit with author, timestamp, and PR description
- Easy rollback:
git revertthe commit, ArgoCD deploys the previous version - No
kubectlaccess needed for deployments: developers interact with Git, not the cluster - Drift detection: if someone accidentally deletes a ConfigMap manually, ArgoCD restores it from Git automatically
Answer:
The problem without a service mesh:
With 20 microservices, each service team would need to implement: TLS certificate management and rotation, retry logic, circuit breakers, timeout handling, and distributed tracing instrumentation. That is 20 codebases all implementing the same cross-cutting concerns differently. A security audit would need to verify each independently.
Solution: Istio service mesh
Istio injects a sidecar proxy (Envoy) into every pod. All network traffic between pods goes through these sidecar proxies, not directly between containers. This means security, observability, and traffic management features are handled at the infrastructure layer — application code needs zero changes.
Install Istio on EKS:
istioctl install --set profile=production -y
kubectl label namespace production istio-injection=enabled
The label istio-injection=enabled tells Istio to automatically inject the Envoy sidecar proxy into every new pod in the production namespace.
Automatic mTLS between all services:
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: production
spec:
mtls:
mode: STRICT
This single configuration enforces mutual TLS for all pod-to-pod communication in the production namespace. Every service now authenticates the identity of every other service it communicates with. Istio handles certificate generation and rotation automatically.
Traffic management example — canary deployment:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: payment-service
spec:
hosts:
- payment-service
http:
- route:
- destination:
host: payment-service
subset: v1
weight: 90
- destination:
host: payment-service
subset: v2
weight: 10
Route 10% of traffic to v2 of the payment service while 90% goes to v1. Gradually shift traffic as confidence grows. No changes to application code or Deployments needed.
Answer:
Cluster topology:
Run three separate EKS clusters: production (payment processing), staging (pre-prod testing), and management (CI/CD, monitoring, ArgoCD). Separation prevents a deployment to staging from impacting production infrastructure.
Network isolation for PCI-DSS:
Payment processing pods run in a dedicated namespace with a NetworkPolicy that allows NO ingress or egress except to explicitly whitelisted services. No pod in the payment namespace can make arbitrary external HTTP calls.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: payment-service-isolation
namespace: payment-processing
spec:
podSelector:
matchLabels:
app: payment-processor
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: api-gateway
egress:
- to:
- podSelector:
matchLabels:
app: payment-database
- to:
- namespaceSelector:
matchLabels:
name: kube-system
ports:
- port: 53 # DNS only
Node isolation for PCI scope reduction:
Payment processing pods run on dedicated nodes with a taint so no other workload can be scheduled there. These nodes have enhanced CloudTrail logging and are the only nodes in-scope for PCI audits.
Zero-downtime deployments:
Blue-green deployments using Argo Rollouts:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
strategy:
blueGreen:
activeService: payment-service-active
previewService: payment-service-preview
autoPromotionEnabled: false # Require manual promotion
prePromotionAnalysis:
templates:
- templateName: payment-success-rate
args:
- name: service-name
value: payment-service-preview
New version deploys to the “preview” service. Automated analysis checks error rate and p99 latency. If metrics are healthy, a human approves promotion. If not, automatic rollback. Zero requests lost because traffic switches atomically between the two services.
99.99% availability (52 minutes downtime/year maximum):
Multi-AZ node groups across 3 AZs. PodDisruptionBudget ensuring minimum 2 replicas always running. Pod topology spread constraints ensuring replicas are in different AZs AND different nodes. RDS Aurora Global Database for payment data with failover under 1 second. Route 53 health checks with 10-second TTL for DNS failover.
Answer:
Kubernetes upgrade rules:
You can only upgrade one minor version at a time (1.27 → 1.28, then 1.28 → 1.29). Skipping versions is not supported. Also: the control plane must be upgraded before the worker nodes.
Phase 1 — Upgrade EKS Control Plane (1.27 → 1.28):
# Review the upgrade guide for any breaking changes
# Update one minor version at a time
eksctl upgrade cluster \
--name production-cluster \
--version 1.28 \
--approve
This upgrades the managed control plane (API server, etcd, scheduler). Your worker nodes remain on 1.27 during this step. Kubernetes guarantees backward compatibility — 1.27 nodes work with a 1.28 control plane.
Phase 2 — Check for deprecated API versions in your manifests:
A very common upgrade failure cause: your Deployments use API versions that were deprecated and removed in the new Kubernetes version.
# Install kubent (kube no trouble) to find deprecated APIs
kubent
Fix any flagged resources before upgrading nodes.
Phase 3 — Upgrade worker nodes (1.27 → 1.28):
For Managed Node Groups, AWS does this with a rolling update:
eksctl upgrade nodegroup \
--name standard-workers \
--cluster production-cluster \
--kubernetes-version 1.28
This creates new nodes with 1.28, cordons old nodes, drains them one by one (respecting PodDisruptionBudgets), and terminates them. Your pods are moved to the new nodes during the drain.
Phase 4 — Upgrade add-ons:
After node upgrade, update the add-ons to versions compatible with 1.28:
eksctl update addon --name vpc-cni --cluster production-cluster
eksctl update addon --name coredns --cluster production-cluster
eksctl update addon --name kube-proxy --cluster production-cluster
Phase 5 — Repeat for 1.28 → 1.29.
Real-world timeline: A 20-node cluster upgrade takes approximately 90 minutes per version, mostly waiting for new nodes to provision and old nodes to drain. Total for two minor version upgrades: 3–4 hours.
Answer:
Multi-tenancy in Kubernetes uses namespaces as the isolation boundary, with four enforcement mechanisms:
1. Namespace-per-team:
kubectl create namespace team-payments
kubectl create namespace team-identity
kubectl create namespace team-analytics
kubectl create namespace team-notifications
kubectl create namespace team-frontend
2. RBAC — each team can only see and manage their own namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: team-payments-full-access
namespace: team-payments
subjects:
- kind: Group
name: payments-team
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: admin # admin role scoped to team-payments namespace only
apiGroup: rbac.authorization.k8s.io
The payments team has admin access in team-payments namespace. They can create, update, delete pods, deployments, services in their namespace. They cannot see anything in team-identity or any other namespace.
3. ResourceQuota — prevent one team from consuming all cluster resources:
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-payments-quota
namespace: team-payments
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.cpu: "40"
limits.memory: 80Gi
count/pods: "100"
count/services: "20"
Even if the payments team has a runaway pod that requests unlimited CPU, it cannot consume more than 40 CPU cores or 80 GiB of memory.
4. NetworkPolicy — teams cannot access each other’s services:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: team-isolation
namespace: team-payments
spec:
podSelector: {} # Apply to all pods in this namespace
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: team-payments # Only allow traffic from same namespace
- namespaceSelector:
matchLabels:
name: kube-system # Allow system components (CoreDNS)
This prevents the team-analytics namespace from directly calling the team-payments API. Cross-team communication must go through defined API contracts, not direct pod-to-pod networking.
5. LimitRange — prevent pods without resource requests (which consume unlimited resources):
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: team-payments
spec:
limits:
- default:
cpu: "500m"
memory: "512Mi"
defaultRequest:
cpu: "100m"
memory: "128Mi"
type: Container
If a developer in the payments team deploys a pod without resource requests, the LimitRange automatically applies default requests and limits. This prevents uncontrolled resource consumption.
📝 Quick Reference Cheat Sheet
EKS Key Commands
# Cluster info
kubectl cluster-info
eksctl get cluster
# Node status
kubectl get nodes -o wide
kubectl describe node <node-name>
# Pod debugging
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous
kubectl exec -it <pod> -n <ns> -- /bin/sh
# Scaling
kubectl scale deployment <name> --replicas=5
kubectl rollout status deployment/<name>
kubectl rollout undo deployment/<name>
# Resource usage
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top nodes
Pod Status Quick Reference
| Status | Meaning | Common Cause |
|---|---|---|
| Pending | Not scheduled | Insufficient resources, taint/toleration mismatch, PVC unbound |
| ContainerCreating | Pulling image or mounting volume | Image pull slow, PVC not bound |
| CrashLoopBackOff | App crashes repeatedly | App error, OOMKill, wrong config |
| OOMKilled | Out of memory | Memory limit too low |
| ImagePullBackOff | Cannot pull container image | Wrong image name, ECR permissions, private repo credentials |
| Terminating (stuck) | Pod not cleaning up | Finalizers not removed, storage unmount issue |
| Evicted | Removed from node | Node ran out of disk or memory |
Storage Access Modes
| Mode | Description | Use Case |
|---|---|---|
| ReadWriteOnce (RWO) | One node, read-write | Databases, EBS |
| ReadOnlyMany (ROX) | Many nodes, read-only | Configuration, static assets |
| ReadWriteMany (RWX) | Many nodes, read-write | Shared uploads, EFS |
Service Type Summary
| Type | Reachable from | Use case |
|---|---|---|
| ClusterIP | Inside cluster only | Service-to-service |
| NodePort | Outside via node IP:port | Dev/test only |
| LoadBalancer | Outside via AWS LB | Single TCP/UDP services |
| Ingress | Outside via ALB (HTTP/HTTPS) | Multiple web services |
EKS Upgrade Order
1. Update EKS control plane (eksctl upgrade cluster)
2. Update EKS managed add-ons (vpc-cni, coredns, kube-proxy)
3. Update worker nodes (eksctl upgrade nodegroup)
4. Update third-party add-ons (metrics-server, ALB controller, etc.)
🎯 Scenario: One node of your production etcd cluster fails. What is the impact and how do you recover?
Answer:
etcd is a distributed key-value store using the Raft consensus algorithm. It stores ALL cluster state: pods, services, secrets, configmaps, RBAC, and custom resources.
Impact by failure scenario:
| Scenario | Impact |
|---|---|
| 1 of 3 nodes down | Cluster fully operational — quorum maintained (2 of 3) |
| 2 of 3 nodes down | Read-only — no changes possible, existing workloads keep running |
| All nodes down | Complete outage — control plane unresponsive |
| Data corruption | Must restore from snapshot backup |
# Check etcd health
kubectl exec -n kube-system etcd-master -- etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health
# Check etcd member list
etcdctl member list --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Take a snapshot backup
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Verify snapshot
etcdctl snapshot status /backup/etcd-20240101.db --write-out=table
# Restore from snapshot
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-20240101.db \
--data-dir=/var/lib/etcd-restored \
--name=master \
--initial-cluster=master=https://127.0.0.1:2380 \
--initial-advertise-peer-urls=https://127.0.0.1:2380
⚠️ Production requirement: Always run etcd with 3 or 5 nodes (odd number for quorum). Automate etcd snapshots to S3 every 30 minutes. Test restores regularly.
🎯 Scenario: You need to update a production web app from v1.0 to v2.0 with zero service interruption and immediate rollback capability.
Answer:
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # Create 1 extra pod above desired count
maxUnavailable: 0 # Never go below desired count (zero-downtime)
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web
image: myapp:v2.0
# Readiness probe gates traffic — CRITICAL for zero-downtime
readinessProbe:
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
successThreshold: 1
# Liveness probe restarts deadlocked pods
livenessProbe:
httpGet:
path: /health/live
port: 8080
periodSeconds: 10
failureThreshold: 3
# Give in-flight requests time to complete
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
# Must be >= preStop sleep time
terminationGracePeriodSeconds: 30
# Trigger update
kubectl set image deployment/web-app web=myapp:v2.0 -n production
# Watch real-time rollout progress
kubectl rollout status deployment/web-app -n production
# Inspect revision history
kubectl rollout history deployment/web-app -n production
# Instant rollback to previous version
kubectl rollout undo deployment/web-app -n production
# Rollback to specific revision
kubectl rollout undo deployment/web-app --to-revision=3 -n production
# Pause mid-rollout (canary-style manual gate)
kubectl rollout pause deployment/web-app -n production
# Check metrics, error rates...
kubectl rollout resume deployment/web-app -n production
✅ Without a readiness probe, Kubernetes has no way to know if a new pod is actually serving traffic. New pods will receive traffic immediately upon startup — before they’re ready — causing errors.
🎯 Scenario: You want to route 10% of production traffic to v2.0 to validate it, then gradually increase to 100%.
Answer:
# Stable Deployment — 9 replicas = 90% traffic
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app-stable
namespace: production
spec:
replicas: 9
selector:
matchLabels:
app: web-app
track: stable
template:
metadata:
labels:
app: web-app # ← shared label
track: stable
spec:
containers:
- name: web
image: myapp:v1.0
---
# Canary Deployment — 1 replica = 10% traffic
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app-canary
namespace: production
spec:
replicas: 1
selector:
matchLabels:
app: web-app
track: canary
template:
metadata:
labels:
app: web-app # ← same shared label
track: canary
spec:
containers:
- name: web
image: myapp:v2.0
---
# Service selects ALL pods with app=web-app
# Traffic split is proportional to replica count (9:1 = 90%:10%)
apiVersion: v1
kind: Service
metadata:
name: web-app
spec:
selector:
app: web-app # Matches BOTH stable and canary pods
ports:
- port: 80
targetPort: 8080
# Step 1: Deploy canary at 10%
kubectl apply -f canary-deployment.yaml
# Step 2: Monitor error rate and latency
kubectl logs -l track=canary --tail=200 -n production
kubectl top pods -l track=canary -n production
# Step 3a: Promote — gradually increase canary, decrease stable
kubectl scale deployment/web-app-canary --replicas=3 # 30%
kubectl scale deployment/web-app-stable --replicas=7 # 70%
# ... eventually
kubectl scale deployment/web-app-canary --replicas=10 # 100%
kubectl scale deployment/web-app-stable --replicas=0
# Step 3b: Rollback if issues found
kubectl scale deployment/web-app-canary --replicas=0
kubectl delete deployment/web-app-canary
💡 For header/cookie-based traffic splitting, use NGINX Ingress canary annotations or a service mesh (Istio, Linkerd). The replica-ratio approach splits randomly which is fine for simple cases.
🎯 Scenario: Your team wants to search and correlate logs across hundreds of pods in Grafana (same tool you use for metrics).
Answer:
# Install Loki + Promtail + Grafana (Loki Stack)
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki-stack grafana/loki-stack \
--namespace logging \
--create-namespace \
--set loki.enabled=true \
--set promtail.enabled=true \
--set grafana.enabled=true \
--set loki.persistence.enabled=true \
--set loki.persistence.size=50Gi
# Promtail ConfigMap — scrape pod logs with metadata enrichment
apiVersion: v1
kind: ConfigMap
metadata:
name: promtail-config
namespace: logging
data:
promtail.yaml: |
server:
http_listen_port: 3101
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
pipeline_stages:
- cri: {}
- labeldrop:
- filename
relabel_configs:
# Enrich logs with K8s metadata
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: app
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
# LogQL queries (Loki Query Language) in Grafana:
# View all logs from production namespace
{namespace="production"}
# View error logs from specific app
{app="api-server", namespace="production"} |= "ERROR"
# Count error rate
rate({app="api-server"} |= "ERROR" [5m])
# Parse JSON logs and filter by field
{app="api-server"} | json | status_code >= 500
# View logs from specific pod
{pod="api-server-7d9f4b-xxxx"}
# Correlate with trace ID
{namespace="production"} | json | traceID = "abc123def456"
🎯 Scenario: Your team wants every change to Kubernetes to go through Git — no manual
kubectl applyin production.
Answer:
# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd \
-f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Get initial admin password
argocd admin initial-password -n argocd
# Expose UI via LoadBalancer
kubectl patch svc argocd-server -n argocd \
-p '{"spec":{"type":"LoadBalancer"}}'
# ArgoCD Application — watches Git and syncs to cluster
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-app-production
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io # Cascade delete
spec:
project: production-apps
source:
repoURL: https://github.com/my-org/k8s-manifests.git
targetRevision: main
path: apps/web-app/overlays/production
# For Helm:
# chart: web-app
# helm:
# releaseName: web-app
# valueFiles: [values-prod.yaml]
# parameters:
# - name: image.tag
# value: v2.1.0
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true # Delete resources removed from Git
selfHeal: true # Revert manual changes back to Git state
allowEmpty: false
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- ApplyOutOfSyncOnly=true # Only sync changed resources
retry:
limit: 5
backoff:
duration: 10s
factor: 2
maxDuration: 5m
# ArgoCD AppProject — group apps and restrict permissions
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: production-apps
namespace: argocd
spec:
description: Production applications
# Which repos this project can use
sourceRepos:
- https://github.com/my-org/k8s-manifests.git
# Which clusters/namespaces this project can deploy to
destinations:
- namespace: production
server: https://kubernetes.default.svc
# Which K8s resources this project can create
clusterResourceWhitelist:
- group: ''
kind: Namespace
namespaceResourceWhitelist:
- group: apps
kind: Deployment
- group: ''
kind: Service
# Prevent deletion in production
orphanedResources:
warn: true
🎯 Scenario: After an image passes tests in staging, you want to automatically promote it to production without changing the manifest files.
Answer:
# CI/CD image promotion workflow
# .github/workflows/promote.yml
name: Promote to Production
on:
workflow_dispatch:
inputs:
image_tag:
description: 'Image tag to promote to production'
required: true
# Or trigger automatically when staging tests pass:
# workflow_run:
# workflows: ["Staging Tests"]
# types: [completed]
# branches: [main]
jobs:
promote:
runs-on: ubuntu-latest
if: github.event.workflow_run.conclusion == 'success'
steps:
- uses: actions/checkout@v4
- name: Install Kustomize
run: curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
- name: Update production image tag
run: |
cd overlays/production
kustomize edit set image myapp=myapp:${{ inputs.image_tag }}
- name: Create Pull Request
uses: peter-evans/create-pull-request@v5
with:
title: "Promote ${{ inputs.image_tag }} to production"
body: |
Promoting image tag `${{ inputs.image_tag }}` to production.
Tested and validated in staging.
branch: promote/${{ inputs.image_tag }}
base: main
labels: ["promotion", "production"]
# ArgoCD Image Updater — automatic image promotion based on tag policy
helm install argocd-image-updater \
argo/argocd-image-updater \
--namespace argocd
# ArgoCD Application with Image Updater annotations
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-app-staging
annotations:
argocd-image-updater.argoproj.io/image-list: "myapp=123456.dkr.ecr.us-east-1.amazonaws.com/myapp"
argocd-image-updater.argoproj.io/myapp.update-strategy: semver
argocd-image-updater.argoproj.io/myapp.allow-tags: "~1.x.x" # Only 1.x.x tags
argocd-image-updater.argoproj.io/write-back-method: git # Write tag to Git
🎯 Scenario: Your team manages both AWS infrastructure (Terraform) and Kubernetes workloads (Helm charts) in the same repository. How do you automate deployments safely?
Answer:
# .github/workflows/deploy.yml
name: Deploy to Production
on:
push:
branches: [main]
pull_request:
branches: [main]
permissions:
id-token: write # OIDC
contents: read
pull-requests: write
jobs:
# ─── TERRAFORM ──────────────────────────────────────────
terraform:
name: Terraform Plan/Apply
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./infrastructure
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/github-actions-role
aws-region: us-east-1
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.7.0
- run: terraform init
- run: terraform validate
- run: terraform fmt -check
- name: Terraform Plan
id: plan
run: terraform plan -out=tfplan -no-color 2>&1 | tee plan.txt
- name: Comment Plan on PR
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const plan = require('fs').readFileSync('./infrastructure/plan.txt','utf8')
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '## Terraform Plan\n```\n' + plan.slice(0,60000) + '\n```'
})
- name: Terraform Apply
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: terraform apply -auto-approve tfplan
# ─── HELM DEPLOY ────────────────────────────────────────
helm-deploy:
name: Deploy to Kubernetes
runs-on: ubuntu-latest
needs: terraform
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/github-actions-role
aws-region: us-east-1
- name: Update kubeconfig
run: aws eks update-kubeconfig --name my-cluster --region us-east-1
- name: Build and push image
run: |
aws ecr get-login-password | docker login --username AWS \
--password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com
docker build -t myapp:${{ github.sha }} .
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/myapp:${{ github.sha }}
- name: Helm upgrade
run: |
helm upgrade --install web-app ./charts/web-app \
--namespace production \
--set image.tag=${{ github.sha }} \
--set image.repository=123456789.dkr.ecr.us-east-1.amazonaws.com/myapp \
--values charts/web-app/values-prod.yaml \
--wait \
--timeout 10m \
--atomic # Auto-rollback if upgrade fails
- name: Verify deployment
run: |
kubectl rollout status deployment/web-app -n production
kubectl get pods -n production -l app=web-app
🎯 Scenario: You need to manage Helm deployments across dev, staging, and production with different values and encrypted secrets in each.
Answer:
Chart structure:
my-app/
├── Chart.yaml
├── values.yaml # Defaults
├── values-dev.yaml # Dev overrides
├── values-staging.yaml # Staging overrides
├── values-prod.yaml # Production overrides
└── templates/
├── deployment.yaml
├── service.yaml
├── ingress.yaml
└── hpa.yaml
# values.yaml (base defaults)
replicaCount: 1
image:
repository: myapp
tag: latest
pullPolicy: IfNotPresent
resources:
requests:
cpu: 100m
memory: 128Mi
autoscaling:
enabled: false
ingress:
enabled: false
# values-prod.yaml (production overrides)
replicaCount: 5
image:
repository: 123456789.dkr.ecr.us-east-1.amazonaws.com/myapp
tag: "v2.1.0"
pullPolicy: Always
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "2"
memory: 2Gi
autoscaling:
enabled: true
minReplicas: 5
maxReplicas: 20
targetCPUUtilizationPercentage: 70
ingress:
enabled: true
host: api.example.com
tlsEnabled: true
podDisruptionBudget:
enabled: true
minAvailable: 3
# Deploy to production
helm upgrade --install web-app ./my-app \
--namespace production \
--values values.yaml \
--values values-prod.yaml \
--set image.tag=$IMAGE_TAG \
--atomic \
--wait \
--timeout 10m
# Helm Secrets plugin — encrypt secret values with SOPS
helm secrets enc secrets-prod.yaml # Encrypt
helm secrets dec secrets-prod.yaml # Decrypt
helm upgrade --install web-app ./my-app \
--values values-prod.yaml \
--values secrets-prod.yaml # Auto-decrypted by helm-secrets plugin
# Helmfile — manage multiple Helm releases declaratively
helmfile sync # Apply all releases
helmfile diff # Show pending changes
helmfile apply --selector app=web-app # Apply specific release
🎯 Scenario: Your team wants to run Kafka in Kubernetes with automatic partition rebalancing, rolling upgrades, and self-healing. Should you write a StatefulSet or use an Operator?
Answer:
Operators encode human operational knowledge into code — they extend Kubernetes with domain-specific controllers.
# Install Strimzi Kafka Operator
helm repo add strimzi https://strimzi.io/charts
helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
--namespace kafka --create-namespace
# Kafka cluster managed by Strimzi Operator
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: production-kafka
namespace: kafka
spec:
kafka:
version: 3.6.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
default.replication.factor: 3
min.insync.replicas: 2
storage:
type: persistent-claim
size: 100Gi
class: fast-ssd
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
cpu: "2"
memory: 4Gi
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 10Gi
class: fast-ssd
entityOperator:
topicOperator: {}
userOperator: {}
# Strimzi manages Topic creation declaratively
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: orders-topic
namespace: kafka
labels:
strimzi.io/cluster: production-kafka
spec:
partitions: 12
replicas: 3
config:
retention.ms: 604800000 # 7 days
segment.bytes: 1073741824 # 1GB segments
💡 When to use an Operator: For complex stateful applications (Kafka, PostgreSQL, Elasticsearch, Redis) where the operational runbook has many steps. Check OperatorHub.io before writing your own.
🎯 Scenario: You’re building a SaaS platform where each customer gets an isolated environment on your Kubernetes cluster.
Answer:
Multi-tenancy approaches:
1. Namespace-per-tenant (soft isolation)
├── Pro: Simple, low overhead
├── Con: Shared kernel, shared cluster DNS
└── Use: Internal teams, trusted tenants
2. Cluster-per-tenant (hard isolation)
├── Pro: Complete isolation
├── Con: Expensive, high operational overhead
└── Use: Highly regulated industries, untrusted code
3. vCluster (virtual clusters)
├── Pro: Near-complete isolation, cheaper than full clusters
├── Con: Complexity of nested Kubernetes
└── Use: Best balance for SaaS multi-tenancy
# vCluster — virtual Kubernetes clusters inside namespaces
helm repo add loft-sh https://charts.loft.sh
helm install vcluster-tenant-a vcluster \
--repo https://charts.loft.sh \
--namespace tenant-a \
--create-namespace \
--values - <<EOF
sync:
ingresses:
enabled: true
storage:
size: 5Gi
isolation:
enabled: true
podSecurityStandard: baseline
resourceQuota:
enabled: true
quota:
requests.cpu: "10"
requests.memory: 20Gi
pods: "50"
EOF
# Connect to vCluster as tenant-a admin
vcluster connect vcluster-tenant-a -n tenant-a
kubectl get pods # Connected to tenant's virtual cluster
# HNC (Hierarchical Namespace Controller) — namespace trees
# parent namespace propagates RBAC and policies to children
apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchyConfiguration
metadata:
name: hierarchy
namespace: tenant-a-prod
spec:
parent: tenant-a # Inherits RBAC from tenant-a namespace
🎯 Scenario: Your AWS EKS bill is $60K/month. How do you reduce it by 30% without impacting production performance?
Answer:
# 1. Install Kubecost for cost visibility
helm repo add kubecost https://kubecost.github.io/cost-analyzer/
helm install kubecost kubecost/cost-analyzer \
--namespace kubecost --create-namespace \
--set global.prometheus.enabled=true
# Access Kubecost UI to see cost breakdown per namespace/workload
kubectl port-forward svc/kubecost-cost-analyzer 9090:9090 -n kubecost
# 2. Use Spot instances for non-critical workloads
# Node group with mixed instances (EKS Managed Node Group)
# 0% on-demand base, 100% Spot above base
# Multiple instance types for Spot diversity
# Deployment that tolerates spot interruption
spec:
template:
spec:
tolerations:
- key: "eks.amazonaws.com/capacityType"
operator: "Equal"
value: "SPOT"
effect: "NoSchedule"
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: eks.amazonaws.com/capacityType
operator: In
values: ["SPOT"]
# 3. Scale to zero dev/staging at night with KEDA
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: dev-scale-to-zero
namespace: development
spec:
scaleTargetRef:
name: dev-api
minReplicaCount: 0 # Scales to ZERO
maxReplicaCount: 5
triggers:
- type: cron
metadata:
timezone: America/New_York
start: "0 9 * * 1-5" # Up at 9 AM weekdays
end: "0 19 * * 1-5" # Down at 7 PM weekdays
desiredReplicas: "3"
# 4. Right-size with VPA recommendations
kubectl get vpa -A -o json | jq '.items[] | {
name: .metadata.name,
namespace: .metadata.namespace,
current: .spec.resourcePolicy,
recommended: .status.recommendation.containerRecommendations
}'
# 5. Remove unused resources
# Identify unused PVCs (no pod mounting them)
kubectl get pvc -A -o json | jq -r '
.items[] |
select(.status.phase == "Bound") |
select(.metadata.annotations["pv.kubernetes.io/bind-completed"] == "yes") |
.metadata.namespace + "/" + .metadata.name
'
# 6. Use Descheduler to rebalance pods after scale-down
helm install descheduler kubernetes-sigs/descheduler \
--namespace kube-system \
--set cronJobApiVersion=batch/v1 \
--set schedule="0 */2 * * *"
Cost saving levers:
| Action | Expected Savings |
|---|---|
| Right-size over-provisioned pods via VPA | 20–40% |
| Spot instances for dev/batch workloads | 60–80% on those nodes |
| Scale dev/staging to zero at night | 50–70% on those envs |
| Cluster Autoscaler aggressive scale-down | 15–25% |
| Spot + Cluster Autoscaler for production batch | 40–60% |
| Remove unused PVs and idle LoadBalancers | 5–10% |
🎯 Scenario: Your microservices need automatic mTLS, circuit breaking, retry logic, and distributed tracing without changing application code.
Answer:
# Install Istio
istioctl install --set profile=production
# Enable automatic sidecar injection for production namespace
kubectl label namespace production istio-injection=enabled
# All new pods in "production" now automatically get an Envoy proxy sidecar
kubectl rollout restart deployment -n production
# Traffic management — canary with header-based routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: web-app
namespace: production
spec:
hosts:
- web-app
http:
# Header-based routing for internal testers
- match:
- headers:
x-canary:
exact: "true"
route:
- destination:
host: web-app
subset: v2
# Weight-based traffic split (10% canary)
- route:
- destination:
host: web-app
subset: v1
weight: 90
- destination:
host: web-app
subset: v2
weight: 10
retries:
attempts: 3
perTryTimeout: 2s
retryOn: "gateway-error,connect-failure,retriable-4xx"
timeout: 10s
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: web-app
spec:
host: web-app
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
http1MaxPendingRequests: 100
http2MaxRequests: 1000
outlierDetection: # Circuit breaker
consecutive5xxErrors: 5
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 50
subsets:
- name: v1
labels:
version: v1.0
- name: v2
labels:
version: v2.0
🎯 Scenario: Your EKS cluster is on K8s 1.26 and needs to upgrade to 1.28. How do you do it with zero downtime?
Answer:
# Pre-upgrade checklist:
# 1. Check deprecated APIs being used
# Install pluto
helm install pluto --repo https://fairwinds.github.io/pluto pluto
kubectl pluto detect-helm # Check Helm releases
pluto detect-files -d ./k8s # Check manifest files
# Common API removals in recent versions:
# K8s 1.25: PodSecurityPolicy removed
# K8s 1.26: FlowSchema v1beta1 removed
# K8s 1.27: CSIStorageCapacity v1beta1 removed
# 2. Check addon compatibility (CoreDNS, kube-proxy, CSI drivers)
kubectl get pods -n kube-system -o wide
# 3. Backup etcd
ETCDCTL_API=3 etcdctl snapshot save /backup/pre-upgrade.db ...
# 4. Ensure PodDisruptionBudgets exist for critical workloads
kubectl get pdb -A
# 5. Test upgrade in staging first!
# EKS upgrade process:
# Step 1: Upgrade control plane (AWS manages this)
aws eks update-cluster-version \
--name my-cluster \
--kubernetes-version 1.28
# Wait for control plane upgrade
aws eks wait cluster-active --name my-cluster
# Step 2: Update managed addons
aws eks update-addon --cluster-name my-cluster --addon-name kube-proxy \
--addon-version v1.28.0-eksbuild.1
aws eks update-addon --cluster-name my-cluster --addon-name coredns \
--addon-version v1.10.1-eksbuild.1
aws eks update-addon --cluster-name my-cluster --addon-name aws-ebs-csi-driver \
--addon-version v1.25.0-eksbuild.1
# Step 3: Upgrade node groups (rolling replacement of nodes)
aws eks update-nodegroup-version \
--cluster-name my-cluster \
--nodegroup-name main \
--kubernetes-version 1.28
# Monitor node group update
aws eks wait nodegroup-active --cluster-name my-cluster --nodegroup-name main
# Verify cluster is healthy
kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed
🎯 Scenario: You want to test your system’s resilience by intentionally causing failures and verifying your application handles them gracefully.
Answer:
# Install Chaos Mesh
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
--namespace chaos-mesh \
--create-namespace \
--set chaosDaemon.runtime=containerd \
--set chaosDaemon.socketPath=/run/containerd/containerd.sock
# PodChaos — randomly kill pods (simulates node failures)
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: pod-failure-test
namespace: production
spec:
action: pod-kill # pod-kill / pod-failure / container-kill
mode: random-max-percent
value: "30" # Kill up to 30% of matching pods
selector:
namespaces:
- production
labelSelectors:
app: web-app
scheduler:
cron: "@every 10m" # Run every 10 minutes
# NetworkChaos — simulate network latency (test timeout handling)
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: network-delay-test
spec:
action: delay
mode: all
selector:
namespaces:
- production
labelSelectors:
app: api-server
delay:
latency: "200ms" # Add 200ms latency
correlation: "25"
jitter: "50ms"
direction: to # Affects outgoing traffic
duration: "5m"
# StressChaos — memory pressure test (verify OOMKill behavior)
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
name: memory-stress-test
spec:
mode: one
selector:
namespaces: [production]
labelSelectors:
app: api-server
stressors:
memory:
workers: 1
size: "512MB" # Allocate 512MB in the target container
duration: "1m"
🎯 Scenario: Your team wants a lightweight GitOps solution that automatically syncs Git to the cluster without a UI dependency.
Answer:
# Install Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash
# Bootstrap Flux — installs controllers + configures Git repo
flux bootstrap github \
--owner=my-org \
--repository=k8s-manifests \
--branch=main \
--path=clusters/production \
--personal
# GitRepository — tells Flux where to pull manifests from
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: k8s-manifests
namespace: flux-system
spec:
interval: 1m # Check for changes every minute
url: https://github.com/my-org/k8s-manifests
ref:
branch: main
secretRef:
name: github-token
# Kustomization — applies manifests from the GitRepository
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: production-apps
namespace: flux-system
spec:
interval: 5m
path: ./apps/production
prune: true # Delete resources removed from Git
sourceRef:
kind: GitRepository
name: k8s-manifests
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: web-app
namespace: production
timeout: 5m
# HelmRelease — manage Helm charts through Flux
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: web-app
namespace: production
spec:
interval: 10m
chart:
spec:
chart: web-app
version: ">=1.0.0 <2.0.0"
sourceRef:
kind: HelmRepository
name: my-charts
namespace: flux-system
values:
replicaCount: 5
image:
tag: v2.1.0
upgrade:
remediation:
retries: 3
rollback:
timeout: 5m
🎯 Scenario: You’re launching a new production Kubernetes cluster. What does your production-readiness checklist include?
Answer:
☐ CLUSTER SETUP
☐ Multi-master control plane (3+ masters for HA)
☐ etcd with 3+ nodes, automated backups to S3 every 15 min
☐ Worker nodes across 3+ availability zones
☐ Cluster Autoscaler configured for node scaling
☐ Managed K8s (EKS/GKE/AKS) preferred over self-managed
☐ NETWORKING
☐ CNI plugin with NetworkPolicy support (Calico/Cilium)
☐ NGINX or AWS Load Balancer Ingress controller
☐ cert-manager for automatic TLS certificate management
☐ CoreDNS with anti-affinity (spread across nodes)
☐ Default-deny NetworkPolicies per namespace
☐ SECURITY
☐ RBAC configured — least privilege for all service accounts
☐ Pod Security Standards enforced (restricted level for production)
☐ Secrets encryption at rest (EncryptionConfiguration or KMS)
☐ External Secrets Operator with AWS Secrets Manager
☐ Image vulnerability scanning in CI (Trivy, Grype)
☐ OPA/Gatekeeper policies (resource limits required, allowed registries)
☐ Audit logging enabled with 30-day retention
☐ No hostNetwork, hostPID, privileged containers in production
☐ Container images built FROM non-root base images
☐ WORKLOADS
☐ All Deployments have resource requests AND limits
☐ All Deployments have liveness + readiness probes
☐ All Deployments have PodDisruptionBudgets (minAvailable: 2+)
☐ maxUnavailable: 0 in rolling update strategy
☐ terminationGracePeriodSeconds >= 30 + preStop hook
☐ Pods spread across AZs with topologySpreadConstraints
☐ No bare pods (use Deployment/StatefulSet/DaemonSet)
☐ STORAGE
☐ StorageClass with WaitForFirstConsumer binding mode
☐ allowVolumeExpansion: true on StorageClasses
☐ reclaimPolicy: Retain for production databases
☐ Regular PVC snapshots / application-level backups
☐ Backup restore tested regularly
☐ OBSERVABILITY
☐ Prometheus + Alertmanager + Grafana (kube-prometheus-stack)
☐ Loki or EFK for centralized log aggregation
☐ Jaeger or Tempo for distributed tracing
☐ Alerts: PodCrashLooping, HighErrorRate, NodeNotReady, DiskPressure
☐ Grafana dashboards for SLIs (latency, error rate, saturation)
☐ SLO tracking and error budget monitoring
☐ CI/CD
☐ GitOps (ArgoCD/Flux) — all changes via Git
☐ Manifest validation in CI (kubeval, kubeconform, Checkov)
☐ Image scanning before push
☐ Helm or Kustomize for environment-specific configs
☐ Automated rollback on health check failure
☐ CAPACITY & COST
☐ VPA in recommendation mode — review monthly
☐ HPA on all customer-facing deployments
☐ Spot instances for dev/staging/batch
☐ Kubecost or AWS Cost Explorer for K8s cost allocation
☐ ResourceQuotas per namespace to prevent runaway costs
☐ Descheduler for pod bin-packing
📋 Quick Reference Cheatsheet
Pod Debugging
kubectl get pods -A -o wide # All pods + node/IP info
kubectl describe pod <pod> -n <ns> # Full details + events
kubectl logs <pod> --previous -c <container> # Pre-crash logs
kubectl logs -l app=web --all-containers --prefix -f # Tail multi-pod
kubectl exec -it <pod> -c <container> -- bash # Shell into container
kubectl debug -it <pod> --image=nicolaka/netshoot # Inject debug container
kubectl debug node/<node> -it --image=ubuntu:22.04 # Debug a node
kubectl port-forward pod/<pod> 8080:8080 # Local port forwarding
kubectl cp <pod>:/path/file ./local # Copy from pod
kubectl get events --sort-by='.lastTimestamp' -n <ns> # Recent events
kubectl get events --field-selector reason=Failed # Only failures
Deployments & Rollouts
kubectl apply -f manifest.yaml --dry-run=server # Preview apply
kubectl diff -f manifest.yaml # Show pending changes
kubectl rollout status deployment/<name> # Watch rollout
kubectl rollout history deployment/<name> # Revision history
kubectl rollout undo deployment/<name> # Rollback
kubectl rollout undo deployment/<name> --to-revision=3 # Specific revision
kubectl rollout restart deployment/<name> # Force restart all pods
kubectl scale deployment/<name> --replicas=5 # Manual scale
kubectl set image deployment/<name> app=myapp:v2.0 # Update image
kubectl autoscale deployment/<name> --min=2 --max=10 # Quick HPA
Nodes & Cluster
kubectl get nodes -o wide # Node status
kubectl describe node <node> # Node details + capacity
kubectl drain <node> --ignore-daemonsets \
--delete-emptydir-data # Drain for maintenance
kubectl cordon <node> # Stop scheduling
kubectl uncordon <node> # Resume scheduling
kubectl top nodes # CPU/memory usage
kubectl top pods --containers --sort-by=memory # Container metrics
kubectl get componentstatuses # Control plane health
kubectl cluster-info # Cluster endpoints
Resources & Config
kubectl get all -n <namespace> # All resources in ns
kubectl get pv,pvc -A # Storage overview
kubectl get networkpolicy -A # All network policies
kubectl get ingress -A # All ingresses
kubectl get hpa,vpa,scaledobject -A # All autoscalers
kubectl api-resources # All resource types
kubectl explain deployment.spec.strategy # Inline docs
kubectl get quota -A # Resource quotas
kubectl get limitrange -A # Limit ranges
Security & RBAC
kubectl auth can-i create pods -n prod --as=[email protected] # Check permissions
kubectl auth can-i --list -n prod --as=[email protected] # List all permissions
kubectl get rolebindings,clusterrolebindings -A # All RBAC bindings
kubectl get serviceaccounts -A # All service accounts
kubectl get secrets -A --field-selector type=Opaque # Opaque secrets
State & Troubleshooting
kubectl get pods --field-selector=status.phase=Pending # Pending pods
kubectl get pods --field-selector=status.phase=Failed # Failed pods
KUBECONFIG=~/.kube/config kubectl config get-contexts # List clusters
kubectl config use-context <context> # Switch cluster
kubectl config set-context --current --namespace=<ns> # Set default ns
kubectl delete pod <pod> --grace-period=0 --force # Force delete
TF_LOG=DEBUG kubectl apply -f file.yaml # Verbose kubectl
🎯 Interview Tips
| Topic | What Interviewers Want to Hear |
|---|---|
| Architecture | Control plane vs data plane, API server is the hub, etcd quorum |
| Scheduling | Filters → Scores → Binds; taints/tolerations; affinity |
| Probes | startup → liveness → readiness; consequences of each failing |
| Services | ClusterIP/NodePort/LoadBalancer; headless for StatefulSets |
| Networking | CNI; CoreDNS; NetworkPolicy requires compatible CNI |
| Storage | PV/PVC/StorageClass abstraction; WaitForFirstConsumer for multi-AZ |
| Security | RBAC least privilege; PSS; non-root; secrets encryption; IRSA |
| Scaling | HPA (stateless) + VPA (right-size) + KEDA (event-driven) + CA (nodes) |
| GitOps | ArgoCD/Flux; declarative; self-healing; drift detection |
| Resilience | PDBs; maxUnavailable=0; multi-AZ spread; graceful shutdown |
| Observability | Prometheus/Grafana metrics; Loki logs; Jaeger traces |
| Troubleshooting | describe → logs –previous → events → exec → network debug |
| Cost | Spot instances; scale-to-zero; right-sizing; Kubecost |
| Upgrades | Check deprecated APIs; PDBs first; control plane then nodes |
Good luck with your Kubernetes interviews! ☸️
Answer:
A CRD extends the Kubernetes API by defining new resource types. Once a CRD is registered, you can create instances of it using kubectl like any built-in resource.
# Define a CRD
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.mycompany.io
spec:
group: mycompany.io
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine:
type: string
enum: [postgres, mysql]
replicas:
type: integer
minimum: 1
scope: Namespaced
names:
plural: databases
singular: database
kind: Database
# Use the custom resource
apiVersion: mycompany.io/v1
kind: Database
metadata:
name: my-db
spec:
engine: postgres
replicas: 3
Answer:
A Service Mesh is a dedicated infrastructure layer that manages service-to-service communication (traffic management, observability, security) using sidecar proxies without changing application code.
Istio architecture:
- Data Plane: Envoy sidecar proxies (injected automatically into Pods) handle all traffic
- Control Plane (Istiod): Manages proxy configuration, certificate lifecycle, and traffic policies
Key Istio features:
- Traffic management: canary deployments, circuit breaking, retries, timeouts
- mTLS: automatic mutual TLS between services
- Observability: distributed tracing (Jaeger), metrics (Prometheus), logging
- Authorization policies: fine-grained L7 access control
# VirtualService — traffic splitting (canary)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-app
spec:
hosts:
- my-app
http:
- route:
- destination:
host: my-app
subset: v1
weight: 90
- destination:
host: my-app
subset: v2
weight: 10
Answer:
Kubernetes is not inherently multi-tenant but can be made so using multiple isolation mechanisms:
Soft multi-tenancy (shared cluster):
- Namespaces — logical isolation
- RBAC — access control per team/namespace
- ResourceQuotas — limit resource consumption per namespace
- LimitRanges — default/max resources per Pod in a namespace
- Network Policies — isolate network traffic between namespaces
- Pod Security Standards — enforce security contexts
Hard multi-tenancy (strong isolation):
- Separate clusters per tenant (VCluster, separate EKS clusters)
- VCluster — virtual Kubernetes clusters inside a namespace
- Capsule / HNC — multi-tenancy frameworks
# ResourceQuota per tenant namespace
apiVersion: v1
kind: ResourceQuota
metadata:
name: tenant-a-quota
namespace: tenant-a
spec:
hard:
requests.cpu: "10"
requests.memory: 20Gi
limits.cpu: "20"
limits.memory: 40Gi
pods: "50"
services: "10"
rk?
Answer:
Kubernetes garbage collection automatically removes objects that are no longer needed:
1. Owner References & Cascading Deletion:
- Resources have
ownerReferencespointing to their owner (e.g., Pod → ReplicaSet → Deployment) - When an owner is deleted, dependents are deleted via Foreground or Background cascading deletion
# Delete with cascade (default: background)
kubectl delete deployment my-app
# Orphan dependents (don't delete ReplicaSet/Pods)
kubectl delete deployment my-app --cascade=orphan
2. Image Garbage Collection:
kubeletremoves unused container images when disk usage exceedsimageGCHighThresholdPercent(default 85%)
3. Container Garbage Collection:
- Removes terminated containers based on
MaxContainerCountandMaxDeadContainerAge
4. API Resource GC:
- Removes completed Jobs, finished Pods (based on
ttlSecondsAfterFinished)
# Auto-delete Job after 60 seconds
spec:
ttlSecondsAfterFinished: 60
Answer:
GitOps is an operational framework where Git is the single source of truth for infrastructure and application configuration. Changes are made via Git commits/PRs, and automated agents reconcile the cluster state to match.
GitOps tools for Kubernetes:
- ArgoCD — declarative, Git-based continuous delivery
- Flux CD — lightweight, GitOps toolkit for Kubernetes
ArgoCD workflow:
- Developer commits Kubernetes manifests to Git
- ArgoCD detects the diff between Git and cluster state
- ArgoCD syncs the cluster (applies manifests)
- Health status is reported back
# ArgoCD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/my-org/my-app
targetRevision: main
path: k8s/
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
Answer:
Multi-cluster architectures improve availability, separate concerns, and meet compliance requirements.
Common patterns:
1. Active-Active (Global Load Balancing):
- Multiple EKS clusters in different regions
- Route 53 latency/geolocation routing between clusters
- Data synchronization via CRDTs or database replication
2. Active-Passive (Disaster Recovery):
- Primary cluster in one region, standby in another
- Velero for backup/restore
- Route 53 failover routing
3. Hub-Spoke (Management Cluster):
- Central management cluster running ArgoCD/Flux
- Spoke clusters receive workloads from the hub
Tools for multi-cluster:
- ArgoCD — multi-cluster GitOps
- Cluster API (CAPI) — manage cluster lifecycle
- AWS App Mesh — cross-cluster service mesh
- Velero — backup and DR
# Register multiple clusters in ArgoCD
argocd cluster add --kubeconfig ./cluster2-kubeconfig arn:aws:eks:us-west-2:123456789:cluster/cluster2
Answer:
Cost optimization strategies:
1. Right-size workloads:
- Use VPA recommendations to set appropriate resource requests
- Avoid over-provisioning CPU/memory
2. Spot Instances:
- Use Karpenter or CA with mixed instance types and Spot
- Design apps to handle interruptions gracefully (2-minute notice)
3. Node consolidation:
- Enable Karpenter’s
consolidationpolicy to bin-pack Pods
4. Fargate for variable workloads:
- Only pay for actual Pod CPU/memory
5. Cluster Autoscaler / Karpenter:
- Scale down idle nodes automatically
6. Savings Plans & Reserved Instances:
- Commit to 1 or 3 years for baseline workloads
# Karpenter — prefer Spot, fall back to On-Demand
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
disruption:
consolidationPolicy: WhenUnderutilized
consolidateAfter: 30s
# Monitor costs with Kubecost
helm install kubecost cost-analyzer \
--repo https://kubecost.github.io/cost-analyzer/ \
--namespace kubecost --create-namespace
Answer:
Blue/Green deployment runs two identical environments (blue = current, green = new) and switches traffic instantaneously.
Method 1: Kubernetes Services + Label Switching
# Switch traffic from blue to green by updating service selector
kubectl patch service my-service \
-p '{"spec":{"selector":{"version":"green"}}}'
Method 2: AWS ALB Weighted Target Groups
# Ingress with traffic splitting (AWS Load Balancer Controller)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.blue-green: |
{
"type": "forward",
"forwardConfig": {
"targetGroups": [
{"serviceName": "blue-service", "servicePort": 80, "weight": 0},
{"serviceName": "green-service", "servicePort": 80, "weight": 100}
]
}
}
Method 3: ArgoCD Rollouts (Argo Rollouts)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
strategy:
blueGreen:
activeService: my-active-service
previewService: my-preview-service
autoPromotionEnabled: false
Answer:
In EKS, etcd is fully managed by AWS. You do not have direct access to etcd. AWS automatically handles:
- etcd backups (multiple times per day)
- Multi-AZ replication for etcd
- Automatic etcd recovery
For application-level DR:
- Velero — backs up Kubernetes resources and PersistentVolumes to S3
# Install Velero with AWS S3 backend
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.8.0 \
--bucket my-velero-bucket \
--backup-location-config region=us-east-1 \
--snapshot-location-config region=us-east-1 \
--secret-file ./credentials-velero
# Backup all resources in a namespace
velero backup create my-backup --include-namespaces production
# Schedule daily backups
velero schedule create daily-backup \
--schedule="0 1 * * *" \
--include-namespaces production
# Restore from backup
velero restore create --from-backup my-backup
Answer:
Three pillars of observability: Metrics, Logs, Traces
Metrics:
- Amazon CloudWatch Container Insights — native AWS monitoring for EKS
- Prometheus + Grafana — open-source, highly flexible
- Datadog / New Relic — enterprise observability platforms
Logs:
- Fluent Bit (DaemonSet) → CloudWatch Logs / OpenSearch
- Fluentd — more plugins, slightly heavier
- EKS control plane logging: enable in AWS Console/CLI
Traces:
- AWS X-Ray — native AWS distributed tracing
- OpenTelemetry (ADOT) — standard collection pipeline
- Jaeger / Tempo — open-source tracing
# Enable EKS Control Plane logging
aws eks update-cluster-config \
--name my-cluster \
--logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
# Install Prometheus stack via Helm
helm install kube-prometheus-stack \
prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace \
--set grafana.adminPassword=admin123
# CloudWatch agent as a DaemonSet (Container Insights)
# Install via add-on
aws eks create-addon \
--cluster-name my-cluster \
--addon-name amazon-cloudwatch-observability
Answer:
EKS upgrade process (recommended steps):
Phase 1: Preparation
# 1. Review EKS release notes and deprecated APIs
# 2. Test upgrade in lower environments first
# 3. Backup with Velero
# Check current version
aws eks describe-cluster --name my-cluster --query cluster.version
# Check deprecated API usage
kubectl convert --help
# Use Pluto to detect deprecated APIs
pluto detect-all-in-cluster --target-versions k8s=v1.29.0
Phase 2: Upgrade the Control Plane
# Upgrade control plane (15-25 min, no downtime)
aws eks update-cluster-version \
--name my-cluster \
--kubernetes-version 1.29
# Wait for completion
aws eks wait cluster-active --name my-cluster
Phase 3: Upgrade Add-ons
# Update EKS add-ons (vpc-cni, coredns, kube-proxy)
aws eks update-addon \
--cluster-name my-cluster \
--addon-name vpc-cni \
--resolve-conflicts OVERWRITE
Phase 4: Upgrade Node Groups
# For Managed Node Groups
aws eks update-nodegroup-version \
--cluster-name my-cluster \
--nodegroup-name standard-nodes
# The process: new nodes → cordon old nodes → drain → terminate
# PodDisruptionBudgets are respected during drain
Phase 5: Validate
kubectl get nodes
kubectl get pods -A
kubectl get events -A | grep Warning
Key tip: Upgrade one minor version at a time (e.g., 1.27 → 1.28 → 1.29). Skipping versions is not supported.
Add More Questions to This Guide
Know a question that should be here? Share it and help the community!
Open Google Form