Kubernetes Interview Questions & Answers — Networking & Security
Kubernetes interview questions covering Services, Ingress, DNS, CNI, NetworkPolicies, RBAC, ServiceAccounts, Secrets, Pod Security, TLS, and admission controllers.
Ans: Pods come and go all the time - they crash, get replaced, move to different machines, and get a new IP address every single time that happens. A Service is Kubernetes’ fix for that: it’s one steady, unchanging address that always points to whichever Pods are healthy right now, so nothing else has to keep track of individual Pod IPs.
That’s the part every Service type has in common. The only thing that changes between the four types is who’s allowed to reach that address, and from where.
- ClusterIP - the default one. It only works inside the cluster, like an internal office phone extension - other Pods can dial it, but nobody outside the building can.
- Use it for: one Pod talking to another, like your backend calling your database. This is what you’ll use most of the time.
- NodePort - opens a specific door number (a port) on every machine in your cluster, so someone outside can walk straight up to any node and get in. It’s simple, but a bit rough around the edges.
- Use it for: quick testing or a demo, not something you’d want to rely on in production.
- LoadBalancer - asks your cloud provider (AWS, GCP, etc.) to set up a real, proper load balancer out front, with one clean public address that spreads traffic across your Pods.
- Use it for: real users hitting your app over the internet. This is the standard choice for production.
- ExternalName - doesn’t actually send traffic anywhere itself. It’s more like a forwarding note that says “hey, what you’re looking for isn’t in here, go to this outside address instead.”
- Use it for: letting your app talk to something outside the cluster (an external database, a third-party API) using a normal-looking Kubernetes Service name.
Ans:
A ConfigMap and a Secret in Kubernetes are used to store configuration data separately from application code so that Pods can use them without being hardcoded.
A ConfigMap is used to store non-sensitive configuration data like environment variables, configuration files, or command-line arguments. For example, you might store a database URL or application settings in a ConfigMap and inject it into a Pod as environment variables or mounted files.
A Secret is similar to a ConfigMap but is used for sensitive data like passwords, API keys, or tokens. Secrets are encoded (not fully encrypted by default) and are designed to reduce exposure of confidential information inside Kubernetes workloads.
To use them in a Pod, you can either pass them as environment variables or mount them as volumes. For example, a ConfigMap can be injected into a container using envFrom, and a Secret can be referenced using secretKeyRef so the application can securely access credentials at runtime without hardcoding them in the container image.
Ans:
| Type | Access | Use Case |
|---|---|---|
| ClusterIP | Only inside cluster | Internal service-to-service communication |
| NodePort | <NodeIP>:<30000-32767> from outside | Development/testing |
| LoadBalancer | Cloud LB public IP | Production external access |
| ExternalName | DNS CNAME alias | Point to external DNS |
# ClusterIP (default)
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 8080
# NodePort
spec:
type: NodePort
ports:
- port: 80
targetPort: 8080
nodePort: 30080 # Optional, auto-assigned if omitted
# LoadBalancer (creates cloud LB)
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
kubectl get service my-service
# EXTERNAL-IP column shows: <none> for ClusterIP, node IPs for NodePort, real IP for LB
Ans:
The default NodePort range is 30000–32767.
# Check configured range on API server
kube-apiserver --help | grep node-port-range
# or check /etc/kubernetes/manifests/kube-apiserver.yaml
grep -i nodeport /etc/kubernetes/manifests/kube-apiserver.yaml
# Custom range (set in kube-apiserver startup args):
--service-node-port-range=30000-32767
If you try to assign a port outside this range, the API server will reject it.
Ans:
Secrets — Kubernetes objects that store sensitive data like passwords, API tokens, TLS certificates.
Types of secrets:
Opaque— Generic key-value data (most common)kubernetes.io/service-account-token— Service account JWT tokenskubernetes.io/tls— TLS certificate and keykubernetes.io/dockerconfigjson— Docker registry credentials
# List secrets
kubectl get secrets
# Create opaque secret
kubectl create secret generic my-secret \
--from-literal=token=abc123
# Create docker registry secret
kubectl create secret docker-registry regcred \
--docker-server=docker.io \
--docker-username=myuser \
--docker-password=mypass
# View secret (base64 encoded)
kubectl get secret my-secret -o jsonpath='{.data.token}' | base64 -d
Note: K8s Secrets are base64-encoded, NOT encrypted by default. Use
etcdencryption at rest + Vault/AWS Secrets Manager for production.
Answer:
A Service is an abstraction that defines a logical set of Pods and a policy to access them. Since Pods are ephemeral and their IPs change, a Service provides a stable IP address and DNS name to access them.
How it works:
- Services use label selectors to find matching Pods
kube-proxymaintains network rules to route traffic- Each Service gets a ClusterIP and a DNS entry (e.g.,
my-service.default.svc.cluster.local)
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: my-app
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: ClusterIP
Answer:
| Type | Description | Use Case |
|---|---|---|
| ClusterIP | Default; exposes Service on internal cluster IP | Internal microservice communication |
| NodePort | Exposes Service on each Node’s IP at a static port (30000-32767) | External access in development |
| LoadBalancer | Exposes Service externally using a cloud load balancer | Production external access |
| ExternalName | Maps Service to a DNS name (e.g., external DB) | Connecting to external services |
# LoadBalancer Service Example (EKS)
apiVersion: v1
kind: Service
metadata:
name: my-lb-service
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
selector:
app: my-app
ports:
- port: 80
targetPort: 8080
type: LoadBalancer
Answer:
A ConfigMap stores non-confidential configuration data as key-value pairs, decoupling configuration from container images. This allows you to change application behavior without rebuilding images.
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
APP_ENV: "production"
APP_PORT: "8080"
config.json: |
{
"logLevel": "info",
"retries": 3
}
Using ConfigMap in a Pod:
spec:
containers:
- name: app
image: my-app
envFrom:
- configMapRef:
name: app-config
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: app-config
Answer:
A Secret stores sensitive data such as passwords, tokens, and SSH keys. Data is stored base64-encoded (not encrypted by default, but can be encrypted at rest with KMS).
# Create a secret from literal values
kubectl create secret generic db-secret \
--from-literal=username=admin \
--from-literal=password=s3cr3t
apiVersion: v1
kind: Secret
metadata:
name: db-secret
type: Opaque
data:
username: YWRtaW4= # base64("admin")
password: czNjcjN0 # base64("s3cr3t")
# Using secrets as environment variables
spec:
containers:
- name: app
image: my-app
env:
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: db-secret
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
Answer:
EKS uses AWS IAM for authentication and Kubernetes RBAC for authorization.
Authentication flow:
kubectlcalls the AWS CLI/SDK to get a pre-signed token via STS- The token is passed to the Kubernetes API server
- The EKS cluster verifies the token with AWS IAM
- Kubernetes RBAC is checked for authorization
# Configure kubectl for EKS
aws eks update-kubeconfig \
--name my-cluster \
--region us-east-1 \
--role-arn arn:aws:iam::123456789:role/eks-admin-role
# Verify access
kubectl auth can-i get pods
# Check kubeconfig
kubectl config view
Ans:
Ingress:
- An Ingress in Kubernetes is an API object that manages external HTTP/HTTPS access to services inside a cluster, usually through a single entry point and rules like routing based on hostnames or URL paths.
- Provides smart routing for HTTP/HTTPS traffic, such as:
- Routing based on domain (example.com/api → backend service)
- Routing based on path (/login → auth service)
Service:
- A Service, on the other hand, is used to expose a set of Pods either internally or externally, but it typically works at a simpler level like exposing a fixed IP or port without advanced routing rules.
- Exposes applications using a stable IP/port (ClusterIP, NodePort, LoadBalancer) and forwards traffic to Pods.
Kubernetes networking follows 3 fundamental rules:
- Every pod gets its own IP address
- Pods can communicate with all other pods without NAT
- Nodes can communicate with all pods without NAT
CNI (Container Network Interface) plugins implement these rules:
| CNI Plugin | Use Case | Features |
|---|---|---|
| Calico | Most popular | NetworkPolicy, BGP, eBPF |
| Flannel | Simple, lightweight | Basic overlay network |
| Cilium | High performance | eBPF, L7 policies, observability |
| Weave | Easy setup | Encrypted by default |
| AWS VPC CNI | EKS native | Pods get real VPC IPs |
Pod-to-Pod communication:
Same node: Pod A → veth → cbr0 bridge → veth → Pod B
Across nodes: Pod A → veth → cbr0 → eth0 → [overlay/BGP] → eth0 → cbr0 → Pod B
DNS resolution in the cluster:
# Pod DNS format
<pod-ip-dashes>.<namespace>.pod.cluster.local
# Example: 10-0-0-1.default.pod.cluster.local
# Service DNS format
<service-name>.<namespace>.svc.cluster.local
# Example: my-svc.production.svc.cluster.local
# Test DNS from inside a pod
kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup my-svc.production
NetworkPolicy — restrict traffic between pods:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backend-netpol
namespace: production
spec:
podSelector:
matchLabels:
app: backend # Apply to backend pods
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend # Only allow traffic from frontend pods
ports:
- protocol: TCP
port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: postgres # Backend can only talk to postgres
ports:
- protocol: TCP
port: 5432
RBAC (Role-Based Access Control) controls who can do what in Kubernetes. It uses 4 objects:
| Object | Scope | Purpose |
|---|---|---|
Role | Namespace | Defines permissions within a namespace |
ClusterRole | Cluster-wide | Defines cluster-wide permissions |
RoleBinding | Namespace | Binds Role/ClusterRole to users/groups/SAs in a namespace |
ClusterRoleBinding | Cluster-wide | Binds ClusterRole cluster-wide |
Complete read-only setup for a developer:
# 1. Role — read-only in 'development' namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: developer-readonly
namespace: development
rules:
- apiGroups: [""] # Core API group
resources: ["pods", "pods/log", "services", "configmaps", "endpoints"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["get", "list", "watch"]
---
# 2. RoleBinding — attach role to a user
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: developer-readonly-binding
namespace: development
subjects:
- kind: User
name: [email protected] # IAM user or OIDC user
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: developer-readonly
apiGroup: rbac.authorization.k8s.io
Test permissions:
# Check what a user can do
kubectl auth can-i list pods -n development --as=[email protected]
# yes
kubectl auth can-i delete pods -n development --as=[email protected]
# no
# List all permissions for a user
kubectl auth can-i --list -n development --as=[email protected]
A ServiceAccount provides an identity for processes running inside a Pod to interact with the Kubernetes API. Every pod automatically gets the default ServiceAccount if not specified.
Why use custom ServiceAccounts?
- Grant specific pods only the permissions they need (least privilege)
- Use with IRSA (IAM Roles for Service Accounts) on EKS for AWS access
- Audit trail — know which pod made which API call
Create a ServiceAccount with RBAC:
# 1. Create ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: deployment-manager
namespace: production
---
# 2. Create Role with needed permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deployment-role
namespace: production
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "update", "patch"]
---
# 3. Bind the Role to the ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: deployment-role-binding
namespace: production
subjects:
- kind: ServiceAccount
name: deployment-manager
namespace: production
roleRef:
kind: Role
name: deployment-role
apiGroup: rbac.authorization.k8s.io
---
# 4. Use ServiceAccount in Pod
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
serviceAccountName: deployment-manager # Attach the SA here
automountServiceAccountToken: true
containers:
- name: app
image: my-app:v1
EKS IRSA — give pods AWS IAM permissions:
# Associate ServiceAccount with IAM role (no access keys needed in pods!)
eksctl create iamserviceaccount \
--cluster my-cluster \
--namespace production \
--name s3-access-sa \
--attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
--approve
By default, Kubernetes Secrets are stored base64-encoded (NOT encrypted) in etcd. This means anyone with etcd access can read them.
Best practices for Kubernetes Secret management:
1. Enable Encryption at Rest:
# /etc/kubernetes/encryption-config.yaml (on API server)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {} # Fallback for unencrypted data
2. Use External Secret Managers (recommended for production):
# External Secrets Operator — syncs AWS Secrets Manager → K8s Secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secretsmanager
kind: ClusterSecretStore
target:
name: db-secret # Creates this K8s Secret
data:
- secretKey: DB_PASSWORD
remoteRef:
key: prod/myapp/database # AWS Secrets Manager path
property: password
3. RBAC to limit Secret access:
# Only allow specific service accounts to read secrets
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["db-secret", "api-key"] # Specific secrets only
verbs: ["get"]
4. Never do these:
# ❌ Never put secrets in ConfigMaps
# ❌ Never commit secrets to Git
# ❌ Never echo secrets in logs
# ❌ Never use environment variables for very sensitive data
# (they can appear in process lists)
# ✅ Mount secrets as files instead
volumeMounts:
- name: db-credentials
mountPath: /etc/secrets
readOnly: true
volumes:
- name: db-credentials
secret:
secretName: db-secret
defaultMode: 0400 # Read-only for owner only
# Audit who accessed a secret
kubectl get events | grep secret
# Enable audit logs in kube-apiserver for full audit trail
Ans:
Ingress is a Kubernetes resource that manages external HTTP/HTTPS access to services, providing routing, SSL termination, and name-based virtual hosting.
Internet → Ingress Controller (nginx/traefik) → Ingress Rules → Services → Pods
Ingress vs Service:
Service type: LoadBalancer= one cloud LB per service (expensive)Ingress= one LB for all services with path/host routing
# Ingress resource example
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: myapp.example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: frontend-service
port:
number: 80
tls:
- hosts:
- myapp.example.com
secretName: tls-secret
# Install nginx ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml
# Check ingress
kubectl get ingress
kubectl describe ingress app-ingress
Ans:
# Create secret
kubectl create secret generic db-secret \
--from-literal=username=admin \
--from-literal=password=mysecret
# Create from file
kubectl create secret generic tls-certs \
--from-file=tls.crt=./cert.crt \
--from-file=tls.key=./cert.key
# View (base64 encoded)
kubectl get secret db-secret -o yaml
echo "bXlzZWNyZXQ=" | base64 -d # decode
# Use in pod - as environment variable
spec:
containers:
- name: app
envFrom:
- secretRef:
name: db-secret
# Use as individual env var
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
# Mount as file (more secure - not in process env)
volumeMounts:
- name: secret-vol
mountPath: /etc/secrets
readOnly: true
volumes:
- name: secret-vol
secret:
secretName: db-secret
Production best practice: Use External Secrets Operator to sync from AWS Secrets Manager / Vault into K8s secrets.
🎯 Scenario: Your app needs both non-sensitive configuration (feature flags, timeouts) and sensitive data (DB credentials, API keys). How do you manage both?
Answer:
# ConfigMap — non-sensitive configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: production
data:
LOG_LEVEL: "info"
MAX_CONNECTIONS: "100"
CACHE_TTL: "300"
FEATURE_NEW_UI: "true"
# Multi-line config file
app.yaml: |
server:
port: 8080
timeout: 30s
database:
pool_size: 20
max_idle: 5
nginx.conf: |
upstream backend { server 127.0.0.1:8080; }
server {
listen 80;
location / { proxy_pass http://backend; }
}
# Secret — sensitive data (base64-encoded, but NOT encrypted by default)
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
namespace: production
type: Opaque
stringData: # stringData auto-base64-encodes
DB_PASSWORD: "super-secret-password"
JWT_SIGNING_KEY: "very-long-random-secret-key"
tls.crt: |
-----BEGIN CERTIFICATE-----
MIID...
-----END CERTIFICATE-----
# Use in a Deployment
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: app
image: myapp:v2.0
# Inject all ConfigMap keys as env vars
envFrom:
- configMapRef:
name: app-config
# Inject all Secret keys as env vars
- secretRef:
name: app-secrets
# Or inject individual keys
env:
- name: DB_URL
valueFrom:
secretKeyRef:
name: app-secrets
key: DB_PASSWORD
# Mount ConfigMap as config files
volumeMounts:
- name: app-config-vol
mountPath: /etc/app
readOnly: true
- name: secrets-vol
mountPath: /etc/secrets
readOnly: true
volumes:
- name: app-config-vol
configMap:
name: app-config
items: # Mount only specific keys
- key: app.yaml
path: config.yaml # Filename in container
- name: secrets-vol
secret:
secretName: app-secrets
defaultMode: 0400 # Owner read-only
⚠️ Kubernetes Secrets are only base64-encoded, NOT encrypted at rest! Anyone with RBAC access to read secrets can decode them. For production: enable EncryptionConfiguration on etcd AND use External Secrets Operator with AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault.
🎯 Scenario: A developer needs to view pods, logs, and deployments in the staging namespace but must not modify anything.
Answer:
# Role — namespace-scoped permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: developer-readonly
namespace: staging
rules:
# View pods and their logs
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/status"]
verbs: ["get", "list", "watch"]
# View services, configmaps, events
- apiGroups: [""]
resources: ["services", "endpoints", "configmaps",
"events", "persistentvolumeclaims", "replicationcontrollers"]
verbs: ["get", "list", "watch"]
# View deployments, replicasets
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets",
"daemonsets", "replicationcontrollers"]
verbs: ["get", "list", "watch"]
# View jobs and cronjobs
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["get", "list", "watch"]
# View HPA
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list", "watch"]
# Explicitly NO exec, portforward, or secret access
# RoleBinding — assigns Role to user/group
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: developer-readonly-binding
namespace: staging
subjects:
- kind: User
name: [email protected]
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: dev-team # All members of this group
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: developer-readonly
apiGroup: rbac.authorization.k8s.io
# Verify permissions
kubectl auth can-i list pods \
--namespace=staging --as=[email protected] # → yes
kubectl auth can-i delete deployments \
--namespace=staging --as=[email protected] # → no
kubectl auth can-i exec pods \
--namespace=staging --as=[email protected] # → no
# List all permissions for a user
kubectl auth can-i --list --namespace=staging \
--as=[email protected]
Answer:
Role-Based Access Control (RBAC) regulates access to Kubernetes resources based on the roles of users or service accounts.
Key objects:
| Object | Scope | Purpose |
|---|---|---|
Role | Namespace | Grants permissions within a namespace |
ClusterRole | Cluster-wide | Grants permissions across all namespaces |
RoleBinding | Namespace | Binds Role to user/group/service account |
ClusterRoleBinding | Cluster-wide | Binds ClusterRole cluster-wide |
# Role — allows reading pods in default namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods-binding
namespace: default
subjects:
- kind: ServiceAccount
name: my-service-account
namespace: default
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Answer:
An Ingress resource defines HTTP/HTTPS routing rules to Services. An Ingress Controller implements those rules (e.g., NGINX, Traefik, AWS ALB Ingress Controller).
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
tls:
- hosts:
- myapp.example.com
secretName: tls-secret
rules:
- host: myapp.example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: frontend-service
port:
number: 80
In EKS, the AWS Load Balancer Controller creates ALBs automatically from Ingress resources using annotations.
Answer:
| Feature | Role | ClusterRole |
|---|---|---|
| Scope | Single namespace | Cluster-wide |
| Use case | Namespace-scoped resources | Cluster-scoped resources (Nodes, PVs) or all namespaces |
| Bound by | RoleBinding | ClusterRoleBinding (or RoleBinding for namespace scope) |
A ClusterRole can be bound in two ways:
- ClusterRoleBinding → grants access across all namespaces
- RoleBinding → grants ClusterRole access within a specific namespace only
This is useful when you want to reuse a ClusterRole definition across multiple namespaces.
Answer:
Kubernetes uses CoreDNS as the cluster DNS server. Every Pod gets a /etc/resolv.conf pointing to the CoreDNS service IP. Services and Pods are accessible via DNS.
DNS naming format:
# Service
<service-name>.<namespace>.svc.cluster.local
# Pod
<pod-ip-dashes>.<namespace>.pod.cluster.local
# Example: 10-244-1-5.default.pod.cluster.local
# Headless service Pods (StatefulSet)
<pod-name>.<service-name>.<namespace>.svc.cluster.local
# Example: mysql-0.mysql.default.svc.cluster.local
# Test DNS from inside a Pod
kubectl exec -it my-pod -- nslookup kubernetes.default
kubectl exec -it my-pod -- curl http://my-service.my-namespace.svc.cluster.local
Answer:
EKS uses a webhook token authenticator that verifies AWS IAM identities:
kubectlrequests a pre-signed STS token viaaws eks get-token- The token is sent to the Kubernetes API server
- The API server passes it to the AWS IAM Authenticator webhook
- The webhook calls STS to validate the token and returns the IAM identity
- Kubernetes maps the IAM identity to a Kubernetes RBAC user/group via the
aws-authConfigMap (or EKS Access Entries)
# Get token manually (for debugging)
aws eks get-token --cluster-name my-cluster
# Check what identity kubectl uses
kubectl auth whoami
aws sts get-caller-identity
Answer:
The aws-auth ConfigMap in the kube-system namespace maps AWS IAM principals (users, roles) to Kubernetes RBAC users and groups. This controls who can access the cluster and with what permissions.
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam::123456789:role/eks-node-group-role
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam::123456789:role/eks-admin-role
username: admin
groups:
- system:masters
mapUsers: |
- userarn: arn:aws:iam::123456789:user/john
username: john
groups:
- developers
Note: AWS now recommends EKS Access Entries (API-based) as the preferred alternative to the aws-auth ConfigMap.
Answer:
The Amazon VPC CNI (Container Network Interface) plugin is the default networking plugin for EKS. It assigns real VPC IP addresses to Pods from the node’s subnet, enabling direct communication between Pods and other AWS resources.
Key features:
- Each Pod gets a real VPC IP address (not a virtual overlay network)
- Pods can communicate directly with RDS, ElastiCache, and other AWS services
- Security Groups can be applied directly to Pods (
SecurityGroupPolicy) - Supports IPv4 and IPv6
# Check VPC CNI version
kubectl describe daemonset aws-node -n kube-system | grep Image
# Check IP address allocation per node
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,MAX_PODS:.status.capacity.pods'
IP address calculation:
Each EC2 instance type has a limit on ENIs and IPs per ENI. Max Pods = (ENIs × (IPs per ENI - 1)) + 2
Answer:
The AWS Load Balancer Controller is a controller that manages AWS Elastic Load Balancers for Kubernetes clusters. It provisions:
- Application Load Balancers (ALBs) for Ingress resources
- Network Load Balancers (NLBs) for Service type LoadBalancer
# ALB Ingress via AWS Load Balancer Controller
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
alb.ingress.kubernetes.io/ssl-redirect: "443"
spec:
rules:
- host: myapp.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service
port:
number: 80
# Install AWS Load Balancer Controller via Helm
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=my-cluster \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller
Answer:
Options for secret management in EKS:
- Kubernetes Secrets (base64 encoded; encrypt with AWS KMS for security)
- AWS Secrets Manager + Secrets Store CSI Driver (mount secrets as volumes)
- AWS Systems Manager Parameter Store (same CSI driver)
- External Secrets Operator (sync external secrets to Kubernetes Secrets)
# Using Secrets Store CSI Driver with AWS Secrets Manager
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-secrets
spec:
provider: aws
parameters:
objects: |
- objectName: "prod/myapp/db-password"
objectType: secretsmanager
objectAlias: db-password
---
spec:
containers:
- name: app
volumeMounts:
- name: secrets
mountPath: /mnt/secrets
readOnly: true
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aws-secrets
# Enable EKS secrets encryption with KMS
aws eks create-cluster \
--name my-cluster \
--encryption-config "resources=[secrets],provider={keyArn=arn:aws:kms:...}"
kubectl expose deployment main-app --type=LoadBalancer --port=80 --target-port=8080 so it gets a cloud load balancer with a public IP.
2) Expose the database with kubectl expose deployment database --type=ClusterIP --port=5432 --target-port=5432 so it only gets an internal cluster IP and DNS name.
3) The main app connects to the database using the internal service name,
e.g. database:5432, which Kubernetes resolves via its internal DNS.
This separation keeps the database private and secure while making only the necessary application public-facing.Answer:
ClusterIP is the default Kubernetes Service type — it gives a Service a stable, virtual IP address that’s only reachable from inside the cluster, never from outside.
apiVersion: v1
kind: Service
metadata:
name: backend-svc
spec:
selector:
app: backend
ports:
- port: 80 # port the Service listens on
targetPort: 8080 # port the Pod's container actually listens on
type: ClusterIP # default — can be omitted
How traffic actually reaches a Pod:
kube-proxyon every node watches the API server for Services/Endpoints and programs local iptables (or IPVS) rules- When another Pod sends traffic to the ClusterIP, those rules intercept it and load-balance it across the Service’s healthy backend Pods (matched via the
selectorlabels) - The ClusterIP itself is virtual — it doesn’t exist on any real network interface, it’s purely an iptables/IPVS forwarding rule
DNS: CoreDNS automatically creates a record so other Pods can reach it by name instead of IP:
backend-svc.default.svc.cluster.local → resolves to the ClusterIP
Headless variant (clusterIP: None): skips the virtual IP and load-balancing entirely — DNS returns the individual Pod IPs directly instead. This is what StatefulSets use, since each Pod needs its own stable, addressable identity (e.g., postgres-0.postgres.default.svc.cluster.local) rather than being load-balanced as an interchangeable group.
When to use it: internal-only communication — a backend API that only the frontend Service should call, or a database that should never be reachable from outside the cluster. For external access you’d put a LoadBalancer or Ingress in front of it instead.
Kubernetes security is a defence-in-depth approach with multiple layers:
Layer 1 — API Server security:
# Restrict anonymous access
--anonymous-auth=false
# Enable audit logging
--audit-log-path=/var/log/kubernetes/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
# Disable insecure port
--insecure-port=0
Layer 2 — RBAC (least privilege):
# Never use cluster-admin in applications
# Create minimal roles per service
# Audit RBAC permissions
kubectl auth can-i --list --as=system:serviceaccount:production:my-sa
Layer 3 — Network Policies (zero-trust networking):
# Deny all traffic by default, then allow explicitly
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: production
spec:
podSelector: {} # Applies to ALL pods in namespace
policyTypes:
- Ingress
- Egress
# No rules = deny all
Layer 4 — Pod Security (Security Context):
spec:
securityContext:
runAsNonRoot: true # Never run as root
runAsUser: 1000
fsGroup: 2000
seccompProfile:
type: RuntimeDefault # Enable seccomp filtering
containers:
- name: app
image: my-app:v1
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true # Container cannot write to filesystem
capabilities:
drop:
- ALL # Drop all Linux capabilities
add:
- NET_BIND_SERVICE # Add only what you need
Layer 5 — Image Security:
# Scan images before pushing
trivy image my-app:v1
grype my-app:v1
# Use Image Policy Webhook to block vulnerable images
# Use private registry — never use :latest tag in production
Layer 6 — Secrets Management:
# Enable encryption at rest for etcd
# Use External Secrets Operator with AWS Secrets Manager / Vault
# Rotate secrets regularly
Layer 7 — Runtime Security:
# Use Falco for runtime threat detection
helm install falco falcosecurity/falco \
--namespace falco-system \
--create-namespace
# Falco detects: shell in containers, privilege escalation,
# unexpected network connections, file system changes
Pod Security Admission (replaces deprecated PodSecurityPolicy):
# Label namespace to enforce security standards
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted # Most strict
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
Ans:
Using CSI Secrets Store Driver + Azure Key Vault Provider:
# Step 1: Install CSI driver
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
# Step 2: Install Azure Key Vault provider
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install azure-csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure
# Step 3: Create SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-keyvault-secrets
spec:
provider: azure
parameters:
vaultName: my-keyvault
clientID: <managed-identity-client-id>
tenantID: <azure-tenant-id>
objects: |
array:
- |
objectName: db-password
objectType: secret
# Step 4: Mount in pod
spec:
containers:
- name: app
volumeMounts:
- name: secrets-store
mountPath: /mnt/secrets-store
readOnly: true
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: azure-keyvault-secrets
Ans:
# Default deny all ingress in 'production' namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: production
spec:
podSelector: {} # Applies to all pods
policyTypes:
- Ingress
- Egress
---
# Allow specific namespace to reach production
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-staging
namespace: production
spec:
podSelector:
matchLabels:
app: myapp
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: staging
ports:
- port: 8080
# Apply the policy
kubectl apply -f netpol.yaml
# Test: pod in staging can reach production
kubectl exec -it <staging-pod> -- curl http://myapp.production:8080
# Test: pod in default namespace cannot reach production
kubectl exec -it <default-pod> -- curl http://myapp.production:8080
# Should fail/timeout
Note: NetworkPolicy requires a CNI plugin that supports it (Calico, Cilium, Weave). Flannel alone does NOT enforce NetworkPolicies.
Answer:
The security problem:
Worker nodes in public subnets have public IP addresses assigned to them. This means:
- Each worker node EC2 instance is directly reachable from the internet
- If a container escapes (container breakout exploit), the underlying node is exposed to the internet
- Kubernetes NodePort services on those nodes are potentially reachable from anywhere
- The attack surface is dramatically larger than necessary
The correct architecture — private worker nodes:
Internet
|
v
Internet Gateway
|
v
Public Subnets (AZ-a, AZ-b, AZ-c)
[ALB / NLB Load Balancers only]
|
v
Private Subnets (AZ-a, AZ-b, AZ-c)
[All EKS Worker Nodes]
[NAT Gateway for outbound internet (to pull images)]
Worker nodes should be in private subnets. They have no public IPs. Inbound traffic reaches them only through the load balancer. Outbound traffic (to pull container images from ECR, call AWS APIs) goes through a NAT Gateway.
How to migrate existing nodes:
You cannot move existing nodes between subnets. The fix is to create a new node group in private subnets and migrate workloads:
# Create new node group in private subnets
eksctl create nodegroup \
--cluster production-cluster \
--name private-workers \
--node-private-networking \
--nodes 3 \
--nodes-min 2 \
--nodes-max 10
# Taint the old public nodes to prevent new pods being scheduled there
kubectl taint nodes <public-node-name> dedicated=old:NoSchedule
# Drain each old node (moves pods to new private nodes)
kubectl drain <public-node-name> --ignore-daemonsets --delete-emptydir-data
# Delete old node group after pods are migrated
eksctl delete nodegroup --cluster production-cluster --name public-workers
EKS API endpoint access:
Also configure the EKS cluster API endpoint to be private-only for maximum security:
EKS Console → Cluster → Networking → Endpoint access → Private
This means kubectl commands only work from within the VPC (via VPN or bastion host), not from the public internet.
Answer:
The VPC CNI plugin — EKS’s networking foundation:
EKS uses the Amazon VPC CNI (Container Network Interface) plugin. This is fundamentally different from how most Kubernetes CNI plugins work.
Most CNI plugins (Flannel, Calico overlay mode): Create a virtual overlay network. Pods get IPs from a virtual CIDR that is separate from the VPC CIDR. Traffic between pods is encapsulated in packets (VXLAN tunnels) and sent across the real network. The VPC network sees EC2 instance IPs only — pod IPs are “hidden” inside tunnels.
Amazon VPC CNI: Every pod gets a REAL VPC IP address. Not a virtual IP — an actual IP from your VPC subnet CIDR. The pod’s IP is routable anywhere in the VPC (and by extension, in any VPC that is peered or connected via Transit Gateway).
How the VPC CNI allocates IPs:
Each EC2 worker node has multiple Elastic Network Interfaces (ENIs). Each ENI can have multiple private IP addresses. The VPC CNI pre-allocates a pool of IP addresses from ENIs and assigns one to each pod when it starts.
Worker Node (t3.large):
ENI 1: 10.0.1.5 (node's primary IP)
10.0.1.10 (assigned to pod-1)
10.0.1.11 (assigned to pod-2)
10.0.1.12 (assigned to pod-3)
ENI 2: 10.0.1.20 (assigned to pod-4)
10.0.1.21 (assigned to pod-5)
Why pod-to-pod pinging works directly:
Because pod-2 (IP 10.0.1.11) and pod-4 (IP 10.0.1.20) are both real VPC IPs, the VPC routing tables already know how to route between them — no tunnel or overlay needed. It’s just VPC routing.
Real-world implication:
Your VPC CIDR must be large enough to accommodate all your pod IPs. If you have 100 nodes each running 30 pods, you need at least 3,000 IPs. A /24 subnet only has 256 addresses. Plan your VPC CIDR with pod density in mind — a /16 VPC with multiple /20 subnets per AZ is a common production choice.
Answer:
ClusterIP (default):
Creates a virtual IP address that is only reachable from within the cluster. Pods in the cluster can reach the service at this stable IP, even as the underlying pods change.
Use when: Service-to-service communication within the cluster. Your frontend pods talking to your backend API pods. The backend API should never be exposed externally — ClusterIP is the right choice.
apiVersion: v1
kind: Service
metadata:
name: backend-api
spec:
type: ClusterIP
selector:
app: backend-api
ports:
- port: 8080
targetPort: 8080
NodePort:
Opens a port (30000–32767) on EVERY worker node. Traffic hitting <any-node-IP>:<nodeport> is forwarded to the pods. Technically reachable from outside the cluster but requires knowing node IPs.
Use when: Development and testing only. Never for production — it requires knowing node IPs, bypasses the load balancer, and exposes a port on every node unnecessarily.
LoadBalancer:
Creates an AWS Load Balancer (NLB or CLB by default) and points it at the pods. One LoadBalancer Service = one AWS Load Balancer = one AWS cost item.
Use when: You need to expose a single TCP/UDP service directly. Good for non-HTTP services like a game server or database proxy. For HTTP services, Ingress is better because one Ingress can route to multiple services, whereas LoadBalancer creates a separate load balancer per service.
Ingress:
An Ingress is a Layer 7 (HTTP/HTTPS) routing resource that uses one load balancer to route to multiple services based on hostname and path.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
kubernetes.io/ingress.class: alb
spec:
rules:
- host: api.mycompany.com
http:
paths:
- path: /users
pathType: Prefix
backend:
service:
name: user-service
port:
number: 80
- path: /orders
pathType: Prefix
backend:
service:
name: order-service
port:
number: 80
- host: admin.mycompany.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: admin-service
port:
number: 80
One ALB, three services, two hostnames. Compare this to LoadBalancer type which would create three separate load balancers at 3× the cost.
Answer:
Why hardcoded credentials are dangerous:
Credentials in a container image or environment variable can be extracted from the running pod by anyone with kubectl exec access. They also appear in pod specs, CI/CD logs, and potentially in git history. There is no auto-rotation — if leaked, they are valid until manually revoked.
The correct approach: IAM Roles for Service Accounts (IRSA)
IRSA uses the Kubernetes Service Account mechanism combined with AWS IAM OIDC federation. The pod receives temporary, auto-rotating AWS credentials that are scoped to exactly the permissions it needs.
Step 1 — Create an IAM Policy with least-privilege permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::my-app-bucket",
"arn:aws:s3:::my-app-bucket/*"
]
},
{
"Effect": "Allow",
"Action": ["secretsmanager:GetSecretValue"],
"Resource": "arn:aws:secretsmanager:ap-south-1:123456789:secret:my-app-secret-*"
}
]
}
Step 2 — Create IAM Role for the Service Account:
eksctl create iamserviceaccount \
--cluster production-cluster \
--namespace my-app \
--name my-app-sa \
--attach-policy-arn arn:aws:iam::123456789:policy/MyAppPolicy \
--approve
Step 3 — Use the Service Account in the pod:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: my-app
spec:
template:
spec:
serviceAccountName: my-app-sa # use the annotated service account
containers:
- name: app
image: my-app:v1.0
# NO environment variables for AWS credentials
# The AWS SDK automatically discovers credentials from the token file
How it works at runtime:
When the pod starts, Kubernetes mounts a JWT token file into the pod at a well-known path. When the AWS SDK makes an API call, it exchanges this JWT token with AWS STS for temporary credentials. The credentials are scoped to exactly the IAM policy you defined. They expire every hour and are automatically refreshed. If the pod is compromised and the token is stolen, it expires soon and can only access the specific resources defined in the policy.
Answer:
What is RBAC:
RBAC (Role-Based Access Control) in Kubernetes defines who can do what to which resources. It has four components: Role (permissions within a namespace), ClusterRole (permissions across all namespaces), RoleBinding (assigns a Role to a user/group), ClusterRoleBinding (assigns a ClusterRole to a user/group).
Step 1 — Create a Role with view-only permissions:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: pod-viewer
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
# Intentionally omitting: delete, create, update, patch
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "watch"]
Step 2 — Create a RoleBinding connecting the Role to the developer:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: junior-dev-pod-viewer
namespace: production
subjects:
- kind: User
name: junior-developer
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-viewer
apiGroup: rbac.authorization.k8s.io
Step 3 — Map the IAM user to the Kubernetes user:
In EKS, authentication uses IAM. Authorization uses RBAC. You must map the IAM user to a Kubernetes username:
eksctl create iamidentitymapping \
--cluster production-cluster \
--arn arn:aws:iam::123456789:user/junior-developer \
--username junior-developer \
--group pod-readers
Verification:
# Test what the user can do
kubectl auth can-i get pods --namespace production --as junior-developer
# Output: yes
kubectl auth can-i delete pods --namespace production --as junior-developer
# Output: no
Real-world RBAC structure for a team:
| Role | Permissions | Assigned to |
|---|---|---|
| developer | get, list, watch pods/logs/events | All developers |
| deployer | create, update deployments | CI/CD service account |
| operator | everything in namespace | Senior engineers |
| cluster-admin | everything everywhere | Only DevOps leads, via MFA |
🎯 Scenario: A developer asks when to use ClusterIP vs NodePort vs LoadBalancer.
Answer:
# ClusterIP (default) — cluster-internal only
apiVersion: v1
kind: Service
metadata:
name: backend-api
spec:
type: ClusterIP
selector:
app: backend
ports:
- port: 8080
targetPort: 8080
# DNS: backend-api.namespace.svc.cluster.local:8080
# NodePort — expose on every node's IP:port
apiVersion: v1
kind: Service
metadata:
name: web-nodeport
spec:
type: NodePort
selector:
app: web
ports:
- port: 80
targetPort: 8080
nodePort: 30080 # Valid range: 30000-32767
# Access: <any-node-ip>:30080
# LoadBalancer — provisions cloud load balancer (AWS NLB/ALB)
apiVersion: v1
kind: Service
metadata:
name: web-lb
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
type: LoadBalancer
selector:
app: web
ports:
- port: 443
targetPort: 8080
protocol: TCP
# AWS provisions an NLB; service gets external DNS/IP
# Headless — no ClusterIP; DNS returns individual pod IPs
apiVersion: v1
kind: Service
metadata:
name: postgres-headless
spec:
clusterIP: None
selector:
app: postgres
ports:
- port: 5432
# DNS: postgres-0.postgres-headless.ns.svc.cluster.local → pod-0 IP
# postgres-1.postgres-headless.ns.svc.cluster.local → pod-1 IP
| Type | Scope | Best For |
|---|---|---|
ClusterIP | Inside cluster | Microservice communication |
NodePort | Node IPs | Dev, on-prem without cloud LB |
LoadBalancer | Internet | Production external services |
Headless | Direct pod DNS | StatefulSets, Cassandra, Kafka |
ExternalName | DNS alias | Route to external service by name |
🎯 Scenario: You have 5 microservices and want them at
api.example.com/users,api.example.com/orders, etc. with auto-renewed HTTPS certificates.
Answer:
# Install NGINX Ingress Controller
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--set controller.metrics.enabled=true
# Install cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager --create-namespace \
--set installCRDs=true
# ClusterIssuer — Let's Encrypt certificate authority
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-prod-key
solvers:
- http01:
ingress:
class: nginx
# Ingress — path-based routing + automatic TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-ingress
namespace: production
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
# Rate limiting
nginx.ingress.kubernetes.io/limit-rps: "100"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: api-example-com-tls # cert-manager creates this
rules:
- host: api.example.com
http:
paths:
- path: /users
pathType: Prefix
backend:
service:
name: users-service
port:
number: 8080
- path: /orders
pathType: Prefix
backend:
service:
name: orders-service
port:
number: 8080
- path: /
pathType: Prefix
backend:
service:
name: frontend-service
port:
number: 3000
# Monitor certificate issuance
kubectl get certificate -n production
kubectl describe certificate api-example-com-tls -n production
kubectl get certificaterequest -n production
🎯 Scenario: Pods in your cluster can’t resolve service names. Requests fail with “connection refused” or DNS lookup failures.
Answer:
Kubernetes runs CoreDNS as a cluster DNS server. Every pod’s /etc/resolv.conf points to the CoreDNS ClusterIP.
DNS resolution hierarchy:
my-service → searches: default.svc.cluster.local
my-service.other-ns → searches: svc.cluster.local
my-service.other-ns.svc → searches: cluster.local
my-service.other-ns.svc.cluster.local → full FQDN (resolved directly)
# Launch a debug pod with DNS tools
kubectl run dns-debug \
--image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 \
--restart=Never -it -- bash
# Inside the pod:
# Check resolv.conf
cat /etc/resolv.conf
# nameserver 10.96.0.10 ← CoreDNS ClusterIP
# search default.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5
# Test service DNS
nslookup kubernetes.default.svc.cluster.local
nslookup my-service.production.svc.cluster.local
# Test external DNS
nslookup google.com
# Debug CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --prefix
# Check CoreDNS config
kubectl get configmap coredns -n kube-system -o yaml
# Common CoreDNS issues:
# 1. CoreDNS pods OOMKilled → increase memory limits
# 2. ndots:5 causing slow resolution → external lookups try 6 DNS queries
# Fix: set ndots:1 in pod dnsConfig for external-heavy workloads
# 3. DNS cache poisoning → use separate upstream resolvers
# Override DNS per pod
apiVersion: v1
kind: Pod
spec:
dnsConfig:
options:
- name: ndots
value: "1" # Reduces unnecessary search path queries
nameservers:
- 8.8.8.8 # Additional custom nameservers
🎯 Scenario: Security audit requires that only frontend can reach the API, only API can reach the database, and nothing else.
Answer:
# Step 1: Default-deny ALL ingress AND egress in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {} # Applies to ALL pods
policyTypes:
- Ingress
- Egress
# Step 2: Allow frontend → API (port 8080 only)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-api
namespace: production
spec:
podSelector:
matchLabels:
app: api-server
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
# Step 3: Allow API → PostgreSQL (port 5432 only)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-to-postgres
namespace: production
spec:
podSelector:
matchLabels:
app: postgres
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: api-server
ports:
- protocol: TCP
port: 5432
# Step 4: Allow all pods to query CoreDNS (essential!)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Step 5: Allow API egress to external services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-external-egress
namespace: production
spec:
podSelector:
matchLabels:
app: api-server
policyTypes:
- Egress
egress:
- ports:
- protocol: TCP
port: 443 # HTTPS to external APIs
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
⚠️ NetworkPolicies require a CNI plugin that supports them — Calico, Cilium, or Weave Net. The default
flannelCNI does not enforce NetworkPolicies. Policies are additive: multiple policies combine with OR logic.
🎯 Scenario: Your app in Kubernetes needs to call an RDS database and a third-party payment API by logical names, not hardcoded endpoints.
Answer:
# ExternalName — DNS alias to external service
# App uses: postgres-prod.production.svc.cluster.local
# Resolves to: prod-db.abc123.us-east-1.rds.amazonaws.com
apiVersion: v1
kind: Service
metadata:
name: postgres-prod
namespace: production
spec:
type: ExternalName
externalName: prod-db.abc123.us-east-1.rds.amazonaws.com
# No selector — no pods. Pure DNS CNAME alias.
# Endpoints + Service — point to external IP directly
# Use when external service doesn't have a hostname
apiVersion: v1
kind: Service
metadata:
name: legacy-payment-api
namespace: production
spec:
ports:
- port: 443
targetPort: 443
---
apiVersion: v1
kind: Endpoints
metadata:
name: legacy-payment-api # Must match Service name
subsets:
- addresses:
- ip: 10.20.30.40 # External server IP
- ip: 10.20.30.41
ports:
- port: 443
# ServiceEntry (Istio) — fine-grained external service control
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: stripe-api
spec:
hosts:
- api.stripe.com
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
🎯 Scenario: Your cluster has high traffic and you suspect kube-proxy iptables rules are becoming a performance bottleneck. What are your options?
Answer:
kube-proxy runs on every node and maintains network rules that redirect traffic from Service ClusterIPs to pod IPs.
Three modes:
| Mode | How It Works | Performance | Notes |
|---|---|---|---|
userspace | Proxy in userspace (old) | Slowest | Deprecated |
iptables | Kernel netfilter rules | Good | Default; O(n) rules for n services |
ipvs | Linux Virtual Server | Best | O(1) lookups; for 1000+ services |
# Check current kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
# Switch to IPVS mode
kubectl edit configmap kube-proxy -n kube-system
# Set: mode: "ipvs"
# Restart kube-proxy pods to apply
kubectl rollout restart daemonset/kube-proxy -n kube-system
# Verify IPVS rules
ipvsadm -Ln # Run on a node
# Cilium — can replace kube-proxy entirely (eBPF)
# No iptables, no kube-proxy, direct eBPF kernel datapath
helm install cilium cilium/cilium \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=<api-server-ip>
💡 For clusters with > 500 services, switch to IPVS mode or use Cilium’s eBPF kube-proxy replacement. iptables rule evaluation is O(n) — performance degrades linearly with service count.
🎯 Scenario: Your compliance team requires all service-to-service communication to be encrypted and mutually authenticated.
Answer:
# Option 1: Istio service mesh — automatic mTLS
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm install istio-base istio/base -n istio-system --create-namespace
helm install istiod istio/istiod -n istio-system
# Label namespace for automatic sidecar injection
kubectl label namespace production istio-injection=enabled
# Enforce strict mTLS across namespace
# Istio PeerAuthentication — enforce mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: enforce-mtls
namespace: production
spec:
mtls:
mode: STRICT # STRICT: only mTLS, no plaintext
# PERMISSIVE: both mTLS and plaintext (migration mode)
# DISABLE: no mTLS
# Istio AuthorizationPolicy — service-level access control
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: api-server-policy
namespace: production
spec:
selector:
matchLabels:
app: api-server
action: ALLOW
rules:
- from:
- source:
principals:
# Only allow calls from frontend service account
- "cluster.local/ns/production/sa/frontend-sa"
to:
- operation:
methods: ["GET", "POST"]
paths: ["/api/*"]
# Verify mTLS is working
istioctl authn tls-check api-server-pod-xxx.production
# View certificate details
istioctl proxy-config secret api-server-pod-xxx.production
🎯 Scenario: You have a multi-region cluster and want traffic to prefer pods in the same zone to reduce latency and egress costs.
Answer:
# Topology Aware Routing (K8s 1.27+)
apiVersion: v1
kind: Service
metadata:
name: web-app
annotations:
service.kubernetes.io/topology-mode: "Auto"
# "Auto" — route to same-zone endpoints when possible
# Fallback to cross-zone if no local endpoints are healthy
spec:
selector:
app: web-app
ports:
- port: 80
targetPort: 8080
# TrafficPolicy in Istio (more fine-grained control)
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: web-app
namespace: production
spec:
host: web-app
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
h2UpgradePolicy: UPGRADE
idleTimeout: 10s
loadBalancer:
localityLbSetting:
enabled: true
distribute:
- from: "us-east-1/us-east-1a/*"
to:
"us-east-1/us-east-1a/*": 80 # 80% same-AZ
"us-east-1/us-east-1b/*": 20 # 20% different AZ
outlierDetection:
consecutive5xxErrors: 5
interval: 30s
baseEjectionTime: 30s
🎯 Scenario: Your security policy forbids storing secrets in Kubernetes. All secrets must live in AWS Secrets Manager and be synced automatically.
Answer:
# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace \
--set installCRDs=true
# SecretStore — configure AWS Secrets Manager connection
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secretsmanager
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa # Uses IRSA — no static keys!
# ExternalSecret — sync a specific secret from AWS → K8s
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials
namespace: production
spec:
refreshInterval: 1h # Re-sync from AWS every hour
secretStoreRef:
name: aws-secretsmanager
kind: SecretStore
target:
name: database-credentials # K8s Secret that gets created
creationPolicy: Owner # ESO owns this secret
template:
type: Opaque
engineVersion: v2
data:
- secretKey: DB_PASSWORD # Key in K8s Secret
remoteRef:
key: prod/myapp/database # AWS SM secret name
property: password # JSON field within secret
- secretKey: DB_USERNAME
remoteRef:
key: prod/myapp/database
property: username
# Bulk import all fields from a JSON secret
dataFrom:
- extract:
key: prod/myapp/all-secrets
# Verify sync status
kubectl get externalsecret database-credentials -n production
# STATUS column shows: SecretSynced or error
kubectl describe externalsecret database-credentials -n production
🎯 Scenario: Your database password is compromised. You need to rotate it immediately with zero application downtime.
Answer:
# Rotation strategy for env-var injected secrets (requires pod restart):
# Step 1: Update secret in database first (allow both old and new password)
# Step 2: Update the K8s secret
kubectl create secret generic db-secret \
--from-literal=DB_PASSWORD=new-secure-password-v2 \
--dry-run=client -o yaml | kubectl apply -f -
# Step 3: Rolling restart — zero-downtime with 4 replicas and maxUnavailable=0
kubectl rollout restart deployment/api-server -n production
kubectl rollout status deployment/api-server -n production
# Step 4: Remove old password from database after all pods are updated
# Step 5: Verify no old-password connections in DB
# For volume-mounted secrets — automatic file update (no restart needed)
# K8s automatically updates mounted secret files within ~1 minute
# App must watch for file changes and reload
# Check file was updated in pod
kubectl exec -it api-server-xxx -- cat /etc/secrets/DB_PASSWORD
# Force immediate update (before kubelet sync)
# Annotate pod to trigger kubelet reconcile
kubectl annotate pod api-server-xxx rotation-timestamp=$(date +%s)
# Best practice: version your secrets
apiVersion: v1
kind: Secret
metadata:
name: db-secret-v2 # Include version in name
namespace: production
annotations:
rotation-date: "2024-01-15"
rotated-by: "security-team"
🎯 Scenario: A security audit flags that Kubernetes Secrets are stored in plaintext in etcd. How do you fix this?
Answer:
# /etc/kubernetes/enc/encryption-config.yaml
# Create this on the control plane node
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps # Optionally encrypt ConfigMaps too
providers:
- aescbc: # AES-CBC encryption
keys:
- name: key1
secret: <base64-encoded-32-byte-key> # openssl rand -base64 32
- identity: {} # Fallback: allows reading unencrypted secrets
# Remove this after re-encrypting all existing secrets
# Generate a 32-byte key
head -c 32 /dev/urandom | base64
# Add to kube-apiserver flags (in /etc/kubernetes/manifests/kube-apiserver.yaml)
# --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
# Mount the config file in the apiserver static pod
# volumeMounts:
# - name: enc
# mountPath: /etc/kubernetes/enc
# readOnly: true
# volumes:
# - name: enc
# hostPath:
# path: /etc/kubernetes/enc
# type: DirectoryOrCreate
# After apiserver restarts, re-encrypt all existing secrets
kubectl get secrets -A -o json | kubectl replace -f -
# Verify a secret is encrypted in etcd
ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
| hexdump -C | head
# Should show: k8s:enc:aescbc:v1:key1:... (encrypted, not plaintext)
🎯 Scenario: Your team uses GitOps (all config in Git), but Kubernetes Secrets can’t be committed to Git as they’re only base64-encoded. How do you solve this?
Answer:
Sealed Secrets encrypts K8s Secrets with a public key. The encrypted SealedSecret is safe to commit to Git. Only the in-cluster controller can decrypt it.
# Install Sealed Secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
--namespace kube-system
# Install kubeseal CLI
brew install kubeseal
# Fetch the public key (for offline use)
kubeseal --fetch-cert > pub-cert.pem
# Create a SealedSecret from a regular Secret
kubectl create secret generic db-secret \
--from-literal=DB_PASSWORD=my-super-secret \
--dry-run=client -o yaml \
| kubeseal \
--cert pub-cert.pem \
--format yaml > sealed-db-secret.yaml
# Now commit sealed-db-secret.yaml to Git — it's safe!
git add sealed-db-secret.yaml
git commit -m "Add sealed DB secret"
# Apply to cluster — controller decrypts and creates regular Secret
kubectl apply -f sealed-db-secret.yaml
# Verify Secret was created
kubectl get secret db-secret
# sealed-db-secret.yaml (safe to commit to Git)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: db-secret
namespace: production
spec:
encryptedData:
DB_PASSWORD: AgB7x9... (long base64 encrypted string)
template:
metadata:
name: db-secret
namespace: production
type: Opaque
🎯 Scenario: Your app pod needs to read from an S3 bucket and write to DynamoDB. How do you give it AWS permissions without static credentials?
Answer:
IRSA (IAM Roles for Service Accounts) binds a Kubernetes ServiceAccount to an AWS IAM role using OIDC federation. No static AWS keys in pods.
# Step 1: Create IAM OIDC provider for the EKS cluster
eksctl utils associate-iam-oidc-provider \
--cluster my-cluster \
--region us-east-1 \
--approve
# Step 2: Create IAM role with trust policy for the ServiceAccount
OIDC_ISSUER=$(aws eks describe-cluster --name my-cluster \
--query "cluster.identity.oidc.issuer" --output text)
aws iam create-role \
--role-name MyAppRole \
--assume-role-policy-document "{
\"Version\": \"2012-10-17\",
\"Statement\": [{
\"Effect\": \"Allow\",
\"Principal\": {\"Federated\": \"arn:aws:iam::123456789:oidc-provider/${OIDC_ISSUER#*//}\"},
\"Action\": \"sts:AssumeRoleWithWebIdentity\",
\"Condition\": {
\"StringEquals\": {
\"${OIDC_ISSUER#*//}:sub\": \"system:serviceaccount:production:my-app-sa\"
}
}
}]
}"
# Step 3: Attach minimal IAM policy
aws iam put-role-policy --role-name MyAppRole \
--policy-name MyAppPolicy \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": ["arn:aws:s3:::my-app-bucket", "arn:aws:s3:::my-app-bucket/*"]
},
{
"Effect": "Allow",
"Action": ["dynamodb:PutItem", "dynamodb:GetItem", "dynamodb:Query"],
"Resource": "arn:aws:dynamodb:us-east-1:123456789:table/MyTable"
}
]
}'
# Step 4: Create annotated ServiceAccount in K8s
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-app-sa
namespace: production
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789:role/MyAppRole"
eks.amazonaws.com/token-expiration: "86400" # 24h token expiry
# Step 5: Use ServiceAccount in Deployment
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
serviceAccountName: my-app-sa # Automatically gets AWS credentials
containers:
- name: app
image: myapp:v2.0
# AWS SDK auto-discovers credentials from the projected volume
# No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed!
🎯 Scenario: Security audit requires that no pods in production run as root, use privileged mode, or mount host paths.
Answer:
# Enforce Pod Security Standards at namespace level (K8s 1.25+, built-in)
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
# enforce: deny pods that violate the policy
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.28
# warn: allow but show warning (use during migration)
pod-security.kubernetes.io/warn: restricted
# audit: record in audit log
pod-security.kubernetes.io/audit: restricted
Three built-in policy levels:
| Level | Restrictions |
|---|---|
privileged | No restrictions — for trusted system pods |
baseline | Minimal restrictions — blocks privileged, hostNetwork, hostPID |
restricted | Most secure — non-root, read-only FS, dropped capabilities, seccomp |
# Pod that satisfies the "restricted" policy
apiVersion: v1
kind: Pod
metadata:
name: secure-app
namespace: production
spec:
securityContext:
runAsNonRoot: true # Must not run as root
runAsUser: 10000 # Specific non-root UID
runAsGroup: 10000
fsGroup: 10000
seccompProfile:
type: RuntimeDefault # Default seccomp profile required
containers:
- name: app
image: myapp:v2.0
securityContext:
allowPrivilegeEscalation: false # Cannot gain more privileges
readOnlyRootFilesystem: true # Immutable container filesystem
runAsNonRoot: true
capabilities:
drop: ["ALL"] # Drop all Linux capabilities
add: ["NET_BIND_SERVICE"] # Only add what's needed
volumeMounts:
- name: tmp-dir # Writable temp dir (since rootFS is RO)
mountPath: /tmp
- name: cache-dir
mountPath: /app/cache
volumes:
- name: tmp-dir
emptyDir: {}
- name: cache-dir
emptyDir: {}
🎯 Scenario: Someone deleted a production deployment. How do you trace exactly who did it, when, and from where?
Answer:
# Audit policy — what to log and at what verbosity
# /etc/kubernetes/audit-policy.yaml (on control plane)
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived # Don't log every incoming request
rules:
# Log all modifications to critical resources in detail
- level: RequestResponse
verbs: ["create", "update", "patch", "delete"]
resources:
- group: "apps"
resources: ["deployments", "statefulsets", "daemonsets"]
- group: ""
resources: ["secrets", "configmaps", "serviceaccounts"]
namespaces: ["production", "staging"]
# Log who accessed secrets (metadata only — don't log secret values)
- level: Metadata
verbs: ["get", "list", "watch"]
resources:
- group: ""
resources: ["secrets"]
# Log Node/ServiceAccount auth issues
- level: Metadata
users: ["system:anonymous"]
# Don't log health check noise
- level: None
nonResourceURLs: ["/healthz*", "/readyz*", "/livez*"]
# Default: log metadata for everything else
- level: Metadata
# Enable in kube-apiserver (add to /etc/kubernetes/manifests/kube-apiserver.yaml)
# --audit-policy-file=/etc/kubernetes/audit-policy.yaml
# --audit-log-path=/var/log/kubernetes/audit.log
# --audit-log-maxage=30
# --audit-log-maxbackup=10
# --audit-log-maxsize=100 # 100MB per file
# Search for who deleted the deployment
cat /var/log/kubernetes/audit.log | \
jq 'select(.verb=="delete" and .objectRef.resource=="deployments" and .objectRef.name=="web-app")' | \
jq '{time:.requestReceivedTimestamp, user:.user.username, userAgent:.userAgent, sourceIP:.sourceIPs[0]}'
# Output: {"time":"2024-01-15T14:23:07Z", "user":"john.doe", "userAgent":"kubectl/v1.28.0", "sourceIP":"10.0.1.5"}
🎯 Scenario: You want to enforce that all pods have resource limits set, all images come from your private registry, and all namespaces have a required label.
Answer:
# Install Gatekeeper
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml
# ConstraintTemplate — defines the policy logic in Rego
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlimits
spec:
crd:
spec:
names:
kind: K8sRequiredLimits
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlimits
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.memory
msg := sprintf("Container '%v' must have memory limits set", [container.name])
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.cpu
msg := sprintf("Container '%v' must have CPU limits set", [container.name])
}
# Constraint — applies the template as an actual policy
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLimits
metadata:
name: require-resource-limits
spec:
enforcementAction: deny # deny / warn / dryrun
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production", "staging"]
excludedNamespaces: ["kube-system", "monitoring"]
# Policy: images must come from approved registries
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8sallowedrepos
spec:
crd:
spec:
names:
kind: K8sAllowedRepos
validation:
openAPIV3Schema:
properties:
repos:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8sallowedrepos
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
satisfied := [good | repo := input.parameters.repos[_]; good := startswith(container.image, repo)]
not any(satisfied)
msg := sprintf("Image '%v' is not from an approved registry", [container.image])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
name: allowed-repos
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
parameters:
repos:
- "gcr.io/my-company/"
- "123456789.dkr.ecr.us-east-1.amazonaws.com/"
🎯 Scenario: You want to prevent unscanned or unsigned container images from being deployed.
Answer:
# GitHub Actions: scan image before push
name: Build and Scan
on: [push]
jobs:
build-and-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t myapp:${{ github.sha }} .
- name: Scan with Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: myapp:${{ github.sha }}
format: sarif
output: trivy-results.sarif
severity: CRITICAL,HIGH
exit-code: 1 # Fail pipeline on CRITICAL/HIGH vulns
ignore-unfixed: true # Skip vulns with no fix available
- name: Sign with Cosign (keyless)
uses: sigstore/cosign-installer@v3
- run: |
cosign sign --yes myapp:${{ github.sha }}
# Creates a signature stored in OCI registry
# Policy Controller / Connaisseur — enforce signature verification at admission
apiVersion: connaisseur.io/v1beta1
kind: ValidationPolicy
metadata:
name: require-signed-images
spec:
validators:
- name: cosign
type: cosign
host: https://sigstore.dev
policy:
- pattern: "123456789.dkr.ecr.us-east-1.amazonaws.com/*:*"
validators:
- name: cosign
with:
key: k8s://cosign-keys/cosign-pub-key
🎯 Scenario: You have dev, staging, and production namespaces on the same cluster. Dev pods must not be able to talk to production services.
Answer:
# Block all cross-namespace traffic INTO production
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-cross-namespace
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
# Only allow traffic from WITHIN production namespace
- from:
- podSelector: {} # Any pod in THIS namespace
# AND from monitoring namespace (for Prometheus scraping)
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
# Allow Ingress controller (lives in ingress-nginx namespace) to reach production
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-controller
namespace: production
spec:
podSelector:
matchLabels:
tier: frontend # Only frontend pods accessible externally
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
ports:
- protocol: TCP
port: 8080
🎯 Scenario: Your app needs dynamic database credentials from HashiCorp Vault, rotated every hour, without pod restarts.
Answer:
# Install Vault with HA in Kubernetes
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
--namespace vault \
--create-namespace \
--set server.ha.enabled=true \
--set server.ha.replicas=3 \
--set server.auditStorage.enabled=true
# Vault Agent Injector — sidecars that inject and renew secrets
# Annotate pods to get automatic secret injection
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
spec:
template:
metadata:
annotations:
# Enable Vault Agent sidecar injection
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "web-app"
# Inject DB credentials at /vault/secrets/db-creds
vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/web-app"
# Template to format the secret as a .env file
vault.hashicorp.com/agent-inject-template-db-creds: |
{{- with secret "database/creds/web-app" -}}
export DB_USERNAME="{{ .Data.username }}"
export DB_PASSWORD="{{ .Data.password }}"
{{- end }}
# Renew lease — app gets new creds before they expire
vault.hashicorp.com/agent-pre-populate-only: "false"
spec:
serviceAccountName: web-app-sa # Vault uses K8s SA for auth
containers:
- name: app
image: myapp:v2.0
command:
- /bin/sh
- -c
- |
source /vault/secrets/db-creds # Load dynamic credentials
exec python app.py
# Configure Vault database secrets engine
vault secrets enable database
vault write database/config/postgres \
plugin_name=postgresql-database-plugin \
allowed_roles="web-app" \
connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb?sslmode=disable" \
username="vault-admin" \
password="vault-admin-password"
vault write database/roles/web-app \
db_name=postgres \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
Answer:
Kubernetes enforces a flat networking model with these requirements:
- All Pods can communicate with each other without NAT
- All Nodes can communicate with all Pods without NAT
- The IP a Pod sees for itself is the same IP others see
Network layers:
- Pod-to-Pod — via CNI plugin (VPC CNI, Calico, Cilium, Flannel)
- Pod-to-Service — via kube-proxy (iptables or IPVS rules)
- External-to-Service — via NodePort, LoadBalancer, or Ingress
CNI Plugin responsibilities:
- Assign IP addresses to Pods
- Set up routing rules
- Handle network policy enforcement (Calico, Cilium)
# Inspect CNI config on a node
cat /etc/cni/net.d/10-aws.conflist
# Trace network path
kubectl exec -it my-pod -- traceroute 10.100.0.1
Answer:
Kubernetes implements service discovery in two ways:
1. DNS-based (recommended):
- CoreDNS resolves Service names to ClusterIPs
<service>.<namespace>.svc.cluster.local
2. Environment variables:
- At Pod start, Kubernetes injects env vars for all Services in the namespace
- e.g.,
MY_SERVICE_SERVICE_HOST,MY_SERVICE_SERVICE_PORT - Limitation: only works for Services created before the Pod
Headless Services (for StatefulSets):
- Set
clusterIP: None - DNS returns individual Pod IPs instead of a single VIP
- Enables direct Pod addressing
# Headless service
apiVersion: v1
kind: Service
metadata:
name: my-stateful-svc
spec:
clusterIP: None
selector:
app: my-stateful-app
ports:
- port: 5432
Answer:
Admission Controllers are plugins that intercept API server requests after authentication and authorization but before persisting objects to etcd. They can validate or mutate requests.
Two types:
- Mutating Admission Webhooks: Modify the request (e.g., inject sidecar, add labels/defaults)
- Validating Admission Webhooks: Allow or reject the request (e.g., enforce policies)
Built-in admission controllers:
LimitRanger— enforces resource limitsResourceQuota— enforces namespace quotasPodSecurity— enforces Pod security standardsMutatingAdmissionWebhook— calls external webhook for mutationsValidatingAdmissionWebhook— calls external webhook for validation
# Validating Webhook configuration
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: my-policy-webhook
webhooks:
- name: validate.mycompany.io
clientConfig:
service:
name: policy-service
namespace: kube-system
path: /validate
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: ["apps"]
resources: ["deployments"]
admissionReviewVersions: ["v1"]
sideEffects: None
Answer:
Network Policies are Kubernetes resources that control traffic flow at the IP/port level between Pods, namespaces, and external endpoints. They require a CNI plugin that supports them (Calico, Cilium, Weave).
# Deny all ingress, allow only from specific namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-frontend
namespace: backend
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: frontend
podSelector:
matchLabels:
role: api
ports:
- protocol: TCP
port: 5432
egress:
- to:
- ipBlock:
cidr: 10.0.0.0/8
Default behavior: Without a NetworkPolicy, all traffic is allowed. Once a NetworkPolicy selects a Pod, that Pod follows the policy’s rules.
Answer:
kube-proxy manages network rules on nodes for Service routing. It supports three modes:
| Feature | iptables | IPVS |
|---|---|---|
| Routing | Sequential rule matching | Hash table lookup |
| Performance | Degrades at scale (O(n)) | Constant time (O(1)) |
| Load balancing algorithms | Round-robin only | RR, least connections, source hash, etc. |
| Scale | Good up to ~1000 Services | Scales to 10,000+ Services |
| Health checking | Limited | Built-in |
# Check current kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
# Switch to IPVS mode (via configmap)
kubectl edit configmap kube-proxy -n kube-system
# Set: mode: "ipvs"
For large clusters (>1000 Services), IPVS mode is strongly recommended. EKS also supports IPVS mode.
Answer:
OPA Gatekeeper is a policy engine for Kubernetes built on Open Policy Agent (OPA). It uses validating admission webhooks and CRDs to enforce custom policies.
Key concepts:
- ConstraintTemplate — defines the policy logic in Rego
- Constraint — an instance of a ConstraintTemplate with specific parameters
# ConstraintTemplate — enforce required labels
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: requirelabels
spec:
crd:
spec:
names:
kind: RequireLabels
validation:
openAPIV3Schema:
type: object
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package requirelabels
violation[{"msg": msg}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("Missing required labels: %v", [missing])
}
---
# Constraint — apply the policy
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireLabels
metadata:
name: must-have-env-label
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment"]
parameters:
labels: ["env", "team", "app"]
Answer:
IRSA (IAM Roles for Service Accounts) allows Kubernetes Pods to assume AWS IAM roles using Kubernetes Service Accounts. This replaces the old pattern of assigning IAM roles to EC2 nodes.
How it works:
- EKS cluster has an OIDC provider configured
- IAM role has a trust policy allowing the OIDC provider and specific service account
- Pod uses a service account annotated with the IAM role ARN
- The EKS Pod Identity Webhook injects AWS credential env vars into the Pod
- AWS SDK in the Pod automatically fetches temporary credentials via OIDC token
# Create OIDC provider for EKS cluster
eksctl utils associate-iam-oidc-provider \
--cluster my-cluster \
--approve
# Create IAM role for service account
eksctl create iamserviceaccount \
--cluster my-cluster \
--namespace my-namespace \
--name my-service-account \
--attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
--approve
# Service Account with IRSA annotation
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-service-account
namespace: my-namespace
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/my-pod-role
Answer:
EKS Pod Identity is a newer, simpler mechanism for granting AWS permissions to Pods. It uses a dedicated Pod Identity Agent DaemonSet instead of OIDC webhooks.
Comparison:
| Feature | IRSA | EKS Pod Identity |
|---|---|---|
| Mechanism | OIDC + webhook | Pod Identity Agent DaemonSet |
| IAM trust policy | Complex (OIDC condition) | Simple (pods.eks.amazonaws.com) |
| Cross-account | Supported | Supported |
| Cluster config | OIDC provider required | Agent add-on required |
| Simplicity | More complex setup | Simpler setup |
# Enable Pod Identity add-on
aws eks create-addon \
--cluster-name my-cluster \
--addon-name eks-pod-identity-agent
# Create Pod Identity association
aws eks create-pod-identity-association \
--cluster-name my-cluster \
--namespace my-namespace \
--service-account my-service-account \
--role-arn arn:aws:iam::123456789:role/my-pod-role
Answer:
1. IAM and RBAC:
- Use IRSA or Pod Identity instead of node-level IAM roles
- Apply least-privilege IAM policies
- Use EKS Access Entries instead of aws-auth ConfigMap
- Regularly audit RBAC bindings
2. Network Security:
- Enable private API server endpoint
- Use Security Groups for Pods
- Implement Network Policies (Calico or Cilium)
- Use VPC endpoints for AWS service traffic
3. Secrets Management:
- Encrypt Kubernetes Secrets with KMS at rest
- Use AWS Secrets Manager via CSI driver or External Secrets Operator
4. Pod Security:
- Enforce Pod Security Standards (
Restrictedprofile) - Disable privilege escalation:
allowPrivilegeEscalation: false - Run containers as non-root users
- Use read-only root filesystems
5. Runtime Security:
- Enable Amazon GuardDuty for EKS (runtime threat detection)
- Use Falco for real-time runtime security
6. Image Security:
- Scan images with Amazon ECR image scanning (or Trivy/Snyk)
- Use immutable image tags
- Sign images with Notary/Cosign
# Restricted Pod Security Standard
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
Answer:
EKS security spans four layers — identity, network, workload, and data:
1. Identity & Access (who can do what):
# Prefer EKS Access Entries (or aws-auth ConfigMap) with least-privilege IAM roles
# Map each team/CI role to a scoped Kubernetes RBAC group — never system:masters for humans
aws eks create-access-entry --cluster-name my-cluster \
--principal-arn arn:aws:iam::123456789:role/dev-team-role \
--kubernetes-groups developers
- Use IRSA (IAM Roles for Service Accounts) or EKS Pod Identity so Pods get scoped AWS permissions — never mount broad node-level IAM roles into every Pod
- Enforce RBAC with least-privilege Roles/ClusterRoles; audit with
kubectl auth can-i --list
2. Network:
# Default-deny NetworkPolicy, then explicitly allow required traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny }
spec:
podSelector: {}
policyTypes: ["Ingress", "Egress"]
- Set the EKS API endpoint to private (or restrict public access to known CIDRs)
- Use Security Groups for Pods to control traffic at the ENI level for sensitive workloads
3. Workload (Pod Security Standards):
# Enforce restricted Pod Security Standard at the namespace level
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
- Run containers as non-root, read-only root filesystem, drop all Linux capabilities by default
- Scan images in the pipeline (Trivy/ECR scan) and block CRITICAL vulnerabilities before deploy
4. Data & Audit:
- Enable secrets encryption with a customer-managed KMS key at cluster creation
- Enable control plane audit logging to CloudWatch Logs and alert on suspicious API calls (e.g.,
execinto prod Pods, RBAC changes) - Keep the EKS version and add-ons (
vpc-cni,coredns,kube-proxy) patched — AWS regularly ships CVE fixes for these
Interview summary line: “I’d treat EKS security as defense-in-depth: IRSA/Pod Identity instead of broad node roles, default-deny network policies, restricted Pod Security Standards enforced per namespace, KMS-encrypted secrets, and audit logging wired to alerts — so no single misconfiguration is enough to compromise the cluster.”
Answer:
Kubernetes has no built-in “user” object — identity comes from an external source (a client certificate, an OIDC token, or in EKS’s case, IAM), and RBAC controls what that identity can do once authenticated. The process has two distinct halves:
1. Give the person an identity (self-managed cluster — client certificate approach):
# Generate a private key and CSR for the new user
openssl genrsa -out john.key 2048
openssl req -new -key john.key -out john.csr -subj "/CN=john/O=developers"
# Submit as a Kubernetes CertificateSigningRequest and approve it
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: john-csr
spec:
request: $(cat john.csr | base64 | tr -d '\n')
signerName: kubernetes.io/kube-apiserver-client
usages: ["client auth"]
EOF
kubectl certificate approve john-csr
# Build a kubeconfig for John using the signed certificate
kubectl config set-credentials john --client-certificate=john.crt --client-key=john.key
On EKS, use IAM instead — map an IAM identity to a Kubernetes username via Access Entries (or the legacy aws-auth ConfigMap):
aws eks create-access-entry --cluster-name my-cluster \
--principal-arn arn:aws:iam::123456789:user/john \
--kubernetes-groups developers
2. Grant SPECIFIC authorization (both cases — this is where RBAC comes in):
# A narrow Role — only what "developers" should be able to do, nothing more
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: developer-access
namespace: staging
rules:
- apiGroups: ["", "apps"]
resources: ["pods", "deployments", "services", "configmaps"]
verbs: ["get", "list", "watch", "create", "update"]
# Notably NOT "delete" or access to "secrets" — least privilege
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: john-developer-binding
namespace: staging
subjects:
- kind: User
name: john # matches CN in the cert, or the mapped IAM username
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: developer-access
apiGroup: rbac.authorization.k8s.io
# Verify what John can actually do
kubectl auth can-i delete pods --as=john -n staging # should be "no"
kubectl auth can-i create deployments --as=john -n staging # should be "yes"
Key point for an interview: “adding a user” and “authorizing a user” are two separate steps — Kubernetes authenticates via an external identity source, then RBAC (Role/RoleBinding, scoped to a namespace like above) is what actually defines “specific authorization.” Always scope with a Role+RoleBinding (namespace-limited) rather than a ClusterRole+ClusterRoleBinding unless the person genuinely needs cluster-wide access.
Answer:
Every Service gets a DNS record, and the namespace determines how much of the name you need to specify — same-namespace lookups can use a short name, cross-namespace lookups need at least the namespace included:
Full form (works from ANY namespace):
backend-svc.orders.svc.cluster.local
From WITHIN the same namespace (orders), the short form also resolves:
backend-svc ← works, thanks to the Pod's search domain
From a DIFFERENT namespace (e.g., a Pod in "frontend" calling "orders"):
backend-svc ← FAILS — resolves to nothing in this namespace
backend-svc.orders ← works — namespace included
backend-svc.orders.svc.cluster.local ← always works, fully qualified
Why the short name only works locally: every Pod’s /etc/resolv.conf has a search list that includes its own namespace (e.g., orders.svc.cluster.local), so an unqualified name gets that suffix appended automatically. A Pod in a different namespace doesn’t have that suffix in its search path, so the same short name resolves to nothing.
# Check a Pod's actual search domains
kubectl exec -it mypod -n frontend -- cat /etc/resolv.conf
# search frontend.svc.cluster.local svc.cluster.local cluster.local ...
What can break cross-namespace DNS specifically:
# 1. A NetworkPolicy blocking egress to kube-dns/CoreDNS on port 53
# — extremely common cause once teams start locking down NetworkPolicies
kubectl get networkpolicy -n frontend -o yaml
# Fix: always allow egress to the kube-system namespace on port 53 (UDP+TCP)
# 2. CoreDNS pods themselves unhealthy or under-resourced
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# 3. Custom Corefile misconfiguration (stub domains, forward rules edited
# incorrectly) breaking resolution for specific zones
kubectl get configmap coredns -n kube-system -o yaml
# 4. The TARGET service genuinely doesn't exist in that namespace, or its
# selector matches zero Pods (Service exists, but has no Endpoints —
# DNS resolves fine, but connecting still fails)
kubectl get endpoints backend-svc -n orders
# 5. dnsPolicy overridden on the Pod itself (rare, but breaks everything)
kubectl get pod mypod -o jsonpath='{.spec.dnsPolicy}'
# Should be "ClusterFirst" (default) for normal in-cluster resolution
Most common root cause in practice: a NetworkPolicy that locks down egress traffic for security but forgets to explicitly allow DNS (port 53) to kube-system — the Pod can’t even resolve the name to attempt a connection, which often gets misdiagnosed as “the other service is down” when it’s actually a DNS-level block.
Add More Questions to This Guide
Know a question that should be here? Share it and help the community!
Open Google Form