Kubernetes Cluster Administration

Scheduling controls

The scheduler picks a node for every unscheduled pod. Administrators steer it with a handful of mechanisms.

• nodeSelector is the simplest: the pod only runs on nodes carrying matching labels.
• Node affinity is a richer version. requiredDuringSchedulingIgnoredDuringExecution is a hard rule, preferredDuringSchedulingIgnoredDuringExecution is a weighted preference.
• Taints and tolerations work in the opposite direction: a taint on a node repels pods unless they carry a matching toleration.

kubectl label node worker-1 disktype=ssd
kubectl taint nodes worker-1 dedicated=batch:NoSchedule
kubectl taint nodes worker-1 dedicated=batch:NoSchedule-   # remove

tolerations:
  - key: dedicated
    operator: Equal
    value: batch
    effect: NoSchedule

Taint effects are NoSchedule (do not place new pods), PreferNoSchedule (try to avoid) and NoExecute (also evict running pods). A toleration only permits placement; to pin pods to a node you need a taint and affinity or selector together. Control-plane nodes are normally tainted so ordinary workloads stay off them.

Other tools: DaemonSets run one pod per node, static pods are started by the kubelet from manifest files on disk (this is how kubeadm runs control-plane components), and a pod can name a custom scheduler with schedulerName.

Resource limits and monitoring

Each container can declare requests (what the scheduler reserves) and limits (the ceiling). Exceeding a memory limit gets the container OOM-killed; exceeding a CPU limit only throttles it. Namespace-level guardrails:

• LimitRange sets default and min/max values per container.
• ResourceQuota caps the total CPU, memory, storage or object counts in a namespace.

The Metrics Server collects short-term CPU and memory usage from kubelets and powers kubectl top:

kubectl top nodes
kubectl top pods -A --sort-by=memory

It keeps no history, so use a proper monitoring stack for trends and alerting.

Upgrades with kubeadm

Upgrade one minor version at a time (for example 1.x to 1.x+1), control plane first, then workers. Component version skew rules mean the kubelet may lag the API server but never lead it.

1. On the first control-plane node, upgrade the kubeadm package, then run kubeadm upgrade plan and kubeadm upgrade apply v1.x.y.
2. Drain the node, upgrade kubelet and kubectl packages, restart the kubelet, and uncordon.
3. On other control-plane nodes use kubeadm upgrade node.
4. For each worker: kubectl drain <node> --ignore-daemonsets, upgrade kubeadm, run kubeadm upgrade node, upgrade and restart the kubelet, then kubectl uncordon <node>.

cordon marks a node unschedulable; drain cordons it and evicts pods. Back up etcd before starting, and check PodDisruptionBudgets so a drain does not stall.

etcd backup and restore

etcd holds all cluster state, so a snapshot is the core of disaster recovery. With etcdctl v3, authenticate with a client certificate signed by the etcd CA (on kubeadm clusters these files live under /etc/kubernetes/pki/etcd):

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

etcdutl snapshot restore /backup/etcd.db --data-dir /var/lib/etcd-restored

To restore, restore into a new data directory, then edit the etcd static pod manifest so its data volume points to that directory; the kubelet recreates the pod. Verify the snapshot with etcdutl snapshot status, copy backups off the node, and rehearse the restore on a schedule.

RBAC and security

Requests pass authentication, authorization and admission. Role-based access control answers the authorization step with four objects: Role and RoleBinding (namespaced), ClusterRole and ClusterRoleBinding (cluster-wide). Rules are purely additive.

kubectl create role pod-reader --verb=get,list,watch --resource=pods -n dev
kubectl create rolebinding read-pods --role=pod-reader --user=dev-user -n dev
kubectl auth can-i list pods -n dev --as dev-user

Practices that matter:

• Give workloads their own ServiceAccount with minimal rights, and avoid binding cluster-admin broadly.
• Use a securityContext: run as non-root, drop capabilities, set a read-only root filesystem.
• Use NetworkPolicies to default-deny and then allow required traffic (needs a CNI that enforces them).
• Pull images from trusted registries, with imagePullSecrets for private ones.
• Know where certificates live and when they expire: kubeadm certs check-expiration.

Storage

A PersistentVolume (PV) is a piece of storage; a PersistentVolumeClaim (PVC) is a request for it; a pod mounts the claim. Claims bind to a PV with enough capacity and compatible access modes (ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod).

A StorageClass enables dynamic provisioning: the claim names a class and a CSI driver creates the volume on demand.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: standard
  resources:
    requests:
      storage: 10Gi

The reclaimPolicy (Delete or Retain) decides what happens to the data when the claim is removed, and volumeBindingMode: WaitForFirstConsumer delays provisioning until a pod is scheduled, avoiding zone mismatches.

Networking and troubleshooting

Pods get cluster-wide IPs from the CNI plugin, Services give them stable virtual IPs through kube-proxy, and CoreDNS resolves names like web.dev.svc.cluster.local. Ingress routes external HTTP traffic. Control-plane nodes need the API server (6443), etcd (2379-2380) and kubelet (10250) ports open between the right peers.

A reliable debugging order:

1. kubectl get nodes and kubectl get pods -A: is anything NotReady or crash-looping?
2. kubectl describe pod: read the Events (scheduling failures, image pulls, probe failures).
3. kubectl logs (add --previous for restarts).
4. Services: does kubectl get endpointslices (or endpoints) list pods? Empty endpoints usually mean a selector mismatch or failing readiness.
5. DNS and connectivity from a throwaway debug pod.
6. Node level: systemctl status kubelet and journalctl -u kubelet; for control-plane faults, check the static pod logs.