Kubernetes Networking, Scaling and Scheduling

Services

Pods are ephemeral and their IPs change. A Service gives a set of pods a stable virtual IP and DNS name, selecting them by label. The matching pod IPs are tracked as endpoints, and only ready pods receive traffic.

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

Inside the cluster, clients use web (same namespace) or web.<namespace>.svc.cluster.local. For HTTP routing by host and path, an Ingress (or the newer Gateway API) sits in front of Services and needs a controller installed; TLS is terminated using a certificate stored in a Secret. A Service with an empty endpoint list almost always means the selector does not match the pod labels.

Pod-to-pod networking

Kubernetes requires that every pod can reach every other pod by IP without NAT, and that nodes can reach pods. It does not implement this itself. A CNI plugin (Calico, Cilium, Flannel and others) assigns pod IPs and wires them across nodes, using overlays such as VXLAN or plain routing.

• Containers in one pod share a network namespace and talk over localhost.
• On one node, pods connect through virtual ethernet pairs and a bridge or routed interface.
• Across nodes, the CNI carries traffic via encapsulation or routes.
• kube-proxy programs packet-forwarding rules (iptables, nftables or IPVS) on each node to translate Service IPs into pod IPs.

By default all traffic is allowed. NetworkPolicies restrict it, but only if the chosen CNI enforces them.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend
spec:
  podSelector:
    matchLabels: {app: api}
  ingress:
    - from:
        - podSelector:
            matchLabels: {app: frontend}

ConfigMaps and Secrets

Keep configuration out of images. A ConfigMap holds non-sensitive key-value data or whole files; a Secret has the same shape for sensitive values. Both can be consumed three ways: as environment variables, as command arguments through variable substitution, or as files in a mounted volume.

kubectl create configmap app-config --from-literal=LOG_LEVEL=info --from-file=app.conf
kubectl create secret generic db --from-literal=password=changeme

env:
  - name: LOG_LEVEL
    valueFrom:
      configMapKeyRef: {name: app-config, key: LOG_LEVEL}
volumes:
  - name: conf
    configMap: {name: app-config}

Pitfalls: environment variables are fixed at container start, whereas mounted files are updated eventually (not when mounted with subPath). Secrets are only base64-encoded by default, not encrypted, so enable encryption at rest, restrict read access with RBAC, and consider an external secret manager. The Downward API can expose pod metadata (name, namespace, labels) to the container as environment variables or files.

Autoscaling

The Horizontal Pod Autoscaler adjusts the replica count of a Deployment (or similar) to hold a target metric. It needs the Metrics Server for resource metrics, and pods must declare CPU requests, because utilisation is measured against them.

kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=10

The rough formula is desired replicas = current replicas x (current metric / target metric), rounded up. The autoscaling/v2 API adds memory, custom metrics per pod, object metrics (such as requests per second on an Ingress) and external metrics, plus behavior settings to slow scale-down and avoid flapping. Do not combine an HPA with a manually managed replica count in the manifest.

The cluster autoscaler works at node level. It adds nodes when pods are unschedulable because of resource shortage, and removes nodes that stay underused when their pods can be moved. PodDisruptionBudgets limit how many replicas can be evicted at once during scale-down or drains. The Vertical Pod Autoscaler adjusts requests rather than replicas.

Advanced scheduling

Placement rules go beyond simple selectors:

• Taints and tolerations reserve nodes (GPU nodes, dedicated pools). The NoExecute effect also evicts, and tolerationSeconds sets how long a pod survives on a failing node.
• Node affinity attracts pods to nodes by label, with required or preferred rules and operators like In, NotIn and Exists.
• Pod affinity co-locates pods (an app next to its cache); pod anti-affinity spreads them apart, using a topologyKey such as the hostname or zone label.
• Topology spread constraints keep replicas evenly distributed across zones or nodes.
• Priority classes let important pods preempt less important ones.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels: {app: web}

Resource requests also determine the pod QoS class: Guaranteed (requests equal limits), Burstable, or BestEffort, which is evicted first under pressure.

API server and node security

Every client of the API server is a user or a ServiceAccount, authorised by RBAC. Give each workload its own ServiceAccount, grant narrow Roles, and disable token auto-mounting where the pod does not call the API (automountServiceAccountToken: false). Inside a pod, the API is reachable at the in-cluster service address using the mounted token and CA.

For nodes and the network:

• Avoid hostNetwork, hostPID, host path mounts and privileged containers. If a pod needs a host port, use hostPort rather than the whole host network.
• Apply Pod Security Standards (restricted or baseline) per namespace.
• Run containers as non-root, drop Linux capabilities, and use read-only root filesystems.
• Never expose the kubelet or etcd ports publicly, and keep the API server behind a firewall or private endpoint.

Best practices

• Always set requests and limits, and add LimitRanges and ResourceQuotas per namespace.
• Use small, pinned image tags (not latest) and one process per container.
• Label everything consistently (app, version, env) and use namespaces to separate teams.
• Keep manifests in version control and apply them through automation.
• Run at least two replicas for important services, spread across zones, with PodDisruptionBudgets.
• Log to stdout and stderr and let a cluster-wide agent collect them.

Extending Kubernetes

Custom Resource Definitions add new object kinds to the API, stored and served like built-ins. A controller watches those objects and reconciles real state toward the declared state; the pairing of CRD plus controller is the operator pattern, used for databases, certificates and more. A minimal CRD declares a group, names and a schema.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.example.com
spec:
  group: example.com
  scope: Namespaced
  names: {kind: Backup, plural: backups}
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          x-kubernetes-preserve-unknown-fields: true

Other extension points include CNI for networking, CSI for storage, device plugins for GPUs, webhooks for validation and mutation, and custom schedulers. Prefer these official hooks over forking Kubernetes.