Istio Service Mesh

Why a service mesh

As an application splits into many services, each one needs the same networking features: load balancing, retries, timeouts, encryption, authentication and metrics. Implementing them in every language and library is repetitive and inconsistent. A service mesh places a proxy next to each workload and handles these concerns uniformly, configured centrally and without code changes.

What you gain: fine-grained traffic control (canaries, mirroring), zero-trust security between services, and consistent telemetry. What you pay: extra proxies consuming CPU and memory, added latency per hop, and more operational complexity. A mesh pays off with many services and strict security or rollout requirements, not for a handful of apps.

Architecture: control plane and data plane

• Data plane: Envoy proxies that carry all traffic between services. In the classic sidecar model, an Envoy container is injected into each pod and intercepts inbound and outbound traffic. Istio also offers an ambient mode that replaces sidecars with a per-node proxy (ztunnel) for L4 traffic and optional per-namespace or per-service waypoint proxies for L7 features.
• Control plane: istiod turns your high-level configuration into proxy configuration and pushes it to every Envoy, discovers services, and acts as the certificate authority issuing workload identities.

Typical install and onboarding:

istioctl install --set profile=demo -y
kubectl label namespace shop istio-injection=enabled
kubectl rollout restart deployment -n shop
istioctl analyze -n shop

Injection happens when a pod is created, so existing pods must restart. Use the default profile (or a customised one) in production; the demo profile is meant for evaluation. istioctl analyze finds common configuration errors, and istioctl proxy-status shows whether proxies are in sync with the control plane.

Traffic management

Three resources do most of the work.

Resource	Purpose
Gateway	Configures an ingress or egress proxy at the edge of the mesh: ports, protocols, hosts, TLS
VirtualService	Routing rules: how requests to a host are matched and where they go
DestinationRule	Policy applied after routing: subsets, load balancing, connection pools, TLS mode

A DestinationRule defines named subsets (usually by pod label) and a VirtualService splits traffic between them, which is the basis of a canary release.

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels: {version: v1}
  - name: v2
    labels: {version: v2}
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts: [reviews]
  http:
  - route:
    - destination: {host: reviews, subset: v1}
      weight: 90
    - destination: {host: reviews, subset: v2}
      weight: 10

To expose a service externally, bind a VirtualService to a Gateway through its gateways and hosts fields. Routes can also match on headers, URI prefixes or methods, enabling header-based routing for testing. The Kubernetes Gateway API is a fully supported alternative to Istio's own Gateway resource.

Resilience: timeouts, retries, circuit breaking, faults

• Timeouts are set on a route (timeout: 3s) so slow upstreams fail fast.
• Retries (attempts, perTryTimeout, retryOn) mask transient failures. Retry only idempotent calls and cap attempts to avoid retry storms.
• Circuit breaking lives in a DestinationRule: connection pool limits plus outlierDetection, which ejects instances that return repeated errors.
• Fault injection deliberately adds delays or HTTP errors to a percentage of requests, so you can test how callers behave before a real outage.

trafficPolicy:
  connectionPool:
    http: {http1MaxPendingRequests: 10}
  outlierDetection:
    consecutive5xxErrors: 5
    interval: 30s
    baseEjectionTime: 60s

Security: mTLS and policy

istiod issues each workload a short-lived certificate tied to its identity (based on its service account), and the proxies use it for mutual TLS: traffic between meshed services is encrypted and both ends are authenticated, with no application change. Certificates rotate automatically; you can plug in your own root CA for production.

Authentication

PeerAuthentication sets the mTLS mode: PERMISSIVE accepts plaintext and mTLS (useful during migration), STRICT requires mTLS. RequestAuthentication validates end-user JWTs.

Authorization

AuthorizationPolicy allows or denies requests by source identity, namespace, method, path or JWT claim.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend
  namespace: shop
spec:
  selector:
    matchLabels: {app: orders}
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/shop/sa/frontend"]
    to:
    - operation:
        methods: ["GET"]

Once any ALLOW policy applies to a workload, requests that match none are denied. A common pattern is an empty-spec deny-all policy per namespace, then explicit allows.

Observability

Because every request passes through Envoy, the mesh produces telemetry without instrumentation:

• Metrics at proxy, service and control-plane level: request rate, errors and latency (the golden signals), scraped by Prometheus and charted in Grafana.
• Distributed traces: proxies emit spans, but the application must forward trace headers (such as traceparent or the B3 headers) so spans join into one trace.
• Access logs: per-request records from the proxy, enabled through mesh configuration.
• Kiali: a console that draws the live service graph, shows traffic and health, and validates Istio configuration.

Practical tips

• Adopt gradually: inject one namespace, keep mTLS permissive, verify, then move to strict.
• Set resource requests for sidecars and watch their overhead.
• Debug with istioctl proxy-config to inspect what a given Envoy actually received.
• Keep VirtualService hosts and DestinationRule hosts consistent; a subset referenced but not defined produces 503 errors.