Consul Service Discovery and Mesh

Architecture and agents

Consul is a networking tool for service discovery, health checking, a key/value store and a service mesh. It runs as a cluster within a datacenter, and every node runs the same binary called an agent in one of two modes.

• Server agents hold cluster state, take part in consensus and answer queries. Run three or five so the cluster tolerates failures; an odd number avoids split votes.
• Client agents run on the machines hosting your services. They register local services, run health checks, and forward requests to servers. They are lightweight and largely stateless.

Servers elect a leader using the Raft consensus protocol. Writes go through the leader and are replicated to a quorum (a majority) before being committed. Losing quorum, for example two of three servers, makes the cluster unavailable for writes, so spread servers across failure domains.

Separately, all agents use a gossip protocol (Serf) to track membership and detect failed nodes quickly. Multiple datacenters are linked through WAN federation or cluster peering, with each keeping its own consensus group. Enterprise editions add features such as non-voting servers, redundancy zones, namespaces and automated upgrades.

Running and joining a cluster

A minimal server configuration looks like this (HCL or JSON):

datacenter = "dc1"
data_dir   = "/opt/consul"
server     = true
bootstrap_expect = 3
bind_addr  = "10.0.0.11"
client_addr = "127.0.0.1"
retry_join = ["10.0.0.11", "10.0.0.12", "10.0.0.13"]
ui_config { enabled = true }

Clients omit server and use retry_join to find the cluster. Cloud auto-join can discover members by instance tags instead of fixed addresses. Check membership with consul members, and remove a node cleanly with consul leave or consul force-leave for one that has died. Open the required ports between agents (server RPC, gossip on both TCP and UDP, and the HTTP/HTTPS API), and apply configuration changes with consul reload or a restart.

Service discovery

Services register with their local agent, either through a definition file or the HTTP API. Consumers then find healthy instances by name through DNS or the API, instead of using hard-coded addresses.

service {
  name = "web"
  port = 8080
  tags = ["v1"]
  check {
    http     = "http://localhost:8080/health"
    interval = "10s"
    timeout  = "2s"
  }
}

Load it with consul services register web.hcl or by placing it in the agent's config directory and reloading. Query it like so:

dig @127.0.0.1 -p 8600 web.service.consul
curl 'http://127.0.0.1:8500/v1/health/service/web?passing'

DNS omits instances with critical checks (and the ?passing API filter omits anything not passing), so failed nodes drop out automatically. Tags allow subsets such as v1.web.service.consul. Prepared queries add smarter lookups, including failover to another datacenter when no local instance is healthy.

Health checks

Checks are attached to a service or node and report passing, warning or critical. Supported types include HTTP, TCP, gRPC, script, TTL, and Docker. With a TTL check the application must report in before the time-to-live expires. Use deregister_critical_service_after to clean up instances that stay critical, and avoid expensive script checks at short intervals. List registered services with consul catalog services, and view check status in the UI or through the health API.

Key/value store and watches

The built-in KV store is useful for small pieces of dynamic configuration and coordination, such as feature flags. It is not a general database; values are limited in size.

consul kv put config/web/log_level info
consul kv get config/web/log_level
consul kv get -recurse config/

Watches and blocking queries notify you when a key, service or node list changes. consul-template renders files from KV and service data and runs a command (such as a reload) on change, which is a common way to update a load balancer config. Back up cluster state with consul snapshot save and restore with consul snapshot restore.

Service mesh

Consul's mesh (often called Connect) secures service-to-service traffic with mutual TLS. Each service gets a sidecar proxy (Envoy by default) that handles encryption and identity; the application talks to its local proxy on localhost.

• A built-in or external certificate authority issues each service a short-lived identity certificate and rotates it automatically.
• Intentions declare which services may talk to which, such as allowing web to call api and denying everything else. They are enforced on identity, not IP address.
• Upstreams define which remote services the sidecar exposes locally.

service {
  name = "web"
  port = 8080
  connect {
    sidecar_service {
      proxy {
        upstreams = [{ destination_name = "api", local_bind_port = 9191 }]
      }
    }
  }
}

consul intention create -allow web api
consul connect envoy -sidecar-for web

The consul intention create command is deprecated in recent releases in favour of service-intentions config entries applied with consul config write, but the model is the same. Newer versions add traffic management (routing, splitting, failover) and gateways for crossing datacenters or networks. Where a default-deny stance is wanted, set a default-deny intention so that only explicitly allowed paths work.

ACLs

The ACL system controls who can read or write services, KV paths, nodes and the API. It is off by default; enable it in agent config by setting enabled = true and default_policy = "deny" in the acl block, then bootstrap once with consul acl bootstrap. The command returns a highly privileged management token. Store it safely and use it only to create narrower credentials.

• Policies are sets of rules, for example service "web" { policy = "write" } or key_prefix "config/" { policy = "read" }.
• Roles bundle policies for reuse.
• Tokens carry policies or roles; requests present them in a header or environment variable.
• Service identities are shorthand grants that let a service register itself and, with the mesh, its sidecar.
• The anonymous token applies to requests with no token, so keep it minimal.

consul acl policy create -name web-policy -rules @web.hcl
consul acl role create -name web-role -policy-name web-policy
consul acl token create -description "web agent" -role-name web-role

Agents also need tokens of their own, for example to register themselves and to read the catalog. Use least privilege and rotate tokens periodically.

Encryption: gossip and TLS

Two separate layers protect cluster traffic.

Gossip encryption

A shared symmetric key encrypts membership gossip. Generate one with consul keygen, set it as encrypt in every agent's config, and rotate with the consul keyring commands (install a new key, make it primary, then remove the old). Never commit the key.

TLS for RPC and HTTP

Use certificates from a private CA (consul tls ca create and consul tls cert create help) and configure verify_incoming so agents reject unauthenticated connections, and verify_outgoing so outgoing connections use TLS and check the server's certificate (recent versions group these settings under a tls block). Set verify_server_hostname so clients only trust servers whose certificate names match the server role, which blocks a rogue client from posing as a server.