Prometheus Monitoring Basics

Architecture

Prometheus is a single server process that pulls metrics over HTTP from targets, stores them in a local time-series database (TSDB), evaluates rules, and answers queries. Around it sit a few optional components:

• Exporters expose metrics for systems that do not speak Prometheus natively (hosts, databases, hardware).
• Client libraries let your own code expose metrics at an HTTP endpoint, usually /metrics.
• Pushgateway accepts pushes from short-lived batch jobs so Prometheus can scrape them later.
• Alertmanager receives firing alerts, then deduplicates, groups, silences and routes them.
• Service discovery finds targets dynamically (Kubernetes, cloud APIs, DNS, files).

Every sample is a timestamp plus a float value, identified by a metric name and a set of labels.

Scraping and exporters

Configuration lives in prometheus.yml. A global scrape interval is set once, then each scrape_config is a job with its targets:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["host1.example.com:9100", "host2.example.com:9100"]

The node exporter is the standard way to expose Linux host metrics (CPU, memory, disk, network) on port 9100. Run it as a service on each host and add it as a target. After editing the config, validate it with promtool check config prometheus.yml, then reload without a restart by sending SIGHUP or, if the server was started with --web.enable-lifecycle, a POST to /-/reload.

Discovery and relabeling

Static lists do not scale. File-based discovery reads targets from JSON or YAML files that automation can rewrite; the platform integrations (Kubernetes, EC2 and others) do the same from an API. Relabeling runs before the scrape and lets you drop targets, rewrite labels or build the address from discovered metadata. metric_relabel_configs applies the same idea to scraped samples, which is the usual way to drop expensive series.

Pushgateway

Use it only for batch jobs that finish before a scrape can happen. It is not a general way to turn Prometheus into a push system: the gateway keeps the last pushed value until it is deleted and hides the health of the job.

Metric types and labels

Type	Behaviour
Counter	Only goes up (resets on restart). Requests served, errors. Query with rate().
Gauge	Goes up and down. Memory in use, queue length.
Histogram	Counts observations into cumulative buckets, plus a sum and a count. Lets you estimate quantiles on the server.
Summary	Computes quantiles in the client. Cannot be aggregated across instances.

Labels turn one metric into many series, for example http_requests_total{method="GET", status="200"}. Every distinct label combination is a separate series, so never put unbounded values (user IDs, full URLs) in labels. That is the main cause of memory blow-ups. Naming conventions help: use base units (seconds, bytes) and a _total suffix for counters.

PromQL essentials

An instant vector is one value per series at a point in time, a range vector is a window of samples per series ([5m]), and a scalar is a plain number. Selectors filter by label with =, !=, =~ and !~.

# per-second request rate over 5 minutes, per job
sum by (job) (rate(http_requests_total[5m]))

# error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# 95th percentile latency from a histogram
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# average latency
rate(http_request_duration_seconds_sum[5m])
  / rate(http_request_duration_seconds_count[5m])

# share of requests faster than one second
sum(rate(http_request_duration_seconds_bucket{le="1"}[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))

Things worth knowing:

• Apply rate() to the counter first, then aggregate. Aggregating first hides counter resets.
• Binary operators match series by identical label sets; use on(), ignoring() and group_left for many-to-one joins.
• offset 1h shifts the evaluation time, and subqueries (expr[30m:1m]) run an instant expression over a range.
• Quantiles from histograms are estimates whose accuracy depends on bucket boundaries.

Recording and alerting rules

Rule files hold groups of rules evaluated every evaluation_interval. Recording rules precompute expensive expressions and store the result as a new series. A common convention is level:metric:operations for the name.

groups:
  - name: http
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - alert: HighErrorRatio
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Error ratio above 5% for {{ $labels.job }}"

An alert is pending until its condition has held for the for duration, then firing. Always set for to avoid flapping on brief spikes. Check files with promtool check rules and unit-test them with promtool test rules.

Alertmanager

Prometheus only decides that an alert fires; Alertmanager decides who hears about it. Its config has a routing tree and a list of receivers:

route:
  receiver: team-default
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: oncall

receivers:
  - name: team-default
    email_configs:
      - to: "team@example.com"
  - name: oncall
    webhook_configs:
      - url: "https://alerts.example.com/hook"

Child routes inherit from their parent and the first matching route wins unless continue: true is set. Grouping bundles related alerts into one notification, inhibition mutes alerts that are implied by a bigger one (a dead node silences its service alerts), and silences mute alerts for planned work. Notification text is customised with Go templates. Keep SMTP credentials and webhook URLs in secret files or environment injection, not in committed config.

Auth and TLS between targets

By default scrapes are plain HTTP and unauthenticated, which is fine only on a trusted network. To harden them:

• Exporters that support the Prometheus web configuration file accept a --web.config.file containing tls_server_config (certificate and key paths) and basic_auth_users (bcrypt-hashed passwords).
• The matching scrape job sets scheme: https, a tls_config with the CA to trust, and basic_auth or an authorization credentials file.

scrape_configs:
  - job_name: node-secure
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/ca.pem
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/node-password
    static_configs:
      - targets: ["node1.example.com:9100"]

Finally, remember the retention trade-off: local storage is bounded by the --storage.tsdb.retention.time flag, so use remote write to a long-term store if you need months of history.