AlertmanagerClusterDown

criticalAlertmanager self-monitoring

Half or more of the Alertmanager replicas in a cluster are failing their Prometheus scrapes.
Severity critical
Source Prometheus scrape of Alertmanager
Key metric up for the Alertmanager job

What it means

Prometheus scrapes each Alertmanager replica. When at least half of a cluster's instances have been down for most of the recent window, the alert fires. A single replica outage in a three-node cluster does not trigger it; losing a majority does.

With most replicas gone, notification delivery depends on the survivors. If all are down, alerts are queued in Prometheus and nobody is paged. Note that this alert may itself not be delivered, which is why an external dead man's switch matters.

Common causes

  • Pods evicted or stuck Pending (node pressure, insufficient resources, PVC for the data volume not bound).
  • A bad config or image upgrade that prevents startup on every replica.
  • All replicas scheduled on the same node or zone that just failed (no anti-affinity).
  • Scrape-side problems only: changed port name, ServiceMonitor selector mismatch, NetworkPolicy blocking Prometheus.
  • OOM kills during a large alert storm.

First checks

  1. Confirm which instances are down:
    up{job=~".*alertmanager.*"} == 0
  2. Check pod state and recent events:
    kubectl -n monitoring get pods -l app.kubernetes.io/name=alertmanager -o wide
kubectl -n monitoring describe pod <alertmanager-pod> | tail -n 30
  3. Look at logs from the previous container if it restarted:
    kubectl -n monitoring logs <alertmanager-pod> -c alertmanager --previous
  4. If pods look healthy, test the scrape path directly:
    kubectl -n monitoring port-forward <alertmanager-pod> 9093 &
curl -s localhost:9093/-/healthy
    Then check the target's error in Prometheus under Status > Targets.

Fixing it

Restore capacity first: free node resources, fix the PVC, or roll back the config or image that broke startup. If only scraping is broken, fix the ServiceMonitor or NetworkPolicy. Afterwards, add pod anti-affinity or topology spread constraints and a PodDisruptionBudget.

Related alerts

← all alert runbooks