NodeLoadHigh

warningHosts (node_exporter)

The host's long-term load average has been far above the number of CPUs it has, so work is queuing up.
Severity warning
Source node_exporter 1.x, loadavg and cpu collectors
Key metrics node_load15, node_cpu_seconds_total (used to count CPUs)

What it means

On Linux, load average counts tasks that are running, waiting for a CPU, or in uninterruptible sleep (usually waiting for disk or NFS). The alert compares the 15-minute load to the host's CPU count and fires when load has been a clear multiple of it for a long period, so a short spike will not page you.

High load is a symptom, not a diagnosis. It can mean "not enough CPU" or "everything is stuck waiting on storage", and the fixes are different.

Common causes

  • CPU-bound workload larger than the host: traffic growth, a runaway process, a busy-looping thread.
  • Tasks stuck in D state on a slow disk or hung NFS mount.
  • Noisy neighbours or CPU steal on an oversubscribed VM.
  • Heavy swapping due to memory pressure.
  • Kubernetes node packed with pods that have no CPU limits.

First checks

  1. Compare short and long load to see if it is getting better or worse:
    node_load1{instance="<instance>"}
node_load15{instance="<instance>"}
count by (instance) (node_cpu_seconds_total{mode="idle", instance="<instance>"})
  2. Break CPU time down by mode. High user/system means CPU contention; high iowait means blocked I/O; high steal means the hypervisor:
    sum by (mode) (rate(node_cpu_seconds_total{instance="<instance>", mode!="idle"}[5m]))
  3. On the host, find the top consumers:
    top -b -n1 -o %CPU | head -20
  4. Count processes in uninterruptible sleep and what they wait on:
    ps -eo state,pid,wchan:32,cmd | awk '$1=="D"'
  5. Check run queue and swap activity:
    vmstat 2 5   # r = runnable, b = blocked, si/so = swap

Fixing it

For CPU contention, stop or throttle the runaway process, scale out, or move to a larger instance; on Kubernetes, set CPU requests and limits and rebalance pods. For D-state tasks, fix the underlying storage or remount the hung filesystem. For steal, move the VM or change instance type. For swapping, address memory first.

Related alerts

← all alert runbooks