Where states come from: plugin exit codes
Every state in Nagios starts life as a humble process exit code. A check plugin runs, prints one line of output, and exits with:
| Exit code | Service state | Meaning |
|---|---|---|
0 | OK | Everything within thresholds. |
1 | WARNING | Above the warning threshold, below critical. |
2 | CRITICAL | Above the critical threshold, or the check itself failed hard (timeout, refused connection). |
3 | UNKNOWN | The plugin couldn't determine a state at all. |
That's the entire contract. The plugin's output line — DISK CRITICAL - free space: / 512 MB (2%) — travels with the state, which is why a good client always shows it: the state says that something's wrong, the output says what.
Service states
- OK — nothing to do.
- WARNING — degraded or heading toward trouble. Whether this deserves a 3 a.m. email is a notification-tuning decision.
- CRITICAL — broken or past the critical threshold. Triage these first.
- UNKNOWN — the check failed, not necessarily the service: bad arguments, missing plugin dependency, internal timeout. Treat a burst of UNKNOWNs as a monitoring bug to fix, or you'll learn to ignore them — and an ignored state is a blind spot.
- PENDING — no result yet; normal right after adding a check or restarting Nagios.
Host states
- UP — the host check (usually a ping) succeeds.
- DOWN — the host itself failed its check.
- UNREACHABLE — Nagios can't see the host, because a parent on the network path (defined with the
parentsdirective — a router, switch, or hypervisor) is down. The host itself may be perfectly healthy.
The DOWN/UNREACHABLE distinction is Nagios's network topology awareness, and it's what makes a good outage display possible: one dead switch shows as one DOWN host plus a cluster of UNREACHABLE dependents — one problem, not twenty. This is exactly what the outages view in NagMon surfaces: fix the blocking host and the rest clears itself.
Soft vs. hard states
Nagios doesn't believe the first bad check result. When a state changes, the object enters a SOFT state and Nagios re-checks at retry_interval until max_check_attempts results agree — only then does the object enter a HARD state.
With max_check_attempts 3 and retry_interval 1, a timeline:
10:00:00 check fails → SOFT CRITICAL (attempt 1/3) — no notification
10:01:00 check fails → SOFT CRITICAL (attempt 2/3) — no notification
10:02:00 check fails → HARD CRITICAL (attempt 3/3) — notification sent
10:07:00 check succeeds → HARD OK — recovery notificationTwo consequences worth internalizing:
- Notifications fire only on hard state changes. "The dashboard showed red but I got no email" almost always means the state was still soft — see the notification pipeline.
- Your effective detection time is
check_interval + (max_check_attempts − 1) × retry_interval. Tune per service: aggressive for revenue-critical checks, relaxed for flappy ones.
Acknowledgements, downtime, and flapping
- Acknowledged — a human has claimed the problem; repeat notifications stop and the acknowledgement (with its comment) is visible to everyone. The state is still a problem state — acknowledging documents it, it doesn't fix it. (Acknowledging from your phone →)
- Scheduled downtime — planned maintenance windows suppress problem notifications for their duration, then alerting resumes automatically.
- Flapping — an object bouncing between states faster than anyone can respond. Nagios's flap detection measures the rate of recent state changes and suppresses notifications while it persists. Flapping is always a signal: either the service is genuinely unstable or the threshold sits exactly on the workload's natural variance.
Reading a dashboard: a triage order
- Blocking outages / UNREACHABLE clusters — one fix here clears the most red.
- DOWN hosts — everything on them is implicitly broken; their service alerts are noise until the host is back.
- CRITICAL services on healthy hosts — real, individual problems.
- WARNING and UNKNOWN — schedule them; fix UNKNOWNs before they train you to ignore states.
Skip anything already acknowledged or in downtime — that work is claimed. (This ordering is why NagMon's Dashboard puts current problems in priority order and lets you sort by severity or state duration.)
FAQ
DOWN vs. UNREACHABLE — which do I fix?
Fix DOWN. UNREACHABLE hosts are usually casualties of a down parent and recover on their own once the path is restored.
UNKNOWN vs. CRITICAL — how different are they really?
Completely. CRITICAL: the check worked and the news is bad. UNKNOWN: the check itself failed, so you have no news at all — a monitoring problem to fix, not a service outage to escalate.
Why did I see red in the UI before any email arrived?
The UI (and the JSON API) shows soft states immediately; email waits for the hard state. That head start is one of the best reasons to keep live status in your pocket.
All of these states, at a glance
NagMon puts state counts, priority-ordered problems, and full plugin output on your iPhone — with filters for every state on this page.