How the notification pipeline works
Every Nagios email is the end of a chain, and debugging is easiest when you know the links:
- A check fails, then keeps failing through its retries until the object enters a hard problem state. Soft states never notify — that's by design, and it's the number-one source of "why didn't I get an email?" confusion. (Refresher: soft vs. hard states.)
- Nagios evaluates notification logic: are notifications enabled globally and on the object? Does the state match
notification_options? Is the current time insidenotification_period? Is the object acknowledged or in downtime? - Each eligible contact is matched against their own notification options and timeperiod.
- The contact's notification command runs — for email, a shell pipeline that formats a message and hands it to the local mail command.
- Your MTA (Postfix, or a relay like msmtp) actually delivers it.
(The authoritative reference for every setting below is the official Nagios Core notifications documentation.)
Step 1: a working mail path
Before touching Nagios config, prove the server can send mail as the nagios user:
sudo -u nagios /bin/mail -s "Nagios mail test" you@example.com <<< "It works."
If that doesn't arrive, no amount of Nagios configuration will help. On a modern network you'll usually relay through an authenticated smarthost (your provider's SMTP, or a transactional mail service) rather than delivering straight to the internet — configure Postfix's relayhost or use msmtp as a lightweight sendmail replacement, and check the spam folder before blaming the config.
Step 2: contacts
A minimal contact and contact group in contacts.cfg:
define contact {
contact_name oncall1
alias On-Call Admin
email you@example.com
host_notification_period 24x7
service_notification_period 24x7
host_notification_options d,u,r
service_notification_options w,c,r
host_notification_commands notify-host-by-email
service_notification_commands notify-service-by-email
}
define contactgroup {
contactgroup_name admins
alias Nagios Administrators
members oncall1
}
The notification_options letters decide which transitions email you: d down, u unreachable (hosts) / unknown (services), w warning, c critical, r recovery, f flapping, s scheduled downtime, n none. Then reference the contact group from your hosts and services (typically via a template's contact_groups admins).
Step 3: the notification command
Nagios Core's sample config ships working notify-host-by-email and notify-service-by-email commands in commands.cfg — a printf pipeline into /bin/mail expanding macros like $HOSTNAME$, $SERVICEDESC$, $SERVICESTATE$, and $SERVICEOUTPUT$. Two upgrades worth making:
- Put the state and object in the subject line.
** PROBLEM: web01/HTTP is CRITICAL **lets you triage from the lock screen without opening the message. - Include the plugin output. The output line usually contains the actual error; an email without it just sends you hunting.
Finally, make sure the master switch is on in nagios.cfg: enable_notifications=1, and that hosts/services have notifications_enabled 1 (the default).
Tuning out the noise
An inbox you've learned to ignore is worse than no alerting. The levers, in order of impact:
- Notify on what's actionable. If nobody will act on Warnings at 3 a.m., don't send them at 3 a.m. — use a business-hours
notification_periodfor warnings and 24×7 for criticals (via two service templates, or per-contact timeperiods). notification_interval— minutes between repeat notifications while a problem persists.0means "tell me once." Repeats every 15 minutes for a known, acknowledged issue are pure noise — which is what acknowledgements are for.first_notification_delay— wait N minutes after a hard failure before the first email; self-healing blips never page anyone.- Retries before hard state —
max_check_attemptsandretry_intervaldecide how long a problem must persist before it's real. Flapping checks should be fixed or have flap detection do its job, not fill inboxes. - Scheduled downtime for planned work — suppress notifications for the maintenance window instead of ignoring your phone. You can schedule downtime from NagMon in a few taps.
Troubleshooting "Nagios isn't sending emails"
Work down the pipeline — nagios.log tells you exactly how far a notification got:
grep 'NOTIFICATION' /usr/local/nagios/var/nagios.log | tail
Nagios isn't sending any emails — no NOTIFICATION entries in the log at all
Nagios decided not to notify; the mail system was never involved. Check, in order: is the state hard yet (soft states never notify)? Is enable_notifications=1 in nagios.cfg? Does the object have notifications_enabled? Do its notification_options include this state? Is the current time inside the notification_period? Is the object acknowledged or in scheduled downtime?
Notifications are logged but the email never arrives
Nagios did its job; the command or mail path failed. Re-run the mail test as the nagios user, check the MTA log (/var/log/mail.log) for deferred or bounced deliveries, and check spam filtering on the receiving end — monitoring emails trip content filters more often than you'd think.
Only some contacts receive the emails
Contact-level filters are the culprit: compare each contact's own notification_options and notification timeperiods. They filter independently of the object's settings — a contact with service_notification_options c,r will never hear about Warnings no matter what the service allows.
From email to fixed
The email is the starting gun, not the finish line. A workflow that works well for on-call admins: the notification lands on your phone with state and host in the subject; you open a mobile client like NagMon to see the blast radius — is it one service, or is the whole rack unreachable? — read the live plugin output, and acknowledge so repeat notifications stop and colleagues see it's handled. Then fix the thing.
Triage Nagios alerts from your pocket
NagMon shows live scope, plugin output, and lets you acknowledge or schedule downtime — the perfect follow-through to a well-tuned email alert.