Andrew Moss 68f1888227 Change streaming terminology to parent-child in the code (#9323) 4 years ago
..
alerta 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
awssns 74fda3de8e Docs: Fix internal links and remove obsolete admonitions (#8946) 4 years ago
custom e99692f145 Docs: Standardize links between documentation (#8638) 4 years ago
discord 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
dynatrace 74fda3de8e Docs: Fix internal links and remove obsolete admonitions (#8946) 4 years ago
email 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
flock 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
hangouts 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
irc 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
kavenegar 74fda3de8e Docs: Fix internal links and remove obsolete admonitions (#8946) 4 years ago
matrix ecd9c48238 Add revisions to Matrix doc (#9295) 4 years ago
messagebird 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
pagerduty 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
prowl 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
pushbullet 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
pushover 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
rocketchat 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
slack 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
smstools3 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
syslog 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
telegram 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
twilio 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
web 9342704a41 Bulk add frontmatter to all documentation (#8354) 5 years ago
Makefile.am dc229f7ed7 Support for matrix notifications (#9196) 4 years ago
README.md e99692f145 Docs: Standardize links between documentation (#8638) 4 years ago
alarm-email.sh e76aac74e6 moved related wiki pages into the repo (#4428) 6 years ago
alarm-notify.sh.in 68f1888227 Change streaming terminology to parent-child in the code (#9323) 4 years ago
alarm-test.sh e76aac74e6 moved related wiki pages into the repo (#4428) 6 years ago
health_alarm_notify.conf 68f1888227 Change streaming terminology to parent-child in the code (#9323) 4 years ago
health_email_recipients.conf e76aac74e6 moved related wiki pages into the repo (#4428) 6 years ago

README.md

Netdata alarm notifications

The exec line in health configuration defines an external script that will be called once the alarm is triggered. The default script is alarm-notify.sh.

You can change the default script globally by editing /etc/netdata/netdata.conf.

alarm-notify.sh is capable of sending notifications:

  • to multiple recipients
  • using multiple notification methods
  • filtering severity per recipient

It uses roles. For example sysadmin, webmaster, dba, etc.

Each alarm is assigned to one or more roles, using the to line of the alarm configuration. Then alarm-notify.sh uses its own configuration file /etc/netdata/health_alarm_notify.conf. To edit it on your system, run /etc/netdata/edit-config health_alarm_notify.conf and find the destination address of the notification for each method.

Each role may have one or more destinations.

So, for example the sysadmin role may send:

  1. emails to admin1@example.com and admin2@example.com
  2. pushover.net notifications to USERTOKENS A, B and C.
  3. pushbullet.com push notifications to admin1@example.com and admin2@example.com
  4. messages to slack.com channel #alarms and #systems.
  5. messages to Discord channels #alarms and #systems.

Configuration

Edit /etc/netdata/health_alarm_notify.conf by running /etc/netdata/edit-config health_alarm_notify.conf:

  • settings per notification method:

    all notification methods except email, require some configuration (i.e. API keys, tokens, destination rooms, channels, etc).

  • recipients per role per notification method

    grep sysadmin /etc/netdata/health_alarm_notify.conf
    
    role_recipients_email[sysadmin]="${DEFAULT_RECIPIENT_EMAIL}"
    role_recipients_pushover[sysadmin]="${DEFAULT_RECIPIENT_PUSHOVER}"
    role_recipients_pushbullet[sysadmin]="${DEFAULT_RECIPIENT_PUSHBULLET}"
    role_recipients_telegram[sysadmin]="${DEFAULT_RECIPIENT_TELEGRAM}"
    role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"
    ...
    

Testing Notifications

You can run the following command by hand, to test alarms configuration:

# become user netdata
su -s /bin/bash netdata

# enable debugging info on the console
export NETDATA_ALARM_NOTIFY_DEBUG=1

# send test alarms to sysadmin
/usr/libexec/netdata/plugins.d/alarm-notify.sh test

# send test alarms to any role
/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"

Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under /usr/lib/netdata). You can always find the location of the alarm-notify.sh script in netdata.conf.

If you need to dig even deeper, you can trace the execution with bash -x. Note that in test mode, alarm-notify.sh calls itself with many more arguments. So first do

bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test

Then look in the output for the alarm-notify.sh calls and run the one you want to trace with bash -x. [analytics](<>)