netdata/health/notifications/pagerduty

PagerDuty

PagerDuty is an enterprise incident resolution service that integrates with ITOps and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into incidents, PagerDuty streamlines the incident management process by reducing alert noise and resolution times. You can send notifications to PagerDuty using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

Setup

Prerequisites

• An installation of the PagerDuty agent on the node running the Netdata Agent
• A PagerDuty Generic API service using either the Events API v2 or Events API v1
• Add a new service to PagerDuty. Click Use our API directly and select either Events API v2 or Events API v1. Once you finish creating the service, click on the Integrations tab to find your Integration Key.
• Access to the terminal where Netdata Agent is running

Configuration

File

The configuration file name for this integration is health_alarm_notify.conf.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config health_alarm_notify.conf

Options

The following options can be defined for this notification

Config Options

| Name | Description | Default | Required | |:----|:-----------|:-------|:--------:| | SEND_PD | Set `SEND_PD` to YES | YES | yes | | DEFAULT_RECIPIENT_PD | Set `DEFAULT_RECIPIENT_PD` to the PagerDuty service key you want the alert notifications to be sent to. You can define multiple service keys like this: `pd_service_key_1` `pd_service_key_2`. | | yes | ##### DEFAULT_RECIPIENT_PD All roles will default to this variable if left unconfigured. The `DEFAULT_RECIPIENT_PD` can be edited in the following entries at the bottom of the same file: ```conf role_recipients_pd[sysadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa" role_recipients_pd[domainadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxb" role_recipients_pd[dba]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc" role_recipients_pd[webmaster]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxd" role_recipients_pd[proxyadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxe" role_recipients_pd[sitemgr]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxf" ```

Examples

Basic Configuration

#------------------------------------------------------------------------------
# pagerduty.com notification options

SEND_PD="YES"
DEFAULT_RECIPIENT_PD="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
USE_PD_VERSION="2"

Troubleshooting

Test Notification

You can run the following command by hand, to test alerts configuration:

# become user netdata
sudo su -s /bin/bash netdata

# enable debugging info on the console
export NETDATA_ALARM_NOTIFY_DEBUG=1

# send test alarms to sysadmin
/usr/libexec/netdata/plugins.d/alarm-notify.sh test

# send test alarms to any role
/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"

Note that this will test all alert mechanisms for the selected role.