netdata/health/notifications/stackpulse

StackPulse agent alert notifications

StackPulse is a software-as-a-service platform for site reliability engineering. It helps SREs, DevOps Engineers and Software Developers reduce toil and alert fatigue while improving reliability of software services by managing, analyzing and automating incident response activities.

Sending Netdata alarm notifications to StackPulse allows you to create smart automated response workflows (StackPulse playbooks) that will help you drive down your MTTD and MTTR by performing any of the following:

• Enriching the incident with data from multiple sources
• Performing triage actions and analyzing their results
• Orchestrating incident management and notification flows
• Performing automatic and semi-automatic remediation actions
• Analyzing incident data and remediation patterns to improve reliability of your services

To send the notification you need:

1. Create a Netdata integration in the StackPulse Administration Portal, and copy the Endpoint URL.

1. On your node, navigate to /etc/netdata/ and run the following command:

   $ ./edit-config health_alarm_notify.conf
   

2. Set the STACKPULSE_WEBHOOK variable to Endpoint URL you copied earlier:

   SEND_STACKPULSE="YES"
   STACKPULSE_WEBHOOK="https://hooks.stackpulse.io/v1/webhooks/YOUR_UNIQUE_ID"
   

3. Now restart Netdata using sudo systemctl restart netdata, or the appropriate method for your system. When your node creates an alarm, you can see the associated notification on your StackPulse Administration Portal

React to alarms with playbooks

StackPulse allow users to create Playbooks giving additional information about events that happen in specific scenarios. For example, you could create a Playbook that responds to a "low disk space" alarm by compressing and cleaning up storage partitions with dynamic data.

Create Playbooks for Netdata alarms

To create a Playbook, you need to access the StackPulse Administration Portal. After the initial setup, you need to access the TRIGGER tab to define the scenarios used to trigger the event. The following variables are available:

• Hostname: The host that generated the event.
• Chart: The name of the chart.
• OldValue : The previous value of the alarm.
• Value: The current value of the alarm.
• Units : The units of the value.
• OldStatus : The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL.
• State: The current alarm status, the acceptable values are the same of OldStatus.
• Alarm : The name of the alarm, as given in Netdata's health.d entries.
• Date : The timestamp this event occurred.
• Duration : The duration in seconds of the previous alarm state.
• NonClearDuration : The total duration in seconds this is/was non-clear.
• Description : A short description of the alarm copied from the alarm definition.
• CalcExpression : The expression that was evaluated to trigger the alarm.
• CalcParamValues : The values of the parameters in the expression, at the time of the evaluation.
• TotalWarnings : Total number of alarms in WARNING state.
• TotalCritical : Total number of alarms in CRITICAL state.
• ID : The unique id of the alarm that generated this event.

For more details how to create a scenario, take a look at the StackPulse documentation.