netdata/health/guides/haproxy/haproxy_backend_status.md

Understand the alert

This alert monitors the average number of failed HAProxy backends over the last 10 seconds. If you receive this alert in a critical state, it means that one or more HAProxy backends are inaccessible or offline.

HAProxy is a reverse-proxy that provides high availability, load balancing, and proxying for TCP and HTTP-based applications. A backend in HAProxy is a set of servers that receive forwarded requests and are defined in the backend section of the configuration.

Troubleshoot the alert

• Check the HAProxy configuration file for errors

Making changes in the configuration file may introduce errors. Always validate the correctness of the configuration file. In most Linux distros, you can run the following check:

  haproxy -c -f /etc/haproxy/haproxy.cfg

• Check the HAProxy service for errors

  1. Use journalctl and inspect the log:

     journalctl -u haproxy.service  --reverse
     

• Check the HAProxy log

  1. By default, HAProxy logs under /var/log/haproxy.log:

     cat /var/log/haproxy.log | grep 'emerg\|alert\|crit\|err\|warning\|notice'
     

  You can also search for log messages with info and debug tags.

• Investigate the backend servers

  1. Verify that the backend servers are online and accepting connections.
  2. Check the backend server logs for any errors or issues.
  3. Ensure that firewall rules or security groups are not blocking traffic from HAProxy to the backend servers.

• Review the HAProxy load balancing algorithm and configuration

  1. Analyze the load balancing algorithm used in the configuration to ensure it is suitable for your setup.
  2. Check for any misconfigurations, such as incorrect server addresses, ports, or weights.

Useful resources

1. The Four Essential Sections of an HAProxy Configuration
2. HAProxy Explained in DigitalOcean