Netdata Logging

This document describes how Netdata generates its own logs, not how Netdata manages and queries logs databases.

Log sources

Netdata supports the following log sources:

daemon, logs generated by Netdata daemon.
collector, logs generated by Netdata collectors, including internal and external ones.
access, API requests received by Netdata
health, all alert transitions and notifications

Log outputs

For each log source, Netdata supports the following output methods:

off, to disable this log source
journal, to send the logs to systemd-journal.
syslog, to send the logs to syslog.
system, to send the output to stderr or stdout depending on the log source.
stdout, to write the logs to Netdata's stdout.
stderr, to write the logs to Netdata's stderr.
filename, to send the logs to a file.

For daemon and collector the default is journal when systemd-journal is available. To decide if systemd-journal is available, Netdata checks:

stderr is connected to systemd-journald
/run/systemd/journal/socket exists
/host/run/systemd/journal/socket exists (/host is configurable in containers)

If any of the above is detected, Netdata will select journal for daemon and collector sources.

All other sources default to a file.

Log formats

Format	Description
journal	journald-specific log format. Automatically selected when logging to systemd-journal.
logfmt	logs data as a series of key/value pairs. The default when logging to any output other than `journal`.
json	logs data in JSON format.

Log levels

Each time Netdata logs, it assigns a priority to the log. It can be one of this (in order of importance):

Level	Description
emergency	a fatal condition, Netdata will most likely exit immediately after.
alert	a very important issue that may affect how Netdata operates.
critical	a very important issue the user should know which, Netdata thinks it can survive.
error	an error condition indicating that Netdata is trying to do something, but it fails.
warning	something unexpected has happened that may or may not affect the operation of Netdata.
notice	something that does not affect the operation of Netdata, but the user should notice.
info	the default log level about information the user should know.
debug	these are more verbose logs that can be ignored.

Logs Configuration

In netdata.conf, there are the following settings:

[logs]
	# logs to trigger flood protection = 1000
	# logs flood protection period = 60
	# facility = daemon
	# level = info
	# daemon = journal
	# collector = journal
	# access = /var/log/netdata/access.log
	# health = /var/log/netdata/health.log

logs to trigger flood protection and logs flood protection period enable logs flood protection for daemon and collector sources. It can also be configured per log source.
facility is used only when Netdata logs to syslog.
level defines the minimum log level of logs that will be logged. This setting is applied only to daemon and collector sources. It can also be configured per source.

Configuring log sources

Each for the sources (daemon, collector, access, health), accepts the following:

source = {FORMAT},level={LEVEL},protection={LOG}/{PERIOD}@{OUTPUT}

Where:

{FORMAT}, is one of the log formats,
{LEVEL}, is the minimum log level to be logged,
{LOGS} is the number of logs to trigger flood protection configured per output,
{PERIOD} is the equivalent of logs flood protection period configured per output,
{OUTPUT} is one of the `log outputs,

All parameters can be omitted, except {OUTPUT}. If {OUTPUT} is the only given parameter, @ can be omitted.

Logs rotation

Netdata comes with logrotate configuration to rotate its log files periodically.

The default is usually found in /etc/logrotate.d/netdata.

Sending a SIGHUP to Netdata, will instruct it to re-open all its log files.

Log Fields

All fields exposed by Netdata

Message IDs

Netdata assigns specific message IDs to certain events:

ed4cdb8f1beb4ad3b57cb3cae2d162fa when a Netdata child connects to this Netdata
6e2e3839067648968b646045dbf28d66 when this Netdata connects to a Netdata parent
9ce0cb58ab8b44df82c4bf1ad9ee22de when alerts change state
6db0018e83e34320ae2a659d78019fb7 when notifications are sent

You can view these events using the Netdata systemd-journal.plugin at the MESSAGE_ID filter, or using journalctl like this:

# query children connection
journalctl MESSAGE_ID=ed4cdb8f1beb4ad3b57cb3cae2d162fa

# query parent connection
journalctl MESSAGE_ID=6e2e3839067648968b646045dbf28d66

# query alert transitions
journalctl MESSAGE_ID=9ce0cb58ab8b44df82c4bf1ad9ee22de

# query alert notifications
journalctl MESSAGE_ID=6db0018e83e34320ae2a659d78019fb7

Using journalctl to query Netdata logs

The Netdata service's processes execute within the netdata journal namespace. To view the Netdata logs, you should specify the --namespace=netdata option.

# Netdata logs since the last time the service was started
journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata

# All netdata logs, the oldest entries are displayed first  
journalctl -u netdata --namespace=netdata

# All netdata logs, the newest entries are displayed first  
journalctl -u netdata --namespace=netdata -r

README.md 19 KB История Исходник