Ilya Mashchenko c6cce6bd2a Disable python sudo modules by default (#4477) 6 years ago
..
adaptec_raid c6cce6bd2a Disable python sudo modules by default (#4477) 6 years ago
apache d802f83e4a Fix make dist (#4401) 6 years ago
beanstalk d802f83e4a Fix make dist (#4401) 6 years ago
bind_rndc d802f83e4a Fix make dist (#4401) 6 years ago
boinc d802f83e4a Fix make dist (#4401) 6 years ago
ceph d802f83e4a Fix make dist (#4401) 6 years ago
chrony d802f83e4a Fix make dist (#4401) 6 years ago
couchdb d802f83e4a Fix make dist (#4401) 6 years ago
cpufreq d802f83e4a Fix make dist (#4401) 6 years ago
cpuidle d802f83e4a Fix make dist (#4401) 6 years ago
dns_query_time d802f83e4a Fix make dist (#4401) 6 years ago
dnsdist d802f83e4a Fix make dist (#4401) 6 years ago
dockerd d802f83e4a Fix make dist (#4401) 6 years ago
dovecot d802f83e4a Fix make dist (#4401) 6 years ago
elasticsearch d802f83e4a Fix make dist (#4401) 6 years ago
example d802f83e4a Fix make dist (#4401) 6 years ago
exim d802f83e4a Fix make dist (#4401) 6 years ago
fail2ban d802f83e4a Fix make dist (#4401) 6 years ago
freeradius d802f83e4a Fix make dist (#4401) 6 years ago
go_expvar 8cff4f255e fixed documentation links (#4418) 6 years ago
haproxy d802f83e4a Fix make dist (#4401) 6 years ago
hddtemp d802f83e4a Fix make dist (#4401) 6 years ago
httpcheck d802f83e4a Fix make dist (#4401) 6 years ago
icecast 868d95de17 icecast fix (#4448) 6 years ago
ipfs d802f83e4a Fix make dist (#4401) 6 years ago
isc_dhcpd d802f83e4a Fix make dist (#4401) 6 years ago
linux_power_supply d802f83e4a Fix make dist (#4401) 6 years ago
litespeed d802f83e4a Fix make dist (#4401) 6 years ago
logind d802f83e4a Fix make dist (#4401) 6 years ago
mdstat d802f83e4a Fix make dist (#4401) 6 years ago
megacli c6cce6bd2a Disable python sudo modules by default (#4477) 6 years ago
memcached d802f83e4a Fix make dist (#4401) 6 years ago
mongodb 2c580e1c9c mongodb: add min pymongo package version (#4449) 6 years ago
monit d802f83e4a Fix make dist (#4401) 6 years ago
mysql d802f83e4a Fix make dist (#4401) 6 years ago
nginx d802f83e4a Fix make dist (#4401) 6 years ago
nginx_plus d802f83e4a Fix make dist (#4401) 6 years ago
nsd d802f83e4a Fix make dist (#4401) 6 years ago
ntpd d802f83e4a Fix make dist (#4401) 6 years ago
ovpn_status_log d802f83e4a Fix make dist (#4401) 6 years ago
phpfpm d802f83e4a Fix make dist (#4401) 6 years ago
portcheck d47ca7429d remove unused variables (#4437) 6 years ago
postfix d802f83e4a Fix make dist (#4401) 6 years ago
postgres d47ca7429d remove unused variables (#4437) 6 years ago
powerdns d802f83e4a Fix make dist (#4401) 6 years ago
proxysql 9f681f10af Add ProxySQL python plugin (#4112) 6 years ago
puppet d802f83e4a Fix make dist (#4401) 6 years ago
python_modules d47ca7429d remove unused variables (#4437) 6 years ago
rabbitmq d802f83e4a Fix make dist (#4401) 6 years ago
redis d802f83e4a Fix make dist (#4401) 6 years ago
rethinkdbs d802f83e4a Fix make dist (#4401) 6 years ago
retroshare d802f83e4a Fix make dist (#4401) 6 years ago
samba c6cce6bd2a Disable python sudo modules by default (#4477) 6 years ago
sensors 37dc871a22 fixed README.md links (#4433) 6 years ago
smartd_log d802f83e4a Fix make dist (#4401) 6 years ago
spigotmc d802f83e4a Fix make dist (#4401) 6 years ago
springboot 37dc871a22 fixed README.md links (#4433) 6 years ago
squid d802f83e4a Fix make dist (#4401) 6 years ago
tomcat d802f83e4a Fix make dist (#4401) 6 years ago
traefik d802f83e4a Fix make dist (#4401) 6 years ago
unbound d802f83e4a Fix make dist (#4401) 6 years ago
uwsgi ad3ff17ac1 Added uwsgi plugin (#4404) 6 years ago
varnish d802f83e4a Fix make dist (#4401) 6 years ago
w1sensor 37dc871a22 fixed README.md links (#4433) 6 years ago
web_log d802f83e4a Fix make dist (#4401) 6 years ago
Makefile.am b85833f081 adaptec_raid python module (#4429) 6 years ago
README.md 8fbf817ef8 modularized all source code (#4391) 6 years ago
python.d.conf d946615d1a Remove duplicated entry and put modules in order in python.d.conf (#4460) 6 years ago
python.d.plugin.in 4f84553396 Fix linters in CI (#4399) 6 years ago

README.md

python.d.plugin

python.d.plugin is a netdata external plugin. It is an orchestrator for data collection modules written in python.

  1. It runs as an independent process ps fax shows it
  2. It is started and stopped automatically by netdata
  3. It communicates with netdata via a unidirectional pipe (sending data to the netdata daemon)
  4. Supports any number of data collection modules
  5. Allows each module to have one or more data collection jobs
  6. Each job is collecting one or more metrics from a single data source

Disclaimer

Every module should be compatible with python2 and python3. All third party libraries should be installed system-wide or in python_modules directory. Module configurations are written in YAML and pyYAML is required.

Every configuration file must have one of two formats:

  • Configuration for only one job:

    update_every : 2 # update frequency
    retries      : 1 # how many failures in update() is tolerated
    priority     : 20000 # where it is shown on dashboard
    
    other_var1   : bla  # variables passed to module
    other_var2   : alb
    
  • Configuration for many jobs (ex. mysql):

    # module defaults:
    update_every : 2
    retries      : 1
    priority     : 20000
    
    local:  # job name
    update_every : 5 # job update frequency
    other_var1   : some_val # module specific variable
    
    other_job:
    priority     : 5 # job position on dashboard
    retries      : 20 # job retries
    other_var2   : val # module specific variable
    

update_every, retries, and priority are always optional.


How to write a new module

Writing new python module is simple. You just need to remember to include 5 major things:

  • ORDER global list
  • CHART global dictionary
  • Service class
  • _get_data method
  • all code needs to be compatible with Python 2 (≥ 2.7) and 3 (≥ 3.1)

If you plan to submit the module in a PR, make sure and go through the PR checklist for new modules beforehand to make sure you have updated all the files you need to.

Global variables ORDER and CHART

ORDER list should contain the order of chart ids. Example:

ORDER = ['first_chart', 'second_chart', 'third_chart']

CHART dictionary is a little bit trickier. It should contain the chart definition in following format:

CHART = {
    id: {
        'options': [name, title, units, family, context, charttype],
        'lines': [
            [unique_dimension_name, name, algorithm, multiplier, divisor]
        ]}

All names are better explained in the External Plugins section. Parameters like priority and update_every are handled by python.d.plugin.

Service class

Every module needs to implement its own Service class. This class should inherit from one of the framework classes:

  • SimpleService
  • UrlService
  • SocketService
  • LogService
  • ExecutableService

Also it needs to invoke the parent class constructor in a specific way as well as assign global variables to class variables.

Simple example:

from base import UrlService
class Service(UrlService):
    def __init__(self, configuration=None, name=None):
        UrlService.__init__(self, configuration=configuration, name=name)
        self.order = ORDER
        self.definitions = CHARTS

_get_data collector/parser

This method should grab raw data from _get_raw_data, parse it, and return a dictionary where keys are unique dimension names or None if no data is collected.

Example:

def _get_data(self):
    try:
        raw = self._get_raw_data().split(" ")
        return {'active': int(raw[2])}
    except (ValueError, AttributeError):
        return None

More about framework classes

Every framework class has some user-configurable variables which are specific to this particular class. Those variables should have default values initialized in the child class constructor.

If module needs some additional user-configurable variable, it can be accessed from the self.configuration list and assigned in constructor or custom check method. Example:

def __init__(self, configuration=None, name=None):
    UrlService.__init__(self, configuration=configuration, name=name)
    try:
        self.baseurl = str(self.configuration['baseurl'])
    except (KeyError, TypeError):
        self.baseurl = "http://localhost:5001"

Classes implement _get_raw_data which should be used to grab raw data. This method usually returns a list of strings.

SimpleService

This is last resort class, if a new module cannot be written by using other framework class this one can be used.

Example: mysql, sensors

It is the lowest-level class which implements most of module logic, like:

  • threading
  • handling run times
  • chart formatting
  • logging
  • chart creation and updating

LogService

Examples: apache_cache, nginx_log

Variable from config file: log_path.

Object created from this class reads new lines from file specified in log_path variable. It will check if file exists and is readable. Also _get_raw_data returns list of strings where each string is one line from file specified in log_path.

ExecutableService

Examples: exim, postfix

Variable from config file: command.

This allows to execute a shell command in a secure way. It will check for invalid characters in command variable and won't proceed if there is one of:

  • '&'
  • '|'
  • ';'
  • '>'
  • '<'

For additional security it uses python subprocess.Popen (without shell=True option) to execute command. Command can be specified with absolute or relative name. When using relative name, it will try to find command in PATH environment variable as well as in /sbin and /usr/sbin.

_get_raw_data returns list of decoded lines returned by command.

UrlService

Examples: apache, nginx, tomcat

Variables from config file: url, user, pass.

If data is grabbed by accessing service via HTTP protocol, this class can be used. It can handle HTTP Basic Auth when specified with user and pass credentials.

_get_raw_data returns list of utf-8 decoded strings (lines).

SocketService

Examples: dovecot, redis

Variables from config file: unix_socket, host, port, request.

Object will try execute request using either unix_socket or TCP/IP socket with combination of host and port. This can access unix sockets with SOCK_STREAM or SOCK_DGRAM protocols and TCP/IP sockets in version 4 and 6 with SOCK_STREAM setting.

Sockets are accessed in non-blocking mode with 15 second timeout.

After every execution of _get_raw_data socket is closed, to prevent this module needs to set _keep_alive variable to True and implement custom _check_raw_data method.

_check_raw_data should take raw data and return True if all data is received otherwise it should return False. Also it should do it in fast and efficient way.