# Monitor any process in real-time with Netdata Netdata is more than a multitude of generic system-level metrics and visualizations. Instead of providing only a bird's eye view of your system, leaving you to wonder exactly _what_ is taking up 99% CPU, Netdata also gives you visibility into _every layer_ of your node. These additional layers give you context, and meaningful insights, into the true health and performance of your infrastructure. One of these layers is the _process_. Every time a Linux system runs a program, it creates an independent process that executes the program's instructions in parallel with anything else happening on the system. Linux systems track the state and resource utilization of processes using the [`/proc` filesystem](https://en.wikipedia.org/wiki/Procfs), and Netdata is designed to hook into those metrics to create meaningful visualizations out of the box. While there are a lot of existing command-line tools for tracking processes on Linux systems, such as `ps` or `top`, only Netdata provides dozens of real-time charts, at both per-second and event frequency, without you having to write SQL queries or know a bunch of arbitrary command-line flags. With Netdata's process monitoring, you can: - Benchmark/optimize performance of standard applications, like web servers or databases - Benchmark/optimize performance of custom applications - Troubleshoot CPU/memory/disk utilization issues (why is my system's CPU spiking right now?) - Perform granular capacity planning based on the specific needs of your infrastructure - Search for leaking file descriptors - Investigate zombie processes ... and much more. Let's get started. ## Prerequisites - One or more Linux nodes running [Netdata](/packaging/installer/README.md) - A general understanding of how to [configure the Netdata Agent](/docs/netdata-agent/configuration/README.md) using `edit-config`. - A Netdata Cloud account. [Sign up](https://app.netdata.cloud) if you don't have one already. ## How does Netdata do process monitoring? The Netdata Agent already knows to look for hundreds of [standard applications that we support via collectors](/src/collectors/COLLECTORS.md), and groups them based on their purpose. Let's say you want to monitor a MySQL database using its process. The Netdata Agent already knows to look for processes with the string `mysqld` in their name, along with a few others, and puts them into the `sql` group. This `sql` group then becomes a dimension in all process-specific charts. The process and groups settings are used by two unique and powerful collectors. [**`apps.plugin`**](/src/collectors/apps.plugin/README.md) looks at the Linux process tree every second, much like `top` or `ps fax`, and collects resource utilization information on every running process. It then automatically adds a layer of meaningful visualization on top of these metrics, and creates per-process/application charts. [**`ebpf.plugin`**](/src/collectors/ebpf.plugin/README.md): Netdata's extended Berkeley Packet Filter (eBPF) collector monitors Linux kernel-level metrics for file descriptors, virtual filesystem IO, and process management, and then hands process-specific metrics over to `apps.plugin` for visualization. The eBPF collector also collects and visualizes metrics on an _event frequency_, which means it captures every kernel interaction, and not just the volume of interaction at every second in time. That's even more precise than Netdata's standard per-second granularity. ### Per-process metrics and charts in Netdata With these collectors working in parallel, Netdata visualizes the following per-second metrics for _any_ process on your Linux systems: - CPU utilization (`apps.cpu`) - Total CPU usage - User/system CPU usage (`apps.cpu_user`/`apps.cpu_system`) - Disk I/O - Physical reads/writes (`apps.preads`/`apps.pwrites`) - Logical reads/writes (`apps.lreads`/`apps.lwrites`) - Open unique files (if a file is found open multiple times, it is counted just once, `apps.files`) - Memory - Real Memory Used (non-shared, `apps.mem`) - Virtual Memory Allocated (`apps.vmem`) - Minor page faults (i.e. memory activity, `apps.minor_faults`) - Processes - Threads running (`apps.threads`) - Processes running (`apps.processes`) - Carried over uptime (since the last Netdata Agent restart, `apps.uptime`) - Minimum uptime (`apps.uptime_min`) - Average uptime (`apps.uptime_average`) - Maximum uptime (`apps.uptime_max`) - Pipes open (`apps.pipes`) - Swap memory - Swap memory used (`apps.swap`) - Major page faults (i.e. swap activity, `apps.major_faults`) - Network - Sockets open (`apps.sockets`) - eBPF file - Number of calls to open files. (`apps.file_open`) - Number of files closed. (`apps.file_closed`) - Number of calls to open files that returned errors. - Number of calls to close files that returned errors. - eBPF syscall - Number of calls to delete files. (`apps.file_deleted`) - Number of calls to `vfs_write`. (`apps.vfs_write_call`) - Number of calls to `vfs_read`. (`apps.vfs_read_call`) - Number of bytes written with `vfs_write`. (`apps.vfs_write_bytes`) - Number of bytes read with `vfs_read`. (`apps.vfs_read_bytes`) - Number of calls to write a file that returned errors. - Number of calls to read a file that returned errors. - eBPF process - Number of process created with `do_fork`. (`apps.process_create`) - Number of threads created with `do_fork` or `__x86_64_sys_clone`, depending on your system's kernel version. (`apps.thread_create`) - Number of times that a process called `do_exit`. (`apps.task_close`) - eBPF net - Number of bytes sent. (`apps.bandwidth_sent`) - Number of bytes received. (`apps.bandwidth_recv`) As an example, here's the per-process CPU utilization chart, including a `sql` group/dimension.  ## Configure the Netdata Agent to recognize a specific process To monitor any process, you need to make sure the Netdata Agent is aware of it. As mentioned above, the Agent is already aware of hundreds of processes, and collects metrics from them automatically. But, if you want to change the grouping behavior, add an application that isn't yet supported in the Netdata Agent, or monitor a custom application, you need to edit the `apps_groups.conf` configuration file. Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md) and use `edit-config` to edit the file. ```bash cd /etc/netdata # Replace this with your Netdata config directory if not at /etc/netdata. sudo ./edit-config apps_groups.conf ``` Inside the file are lists of process names, oftentimes using wildcards (`*`), that the Netdata Agent looks for and groups together. For example, the Netdata Agent looks for processes starting with `mysqld`, `mariad`, `postgres`, and others, and groups them into `sql`. That makes sense, since all these processes are for SQL databases. ```text sql: mysqld* mariad* postgres* postmaster* oracle_* ora_* sqlservr ``` These groups are then reflected as [dimensions](/src/web/README.md#dimensions) within Netdata's charts.  See the following two sections for details based on your needs. If you don't need to configure `apps_groups.conf`, jump down to [visualizing process metrics](#visualize-process-metrics). ### Standard applications (web servers, databases, containers, and more) As explained above, the Netdata Agent is already aware of most standard applications you run on Linux nodes, and you shouldn't need to configure it to discover them. However, if you're using multiple applications that the Netdata Agent groups together you may want to separate them for more precise monitoring. If you're not running any other types of SQL databases on that node, you don't need to change the grouping, since you know that any MySQL is the only process contributing to the `sql` group. Let's say you're using both MySQL and PostgreSQL databases on a single node, and want to monitor their processes independently. Open the `apps_groups.conf` file as explained in the [section above](#configure-the-netdata-agent-to-recognize-a-specific-process) and scroll down until you find the `database servers` section. Create new groups for MySQL and PostgreSQL, and move their process queries into the unique groups. ```text # ----------------------------------------------------------------------------- # database servers mysql: mysqld* postgres: postgres* sql: mariad* postmaster* oracle_* ora_* sqlservr ``` Restart Netdata with `sudo systemctl restart netdata`, or the appropriate method for your system, to start collecting utilization metrics from your application. Time to [visualize your process metrics](#visualize-process-metrics). ### Custom applications Let's assume you have an application that runs on the process `custom-app`. To monitor eBPF metrics for that application separate from any others, you need to create a new group in `apps_groups.conf` and associate that process name with it. Open the `apps_groups.conf` file as explained in the [section above](#configure-the-netdata-agent-to-recognize-a-specific-process). Scroll down to `# NETDATA processes accounting`. Above that, paste in the following text, which creates a new `custom-app` group with the `custom-app` process. Replace `custom-app` with the name of your application's Linux process. `apps_groups.conf` should now look like this: ```text ... # ----------------------------------------------------------------------------- # Custom applications to monitor with apps.plugin and ebpf.plugin custom-app: custom-app # ----------------------------------------------------------------------------- # NETDATA processes accounting ... ``` Restart Netdata with `sudo systemctl restart netdata`, or the appropriate method for your system, to start collecting utilization metrics from your application. ## Visualize process metrics Now that you're collecting metrics for your process, you'll want to visualize them using Netdata's real-time, interactive charts. Find these visualizations in the same section regardless of whether you use [Netdata Cloud](https://app.netdata.cloud) for infrastructure monitoring, or single-node monitoring with the local Agent's dashboard at `http://localhost:19999`. If you need a refresher on all the available per-process charts, see the [above list](#per-process-metrics-and-charts-in-netdata). ### Using Netdata's application collector (`apps.plugin`) `apps.plugin` puts all of its charts under the **Applications** section of any Netdata dashboard.  Let's continue with the MySQL example. We can create a [test database](https://www.digitalocean.com/community/tutorials/how-to-measure-mysql-query-performance-with-mysqlslap) in MySQL to generate load on the `mysql` process. `apps.plugin` immediately collects and visualizes this activity `apps.cpu` chart, which shows an increase in CPU utilization from the `sql` group. There is a parallel increase in `apps.pwrites`, which visualizes writes to disk.   Next, the `mysqlslap` utility queries the database to provide some benchmarking load on the MySQL database. It won't look exactly like a production database executing lots of user queries, but it gives you an idea into the possibility of these visualizations. ```bash sudo mysqlslap --user=sysadmin --password --host=localhost --concurrency=50 --iterations=10 --create-schema=employees --query="SELECT * FROM dept_emp;" --verbose ``` The following per-process disk utilization charts show spikes under the `sql` group at the same time `mysqlslap` was run numerous times, with slightly different concurrency and query options.  > 💡 Click on any dimension below a chart in Netdata Cloud (or to the right of a chart on a local Agent dashboard), to > visualize only that dimension. This can be particularly useful in process monitoring to separate one process' > utilization from the rest of the system. ### Using Netdata's eBPF collector (`ebpf.plugin`) Netdata's eBPF collector puts its charts in two places. Of most importance to process monitoring are the **ebpf file**, **ebpf syscall**, **ebpf process**, and **ebpf net** sub-sections under **Applications**, shown in the above screenshot. For example, running the above workload shows the entire "story" how MySQL interacts with the Linux kernel to open processes/threads to handle a large number of SQL queries, then subsequently close the tasks as each query returns the relevant data.  `ebpf.plugin` visualizes additional eBPF metrics, which are system-wide and not per-process, under the **eBPF** section.