how-streaming-works.mdx 5.8 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899
  1. ---
  2. title: "How metrics streaming works"
  3. description: >-
  4. "Netdata's real-time streaming allows you to replicate metrics data
  5. across multiple nodes, or centralize all your metrics data into a single
  6. time-series database (TSDB)."
  7. type: "explanation"
  8. custom_edit_url: "https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/how-streaming-works.mdx"
  9. sidebar_label: "How metrics streaming works"
  10. learn_status: "Published"
  11. learn_topic_type: "Concepts"
  12. learn_rel_path: "Concepts"
  13. ---
  14. # How metrics streaming works
  15. Each node running Netdata can stream the metrics it collects, in real time, to another node. Streaming allows you to
  16. replicate metrics data across multiple nodes, or centralize all your metrics data into a single time-series database
  17. (TSDB).
  18. When one node streams metrics to another, the node receiving metrics can visualize them on the
  19. [dashboard](https://github.com/netdata/netdata/blob/master/docs/visualize/interact-dashboards-charts.md), run health checks to [trigger
  20. alarms](https://github.com/netdata/netdata/blob/master/docs/monitor/view-active-alarms.md) and [send notifications](https://github.com/netdata/netdata/blob/master/docs/monitor/enable-notifications.md), and
  21. [export](https://github.com/netdata/netdata/blob/master/docs/export/external-databases.md) all metrics to an external TSDB. When Netdata streams metrics to another
  22. Netdata, the receiving one is able to perform everything a Netdata instance is capable of.
  23. Streaming lets you decide exactly how you want to store and maintain metrics data. While we believe Netdata's
  24. [distributed architecture](https://github.com/netdata/netdata/blob/master/docs/store/distributed-data-architecture.md) is ideal for speed and scale, streaming
  25. provides centralization options for those who want to maintain only a single TSDB instance.
  26. ## Streaming basics
  27. There are three types of nodes in Netdata's streaming ecosystem.
  28. - **Parent**: A node, running Netdata, that receives streamed metric data.
  29. - **Child**: A node, running Netdata, that streams metric data to one or more parent.
  30. - **Proxy**: A node, running Netdata, that receives metric data from a child and "forwards" them on to a
  31. separate parent node.
  32. Netdata uses API keys, which are just random GUIDs, to authorize the communication between child and parent nodes. We
  33. recommend using `uuidgen` for generating API keys, which can then be used across any number of streaming connections.
  34. Or, you can generate unique API keys for each parent-child relationship.
  35. Once the parent node authorizes the child's API key, the child can start streaming metrics.
  36. It's important to note that the streaming connection uses TCP, UDP, or Unix sockets, _not HTTP_. To proxy streaming
  37. metrics, you need to use a proxy that tunnels [OSI layer 4-7
  38. traffic](https://en.wikipedia.org/wiki/OSI_model#Layer_4:_Transport_Layer) without interfering with it, such as
  39. [SOCKS](https://en.wikipedia.org/wiki/SOCKS) or Nginx's [TCP/UDP load
  40. balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/).
  41. ## Supported streaming configurations
  42. Netdata supports any combination of parent, child, and proxy nodes that you can imagine. Any node can act as both a
  43. parent, child, or proxy at the same time, sending or receiving streaming metrics from any number of other nodes.
  44. Here are a few example streaming configurations:
  45. - **Headless collector**:
  46. - Child `A`, _without_ a database or web dashboard, streams metrics to parent `B`.
  47. - `A` metrics are only available via the local Agent dashboard for `B`.
  48. - `B` generates alarms for `A`.
  49. - **Replication**:
  50. - Child `A`, _with_ a database and web dashboard, streams metrics to parent `B`.
  51. - `A` metrics are available on both local Agent dashboards, and can be stored with the same or different metrics
  52. retention policies.
  53. - Both `A` and `B` generate alarms.
  54. - **Proxy**:
  55. - Child `A`, _with or without_ a database, sends metrics to proxy `C`, also _with or without_ a database. `C` sends
  56. metrics to parent `B`.
  57. - Any node with a database can generate alarms.
  58. ## Viewing streamed metrics
  59. Parent nodes feature a **Replicated Nodes** section in the left-hand panel, which opens with the hamburger icon
  60. ![Hamburger icon](https://raw.githubusercontent.com/netdata/netdata-ui/master/src/components/icon/assets/hamburger.svg)
  61. in the top navigation. The parent node, plus any child nodes, appear here. Click on any of the hostnames to switch
  62. between parent and child dashboards, all served by the parent's [web server](https://github.com/netdata/netdata/blob/master/web/server/README.md).
  63. ![Switching between
  64. ](https://user-images.githubusercontent.com/1153921/110043346-761ec000-7d04-11eb-8e58-77670ba39161.gif)
  65. Each child dashboard is also available directly at the following URL pattern:
  66. `http://PARENT-NODE:19999/host/CHILD-HOSTNAME`.
  67. ## What's next?
  68. Now that you understand the fundamentals of streaming metrics between nodes, go ahead and [enable
  69. streaming](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/enable-streaming.mdx) using a simple `parent-child` relationship. For all
  70. the details, see the [streaming reference](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/reference-streaming.mdx) doc.
  71. Take your streaming setup even further by [exporting metrics](https://github.com/netdata/netdata/blob/master/docs/export/external-databases.md) to an external TSDB.
  72. ### Related information & further reading
  73. - Streaming
  74. - **[How Netdata streams metrics](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/how-streaming-works.mdx)**
  75. - [Enable streaming between nodes](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/enable-streaming.mdx)
  76. - [Streaming reference](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/reference-streaming.mdx)