{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Netdata Anomaly Detection Deepdive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/netdata/netdata/blob/master/ml/notebooks/netdata_anomaly_detection_deepdive.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This notebook will walk through a simplified python based implementation of the C & C++ code in [`netdata/netdata/ml/`](https://github.com/netdata/netdata/tree/master/ml) used to power the [anomaly detection capabilities](https://github.com/netdata/netdata/blob/master/ml/README.md) of the Netdata agent.\n",
    "\n",
    "The main goal here is to help interested users learn more about how the machine learning works under the hood. If you just want to get started by enabling ml on your agent you can check out these [simple configuration steps](https://learn.netdata.cloud/docs/agent/ml#configuration). \n",
    "\n",
    "🚧 **Note**: This functionality is still under active development and considered experimental. Changes might cause the feature to break. We dogfood it internally and among early adopters within the Netdata community to build the feature. If you would like to get involved and help us with some feedback, email us at analytics-ml-team@netdata.cloud or come join us in the [🤖-ml-powered-monitoring](https://discord.gg/4eRSEUpJnc) channel of the Netdata discord. Alternativley, if GitHub is more of your thing, feel free to create a [GitHub discussion](https://github.com/netdata/netdata/discussions?discussions_q=label%3Aarea%2Fml).\n",
    "\n",
    "In this notebook we will:\n",
    "\n",
    "1. [**Get raw data**](#get-raw-data): Pull some recent data from one of our demo agents.\n",
    "2. [**Add some anomalous data**](#add-some-anomalous-data): Be evil and mess up the tail end of the data to make it obviously \"anomalous\".\n",
    "3. [**Lets do some ML!**](#lets-do-some-ml): Implement an unsupervised clustering based approach to anomaly detection.\n",
    "4. [**Lets visualize all this!**](#lets-visualize-all-this): Plot and explore all this visually.\n",
    "5. [**So, how does it _actually_ work?**](#so-how-does-it-actually-work): Dig a little deeper on what's going on under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Imports & Helper Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Uncomment and run the next cell to install [netdata-pandas](https://github.com/netdata/netdata-pandas) which we will use to easily pull data from the [Netdata agent REST API](https://learn.netdata.cloud/docs/agent/web/api) into a nice clean [Pandas](https://pandas.pydata.org/) [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) where it will be easier to work with. \n",
    "\n",
    "Once you have [netdata-pandas](https://github.com/netdata/netdata-pandas) installed you can comment it back out and rerun the cell to clear the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "aL4gm-jUffEx",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# uncomment the line below (when running in google colab) to install the netdata-pandas library, comment it again when done.\n",
    "#!pip install netdata-pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EMZBHjG4mOQh",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from datetime import datetime, timedelta\n",
    "import itertools\n",
    "import random\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import matplotlib.patches as mpatches\n",
    "from sklearn.cluster import KMeans\n",
    "from scipy.spatial.distance import cdist\n",
    "from netdata_pandas.data import get_data\n",
    "\n",
    "# helper functions\n",
    "\n",
    "\n",
    "def preprocess_df(df, lags_n, diffs_n, smooth_n):\n",
    "    \"\"\"Given a pandas dataframe preprocess it to take differences, add smoothing, lags and abs values. \n",
    "    \"\"\"\n",
    "    if diffs_n >= 1:\n",
    "        # take differences\n",
    "        df = df.diff(diffs_n).dropna()\n",
    "    if smooth_n >= 2:\n",
    "        # apply a rolling average to smooth out the data a bit\n",
    "        df = df.rolling(smooth_n).mean().dropna()\n",
    "    if lags_n >= 1:\n",
    "        # for each dimension add a new columns for each of lags_n lags of the differenced and smoothed values for that dimension\n",
    "        df_columns_new = [f'{col}_lag{n}' for n in range(lags_n+1) for col in df.columns]\n",
    "        df = pd.concat([df.shift(n) for n in range(lags_n + 1)], axis=1).dropna()\n",
    "        df.columns = df_columns_new\n",
    "    # sort columns to have lagged values next to each other for clarity when looking at the feature vectors\n",
    "    df = df.reindex(sorted(df.columns), axis=1)\n",
    "    \n",
    "    # take absolute values as last step\n",
    "    df = abs(df)\n",
    "    \n",
    "    return df\n",
    "\n",
    "\n",
    "def add_shading_to_plot(ax, a, b, t, c='y', alpha=0.2):\n",
    "    \"\"\"Helper function to add shading to plot and add legend item.\n",
    "    \"\"\"\n",
    "    plt.axvspan(a, b, color=c, alpha=alpha, lw=0)\n",
    "    handles, labels = ax.get_legend_handles_labels()\n",
    "    patch = mpatches.Patch(color=c, label=t, alpha=alpha)\n",
    "    handles.append(patch) \n",
    "    plt.legend(handles=handles)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inputs & Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A full list of all the anomaly detection configuration parameters, and descriptions of each, can be found in the [configuration](https://github.com/netdata/netdata/blob/master/ml/README.md#configuration) section of the [ml readme](https://github.com/netdata/netdata/blob/master/ml/README.md).\n",
    "\n",
    "Below we will focus on some basic params to decide what data to pull and the main ml params of importance in understanding how it all works.\n",
    "\n",
    "#### training size/scheduling parameters:\n",
    "- `train_every`: How often to train or retrain each model.\n",
    "- `num_samples_to_train`: How much of the recent data to train on, for example 3600 would mean training on the last 1 hour of raw data. The default in the netdata agent currently is 14400, so last 4 hours.\n",
    "\n",
    "#### feature preprocessing related parameters:\n",
    "- `num_samples_to_diff`: This is really just a 1 or 0 flag to turn on or off differencing in the feature preprocessing. It defaults to 1 (to take differences) and generally should be left alone.\n",
    "- `num_samples_to_smooth`: The extent of smoothing (averaging) applied as part of feature preprocessing.\n",
    "- `num_samples_to_lag`: The number of previous values to also include in our feature vector.\n",
    "\n",
    "#### anomaly score related parameters:\n",
    "- `dimension_anomaly_score_threshold`: The threshold on the anomaly score, above which the data it considered anomalous and the [anomaly bit](https://github.com/netdata/netdata/blob/master/ml/README.md#anomaly-bit) is set to 1 (its actually set to 100 in reality but this just to make it behave more like a rate when aggregated in the netdata agent api). By default this is `0.99` which means anything with an anomaly score above 99% is considered anomalous. Decreasing this threshold makes the model more sensitive and will leave to more anomaly bits, increasing it does the opposite.\n",
    "\n",
    "#### model parameters:\n",
    "- `n_clusters_per_dimension`: This is the number of clusters to fit for each model, by default it is set to 2 such that 2 cluster [centroids](https://en.wikipedia.org/wiki/Centroid) will be fit for each model.\n",
    "- `max_iterations`: The maximum number of iterations the fitting of the clusters is allowed to take. In reality the clustering will converge a lot sooner than this.\n",
    "\n",
    "**Note**: There is much more detailed discussion of all there configuration parameters in the [\"Configuration\"](https://github.com/netdata/netdata/blob/master/ml/README.md#configuration) section of the ml readme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "tBUVUpR3fohX"
   },
   "outputs": [],
   "source": [
    "# data params\n",
    "hosts = ['london.my-netdata.io']\n",
    "charts = ['system.cpu']\n",
    "# if want to just focus on a subset of dims, in this case lets just pick one for simplicity\n",
    "dims = ['system.cpu|user'] \n",
    "last_n_hours = 2\n",
    "# based on last_n_hours define the relevant 'before' and 'after' params for the netdata rest api on the agent\n",
    "before = int(datetime.utcnow().timestamp())\n",
    "after = int((datetime.utcnow() - timedelta(hours=last_n_hours)).timestamp())\n",
    "\n",
    "# ml params\n",
    "train_every = 3600\n",
    "num_samples_to_train = 3600\n",
    "num_samples_to_diff = 1\n",
    "num_samples_to_smooth = 3\n",
    "num_samples_to_lag = 5\n",
    "dimension_anomaly_score_threshold = 0.99\n",
    "n_clusters_per_dimension = 2\n",
    "max_iterations = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Get raw data<a id=\"get-raw-data\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will use the `get_data()` function from the [netdata-pandas](https://github.com/netdata/netdata-pandas) library to just pull down our raw data from the agent into a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 660
    },
    "id": "Ypudrfu-fpje",
    "outputId": "b25c7322-03b4-4475-c416-37c3abbe78a4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7200, 1)\n",
      "1647978087 1647985286\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>system.cpu|user</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_idx</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1647978087</th>\n",
       "      <td>1.503759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647978088</th>\n",
       "      <td>0.252525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647978089</th>\n",
       "      <td>0.755668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647978090</th>\n",
       "      <td>0.503778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647978091</th>\n",
       "      <td>0.501253</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            system.cpu|user\n",
       "time_idx                   \n",
       "1647978087         1.503759\n",
       "1647978088         0.252525\n",
       "1647978089         0.755668\n",
       "1647978090         0.503778\n",
       "1647978091         0.501253"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 45608 characters]",
      "text/plain": [
       "<Figure size 1152x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get raw data\n",
    "df = get_data(hosts=hosts, charts=charts, after=after, before=before)\n",
    "\n",
    "# filter df for just the dims if set\n",
    "if len(dims):\n",
    "    df = df[[dim for dim in dims]]\n",
    "\n",
    "# set some variables based on our data\n",
    "df_timestamp_min = df.index.min()\n",
    "df_timestamp_max = df.index.max()\n",
    "\n",
    "# print some info\n",
    "print(df.shape)\n",
    "print(df_timestamp_min, df_timestamp_max)\n",
    "display(df.head())\n",
    "\n",
    "# lets just plot each dimension to have a look at it\n",
    "for col in df.columns:    \n",
    "\n",
    "    # plot dimension, setting index to datetime so its more readable on the plot\n",
    "    df[[col]].set_index(pd.to_datetime(df.index, unit='s')).plot(title=f'Raw Data - {col}', figsize=(16,6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Add some anomalous data<a id=\"add-some-anomalous-data\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we will pick the last `n_tail_anomalous` observations and mess them up in some random but noticeable way. In this case we randomly shuffle the data and then multiply each observation by some integer randomly chosen from `integers_to_pick_randomly`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 405
    },
    "id": "RDuB5PdjOaAX",
    "outputId": "d686cea5-d0a8-4ed4-aa58-64770a063fbb"
   },
   "outputs": [],
   "source": [
    "# size of anomalous data\n",
    "n_tail_anomalous = 500\n",
    "integers_to_pick_randomly = [0,1,5,10]\n",
    "\n",
    "# randomly scramble data and multiply randomly by some numbers to make it anomalous looking\n",
    "anomalous_shape = (n_tail_anomalous, len(df.columns))\n",
    "randomly_scrambled_data = np.random.choice(df.tail(n_tail_anomalous).values.reshape(-1,), anomalous_shape)\n",
    "random_integers = np.random.choice(integers_to_pick_randomly, anomalous_shape)\n",
    "data_anomalous = randomly_scrambled_data * random_integers\n",
    "\n",
    "# create anomalous dataframe\n",
    "df_anomalous = pd.DataFrame(data = data_anomalous, columns = df.columns)\n",
    "# make sure it has the expected index since we don't want to shuffle that\n",
    "df_anomalous.index = df.tail(n_tail_anomalous).index\n",
    "\n",
    "# overwrite last n_tail observations with anomalous data\n",
    "df.update(df_anomalous)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the plot below it should be clear that the light yellow section of the data has been messed with and is now \"anomalous\" or \"strange looking\" in comparison to all the data that comes before it. \n",
    "\n",
    "Our goal now is to create some sort of [anomaly score](https://github.com/netdata/netdata/blob/master/ml/README.md#anomaly-score) that can easily capture this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39496 characters]",
      "text/plain": [
       "<Figure size 1152x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's just plot each dimension now that we have added some anomalous data\n",
    "for col in df.columns:\n",
    "    \n",
    "    ax = df.set_index(pd.to_datetime(df.index, unit='s')).plot(title=f'Anomalous Data Appended - {col}', figsize=(16,6))\n",
    "    add_shading_to_plot(ax, df_timestamp_max - n_tail_anomalous, df_timestamp_max, 'anomalous data')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Lets do some ML!<a id=\"lets-do-some-ml\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "In this notebook we will just use good old [kmeans](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) from [scikit-learn](https://scikit-learn.org/stable/index.html). \n",
    "\n",
    "In reality the Netdata Agent uses the awesome [dlib](https://github.com/davisking/dlib) c++ library and the [`find_clusters_using_kmeans`](http://dlib.net/ml.html#find_clusters_using_kmeans) function along with a few others. You can see the Netdata KMeans code [here](https://github.com/netdata/netdata/blob/master/ml/kmeans/KMeans.cc).\n",
    "\n",
    "The code below:\n",
    "\n",
    "1. Will initialize some empty objects to use during model training and inference.\n",
    "2. Will loop over every observation and run training and inference in a similar way to how the Agent would process each observation.\n",
    "\n",
    "Of course the Agent implementation is a lot more efficient and uses more efficient streaming and buffer based approaches as opposed to the fairly naive implementation below. \n",
    "\n",
    "The idea in this notebook is to make the general approach as readable and understandable as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "W6UL8U04ppmM"
   },
   "outputs": [],
   "source": [
    "# initialize an empty kmeans model for each dimension\n",
    "models = {\n",
    "    dim: {\n",
    "        'model' : KMeans(n_clusters=n_clusters_per_dimension, max_iter=max_iterations),\n",
    "        'fitted': False\n",
    "    } for dim in df.columns\n",
    "}\n",
    "\n",
    "# initialize dictionary for storing anomaly scores for each dim\n",
    "anomaly_scores = {\n",
    "    dim: {\n",
    "        't' : [],\n",
    "        'anomaly_score': []\n",
    "    } for dim in df.columns\n",
    "}\n",
    "\n",
    "# initialize dictionary for storing anomaly bits for each dim\n",
    "anomaly_bits = {\n",
    "    dim: {\n",
    "        't' : [],\n",
    "        'anomaly_bit': []\n",
    "    }\n",
    "    for dim in df.columns\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to just loop over each row of data and produce anomaly scores once we have some trained models and train or retrain periodically as defined by `train_every`. \n",
    "\n",
    "**Note**: The Netdata Agent implementation spreads out the training across each `train_every` window as opposed to trying to train all models in one go like the below implementation. It also avoids some obvious edges cases where there is no need to retrain, for example when the data have not changed at all since last model was trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_wxIeEhGiWYv",
    "outputId": "8fdfad43-917d-42d1-8997-a49daac25b3d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train at t=1647981687, (n=3600, train_after=1647981687, train_before=1647978087)\n"
     ]
    }
   ],
   "source": [
    "# loop over each row of data in dataframe\n",
    "for t, row in df.iterrows():\n",
    "\n",
    "    # get n based on timestamp\n",
    "    n = t - df_timestamp_min\n",
    "\n",
    "    # for each dimension, if we have a fitted model then make predictions\n",
    "    for dim in df.columns:\n",
    "\n",
    "        # if we have a fitted model, get anomaly score\n",
    "        if models[dim]['fitted']:\n",
    "            \n",
    "            #################################\n",
    "            # Inference / Scoring\n",
    "            #################################\n",
    "\n",
    "            # get a buffer of recent data\n",
    "            buffer_size = num_samples_to_diff + num_samples_to_smooth + num_samples_to_lag * 2\n",
    "            df_dim_recent = df[[dim]].loc[(t-buffer_size):t]\n",
    "\n",
    "            # preprocess/featurize recent data\n",
    "            df_dim_recent_preprocessed = preprocess_df(\n",
    "                df_dim_recent,\n",
    "                num_samples_to_lag,\n",
    "                num_samples_to_diff,\n",
    "                num_samples_to_smooth\n",
    "            )\n",
    "\n",
    "            # take most recent feature vector\n",
    "            X = df_dim_recent_preprocessed.tail(1).values\n",
    "            \n",
    "            # get the existing trained cluster centers\n",
    "            cluster_centers = models[dim]['model'].cluster_centers_\n",
    "\n",
    "            # get anomaly score based on the sum of the euclidean distances between the \n",
    "            # feature vector and each cluster centroid\n",
    "            raw_anomaly_score = np.sum(cdist(X, cluster_centers, metric='euclidean'), axis=1)[0]\n",
    "\n",
    "            # normalize anomaly score based on min-max normalization\n",
    "            # https://en.wikipedia.org/wiki/Feature_scaling#Rescaling_(min-max_normalization)\n",
    "            # the idea here is to convert the raw_anomaly_score we just computed into a number on a\n",
    "            # [0, 1] scale such that it behaves more like a percentage. We use the min and max raw scores\n",
    "            # observed during training to achieve this. This would mean that a normalized score of 1 would\n",
    "            # correspond to a distance as big as the biggest distance (most anomalous) observed on the \n",
    "            # training data. So scores that are 99% or higher will tend to be as strange or more strange\n",
    "            # as the most strange 1% observed during training.\n",
    "            \n",
    "            # normalize based on scores observed during training the model\n",
    "            train_raw_anomaly_score_min = models[dim]['train_raw_anomaly_score_min']\n",
    "            train_raw_anomaly_score_max = models[dim]['train_raw_anomaly_score_max']\n",
    "            train_raw_anomaly_score_range = train_raw_anomaly_score_max - train_raw_anomaly_score_min\n",
    "            \n",
    "            # normalize\n",
    "            anomaly_score = (raw_anomaly_score - train_raw_anomaly_score_min) / train_raw_anomaly_score_range\n",
    "            \n",
    "            # The Netdata Agent does not actually store the normalized_anomaly_score since doing so would require more storage\n",
    "            # space for each metric, essentially doubling the amount of metrics that need to be stored. Instead, the Netdata Agent\n",
    "            # makes use of an existing bit (the anomaly bit) in the internal storage representation used by netdata. So if the \n",
    "            # normalized_anomaly_score passed the dimension_anomaly_score_threshold netdata will flip the corresponding anomaly_bit\n",
    "            # from 0 to 1 to signify that the observation the scored feature vector is considered \"anomalous\". \n",
    "            # All without any extra storage overhead required for the Netdata Agent database! Yes it's almost magic :)\n",
    "\n",
    "            # get anomaly bit\n",
    "            anomaly_bit = 100 if anomaly_score >= dimension_anomaly_score_threshold else 0\n",
    "            \n",
    "            # save anomaly score\n",
    "            anomaly_scores[dim]['t'].append(t)\n",
    "            anomaly_scores[dim]['anomaly_score'].append(anomaly_score)\n",
    "\n",
    "            # save anomaly bit\n",
    "            anomaly_bits[dim]['t'].append(t)\n",
    "            anomaly_bits[dim]['anomaly_bit'].append(anomaly_bit)\n",
    "    \n",
    "    # check if the model needs (re)training\n",
    "    if (n >= num_samples_to_train) & (n % train_every == 0):\n",
    "        \n",
    "        #################################\n",
    "        # Train / Re-Train\n",
    "        #################################\n",
    "\n",
    "        train_before = t - num_samples_to_train\n",
    "        train_after = t\n",
    "        print(f'train at t={t}, (n={n}, train_after={train_after}, train_before={train_before})')\n",
    "\n",
    "        # loop over each dimension/model\n",
    "        for dim in df.columns:\n",
    "            \n",
    "            # get training data based on most recent num_samples_to_train\n",
    "            df_dim_train = df[[dim]].loc[(t-num_samples_to_train):t]\n",
    "            \n",
    "            # preprocess/featurize training data\n",
    "            df_dim_train_preprocessed = preprocess_df(\n",
    "                df_dim_train,\n",
    "                num_samples_to_lag,\n",
    "                num_samples_to_diff,\n",
    "                num_samples_to_smooth\n",
    "            )\n",
    "\n",
    "            # fit model using the fit method of kmeans\n",
    "            models[dim]['model'].fit(df_dim_train_preprocessed.values) \n",
    "            models[dim]['fitted'] = True # mark model as fitted\n",
    "            \n",
    "            # get cluster centers of model we just trained\n",
    "            cluster_centers = models[dim]['model'].cluster_centers_\n",
    "\n",
    "            # get training scores, needed to get min and max scores for normalization at inference time\n",
    "            train_raw_anomaly_scores = np.sum(cdist(df_dim_train_preprocessed.values, cluster_centers, metric='euclidean'), axis=1)\n",
    "            # save min and max anomaly score during training, used to normalize all scores to be 0,1 scale\n",
    "            models[dim]['train_raw_anomaly_score_min'] = min(train_raw_anomaly_scores)\n",
    "            models[dim]['train_raw_anomaly_score_max'] = max(train_raw_anomaly_scores)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hard work is now all done. The above cell has processed all the data, trained or retrained models as defined by the initial config, and saved all anomaly scores and anomaly bits.\n",
    "\n",
    "The rest of the notebook will try to help make more sense of all this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "0iN0PCPGiWBx"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>system.cpu|user</th>\n",
       "      <th>system.cpu|user__anomaly_score</th>\n",
       "      <th>system.cpu|user__anomaly_bit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1647981888</th>\n",
       "      <td>0.753769</td>\n",
       "      <td>0.228337</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647984190</th>\n",
       "      <td>0.757576</td>\n",
       "      <td>0.144231</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647983651</th>\n",
       "      <td>0.753769</td>\n",
       "      <td>0.198606</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647982084</th>\n",
       "      <td>0.757576</td>\n",
       "      <td>0.189867</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647983422</th>\n",
       "      <td>1.002506</td>\n",
       "      <td>0.333199</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            system.cpu|user  system.cpu|user__anomaly_score  \\\n",
       "time_idx                                                      \n",
       "1647981888         0.753769                        0.228337   \n",
       "1647984190         0.757576                        0.144231   \n",
       "1647983651         0.753769                        0.198606   \n",
       "1647982084         0.757576                        0.189867   \n",
       "1647983422         1.002506                        0.333199   \n",
       "\n",
       "            system.cpu|user__anomaly_bit  \n",
       "time_idx                                  \n",
       "1647981888                           0.0  \n",
       "1647984190                           0.0  \n",
       "1647983651                           0.0  \n",
       "1647982084                           0.0  \n",
       "1647983422                           0.0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create dataframe of anomaly scores\n",
    "df_anomaly_scores = pd.DataFrame()\n",
    "for dim in anomaly_scores:\n",
    "    df_anomaly_scores_dim = pd.DataFrame(data=zip(anomaly_scores[dim]['t'],anomaly_scores[dim]['anomaly_score']),columns=['time_idx',f'{dim}__anomaly_score']).set_index('time_idx')\n",
    "    df_anomaly_scores = df_anomaly_scores.join(df_anomaly_scores_dim, how='outer')\n",
    "\n",
    "# create dataframe of anomaly bits\n",
    "df_anomaly_bits = pd.DataFrame()\n",
    "for dim in anomaly_bits:\n",
    "    df_anomaly_bits_dim = pd.DataFrame(data=zip(anomaly_bits[dim]['t'],anomaly_bits[dim]['anomaly_bit']),columns=['time_idx',f'{dim}__anomaly_bit']).set_index('time_idx')\n",
    "    df_anomaly_bits = df_anomaly_bits.join(df_anomaly_bits_dim, how='outer')\n",
    "\n",
    "# join anomaly scores to raw df\n",
    "df_final = df.join(df_anomaly_scores, how='outer')\n",
    "\n",
    "# join anomaly bits to raw df\n",
    "df_final = df_final.join(df_anomaly_bits, how='outer')\n",
    "\n",
    "# let's look at a sample of some scored observations\n",
    "display(df_final.tail(num_samples_to_train).sample(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the dataframe above we see that each observation now also has a column with the `__anomaly_score` and one with the `__anomaly_bit`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Lets visualize all this!<a id=\"lets-visualize-all-this\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our raw data, our anomaly scores, and our anomaly bits - we can plot this all side by side to get a clear picture of how it all works together.\n",
    "\n",
    "In the plots below we see that during the light yellow \"anomalous\" period the \"[anomaly scores](https://github.com/netdata/netdata/blob/master/ml/README.md#anomaly-score)\" get elevated to such an extend that many \"[anomaly bits](https://github.com/netdata/netdata/blob/master/ml/README.md#anomaly-bit)\" start flipping from 0 to 1 and essentially \"turning on\" to signal potentially anomalous data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "zVoR1BJ5nCGv",
    "outputId": "ffcc7765-ea39-47c1-da99-ec79647d0871",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43016 characters]",
      "text/plain": [
       "<Figure size 1440x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 33956 characters]",
      "text/plain": [
       "<Figure size 1440x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 17520 characters]",
      "text/plain": [
       "<Figure size 1440x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 63032 characters]",
      "text/plain": [
       "<Figure size 1440x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "figsize = (20,4)\n",
    "\n",
    "for dim in models:\n",
    "\n",
    "    # create a dim with the raw data, anomaly score and anomaly bit for the dim\n",
    "    df_final_dim = df_final[[dim,f'{dim}__anomaly_score',f'{dim}__anomaly_bit']]\n",
    "    \n",
    "    # plot raw data, including the anomalous data\n",
    "    ax = df_final_dim[[dim]].set_index(pd.to_datetime(df_final_dim.index, unit='s')).plot(\n",
    "        title=f'Raw Data (Anomalous Appended) - {dim}', figsize=figsize\n",
    "    )\n",
    "    add_shading_to_plot(ax, df_timestamp_max - n_tail_anomalous, df_timestamp_max, 'Anomalous Data')\n",
    "    \n",
    "    # plat the corresponding anomaly scores\n",
    "    ax = df_final_dim[[f'{dim}__anomaly_score']].set_index(pd.to_datetime(df_final_dim.index, unit='s')).plot(\n",
    "        title=f'Anomaly Score - {dim}', figsize=figsize\n",
    "    )\n",
    "    add_shading_to_plot(ax, df_timestamp_max - n_tail_anomalous, df_timestamp_max, 'Anomalous Data')\n",
    "    \n",
    "    # plot the corresponding anomaly bits\n",
    "    ax = df_final_dim[[f'{dim}__anomaly_bit']].set_index(pd.to_datetime(df_final_dim.index, unit='s')).plot(\n",
    "        title=f'Anomaly Bit - {dim}', figsize=figsize\n",
    "    )\n",
    "    add_shading_to_plot(ax, df_timestamp_max - n_tail_anomalous, df_timestamp_max, 'Anomalous Data')\n",
    "\n",
    "    # finally, plot it all on the same plot (which might not be so easy or clear to read)\n",
    "    df_final_dim_normalized = (df_final_dim-df_final_dim.min())/(df_final_dim.max()-df_final_dim.min())\n",
    "    ax = df_final_dim_normalized.set_index(pd.to_datetime(df_final_dim_normalized.index, unit='s')).plot(\n",
    "        title=f'Combined (Raw, Score, Bit) - {dim}', figsize=figsize\n",
    "    )\n",
    "    add_shading_to_plot(ax, df_timestamp_max - n_tail_anomalous, df_timestamp_max, 'Anomalous Data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last concept to introduce now is the \"[anomaly rate](https://github.com/netdata/netdata/blob/master/ml/README.md#anomaly-rate)\" which is really just an average over \"anomaly bits\".\n",
    "\n",
    "For example, in the next cell we will just average all the anomaly bits across the light yellow window of time to find the anomaly rate for the metric within this window. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_tail_anomalous_anomaly_rate = 96.6%\n",
      "\n",
      "This means the \"anomaly rate\" within the yellow period of anomalous data was 96.6%\n",
      "\n",
      "Another way to think of this is that 96.6% of the observations during the yellow \n",
      "window were considered anomalous based on the latest trained model.\n"
     ]
    }
   ],
   "source": [
    "# average the anomaly bits within the n_tail_anomalous period of the data\n",
    "n_tail_anomalous_anomaly_rate = df_final_dim[[f'{dim}__anomaly_bit']].tail(n_tail_anomalous).mean()[0]\n",
    "\n",
    "print(f'n_tail_anomalous_anomaly_rate = {n_tail_anomalous_anomaly_rate}%')\n",
    "print(f'\\nThis means the \"anomaly rate\" within the yellow period of anomalous data was {n_tail_anomalous_anomaly_rate}%')\n",
    "print(f'\\nAnother way to think of this is that {n_tail_anomalous_anomaly_rate}% of the observations during the yellow \\nwindow were considered anomalous based on the latest trained model.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. So, how does it _actually_ work?<a id=\"so-how-does-it-actually-work\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this final section of the notebook below we will dig in to try understand this a bit more intuitively.\n",
    "\n",
    "First we will \"[featureize](https://brilliant.org/wiki/feature-vector/)\" or \"preprocess\" all the data. Then we will explore what these feature vectors actually are, how they look, and how we derive anomaly scores based on their distance to the models cluster centroids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocess/featurize all data\n",
    "df_preprocessed = preprocess_df(\n",
    "    df,\n",
    "    num_samples_to_lag,\n",
    "    num_samples_to_diff,\n",
    "    num_samples_to_smooth\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have preprocessed all of our data, lets just take a look at it.\n",
    "\n",
    "You will see that we have essentially just added `num_samples_to_lag` additional columns to the dataframe, one for each lag. The numbers themselves also are now longer the original raw metric values, instead they have first been differenced (just take difference of latest value with previous value so that we are working with delta's as opposed to original raw metric) and also smoothed (in this case by just averaging the previous `num_samples_to_smooth` previous differenced values).\n",
    "\n",
    "The idea here is to define the representation that the model will work in. In this case the model will decide if a recent observation is anomalous based on it's corresponding feature vector which is a differenced, smoothed, and lagged array or list of recent values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7192, 6)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>system.cpu|user_lag0</th>\n",
       "      <th>system.cpu|user_lag1</th>\n",
       "      <th>system.cpu|user_lag2</th>\n",
       "      <th>system.cpu|user_lag3</th>\n",
       "      <th>system.cpu|user_lag4</th>\n",
       "      <th>system.cpu|user_lag5</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1647983445</th>\n",
       "      <td>3.330669e-16</td>\n",
       "      <td>0.167293</td>\n",
       "      <td>0.499561</td>\n",
       "      <td>0.167504</td>\n",
       "      <td>0.000633</td>\n",
       "      <td>0.253165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647980613</th>\n",
       "      <td>5.967300e-03</td>\n",
       "      <td>0.000422</td>\n",
       "      <td>0.166665</td>\n",
       "      <td>0.335848</td>\n",
       "      <td>0.083963</td>\n",
       "      <td>0.083542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647984383</th>\n",
       "      <td>2.531518e-01</td>\n",
       "      <td>0.083327</td>\n",
       "      <td>0.001899</td>\n",
       "      <td>0.251886</td>\n",
       "      <td>0.083963</td>\n",
       "      <td>0.082700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647984447</th>\n",
       "      <td>1.696266e-01</td>\n",
       "      <td>0.083542</td>\n",
       "      <td>0.081459</td>\n",
       "      <td>0.082074</td>\n",
       "      <td>0.082280</td>\n",
       "      <td>0.168344</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1647983270</th>\n",
       "      <td>5.101800e-03</td>\n",
       "      <td>0.082498</td>\n",
       "      <td>0.082703</td>\n",
       "      <td>0.004262</td>\n",
       "      <td>0.174051</td>\n",
       "      <td>0.001050</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            system.cpu|user_lag0  system.cpu|user_lag1  system.cpu|user_lag2  \\\n",
       "time_idx                                                                       \n",
       "1647983445          3.330669e-16              0.167293              0.499561   \n",
       "1647980613          5.967300e-03              0.000422              0.166665   \n",
       "1647984383          2.531518e-01              0.083327              0.001899   \n",
       "1647984447          1.696266e-01              0.083542              0.081459   \n",
       "1647983270          5.101800e-03              0.082498              0.082703   \n",
       "\n",
       "            system.cpu|user_lag3  system.cpu|user_lag4  system.cpu|user_lag5  \n",
       "time_idx                                                                      \n",
       "1647983445              0.167504              0.000633              0.253165  \n",
       "1647980613              0.335848              0.083963              0.083542  \n",
       "1647984383              0.251886              0.083963              0.082700  \n",
       "1647984447              0.082074              0.082280              0.168344  \n",
       "1647983270              0.004262              0.174051              0.001050  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df_preprocessed.shape)\n",
    "df_preprocessed.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model works based on these feature vectors. A lot of ML is about training a model to define some [\"compressed representation\"](https://en.wikipedia.org/wiki/Data_compression#Machine_learning) of the training data that can then be useful for new data in some way.\n",
    "\n",
    "This is exactly what our cluster models are trying to do. They process a big bag of preprocessed feature vectors, covering `num_samples_to_train` raw observations, during training to come up with the best, synthetic, `n_clusters_per_dimension` feature vectors as a useful compressed representation of the training data.\n",
    "\n",
    "The cell below will just show you what those `n_clusters_per_dimension` (in this case 2) synthetic (made up by the kemans algo) feature vectors are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>system.cpu|user_lag0</th>\n",
       "      <th>system.cpu|user_lag1</th>\n",
       "      <th>system.cpu|user_lag2</th>\n",
       "      <th>system.cpu|user_lag3</th>\n",
       "      <th>system.cpu|user_lag4</th>\n",
       "      <th>system.cpu|user_lag5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>centroid 0</th>\n",
       "      <td>0.182626</td>\n",
       "      <td>0.169506</td>\n",
       "      <td>0.100484</td>\n",
       "      <td>0.178778</td>\n",
       "      <td>0.177843</td>\n",
       "      <td>0.100711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>centroid 1</th>\n",
       "      <td>0.115532</td>\n",
       "      <td>0.141029</td>\n",
       "      <td>0.276627</td>\n",
       "      <td>0.122611</td>\n",
       "      <td>0.124448</td>\n",
       "      <td>0.276112</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            system.cpu|user_lag0  system.cpu|user_lag1  system.cpu|user_lag2  \\\n",
       "centroid 0              0.182626              0.169506              0.100484   \n",
       "centroid 1              0.115532              0.141029              0.276627   \n",
       "\n",
       "            system.cpu|user_lag3  system.cpu|user_lag4  system.cpu|user_lag5  \n",
       "centroid 0              0.178778              0.177843              0.100711  \n",
       "centroid 1              0.122611              0.124448              0.276112  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# lets pick the first model to look at\n",
    "model = list(models.keys())[0]\n",
    "\n",
    "# get the cluster centroids and put them in a dataframe similar to above\n",
    "df_cluster_centers = pd.DataFrame(models[model]['model'].cluster_centers_, columns=df_preprocessed.columns)\n",
    "df_cluster_centers.index = [f'centroid {i}' for i in df_cluster_centers.index.values]\n",
    "display(df_cluster_centers)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At inference time we can now use our `n_clusters_per_dimension` cluster centers as a sort of set of \"reference\" feature vectors we can compare against. \n",
    "\n",
    "When we see a new feature vector that is very far away from these \"reference\" feature vectors, we can take that as a signal that the recent data the feature vector was derived from may look significantly different than most of the data the clusters where initially train on. And as such it may be \"anomalous\" or \"strange\" in some way that might be meaningful to you are a user trying to monitor and troubleshoot systems based on these metrics.\n",
    "\n",
    "To try make this visually clearer we will take 10 random feature vectors from the first half of our data where things were generally normal and we will also take 10 random feature vectors from the yellow anomalous period of time. Lastly we will also include the cluster centroids themselves to see how they compare to both sets of 10 feature vectors. \n",
    "\n",
    "Basically this is represented in the heatmap below where each row is a processed feature vectors corresponding to some timestamp `t`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 120344 characters]",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# prepare heatmap\n",
    "df_heatmap = pd.concat([df_preprocessed.sample(10),df_preprocessed.tail(n_tail_anomalous).sample(10), df_cluster_centers])\n",
    "df_heatmap = df_heatmap.round(2)\n",
    "\n",
    "# get scores\n",
    "heatmap_scores = np.sum(cdist(df_heatmap, models[dim]['model'].cluster_centers_, metric='euclidean'), axis=1)\n",
    "heatmap_bits = [1 if score >= dimension_anomaly_score_threshold else 0 for score in heatmap_scores]\n",
    "\n",
    "# add anomaly score to index\n",
    "heatmap_index_inputs = list(zip(range(1, len(df_heatmap)+1), df_heatmap.index, heatmap_scores, heatmap_bits))\n",
    "df_heatmap.index = [f'{x[0]}. t={x[1]} (AS={round(x[2]*100)}%, AB={x[3]})' for x in heatmap_index_inputs]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,10))\n",
    "sns.heatmap(df_heatmap, annot=True, ax=ax, cmap='RdYlBu')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see from the above heatmap that the top 10 rows all tend to have lower anomaly scores (AS) and anomaly bits (AB) that are 0. While its the opposite for rows 11-20.\n",
    "\n",
    "The final two rows are the cluster centroids themselves which should look more similar to the first 10 rows, fairly different to rows 11-20. And of course, you would expect that each cluster centroid itself has a low anomaly score and non-anomalous anomaly bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to present this visually is via line or bar plots. Below we just plot each of the rows above as a line. First all at the same time on one plot and then each one individually as a bar plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the plot below the dotted lines correspond to the feature vectors sampled from the yellow anomalous period and as such we expect them to look very different to the solid lines (sampled from the normal period) and the solid circled lines which correspond to the two centroids. (Admittedly its not the clearest of plots to read since the normal lines all bunch together)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 85572 characters]",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "line_styles = ['-' for i in range(10)] + ['--' for i in range(10)] + ['o-' for i in range(2)]\n",
    "df_heatmap.transpose().plot(legend=False, style=line_styles, figsize=(10,5), rot=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also just plot each feature vector itself as a bar plot with one bar for each of our 6 features. We have set the y-axis in the below plots to be fixed such that the differences are more obvious."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 11484 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11440 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11736 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11712 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11600 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11572 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11648 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11752 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11652 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11744 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11800 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11728 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11764 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11884 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11732 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11756 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11900 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11948 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11936 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11808 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11476 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 11340 characters]",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i,row in df_heatmap.iterrows():\n",
    "    ax = row.plot(kind='bar', figsize=(10,3),title=f'{i}', rot=15)\n",
    "    ax.set_ylim(np.min(df_heatmap.values),np.max(df_heatmap.values))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, an alternative way to visualize the distances between the three groups of points from the heatmap would be a series of scatter plots, one for each pair of features. \n",
    "\n",
    "This should give a good intuition for the distance measures (in 6 dimensional feature space, lastest preprocessed observation plus the 5 lagged values) that underpin the raw anomaly score.\n",
    "\n",
    "Generally we would expect to see the blue 'normal' points group closer together and 'near' the green cluster centroids, meanwhile the anomalous orange points should tend to be further away from the other two groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10692 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 10548 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 12068 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 12664 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 13496 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get list of feature pair combinations\n",
    "feature_pairs = list(itertools.combinations(df_heatmap.columns, 2))\n",
    "feature_pairs_sample = random.sample(feature_pairs,5) # just sample 5 pairs\n",
    "\n",
    "# define colors of points\n",
    "point_colors = ['blue' for i in range(10)] + ['orange' for i in range(10)] + ['green' for i in range(2)]\n",
    "\n",
    "# plot each pair of features\n",
    "for feature_pair in feature_pairs_sample:\n",
    "    df_heatmap.plot.scatter(x=feature_pair[0], y=feature_pair[1], color=point_colors, subplots=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You made it! Congratualtions to getting all the way to the end (its ok if you just skipped here too), as a reward [here](https://youtube.com/playlist?list=PL6Zhl9mK2r0Ja7SKX72rD4dfBNnaQbbwR) is a playlist of interesting data science and machine learning youtube videos that you might like considering you just read a jupyter notebook about using kmeans to do unsupervised anomaly detection :)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Netdata Anomaly Detection Deepdive.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
