From 7d693b543363f4e820241f84a952e99e9965a8ea Mon Sep 17 00:00:00 2001 From: Arve Knudsen <arve.knudsen@gmail.com> Date: Fri, 10 Nov 2017 12:03:55 +0100 Subject: [PATCH] kube-prometheus: Update development docs --- README.md | 4 ++- docs/developing-alerts-and-dashboards.md | 32 ++++++++++++++---------- 2 files changed, 22 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index a47d8f41..c8236bf5 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,9 @@ As such, in order to make changes to the dashboard bundle, you need to change th files in assets/grafana, eventually add your own, and then run `make generate` in the kube-prometheus root directory. -To read more in depth about developing dashboards, read the [Developing alerts and dashboards](docs/developing-alerts-and-dashboards.md) documentation. +To read more in depth about developing dashboards, read the +[Developing Prometheus Rules and Grafana Dashboards](docs/developing-alerts-and-dashboards.md) +documentation. ### Reloading of dashboards diff --git a/docs/developing-alerts-and-dashboards.md b/docs/developing-alerts-and-dashboards.md index bfba9f0d..ed3a2a06 100644 --- a/docs/developing-alerts-and-dashboards.md +++ b/docs/developing-alerts-and-dashboards.md @@ -1,35 +1,41 @@ -# Developing Alerts and Dashboards +# Developing Prometheus Rules and Grafana Dashboards -`kube-prometheus` ships with a set of default alerting rules and dashboards. At some point one might like to extend them. This document is intended to explain the workflow of how additional alerting rules and dashboards could be added. +`kube-prometheus` ships with a set of default [Prometheus rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [Grafana](http://grafana.com/) dashboards. At some point one might like to extend them, the purpose of this document is to explain how to do this. -For both, the Prometheus alerting rules as well as the Grafana dashboards, there are Kubernetes `ConfigMap`s, that are generated from content in the `assets/` directory. +For both the Prometheus rules and the Grafana dashboards there are Kubernetes `ConfigMap`s, that are generated from content in the `assets/` directory. The source of truth for the alerts and dashboards are the files in the `assets/` directory. The respective files have to be changed there and then the `make generate` make target is executed to re-generate the Kubernetes manifests. Note: `make generate` should be executed from kube-prometheus base directory. -## Alerts +## Prometheus Rules -The `ConfigMap` that is generated and holds the alerting rule files can be found in `manifests/prometheus/prometheus-k8s-rules.yaml`. +The `ConfigMap` that is generated and holds the Prometheus rule files can be found in `manifests/prometheus/prometheus-k8s-rules.yaml`. -It is generated by taking all the `*.rules` files in the `assets/prometheus/rules/` directory and generate the `ConfigMap`. +It is generated from all the `*.rules.yaml` files in the `assets/prometheus/rules/` directory. -To extend the alerting rules simply add a new `.rules` file into the `assets/prometheus/rules/` directory and re-generate the manifests. To modify the existing rules, simply edit the respective `.rules` file and re-generate the manifest. +To extend the rules simply add a new `.rules.yaml` file into the `assets/prometheus/rules/` directory and re-generate the manifests. To modify the existing rules, simply edit the respective `.rules.yaml` file and re-generate the manifest. Then the generated manifest can be applied against a Kubernetes cluster. ## Dashboards -The `ConfigMap` that is generated and holds the dashboard definitions can be found in `manifests/grafana/grafana-dashboards.yaml`. +The generated `ConfigMap`s holding the Grafana dashboard definitions can be found in `manifests/grafana/grafana-dashboards.yaml`. -As Grafana's support for applying dashboards from files is limited a sidecar (called "grafana-watcher") was implemented. It watches the dashboard definitions provided through the `ConfigMap` and ensures that Grafana's SQLite database is in sync with the dashboard definitions. +The dashboards themselves get generated from Python scripts: assets/grafana/\*.dashboard.py. +These scripts are loaded by the [grafanalib](https://github.com/aknuds1/grafanalib) +Grafana dashboard generator, which turns them into dashboards. -To edit/create a dashboard login to Grafana and modify and save the dashboard. Then download the dashboard definition in Grafana through `Share` -> `Export` -> `Save to file`. Move the file to `assets/grafana/` and re-generate the manifests (executing `make generate` from kube-prometheus base directory). +Bear in mind that we are for now using a fork of grafanalib as we needed to make extensive +changes to it, in order to be able to generate our dashboards. We are hoping to be able to +consolidate our version with the original. -Note: The dashboard json file to be copied in `assets/grafana/` should be suffixed with `-dashboard.json`, otherwise it won't be processed by `make generate`. +After changing grafanalib scripts in assets/grafana, or adding your own, you'll have to run +`make generate` in the kube-prometheus root directory in order to re-generate the dashboards +manifest. You can deploy the latter with kubectl similar to the following: -Then the generated manifest can be applied against a Kubernetes cluster with something like: ``` kubectl -n monitoring apply -f manifests/grafana/grafana-dashboards.yaml ``` -That will update the ConfigMap `grafana-dashboards`. Change should be automatically detected by grafana-watcher and dashboards reloaded. + +This should cause Grafana to re-load its dashboards automatically. -- GitLab