From 817bd1da972610caa0d822100cd4782d9b19db16 Mon Sep 17 00:00:00 2001 From: Frederic Branczyk <fbranczyk@gmail.com> Date: Mon, 24 Jul 2017 15:25:31 +0200 Subject: [PATCH] kube-prometheus/docs: add docs on how to modify assets --- Makefile | 3 +++ docs/developing-alerts-and-dashboards.md | 27 ++++++++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 Makefile create mode 100644 docs/developing-alerts-and-dashboards.md diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..04bd205a --- /dev/null +++ b/Makefile @@ -0,0 +1,3 @@ +generate: + @echo ">> Compiling assets and generating Kubernetes manifests" + @hack/scripts/generate-manifests.sh diff --git a/docs/developing-alerts-and-dashboards.md b/docs/developing-alerts-and-dashboards.md new file mode 100644 index 00000000..80630940 --- /dev/null +++ b/docs/developing-alerts-and-dashboards.md @@ -0,0 +1,27 @@ +# Developing Alerts and Dashboards + +`kube-prometheus` ships with a set of default alerting rules and dashboards. At some point one might like to extend them. This document is intended to explain the workflow of how additional alerting rules and dashboards could be added. + +For both, the Prometheus alerting rules as well as the Grafana dashboards, there are Kubernetes `ConfigMap`s, that are generated from content in the `assets/` directory. + +The source of truth for the alerts and dashboards are the files in the `assets/` directory. The respective files have to be changed there and then the `make generate` make target is executed to re-generate the Kubernetes manifests. + +## Alerts + +The `ConfigMap` that is generated and holds the alerting rule files can be found in `manifests/prometheus/prometheus-k8s-rules.yaml`. + +It is generated by taking all the `*.rules` files in the `assets/prometheus/rules/` directory and generate the `ConfigMap`. + +To extend the alerting rules simply add a new `.rules` file into the `assets/prometheus/rules/` directory and re-generate the manifests. To modify the existing rules, simply edit the respective `.rules` file and re-generate the manifest. + +Then the generated manifest can be applied against a Kubernetes cluster. + +## Dashboards + +The `ConfigMap` that is generated and holds the dashboard definitions can be found in `manifests/grafana/grafana-dashboards.yaml`. + +As Grafana's support for applying dashboards from files is limited a sidecar (called "grafana-watcher") was implemented. It watches the dashboard definitions provided through the `ConfigMap` and ensures that Grafana's SQLite database is in sync with the dashboard definitions. + +To edit/create a dashboard login to Grafana and modify and save the dashboard. Then download the dashboard definition in Grafana through `Share` -> `Export` -> `Save to file`. Move the file to `assets/grafana/` and re-generate the manifests. + +Then the generated manifest can be applied against a Kubernetes cluster. -- GitLab