diff --git a/README.md b/README.md index 6dbc5629acdb09b00f7bad5174a2c93d398cfeab..82cb7a800b1be635c0e681d7458785f6566a0294 100644 --- a/README.md +++ b/README.md @@ -79,6 +79,8 @@ sidecar container aims to emulate the behavior, by keeping the Grafana database with the provided ConfigMap. Hence, the Grafana pod is effectively stateless. This allows managing dashboards via `git` etc. and easily deploying them via CD pipelines. +For information about how to update/handle the dashboards check [Developing alerts and dashboards](docs/developing-alerts-and-dashboards.md) doc. + In the future, a separate Grafana operator will support gathering dashboards from multiple ConfigMaps based on label selection. diff --git a/docs/developing-alerts-and-dashboards.md b/docs/developing-alerts-and-dashboards.md index 80630940a1fa2fb8bfb19cf66150871c5a5b9ae9..bfba9f0d2c51300d947aeca09872dd10f085c9c5 100644 --- a/docs/developing-alerts-and-dashboards.md +++ b/docs/developing-alerts-and-dashboards.md @@ -6,6 +6,8 @@ For both, the Prometheus alerting rules as well as the Grafana dashboards, there The source of truth for the alerts and dashboards are the files in the `assets/` directory. The respective files have to be changed there and then the `make generate` make target is executed to re-generate the Kubernetes manifests. +Note: `make generate` should be executed from kube-prometheus base directory. + ## Alerts The `ConfigMap` that is generated and holds the alerting rule files can be found in `manifests/prometheus/prometheus-k8s-rules.yaml`. @@ -22,6 +24,12 @@ The `ConfigMap` that is generated and holds the dashboard definitions can be fou As Grafana's support for applying dashboards from files is limited a sidecar (called "grafana-watcher") was implemented. It watches the dashboard definitions provided through the `ConfigMap` and ensures that Grafana's SQLite database is in sync with the dashboard definitions. -To edit/create a dashboard login to Grafana and modify and save the dashboard. Then download the dashboard definition in Grafana through `Share` -> `Export` -> `Save to file`. Move the file to `assets/grafana/` and re-generate the manifests. +To edit/create a dashboard login to Grafana and modify and save the dashboard. Then download the dashboard definition in Grafana through `Share` -> `Export` -> `Save to file`. Move the file to `assets/grafana/` and re-generate the manifests (executing `make generate` from kube-prometheus base directory). -Then the generated manifest can be applied against a Kubernetes cluster. +Note: The dashboard json file to be copied in `assets/grafana/` should be suffixed with `-dashboard.json`, otherwise it won't be processed by `make generate`. + +Then the generated manifest can be applied against a Kubernetes cluster with something like: +``` +kubectl -n monitoring apply -f manifests/grafana/grafana-dashboards.yaml +``` +That will update the ConfigMap `grafana-dashboards`. Change should be automatically detected by grafana-watcher and dashboards reloaded.