diff --git a/README.md b/README.md index c71d0743de665d6550385056536a672ac532901e..0ff7e2017410eebd587e4e06bfbb1dae451d8c22 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,18 @@ This stack is meant for cluster monitoring, so it is pre-configured to collect m * [Customizing Prometheus alerting/recording rules and Grafana dashboards](#customizing-prometheus-alertingrecording-rules-and-grafana-dashboards) * [Exposing Prometheus/Alermanager/Grafana via Ingress](#exposing-prometheusalermanagergrafana-via-ingress) * [Minikube Example](#minikube-example) +* [Troubleshooting](#troubleshooting) + * [Error retrieving kubelet metrics](#error-retrieving-kubelet-metrics) ## Prerequisites You will need a Kubernetes cluster, that's it! By default it is assumed, that the kubelet uses token authN and authZ, as otherwise Prometheus needs a client certificate, which gives it full access to the kubelet, rather than just the metrics. Token authN and authZ allows more fine grained and easier access control. +This means the kubelet configuration must contain these flags: + +* `--authentication-token-webhook=true` This flag enables, that a `ServiceAccount` token can be used to authenticate against the kubelet(s). +* `--authorization-mode=Webhook` This flag enables, that the kubelet will perform an RBAC request with the API to determine, whether the requesting entity (Prometheus in this case) is allow to access a resource, in specific for this project the `/metrics` endpoint. + ### minikube In order to just try out this stack, start minikube with the following command: @@ -269,3 +276,19 @@ local kp = { ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } + { ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } ``` + +## Troubleshooting + +### Error retrieving kubelet metrics + +Should the Prometheus `/targets` page show kubelet targets, but not able to successfully scrape the metrics, then most likely it is a problem with the authentication and authorization setup of the kubelets. + +As described in the [prerequisites](#prerequisites) section, in order to retrieve metrics from the kubelet token authentication and authorization must be enabled. Some Kubernetes setup tools do not enable this by default. + +#### Authentication problem + +The Prometheus `/targets` page will show the kubelet job with the error `403 Unauthorized`, when token authentication is not enabled. Ensure, that the `--authentication-token-webhook=true` flag is enabled on all kubelet configurations. + +#### Authorization problem + +The Prometheus `/targets` page will show the kubelet job with the error `401 Unauthorized`, when token authorization is not enabled. Ensure that the `--authorization-mode=Webhook` flag is enabled on all kubelet configurations.