README.md

kube-prometheus

Note that everything is experimental and may change significantly at any time.

This repository collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

The content of this project is written in jsonnet. This project could both be described as a package as well as a library.

Components included in this package:

• The Prometheus Operator
• Highly available Prometheus
• Highly available Alertmanager
• Prometheus node-exporter
• Prometheus Adapter for Kubernetes Metrics APIs
• kube-state-metrics
• Grafana

This stack is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the kubernetes-mixin project, similar to this project it provides composable jsonnet as a library for users to customize to their needs.

Warning

If you are migrating from release-0.7 branch or earlier please read what changed and how to migrate in our guide.

Table of contents

• kube-prometheus
  • Warning
  • Table of contents
  • Prerequisites
    • minikube
  • Compatibility
    • Kubernetes compatibility matrix
  • Quickstart
    • Access the dashboards
  • Customizing Kube-Prometheus
    • Installing
    • Compiling
    • Apply the kube-prometheus stack
    • Containerized Installing and Compiling
  • Update from upstream project
    • Update jb
    • Update kube-prometheus
    • Compile the manifests and apply
  • Configuration
  • Customization Examples
    • Cluster Creation Tools
    • Internal Registry
    • NodePorts
    • Prometheus Object Name
    • node-exporter DaemonSet namespace
    • Alertmanager configuration
    • Adding additional namespaces to monitor
      • Defining the ServiceMonitor for each additional Namespace
    • Monitoring all namespaces
    • Static etcd configuration
    • Pod Anti-Affinity
    • Stripping container resource limits
    • Customizing Prometheus alerting/recording rules and Grafana dashboards
    • Exposing Prometheus/Alermanager/Grafana via Ingress
    • Setting up a blackbox exporter
  • Minikube Example
  • Continuous Delivery
  • Troubleshooting
    • Error retrieving kubelet metrics
      • Authentication problem
      • Authorization problem
    • kube-state-metrics resource usage
    • Error retrieving kube-proxy metrics
  • Contributing
  • License

Prerequisites

You will need a Kubernetes cluster, that's it! By default it is assumed, that the kubelet uses token authentication and authorization, as otherwise Prometheus needs a client certificate, which gives it full access to the kubelet, rather than just the metrics. Token authentication and authorization allows more fine grained and easier access control.

This means the kubelet configuration must contain these flags:

• --authentication-token-webhook=true This flag enables, that a ServiceAccount token can be used to authenticate against the kubelet(s). This can also be enabled by setting the kubelet configuration value authentication.webhook.enabled to true.
• --authorization-mode=Webhook This flag enables, that the kubelet will perform an RBAC request with the API to determine, whether the requesting entity (Prometheus in this case) is allowed to access a resource, in specific for this project the /metrics endpoint. This can also be enabled by setting the kubelet configuration value authorization.mode to Webhook.

This stack provides resource metrics by deploying the Prometheus Adapter. This adapter is an Extension API Server and Kubernetes needs to be have this feature enabled, otherwise the adapter has no effect, but is still deployed.

minikube

To try out this stack, start minikube with the following command:

$ minikube delete && minikube start --kubernetes-version=v1.20.0 --memory=6g --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0