README.md

kube-prometheus

Note that everything in the contrib/kube-prometheus/ directory is experimental and may change significantly at any time.

This repository collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

The content of this project is written in jsonnet. This project could both be described as a package as well as a library.

Components included in this package:

• The Prometheus Operator
• Highly available Prometheus
• Highly available Alertmanager
• Prometheus node-exporter
• kube-state-metrics
• Grafana

This stack is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the kubernetes-mixin project, similar to this project it provides composable jsonnet as a library for users to customize to their needs.

Table of contents

• Prerequisites
  • minikube
• Quickstart
• Customizing Kube-Prometheus
  • Installing
  • Compiling
  • Containerized Installing and Compiling
• Configuration
• Customization Examples
  • Cluster Creation Tools
  • Internal Registries
  • NodePorts
  • Prometheus Object Name
  • node-exporter DaemonSet namespace
  • Alertmanager configuration
  • Static etcd configuration
  • Pod Anti-Affinity
  • Customizing Prometheus alerting/recording rules and Grafana dashboards
  • Exposing Prometheus/Alermanager/Grafana via Ingress
• Minikube Example
• Troubleshooting
  • Error retrieving kubelet metrics
  • kube-state-metrics resource usage
• Contributing

Prerequisites

You will need a Kubernetes cluster, that's it! By default it is assumed, that the kubelet uses token authentication and authorization, as otherwise Prometheus needs a client certificate, which gives it full access to the kubelet, rather than just the metrics. Token authentication and authorization allows more fine grained and easier access control.

This means the kubelet configuration must contain these flags:

• --authentication-token-webhook=true This flag enables, that a ServiceAccount token can be used to authenticate against the kubelet(s).
• --authorization-mode=Webhook This flag enables, that the kubelet will perform an RBAC request with the API to determine, whether the requesting entity (Prometheus in this case) is allow to access a resource, in specific for this project the /metrics endpoint.

This stack provides resource metrics by deploying the Prometheus Adapter. This adapter is an Extension API Server and Kubernetes needs to be have this feature enabled, otherwise the adapter has no effect, but is still deployed.

minikube

In order to just try out this stack, start minikube with the following command:

$ minikube delete && minikube start --kubernetes-version=v1.10.1 --memory=4096 --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0

Quickstart

This project is intended to be used as a library (i.e. the intent is not for you to create your own modified copy of this repository).

Though for a quickstart a compiled version of the Kubernetes manifests generated with this library (specifically with example.jsonnet) is checked into this repository in order to try the content out quickly. To try out the stack un-customized run: