jsonnet: drop some of prometheus-adapter metrics

The current implementation of prometheus-adapter exposes a lot of
metrics about the health of its aggregated apiserver. The issue is that
the some of these metrics are not very useful in the context of
prometheus-adapter, and we currently can't avoid exposing them since
they are registered to the Kubernetes global Prometheus registry. Until
this is improved in upstream Kubernetes, we could benefit from dropping
some of the metrics that are not very useful.

Before this change, in a default kube-prometheus installation, we would
have 800+ series for prometheus-adapter against 400+, so we divided the
number of series by two will focusing on the most valuable metrics for
prometheus-adapter.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

jsonnet/kube-prometheus/components/prometheus-adapter.libsonnet

@@ -172,6 +172,21 @@ function(params) {
             insecureSkipVerify: true,
           },
           bearerTokenFile: '/var/run/secrets/kubernetes.io/serviceaccount/token',
+          metricRelabelings: [
+            {
+              sourceLabels: ['__name__'],
+              action: 'drop',
+              regex: '(' + std.join('|',
+                                    [
+                                      'apiserver_client_certificate_.*',  // The only client supposed to connect to the aggregated API is the apiserver so it is not really meaningful to monitor its certificate.
+                                      'apiserver_envelope_.*',  // Prometheus-adapter isn't using envelope for storage.
+                                      'apiserver_flowcontrol_.*',  // Prometheus-adapter isn't using flowcontrol.
+                                      'apiserver_storage_.*',  // Prometheus-adapter isn't using the apiserver storage.
+                                      'apiserver_webhooks_.*',  // Prometeus-adapter doesn't make use of apiserver webhooks.
+                                      'workqueue_.*',  // Metrics related to the internal apiserver auth workqueues are not very useful to prometheus-adapter.
+                                    ]) + ')',
+            },
+          ],
         },
       ],
     },

manifests/prometheus-adapter-serviceMonitor.yaml

@@ -12,6 +12,11 @@ spec:
   endpoints:
   - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
     interval: 30s
+    metricRelabelings:
+    - action: drop
+      regex: (apiserver_client_certificate_.*|apiserver_envelope_.*|apiserver_flowcontrol_.*|apiserver_storage_.*|apiserver_webhooks_.*|workqueue_.*)
+      sourceLabels:
+      - __name__
     port: https
     scheme: https
     tlsConfig: