where CREDENTIAL_PATH is the path to your etcd client credentials on your work machine.
(Kube-aws stores them inside the credential folder).
## b - Get Promnetheus Operator to load the secret
In the previous step we have named the secret 'etcd-certs'.
Edit prometheus-operator/contrib/kube-prometheus/manifests/prometheus/prometheus-k8s.yaml and add the secret under the spec of the Prometheus object manifest:
```
secrets:
- etcd-certs
```
The manifest will look like that:
```
apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
name: k8s
labels:
prometheus: k8s
spec:
replicas: 2
secrets:
- etcd-certs
version: v1.7.0
```
If your Prometheus Operator is already in place, update it:
# Step 2 - Create the Service, endpoints and ServiceMonitor
The below manifest creates a Service to expose etcd metrics (port 2379)
Replace IP_OF_YOUR_ETCD_NODE_[0/1/2] with the IP addresses of your etcd nodes. If you have more than one node, add them to the same list.
In this example we use insecureSkipVerify: true as kube-aws default certiicates are not valid against the IP. They were created for the DNS. Depending on your use case, you might want to remove this flag or set it to false. (true required for kube-aws if using default certificate generators method)
You now need to allow the nodes Prometheus are running on to talk to the etcd on the port 2379 (if 2379 is the port used by etcd to expose the metrics)
If using kube-aws, you will need to edit the etcd security group inbound, specifying the security group of your Kubernetes node (worker) as the source.
## kube-aws and EIP or ENI inconsistency
With kube-aws, each etcd node has two IP addresses:
* EC2 instance IP
* EIP or ENI (depending on the chosen method in yuour cluster.yaml)
For some reason, some etcd node answer to :2379/metrics on the intance IP (eth0), some others on the EIP|ENI address (eth1). See issue https://github.com/kubernetes-incubator/kube-aws/issues/923
It would be of course much better if we could hit the EPI/ENI all the time as they don't change even if the underlying EC2 intance goes down.
If specifying the Instance IP (eth0) in the Prometheus Operator ServiceMonitor, and the EC2 intance goes down, one would have to update the ServiceMonitor.
Another idea woud be to use the DNS entries of etcd, but those are not currently supported for EndPoints objects in Kubernetes.
# Step 4: verify
Go to the Prometheus UI on :9090/config and check that you have an etcd job entry:
```
- job_name: monitoring/etcd-k8s/0
scrape_interval: 30s
scrape_timeout: 10s
...
```
On the :9090/targets page, you should see "etcd" with the UP state. If not, check the Error column for more information.
# Step 5: Grafana dashboard
## Find a dashboard you like
Try to load this dashboard:
https://grafana.com/dashboards/3070
## Save the dashboard in the configmap
As documented here, [Developing Alerts and Dashboards](developing-alerts-and-dashboards.md), the Grafana instances are stateless. The dashboards are automatically re-loaded from the ConfigMap.
So if you load a dashboard through the Grafana UI, it won't be kept unless saved in ConfigMap
Read [the document](developing-alerts-and-dashboards.md), but in summary:
### Copy your dashboard:
Once you are happy with the dashboard, export it and move it to `prometheus-operator/contrib/kube-prometheus/assets/grafana/` (ending in "-dashboard.json")
