Skip to content

Prometheus

Prometheus

Warning

This documentation is deprecated, please check here for its new home

Kubernetes

Clusters on OpenStack can be created with advanced metrics capabilities. These metrics are collected with a resolution of 30 seconds and include resources such as CPU, Memory, Disk and Network IO as well as R/W rates. Metrics for cluster, nodes and pods are gathered using Prometheus. These metrics of fine granularity are available on your cluster for up to a period of 14 days.

Prometheus is not installed by default on the cluster and requires additional labels to be specified during cluster creation:

--labels monitoring_enabled=true

For Grafana instance available on the cluster, you can also specify the desired dashboard password with the following label:

--labels grafana_admin_passwd=admin

NOTE: Do the following procedure from your own machine or VM, not from a shared cluster like lxplus. That would expose your cluster

To access it, start by creating a proxy connection to your cluster from your host:

$ kubectl -n kube-system port-forward service/prometheus-operator-grafana 3000:80 &
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

The dashboards are predefined and loaded at cluster creation time. The dashboards are then accessible by accessing (on your host): http://localhost:3000

In the background, Grafana interacts with the Prometheus server by means of the PromQL. PromQL is a querying language specific to the Prometheus server. If you want to do custom dashboards you can start by checking the querying rules with this examples.

CONSIDERATIONS: Before creating your cluster take into account the scale of the cluster. This is important as the Prometheus server pod might not fit your nodes. This is particularly important if you are using Cluster Autoscaling as the Prometheus server will schedule resources needed to meet the maximum number of nodes that your cluster can scale up to.

The Prometheus server will consume the following resources:

  • RAM:: 256 (base) + Nodes * 40 [MB]
  • CPU:: 128 (base) + Nodes * 7 [mCPU]
  • Disk:: 15 GB for 2 weeks (needs refinement)

Check the pre-calculated table below the required minion flavor to support your cluster size.

Maximum Cluster Size* Minion Flavor
40 m2.small (w\ external storage only)
87 m2.medium
180 m2.large
368 m2.xlarge
743 m2.2xlarge

* Cluster size includes minion and master nodes

Note: The prometheus server calculates resource usage based on requested number of nodes at cluster creation but node groups are added after, so this won't be taken into account. To circumvent this if, when you create the cluster, you already plan to use node groups you should also use the label max_node_count to indicate the maximum of nodes your cluster will have. Otherwise you will have to edit statefulset.apps/prometheus-prometheus-prometheus and change the spec.template.spec.resources values according to the formulas above.

Known Issues/Limitations:

  • Grafana Dashboards - Although you can edit grafana dashboards from the browser, these changes are not persisted. In other words, when you exit the dashboard view, all work will be lost. Workaround - To persist your dashboards you can create them, extract the generated JSON and save it into a ConfigMap adding the label grafana_dashboard: "1". See example below. To edit pre-existing dashboards you can edit their respective ConfigMaps. 
kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    grafana_dashboard: "1"
  name: your-new-dashboard
  namespace: kube-system
data:
  your-new-dashboard.json: |-
    {
        [...]
    }
  • Grafana server capacity - Grafana by default is configured to request small amount of CPU and memory. If the number of users accessing the Grafana UI is expected to be considerable, adjust the deployment manifest as needed.
$ kubectl -n kube-system edit deployment.apps/prometheus-operator-grafana

  • Prometheus data persistency - Prometheus monitoring runs as a cluster application on your minions in a priviledge state. If the node where your prometheus pod is running goes down the pod is re-scheduled on another node however your metrics data history is lost. It is possible to store your data on a PVC but, this feature is not available as of yet.

  • Alertmanager - An extra component provided with the prometheus-operator solution is enabled and available. Similar rules apply for the Alertmanager capacity as defined in Grafana server capacity as this service runs under Best-Effort scheme.

Using traefik ingress

You can also expose the Grafana service through traefik ingress. In such a case, kubectl proxy is not needed and you will be able to expose your service externally. For advanced Ingress configuration read the load balancing tutorials.

Create a new ingress resource for the Grafana service using the following definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  labels:
    app: grafana
  name: prometheus-operator-grafana
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/frontend-entry-points: http
    traefik.ingress.kubernetes.io/protocol: http
spec:
  rules:
  - host: monit-preview.cern.ch
    http:
      paths:
      - backend:
          serviceName: prometheus-operator-grafana
          servicePort: 80
        path: /

In this setup, we use the host header to route traffic to the right backend. Here, monit-preview.cern.ch should be replaced by the landb alias created during the ingress node setup in load balancing.

Prometheus Custom Metrics

If you are running an application, chances are that you would like to have specific metrics related to its functioning state. Many applications have already some way of exposing a prometheus endpoint to scrape these metrics, or if not there may be publicly available exporters. You can also make your own exporter using client libraries.

In this example we will assume that the /metrics endpoint is already exposed and we will use MongoDB as an example. We will make all the configurations so that metrics are automatically scraped by prometheus.

  • We can find a helm chart for MongoDB on github. We can also notice that there is an option to instruct the creation of a prometheus sidecar metrics exporter that we will use [1][2]. 
helm install --name my-mongodb stable/mongodb --namespace mongodb \
  --set global.storageClass=geneva-cephfs-testing \
  --set persistence.storageClass=geneva-cephfs-testing \
  --set metrics.enabled=true
  • Validate that you have access to the /metrics endpoint with: 
kubectl -n mongodb port-forward service/my-mongodb \
  8080:$(kubectl -n mongodb get service/my-mongodb \
  -o jsonpath='{.spec.ports[?(@.name=="metrics")].port}') &

xdg-open http://localhost:8080/metrics
  • With our service up and exposing metrics, we need to create a ServiceMonitor manifest with instruction on how prometheus can scrape this service. 
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mongodb-monitor
  namespace: mongodb
  labels:
    release: cern-magnum
    app: mongodb-scraper
spec:
  endpoints:
  - path: /metrics
    port: metrics
    interval: 30s
  jobLabel: "app.kubernetes.io/name"
  namespaceSelector:
    matchNames:
    - mongodb
  selector:
    matchLabels:
      app: mongodb

Details:

  • metadata.labels.release=cern-magnum - This is the label that signals to Prometheus server that it should integrate this configuration. For clusters older than kubernetes-1.22 this label should be 'release: prometheus-operator'
  • spec.endpoints.path=/metrics - The endpoint where the metrics can be reached.
  • spec.endpoints.interval=30s - The interval of time between successive scrapes.
  • spec.namespaceSelector.matchNames=["mongodb"] - Namespace selector when searching Services.
  • spec.selector.matchLabels={app:mongodb} - The service selector label(s).

Install your defined prometheus ServiceMonitor manifest 

kubectl apply -f mongoDBServiceMonitor.yaml
  • To validate that prometheus is scraping this data we expose the prometheus service with: 
$ kubectl -n kube-system port-forward service/prometheus-prometheus 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

  • Validate that your service is being discovered and scraped with:

    • Service Discovery: http://localhost:9090/service-discovery
    • Targets: http://localhost:9090/targets

    You should see:

    • On "Service Discovery" that you have (1/2 active targets)
    • On "Targets" that all the pods that are mapped by the selected kubernetes service have an updated Last Scrape parameter

There are other kinds of monitors other than ServiceMonitor which mainly matches services. You can find more about these objects in the Prometheus documentation

[1] Creating manila shares requires available space and authorization to this service. Manila shares are not available in Personal Projects.
[2] Notice the metrics.serviceMonitor.* section. We could use these parameters to fully set up monitoring for MongoDB but for sake of explanation we will implement a ServiceMonitor ourselves.

Last update: June 1, 2022