Prometheus Service Monitors

· Overview
· Basic Environment Description
· Configuration
· Reflection

Overview

Our team is currently utilizing Prometheus to monitor our environment. Recently, I have been looking to expand our monitoring to include Kubernetes components. I would like to document the process of setting up ServiceMonitors for this purpose.

Service monitors define a set of targets for Prometheus to monitor, eliminating the need to modify Prometheus configurations. Instead, we can use declarative Kubernetes syntax to define these targets.

Basic Environment Description

The installation process of Prometheus will not be further elaborated here. Please refer to the following reference for installing Prometheus.

ref:

• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
• https://github.com/prometheus-operator/kube-prometheus

Configuration

We can refer to the example below for configuration reference:

prometheus-operator/Documentation/user-guides/getting-started.md at main ·…

• Create Service and ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  labels:
    team: devops
    release: prometheus-stack #This label allows Prometheus to find service monitors in the cluster and register them.
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: web

The release: prometheus-stack of the ServiceMonitor needs to be confirmed with the following steps:

• Get CRD

kubectl get crd -n <prometheus_namespaces>

• Get CRD and get label

kubectl get prometheuses.monitoring.coreos.com -o yaml -n <prometheus_namespaces>

and we could find it in items.spec.serviceMonitorSelector.matchLabels.release

We can see our configuration on the Targets page of our Prometheus dashboard.

Reflection

The overall setup isn’t too difficult, but it requires coordination with developers. We need to use the Prometheus library and ensure the exposure of Prometheus metrics path; otherwise, Prometheus couldn’t scrape the metrics properly.

The complete code can be referenced: https://github.com/JacksonChen63/Prometheus-ServiceMonitors