Metriken mit Prometheus

Prometheus ist das de-facto Standard-Tool für Metriken in Kubernetes. Es scraped HTTP-Endpunkte (/metrics) und speichert Zeitreihendaten.

Prometheus im Cluster (kube-prometheus-stack)

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

kubectl get pods -n monitoring
kubectl get svc -n monitoring

Der kube-prometheus-stack enthält:

Prometheus UI

kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring --address=0.0.0.0

PromQL-Beispiele

# CPU-Nutzung aller Nodes
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory-Nutzung
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100

# HTTP-Request-Rate (OTEL rolldice)
rate(http_server_request_duration_seconds_count{service_name="dice-server"}[5m])

# Anzahl laufender Pods
kube_pod_status_phase{phase="Running"}

OTEL Metriken mit Alloy weiterleiten

In config.alloy:

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]
}

prometheus.remote_write "grafana_cloud" {
  endpoint {
    url = env("PROMETHEUS_URL")
    basic_auth {
      username = env("PROMETHEUS_USERNAME")
      password = env("PROMETHEUS_PASSWORD")
    }
  }
}