Prometheus integration

Flex's Prometheus egress endpoints follow the OpenMetrics standard.

This allows you to integrate Flex to your favorite observability tools such as Prometheus, New Relic or Grafana for long-term reporting and alerting.

The Prometheus metric name and label format specifies [a-zA-Z_][a-zA-Z0-9_]* as valid characters. Where Kafka resource names (e.g. groups, topics) contain characters outside of that range Flex will convert non-matching characters to _.

Configuration

To enable Prometheus endpoints set the following environment variable.

PROMETHEUS_EGRESS=true

Security

Prometheus Endpoints are not secure by default.

To secure all metric endpoints you can configure basic authentication:

PROMETHEUS_USERNAME=foo
PROMETHEUS_PASSWORD=bar

Getting Started

See our how-to blogpost on alerting and monitoring with Flex, Prometheus, and AlertManager.

Endpoints

Flex provides Prometheus endpoints for all metrics, topic and group offsets, and streams.

Flex logs the path to each Prometheus endpoint on startup

* GET /metrics/v1 - all metrics
* GET /metrics/v1/cluster/:cluster-id - metrics for a specific cluster-id
* GET /metrics/v1/connect/:connect-id - metrics for a specific connect-id
* GET /metrics/v1/schema/:schema-id - metrics for a specific schema-id
* GET /metrics/v1/ksqldb/:ksqldb-id - metrics for a specific ksqldb-id
* GET /group-offsets/v1 - all group offset metrics
* GET /offsets/v1 - all topic offset metrics

Sample Scraper Configuration

Sample Prometheus scraper configuration that we use to test Flex:

scrape_configs:
  - job_name: 'kpow'
    metrics_path: '/metrics/v1'
    static_configs:
      - targets: ['host.docker.internal:3000']
  - job_name: 'kpow_streams'
    metrics_path: '/streams/v1'
    static_configs:
      - targets: ['host.docker.internal:3000']
  - job_name: 'kpow_offsets'
    metrics_path: '/offsets/v1'
    static_configs:
      - targets: ['host.docker.internal:3000']