Configuration

Environment variables

Kpow is configured with these environment variables.

Kafka

One instance of Kpow can manage multiple Kafka clusters and associated resources.

  • See Kafka cluster for Kafka environment variable reference
  • See Kafka Connect for Connect environment variable reference
  • See Schema Registry for Schema registry environment variable reference
  • See ksqlDB for ksqlDB environment variable reference

Web server

The Kpow UI and Prometheus endpoints are served by Jetty.

The server can be configured to serve the UI via HTTPS or on a different port, or to redirect correctly when fronted with an HTTPS-terminating proxy.

LOG_FORMAT

Type: String, Default: plain

Set to 'json' to have Kpow write application logs in JSON format.

PORT

Type: Long, Default: 3000

The server port of the Kpow UI.

HTTP_FORWARDED

Type: Boolean, Default: false

See Jetty HTTP_FORWARDED module for more information.

Configure when running Kpow with Jetty authentication and behind a reverse-proxy that is performing HTTP termination. When true the Jetty authentication process will respect the HTTPS scheme when redirecting post-authentication.

ENABLE_HTTPS

Type: Boolean, Default: false

See HTTPS connections for more information.

Serve the Kpow UI via HTTPS (requires further configuration, below)

HTTPS_SNI_HOST_CHECK

Type: Boolean, Default: false

When SSL is configured, confirm that the certificate sent to the client matches the Host header.

HTTPS_KEYSTORE_LOCATION

Type: String (e.g. /ssl/https.keystore.jks)

Path to the SSL Keystore.

HTTPS_KEYSTORE_TYPE

Type: String, Default: JKS

Type of the SSL Keystore.

HTTPS_KEYSTORE_PASSWORD

Type: String

Password of the SSL Keystore.

HTTPS_TRUSTSTORE_LOCATION

Type: String (e.g. /ssl/https.truststore.jks)

Path to the SSL Truststore.

HTTPS_TRUSTSTORE_TYPE

Type: String, Default: JKS

Type of the SSL Truststore.

HTTPS_TRUSTSTORE_PASSWORD

Type: String

Password of the SSL Truststore

Authentication

Kpow supports Jetty (File, LDAP, DB, JAAS), SAML, OpenID and OAuth for authentication.

See User authentication for more details.

AUTH_PROVIDER_TYPE

Type: Enum, Values: okta, github, saml, jetty, auth0

Your choice of Authentication provider, specify Jetty for LDAP, DB, File, or JAAS.

OKTA_ORGANISATION

Type: String

When using Okta authentication - the name of your Okta organisation.

AUTH_LANDING_URI

Type: String (e.g. https://staging.kpow.z-corp.com)

The absolute URL to redirect to after successful login.

OPENID_AUTH_URI

Type: String

The OpenID Auth URI, e.g.

OPENID_API_URI

Type: String

The OpenID API URI, e.g.

OPENID_TOKEN_URI

Type: String

The OpenID Token URI, e.g.

OPENID_CLIENT_ID

Type: String

The OpenID Client ID found in your configured OpenID App.

OPENID_CLIENT_SECRET

Type: String

The OpenID Client Secret found in your configured OpenID App.

SAML_RELYING_PARTY_IDENTIFIER

Type: String

Your Kpow Application ID

SAML_ACS_URL

Type: String

The Assertion Consumer Service URL

SAML_METADATA_FILE

Type: String (e.g. /path/to/metadata.xml)

The Metadata File from your SAML provider.

SAML_CERT

Type: String (e.g. /path/to/saml.cert)

Optional SAML Certificate

SAML_SESSION_S

Type: Long, Default: 3600

The duration in seconds before re-authenticating SAML credentials.

DEBUG_AUTH

Type: Boolean, Default: False

Enable auth debug logging

JETTY_AUTH_METHOD

Type: Enum, Values: form, basic, Default: form

When using Jetty authentication, specifies to use form or basic-auth login UX

Authorization

RBAC_CONFIGURATION_FILE

Type: String (e.g. /path/to/rbac.yaml)

See Role Based Access Control for more information

The path to your RBAC configuration file (optional, requires Authentication enabled)

Global Access Controls

See Simple Access Control for more information

Apply global access controls like ALLOW_TOPIC_CREATE, etc.

General

HTTP_PROXY and HTTPS_PROXY

Configures Kpow to access all resources (AWS, Schema Registry, Kafka Connect, ksqlDB etc) through a proxy server.

Type: String

$ export HTTP_PROXY=http://10.15.20.25:1234
$ export HTTP_PROXY=http://proxy.example.com:1234
$ export HTTPS_PROXY=http://10.15.20.25:5678
$ export HTTPS_PROXY=http://proxy.example.com:5678

DATA_POLICY_CONFIGURATION_FILE

Type: String (e.g. /path/to/data-policies.yaml)

See Data policies for more information

The path to your Kpow Data Policy Configuration.

CUSTOM_SERDES

Type: String (e.g. io.kpow.SerdeOne,io.kpow.SerdeTwo)

See Serdes for more information.

Comma separated names of custom Serdes that can be found on the classpath.

DEFAULT_HEADERS_SERDES

Type: String (e.g. JSON)

The default headers Serde to use when inspecting data.

DEFAULT_KEY_SERDES

Type: String (e.g. JSON)

The default key Serde to use when inspecting data.

DEFAULT_VALUE_SERDES

Type: String (e.g. AVRO)

The default value Serde to use when inspecting data.

AVAILABLE_KEY_SERDES

Type: String (e.g. JSON,String)

Comma separated list of key Serdes to present when inspecting data.

AVAILABLE_VALUE_SERDES

Type: String (e.g. JSON,String)

Comma separated list of value Serdes to present when inspecting data.

NUM_PARTITIONS

Type: Long, Default: 12

The number of partitions for Kpow's internal topics.

REPLICATION_FACTOR

Type: Long, Default: 3

The replication factor of Kpow's internal topics.

REQUEST_TIMEOUT_MS

Type: Long, Default: 30000

The request.timeout.ms setting for Kpow's internal consumer groups.

MAX_PRODUCE_REQUEST_SIZE

Type: Long, Default: 1000000

The max.produce.request.size setting for Kpow's internal producers

PERSISTENCE_MODE

Type: Enum, Values: full, audit, none, Default: Full

Kpow stores data in the first cluster in your configuration (the Primary Cluster). This storage takes the form of several internal topics that are tuned to retain only a small amount of data.

In addition, an audit log topic is persisted permanently for data governance purposes.

These internal topics provide considerable feature support to Kpow, but there are circumstances in which you might want to turn them off.

full

This is the current default persistence behavior of Kpow and utilizes the full set of internal topics.

audit

Only internal topic that is created is the audit log.

This mode considerably reduces the amount of data written to Kafka, while retaining a full data governance trail.

When this mode is activated, certain features of Kpow run in a modified manner:

  • Metrics charts are not re-hydrated on a Kpow restart (normally they hydrate from an internal changelog).
  • Activity metrics (e.g. 'this topic was written to 3 minutes ago') are not persisted/maintained through a Kpow restart.
  • Kpow Streams Agent integration is disabled

none

Zero data is written to Kafka.

This mode ensures that no internal topics are created and no data is written by Kpow to your Kafka cluster.

When this mode is activated, certain features of Kpow run in a modified manner:

  • Metrics charts are not re-hydrated on a Kpow restart (normally they hydrate from an internal changelog).
  • Activity metrics (e.g. 'this topic was written to 3 minutes ago') are not persisted/maintained through a Kpow restart.
  • Audit log and user log are not available on a Kpow restart
  • Kpow Streams Agent integration is disabled

PRESENTATION_MODE

Type: Enum, Default: DEFAULT

Controls the global PRESENTATION_MODE. This environment variable controls how the UI will be displayed.

Valid values:

  • DEFAULT - the default presentation mode. No menu items are hidden.
  • HIDE_RESOURCES - if there are no configured Connect/Schema/ksqlDB resources then these menu items are hidden.

PROMETHEUS_EGRESS

Type: Boolean, Default: false

See Prometheus integration for more information.

Enable Prometheus endpoints for metrics and offsets egress.

PROMETHEUS_LABEL_ENV

Type: Boolean, Default: True

Include your ENVIRONMENT_NAME as 'env' label on Prometheus metrics.

PROMETHEUS_PASSWORD

Type: String

Optional. Sets the basic auth password for the Prometheus endpoints.

PROMETHEUS_USERNAME

Type: String

Optional. Sets the basic auth username for the Prometheus endpoints.

SNAPSHOT_SIMPLE_GROUPS

Type: Boolean, Default: True

Take observations of V1 consumer groups (simple groups including Flink consumers).

SNAPSHOT_PARALLELISM

Type: Long, Default: 3

The level of parallelism configured for Kpow telemetry capture and snapshotting.

SNAPSHOT_DEBUG

Type: Boolean, Default: False

Add additional logging messages to help debug snapshotting.

LIVE_MODE_ENABLED

Type: Boolean, Default: True

See Live mode for more information.

Allow your users to switch to live mode.

LIVE_MODE_PERIOD_MS

Type: Long, Default: 60000

Live mode will prompt you to continue after this period has elapsed.

LIVE_MODE_INTERVAL_MS

Type: Long, Default: 3500

The amount of time between live mode snapshots.

LIVE_MODE_MAX_CONCURRENT_USERS

Type: Long, Default: 2

The maximum number of concurrent live mode user sessions.

MUTATION_SCHEDULER_EXPIRES_MS

Type: Long, Default: 900000

The duration (in ms) until a scheduled mutation is expired. Default is 15 minutes.

SAMPLER_TIMEOUT_MS

Type: Long, Default: 7000

The maximum period of a single data inspect query.

SAMPLER_CONSUMER_THREADS

Type: Long, Default: 6

The level of parallelism for a data inspect query.

SLACK_WEBHOOK_URL

Type: String (e.g https://slack/webhook-url)

Send Audit log messages to Slack.

SLACK_WEBHOOK_URL_VERBOSITY

Type: Enum, Values: Mutations, Queries, All, Default: Mutations

Select the type of Audit log messages that are sent to Slack.

STREAMS_ERROR_STRATEGY

Type: Enum, Values: LOG_EXCEPTION_,_LOG___AND_EXIT, Default: LOG EXCEPTION

The strategy to use when Kpow's internal Kafka Streams instance enters an ERROR state.

STREAMS_TASK_TIMEOUT_MS

Type: Long, Default: 300000

Configures Kpow's internal streams task.timeout.ms value. See: KIP-572 for more information.

TEMPORARY_POLICY_MAX_MS

Type: Long, Default: 3600000

Configures the maximum allowed duration a temporary policy can be applied for.

WS_ALLOWED_ORIGINS

Type: String, Default: '*' (eg https://kpow.mycorp:3000)

The Kpow UI makes use of websockets and when a user session starts a websocket connection upgrade occurs in browser. To apply same-origin header checking to the websocket upgrade request configure the user-facing scheme, host, and (optional) port of the Kpow instance. Kpow websockets are also protected by CSRF tokens, same-origin checking is an optional extra setting. The default allowed origins are '*', in part because Kpow is often deployed in a manner that it is impossible for Kpow to determine the user-facing host of its own instance.

ALLOW_CONCURRENT_LOGIN

Type: Boolean, Default: true

Set it to false to detect and block concurrent logins to the same user account.

SESSION_MAX_AGE

Type: Long, Default: -1

The default max age is -1, which means no expiry. You may manage this session age via the identity provide integrated with Kpow (e.g. Okta, etc). However, if you are using Kpow with LDAP this new parameter allows you to evict session after a set period in seconds.

Next
Kafka cluster

Copyright © 2022 Factor House.