Environment variables
Configuration example
You can use these environment variables to configure a community version of Kpow and avoid the wizard UI.
For example, you can create a kpow.env file (~./kpow.env):
# Name this Kpow Installation
ENVIRONMENT_NAME=Kpow Community Demo
# Configure Kafka cluster
BOOTSTRAP=bootstrap-url:9092
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="USERNAME" password="PASSWORD";
# Configure Schema Registry
SCHEMA_REGISTRY_URL=https://your-schema-url
SCHEMA_REGISTRY_AUTH=USER_INFO
SCHEMA_REGISTRY_USER=your-schema-username
SCHEMA_REGISTRY_PASSWORD=your-schema-password
# Configure Connect
CONNECT_NAME=Connect Cluster
CONNECT_REST_URL=https://connect-cluster-url
CONNECT_AUTH=BASIC
CONNECT_BASIC_AUTH_USER=CONNECT_USERNAME
CONNECT_BASIC_AUTH_PASS=CONNECT_PASSWORD
# The license details from your community sign-up email
LICENSE_ID=b2b41e30-a401-4b70-8a36-a830581a85d5
LICENSE_CODE=COMMUNITY
LICENSEE=Factor House
LICENSE_EXPIRY=2024-06-20
LICENSE_SIGNATURE=683013F8BAC1D61560FC0C40C2B340..
And start Kpow like so:
docker run --pull=always -p 3000:3000 --env-file ./kpow.env -m 2G factorhouse/kpow-ce:latest
Configuration options
Kpow is configured with these environment variables.
Kafka
One instance of Kpow can manage multiple Kafka clusters and associated resources.
- See Kafka Cluster for Kafka environment variable reference
- See Kafka Connect for Connect environment variable reference
- See Schema Registry for Schema registry environment variable reference
- See ksqlDB for ksqlDB environment variable reference
Web server
The Kpow UI and Prometheus endpoints are served by Jetty.
The server can be configured to serve the UI via HTTPS or on a different port, or to redirect correctly when fronted with an HTTPS-terminating proxy.
LOG_FORMAT
Type: String, Default: plain
Set to 'json' to have Kpow write application logs in JSON format.
PORT
Type: Long, Default: 3000
The server port of the Kpow UI.
HTTP_FORWARDED
Type: Boolean, Default: false
See Jetty HTTP_FORWARDED module for more information.
Configure when running Kpow with Jetty authentication and behind a reverse-proxy that is performing HTTP termination. When true the Jetty authentication process will respect the HTTPS scheme when redirecting post-authentication.
ENABLE_HTTPS
Type: Boolean, Default: false
See HTTPS Connections for more information.
Serve the Kpow UI via HTTPS (requires further configuration, below)
HTTPS_SNI_HOST_CHECK
Type: Boolean, Default: false
When SSL is configured, confirm that the certificate sent to the client matches the Host header.
HTTPS_STS_MAX_AGE
Type: Long, Default: 31536000
The max age in seconds for a Strict-Transport-Security response header. If set less than zero then no header is sent.
HTTPS_INCLUDE_SUBDOMAINS
Type: Boolean, Default: true
Whether the includeSubdomains attribute is sent with the Strict-Transport-Security response header
HTTPS_KEYSTORE_LOCATION
Type: String (e.g. /ssl/https.keystore.jks)
Path to the SSL Keystore.
HTTPS_KEYSTORE_TYPE
Type: String, Default: JKS
Type of the SSL Keystore.
HTTPS_KEYSTORE_PASSWORD
Type: String
Password of the SSL Keystore.
HTTPS_TRUSTSTORE_LOCATION
Type: String (e.g. /ssl/https.truststore.jks)
Path to the SSL Truststore.
HTTPS_TRUSTSTORE_TYPE
Type: String, Default: JKS
Type of the SSL Truststore.
HTTPS_TRUSTSTORE_PASSWORD
Type: String
Password of the SSL Truststore
Authentication
Kpow supports Jetty (File, LDAP, DB, JAAS), SAML, OpenID and OAuth for authentication.
See User Authentication for more details.
AUTH_PROVIDER_TYPE
Type: Enum, Values: okta, github, saml, jetty, auth0
Your choice of Authentication provider, specify Jetty for LDAP, DB, File, or JAAS.
OKTA_ORGANISATION
Type: String
When using Okta authentication - the name of your Okta organisation.
AUTH_LANDING_URI
Type: String (e.g. https://staging.kpow.z-corp.com)
The absolute URL to redirect to after successful login.
OPENID_AUTH_URI
Type: String
The OpenID Auth URI, e.g.
- GitHub: https://github.com/login/oauth/authorize
- GitHub Enterprise: [Server URL]/login/oauth/authorize
OPENID_API_URI
Type: String
The OpenID API URI, e.g.
- GitHub: https://api.github.com/user
- GitHub Enterprise: [Server URL]/api/v3/user
OPENID_TOKEN_URI
Type: String
The OpenID Token URI, e.g.
- GitHub: https://github.com/login/oauth/access_token
- GitHub Enterprise: [Server URL]/login/oauth/access_token
OPENID_CLIENT_ID
Type: String
The OpenID Client ID found in your configured OpenID App.
OPENID_CLIENT_SECRET
Type: String
The OpenID Client Secret found in your configured OpenID App.
SAML_RELYING_PARTY_IDENTIFIER
Type: String
Your Kpow Application ID
SAML_ACS_URL
Type: String
The Assertion Consumer Service URL
SAML_METADATA_FILE
Type: String (e.g. /path/to/metadata.xml)
The Metadata File from your SAML provider.
SAML_CERT
Type: String (e.g. /path/to/saml.cert)
Optional SAML Certificate
SAML_SESSION_S
Type: Long, Default: 3600
The duration in seconds before re-authenticating SAML credentials.
DEBUG_AUTH
Type: Boolean, Default: False
Enable auth debug logging
JETTY_AUTH_METHOD
Type: Enum, Values: form, basic, Default: form
When using Jetty authentication, specifies to use form or basic-auth login UX
Authorization
RBAC_CONFIGURATION_FILE
Type: String (e.g. /path/to/rbac.yaml)
See Role Based Access Control for more details.
The path to your RBAC configuration file (optional, requires Authentication enabled)
Global access controls
See Simple Access Control for more details.
Apply global access controls like ALLOW_TOPIC_CREATE, etc.
General
HTTP_PROXY and HTTPS_PROXY
Configures Kpow to access all resources (AWS, Schema Registry, Kafka Connect, ksqlDB etc) through a proxy server.
Type: String
$ export HTTP_PROXY=http://10.15.20.25:1234
$ export HTTP_PROXY=http://proxy.example.com:1234
$ export HTTPS_PROXY=http://10.15.20.25:5678
$ export HTTPS_PROXY=http://proxy.example.com:5678
DATA_POLICY_CONFIGURATION_FILE
Type: String (e.g. /path/to/data-policies.yaml)
See Data Policies for more details.
The path to your Kpow Data Policy Configuration.
CUSTOM_SERDES
Type: String (e.g. io.kpow.SerdeOne,io.kpow.SerdeTwo)
See SerDes for more details.
Comma separated names of custom SerDes that can be found on the classpath.
ENABLE_INTERNAL_TENANT
Type: Boolean, Default: true
Enables the internal tenant, which hides Kpow's internal Kafka topics and consumer groups from the UI.
Note: Only applies when using simple access control.
DEFAULT_HEADERS_SERDES
Type: String (e.g. JSON)
The default headers Serde to use when inspecting data.
DEFAULT_KEY_SERDES
Type: String (e.g. JSON)
The default key SerDes to use when inspecting data.
DEFAULT_VALUE_SERDES
Type: String (e.g. AVRO)
The default value SerDes to use when inspecting data.
AVAILABLE_KEY_SERDES
Type: String (e.g. JSON,String)
Comma separated list of key SerDes to present when inspecting data.
AVAILABLE_VALUE_SERDES
Type: String (e.g. JSON,String)
Comma separated list of value SerDes to present when inspecting data.
SAMPLER_DEFAULT_KEY_FORMAT
Type: String, Default PRETTY
Default display preference for formatting keys when inspecting data. Must be one
of PRETTY, PRETTY_SORTED, or RAW.
SAMPLER_DEFAULT_VALUE_FORMAT
Type: String, Default PRETTY
Default display preference for formatting values when inspecting data. Must be
of PRETTY, PRETTY_SORTED, or RAW.
SAMPLER_DEFAULT_TIMESTAMP_FORMAT
Type: String, Default UNIX
Default display preference for formatting timestamps when inspecting data. Must
be one of UNIX, UTC, or LOCAL.
SAMPLER_DEFAULT_SIZE_FORMAT
Type: String, Default HUMAN
Default display preference for formatting field/record sizes when inspecting
data. Must be one of HUMAN or INT.
SAMPLER_DEFAULT_MAX_BYTES
Type: Long, Default 2500
Default display preference for the maximum key/value size (in bytes) before the key/value is collapsed (requiring a click to expand) when inspecting data.
SAMPLER_DEFAULT_VISIBLE_FIELDS
Type: String, Default Topic,Partition,Offset,Timestamp,Headers,Age
Default display preference for fields to display when inspecting data (in
addition to the key and value, which are always displayed). Must be a
comma-separated list of these field names as they appear in the UI: Topic, Partition, Offset,
Timestamp, Headers, Age, Key size, Value size, Size, Registry ID,
Schema ID, and SerDes.
NUM_PARTITIONS
Type: Long, Default: 12
The number of partitions for Kpow's internal topics.
REPLICATION_FACTOR
Type: Long, Default: 3
The replication factor of Kpow's internal topics.
REQUEST_TIMEOUT_MS
Type: Long, Default: 30000
The request.timeout.ms setting for Kpow's internal consumer groups.
MAX_PRODUCE_REQUEST_SIZE
Type: Long, Default: 1000000
The max.produce.request.size setting for Kpow's internal producers
PERSISTENCE_MODE
Type: Enum, Values: full, audit, none, Default: Full
Kpow stores data in the first cluster in your configuration (the Primary Cluster). This storage takes the form of several internal topics that are tuned to retain only a small amount of data.
In addition, an audit log topic is persisted permanently for data governance purposes.
These internal topics provide considerable feature support to Kpow, but there are circumstances in which you might want to turn them off.
full
This is the current default persistence behavior of Kpow and utilizes the full set of internal topics.
audit
Only internal topic that is created is the audit log.
This mode considerably reduces the amount of data written to Kafka, while retaining a full data governance trail.
When this mode is activated, certain features of Kpow run in a modified manner:
- Metrics charts are not re-hydrated on a Kpow restart (normally they hydrate from an internal changelog).
- Activity metrics (e.g. 'this topic was written to 3 minutes ago') are not persisted/maintained through a Kpow restart.
- Kpow Streams Agent integration is disabled
none
Zero data is written to Kafka.
This mode ensures that no internal topics are created and no data is written by Kpow to your Kafka cluster.
When this mode is activated, certain features of Kpow run in a modified manner:
- Metrics charts are not re-hydrated on a Kpow restart (normally they hydrate from an internal changelog).
- Activity metrics (e.g. 'this topic was written to 3 minutes ago') are not persisted/maintained through a Kpow restart.
- Audit log and user log are not available on a Kpow restart
- Kpow Streams Agent integration is disabled
PRESENTATION_MODE
Type: Enum, Default: DEFAULT
Controls the global PRESENTATION_MODE. This environment variable controls how the UI will be displayed.
Valid values:
DEFAULT- the default presentation mode. No menu items are hidden.HIDE_RESOURCES- if there are no configured Connect/Schema/ksqlDB resources then these menu items are hidden.
PROMETHEUS_EGRESS
Type: Boolean, Default: false
See Prometheus Integration for more details.
Enable Prometheus endpoints for metrics and offsets egress.
PROMETHEUS_LABEL_ENV
Type: Boolean, Default: True
Include your ENVIRONMENT_NAME as 'env' label on Prometheus metrics.
PROMETHEUS_PASSWORD
Type: String
Optional. Sets the basic auth password for the Prometheus endpoints.
PROMETHEUS_USERNAME
Type: String
Optional. Sets the basic auth username for the Prometheus endpoints.
SNAPSHOT_SIMPLE_GROUPS
Type: Boolean, Default: True
Take observations of V1 consumer groups (simple groups including Flink consumers).
SNAPSHOT_CONFIG_PERIOD_MS
Type: Long, Default: 600000
Period of time between taking snapshots of Kafka configuration resources, i.e.,
ACL-, broker-, and topic configuration. Must be between 60000 (1 minute) and
900000 (15 minutes). Default is 10 minutes.
SNAPSHOT_PARALLELISM
Type: Long, Default: 3
The level of parallelism configured for Kpow telemetry capture and snapshotting.
SNAPSHOT_DEBUG
Type: Boolean, Default: False
Add additional logging messages to help debug snapshotting.
MUTATION_SCHEDULER_EXPIRES_MS
Type: Long, Default: 900000
The duration (in ms) until a scheduled mutation is expired. Default is 15 minutes.
SAMPLER_N_CONSUMERS
Type: Long, Default: 3
Kpow creates a consumer connection pool for each worker when executing data inspect queries. This variable controls how many consumer threads each worker has available. Increasing this value improves query throughput by allowing more consumers to scan for records in parallel.
SAMPLER_POLL_MS
Type: Long, Default: 3500
The poll duration used by data inspect consumers. For most deployments the default is fine, but if your topics contain large messages you may see performance improvements by increasing this value, as consumers have more time to fetch and batch records per poll.
SAMPLER_TIMEOUT_MS
Type: Long, Default: 7000
A query will stop once it has found the limit of positively matched records (100) or after a timeout (default: 7s). You can always pick up where a query left off by clicking "Continue Consuming", or run the query as a streaming search.
SAMPLER_N_WORKERS
Type: Long, Default: 1
Data inspect and kREPL queries are executed within a worker queue. When a request comes in, Kpow assigns it to an available worker, which either executes the query immediately in its own thread or holds it until capacity frees up. This environment variable controls the number of workers available to process queries concurrently.
SAMPLER_WORKER_BUFFER_SIZE
Type: Long, Default: 1000
The capacity of the sampler worker queue. If the queues buffer is at capacity the request will be rejected and the user will see a "Rejected" state for their query and will have to retry.
WEBHOOK_PROVIDER
Type: Enum, Values: slack, teams, generic
Specifies the type of webhook integration.
WEBHOOK_URL
Type: String (e.g https://slack/webhook-url)
The URL that will receive webhook events via POST requests.
WEBHOOK_VERBOSITY
Type: Enum, Values: Mutations, Queries, All, Default: Mutations
Select the type of Audit log messages that are sent to the webhook provider.
STREAMS_ERROR_STRATEGY
Type: Enum, Values: LOGEXCEPTION,_LOG___AND_EXIT, Default: LOG EXCEPTION
The strategy to use when Kpow's internal Kafka Streams instance enters an ERROR state.
STREAMS_TASK_TIMEOUT_MS
Type: Long, Default: 300000
Configures Kpow's internal streams task.timeout.ms value. See: KIP-572 for more information.
TEMPORARY_POLICY_MAX_MS
Type: Long, Default: 3600000
Configures the maximum allowed duration a temporary policy can be applied for.
WS_ALLOWED_ORIGINS
Type: String, Default: '*' (eg https://kpow.mycorp:3000)
The Kpow UI makes use of websockets and when a user session starts a websocket connection upgrade occurs in browser. To apply same-origin header checking to the websocket upgrade request configure the user-facing scheme, host, and (optional) port of the Kpow instance. Kpow websockets are also protected by CSRF tokens, same-origin checking is an optional extra setting. The default allowed origins are '*', in part because Kpow is often deployed in a manner that it is impossible for Kpow to determine the user-facing host of its own instance.
ALLOW_CONCURRENT_LOGIN
Type: Boolean, Default: true
Set it to false to detect and block concurrent logins to the same user account.
SESSION_MAX_AGE
Type: Long, Default: -1
The default max age is -1, which means no expiry. You may manage this session age via the identity provide integrated with Kpow (e.g. Okta, etc). However, if you are using Kpow with LDAP this new parameter allows you to evict session after a set period in seconds.