Kafka Connect

Access Control

User permissions to Kafka cluster resources are defined by Connect actions. See: User authorization.

Configuration

Kafka Connect

The following connects to a standard Kafka Connect cluster using the standard REST API.

Kpow connects to a Connect cluster with environment variables.

VariableDescription
CONNECT_NAMEUI and logs friendly name for this connect
CONNECT_STARTUP_VALIDATION(Optional) Validate that the Kafka Connect server is reachable at startup. When set to false, and Kpow cannot reach the host, an error message will be displayed in the UI, and Kpow will attempt to reconnect to the specified host. Default: true
CONNECT_REST_URLThe client connection URL for your connect cluster
CONNECT_AUTHBASIC if basic authentication is configured
CONNECT_BASIC_AUTH_USERUsername if basic authentication is configured
CONNECT_BASIC_AUTH_PASSPassword if basic authentication is configured
CONNECT_OFFSET_STORAGE_TOPIC(Optional) Topic that holds connect offsets
CONNECT_GROUP_ID(Optional) Unique string identifying worker cluster group
CONNECT_PERMISSIVE_SSLTrue if SSL certificate validation should be disabled
CONNECT_TIMEOUT_MSThe timeout value in ms for all HTTP requests made to a Kafka Connect cluster. Default: 5000
CONNECT_RESOURCE_IDSOptional, comma separated list of unique ids. Only specify when configuring multiple connect clusters
CONNECT_OBSERVATION_VERSIONOptional, specifies the snapshot version to use. Defaults to 1

Confluent Cloud Managed Connect

To configure Confluent Cloud's managed connect see: Managed Confluent Connect.

MSK Connect

To configure Amazon's MSK Connect view see: MSK Connect.

Configuring multiple Connect Clusters

Kpow supports multiple Kafka Connect clusters associated to a single Kafka cluster.

To configure multiple Kafka Connect clusters, use the environment variable CONNECT_RESOURCE_IDS to define a comma separated list of Connect clusters. Kpow uses the resource ID as a prefix in the environment variable.

Example configuration when configuring two Kafka Connect clusters:

CONNECT_RESOURCE_IDS=DEV1,QA2
DEV1_CONNECT_REST_URL=http://dev1-connect.mycorp.org:8003
QA2_CONNECT_REST_URL=http://qa2-connect.mycorp.org:8003

In this example we have defined connections to two Kafka Connect resources: DEV1 and QA2

Note: the values for CONNECT_RESOURCE_IDS must be upper-case like DEV1 and not dev1.

Observation version

To control how Kpow snapshots Kafka Connect, use the CONNECT_OBSERVATION_VERSION flag to set the observation version.

By default, when CONNECT_OBSERVATION_VERSION is set to 1, Kpow follows a two-step process to capture connect metadata. First, it retrieves all connectors with a single query. Then, with the specified parallelism level, it makes multiple REST calls for each connector: a separate call for task, status and connector information.

This method can be resource-intensive for larger connect clusters, as the number of REST calls increases proportionally to the number of connectors.

Alternatively, when CONNECT_OBSERVATION_VERSION is set to 2 (Kafka Connect only), Kpow uses a single-step process to capture connect metadata. It makes a single REST call to fetch all connectors, including their metadata.

This approach can be much more efficient than version 1, but is only supported by more recent Kafka Connect API versions (version 6.2 and above).

Previous
ksqlDB