Kafka Connect
Access Control
User permissions to Kafka cluster resources are defined by Connect actions. See: Overview.
Configuration
Kafka Connect
The following connects to a standard Kafka Connect cluster using the standard REST API.
Kpow connects to a Connect cluster with environment variables.
| Variable | Description |
|---|---|
| CONNECT_NAME | UI and logs friendly name for this connect |
| CONNECT_STARTUP_VALIDATION | (Optional) Validate that the Kafka Connect server is reachable at startup. When set to false, and Kpow cannot reach the host, an error message will be displayed in the UI, and Kpow will attempt to reconnect to the specified host. Default: true |
| CONNECT_REST_URL | The client connection URL for your connect cluster |
| CONNECT_AUTH | BASIC if basic authentication is configured |
| CONNECT_BASIC_AUTH_USER | Username if basic authentication is configured |
| CONNECT_BASIC_AUTH_PASS | Password if basic authentication is configured |
| CONNECT_OFFSET_STORAGE_TOPIC | (Optional) Topic that holds connect offsets |
| CONNECT_GROUP_ID | (Optional) Unique string identifying worker cluster group |
| CONNECT_PERMISSIVE_SSL | True if SSL certificate validation should be disabled |
| CONNECT_TIMEOUT_MS | The timeout value in ms for all HTTP requests made to a Kafka Connect cluster. Default: 5000 |
| CONNECT_RESOURCE_IDS | Optional, comma separated list of unique ids. Only specify when configuring multiple connect clusters |
| CONNECT_OBSERVATION_VERSION | Optional, specifies the snapshot version to use. Defaults to 1 |
Confluent Cloud Managed Connect
To configure Confluent Cloud's managed connect see: Managed Confluent Connect.
MSK Connect
To configure Amazon's MSK Connect view see: MSK Connect.
Configuring Multiple Connect Clusters
Kpow supports multiple Kafka Connect clusters associated to a single Kafka cluster.
To configure multiple Kafka Connect clusters, use the environment variable CONNECT_RESOURCE_IDS to define a comma separated list of Connect clusters. Kpow uses the resource ID as a prefix in the environment variable.
Example configuration when configuring two Kafka Connect clusters:
CONNECT_RESOURCE_IDS=US1,EU2
US1_CONNECT_REST_URL=http://us1-connect.mycorp.org:8003
EU2_CONNECT_REST_URL=http://eu2-connect.mycorp.org:8003
In this example we have defined a connection to two Kafka Connect resources: US1 and EU2
Note: the values for CONNECT_RESOURCE_IDS must be upper-case like US1 and not us1.
Observation Version
To control how Kpow snapshots Kafka Connect, use the CONNECT_OBSERVATION_VERSION flag to set the observation version.
By default, when CONNECT_OBSERVATION_VERSION is set to 1, Kpow follows a two-step process to capture connect metadata. First, it retrieves all connectors with a single query. Then, with the specified parallelism level, it makes multiple REST calls for each connector: a separate call for task, status and connector information.
This method can be resource-intensive for larger connect clusters, as the number of REST calls increases proportionally to the number of connectors.
Alternatively, when CONNECT_OBSERVATION_VERSION is set to 2 (Kafka Connect only), Kpow uses a single-step process to capture connect metadata. It makes a single REST call to fetch all connectors, including their metadata.
This approach can be much more efficient than version 1, but is only supported by more recent Kafka Connect API versions (version 6.2 and above).