Configuration
ksqlDB
ksqlDB is a database purpose-built for stream processing applications on top of Apache Kafka®.
Kpow can connect to your ksqlDB server like any other Kafka resource.
Use the following configuration to set up Kpow with ksqlDB. For documentation on ksqlDB usage within Kpow, see: %{ link id="ksqldb-usage" product="kpow-ee" /%}.
Configuration
ksqlDB
The following connects to a standard ksqlDB server using the standard REST API.
Kpow connects to a ksqlDB server with environment variables.
| Variable | Required? | Description |
|---|---|---|
| KSQLDB_NAME | false | UI and logs friendly name for this ksqlDB server. |
| KSQLDB_HOST | true | The hostname for this ksqlDB server connection. |
| KSQLDB_PORT | true | The port for this ksqlDB server connection. |
| KSQLDB_QUERY_MAX_ROWS | false | The maximum number of rows to be returned from a pull query (default: 100). |
| KSQLDB_INSERT_MAX_ROWS | false | The maximum number of rows to be buffered when inserting data (default: 200). |
| KSQLDB_USE_ALPN | false | Sets whether ALPN should be used. Defaults to false. |
| KSQLDB_USE_TLS | false | Sets whether TLS should be used. Defaults to false. |
| KSQLDB_VERIFY_HOST | false | Sets whether hostname verification (for TLS) is enabled. Defaults to true. |
| KSQLDB_TRUSTSTORE | false | Sets the trust store path. |
| KSQLDB_TRUSTSTORE_PASSWORD | false | Sets the trust store password. |
| KSQLDB_KEYSTORE | false | Sets the key store path. |
| KSQLDB_KEYSTORE_PASSWORD | false | Sets the key store password. |
| KSQLDB_KEY_PASSWORD | false | Sets the key password. |
| KSQLDB_KEY_ALIAS | false | Sets the key alias. |
| KSQLDB_BASIC_AUTH_USER | false | Sets the username to be used for HTTP basic authentication when connecting to the ksqlDB server. |
| KSQLDB_BASIC_AUTH_PASSWORD | false | Sets the password to be used for HTTP basic authentication when connecting to the ksqlDB server. |
| KSQLDB_RESOURCE_IDS | false | Comma separated list of unique ids. Only specify when configuring multiple ksqlDB servers. |
| KSQLDB_SCHEMA_REGISTRY | false | The schema registry ID associated to this ksqlDB instance. |
Confluent Cloud
Kpow's ksqlDB integration works with Confluent Cloud's hosted ksqlDB server.
Generate API keys
Generate an API key for Kpow to use. You can do this by installing the Confluent command-line tools and running:
confluent login
confluent environment use $ENVIRONMENT_NAME
confluent kafka cluster use $CLUSTER_ID
confluent api-key create --resource $KSQLDB_ID
You can find references to the various IDs of your resources inside Confluent Cloud's web UI.
Configure Kpow
The following example configuration connects Kpow to Confluent Cloud's managed ksqlDB:
KSQLDB_HOST="skdfhfk-x54nr.us-east-2.aws.confluent.cloud"
KSQLDB_PORT="443"
KSQLDB_BASIC_AUTH_USER="$API_KEY"
KSQLDB_BASIC_AUTH_PASSWORD="$API_SECRET"
KSQLDB_USE_TLS="true"
KSQLDB_USE_ALPN="true"
KSQLDB_NAME="Confluent ksqlDB"
You can find the hostname for your ksqlDB server within Confluent Cloud's web UI.
The variables API_KEY and API_SECRET come from the secret key you generated in the previous step.
Secure connections (HTTPS/TLS)
If your ksqlDB server has been secured with TLS, you will need to set the KSQLDB_USE_TLS environment variable:
KSQLDB_HOST="ksqldb.corp.io"
KSQLDB_PORT="443"
KSQLDB_USE_TLS="true"
KSQLDB_NAME="Secure ksqlDB"
Additionally, you may need to configure a keystore or truststore as well.
If you use Application-Layer Protocol Negotiation you need to set the KSQLDB_USE_ALPN environment variable as well.
Refer to the configuration table above for a description of each environment variable.
Configuring multiple ksqlDB servers
Kpow supports multiple ksqlDB servers associated to a single Kafka cluster.
To configure multiple ksqlDB servers, use the environment variable KSQLDB_RESOURCE_IDS to define a comma separated list of ksqlDB servers. Kpow uses the resource ID as a prefix in the environment variable.
Example configuration when configuring two ksqlDB servers:
KSQLDB_RESOURCE_IDS=DEV1,QA2
DEV1_KSQLDB_HOST=dev1.my-deployment.com
DEV1_KSQLDB_PORT=8080
QA2_KSQLDB_HOST=qa2.my-deployment.com
QA2_KSQLDB_PORT=8080
In this example we have defined connections to two ksqlDB resources: DEV1 and QA2
Note: The values for KSQLDB_RESOURCE_IDS must be upper-case like DEV1 and not dev1.
Schema Registry integration
Kpow can link together a ksqlDB server with any configured Schema Registry.
This integration provides inline schema information within the ksqlDB UI, as well quick links to edit schemas and inspect raw topic data using Avro, Protobuf or JSON Schema.
This feature is enabled automatically if you only have a single schema registry resource configured for your cluster.
If you have more than one schema registry configured (using SCHEMA_REGISTRY_RESOURCE_IDS), you can use the KSQLDB_SCHEMA_REGISTRY environment variable to link a ksqlDB server to a schema registry.
Example configuration linking a schema registry named LOCAL to a ksqlDB sever:
SCHEMA_REGISTRY_RESOURCE_IDS=LOCAL,CONFLUENT
LOCAL_SCHEMA_REGISTRY_URL=http://localhost:8001
KSQLDB_HOST=localhost
KSQLDB_PORT=8080
KSQLDB_SCHEMA_REGISTRY=LOCAL
Access control
User permissions to Kafka cluster resources are defined by ksqlDB actions. See: User authorization.
See ksqlDB for more details about configuring RBAC policies for ksqlDB.