Configuration
Schema Registry
Access Control
User permissions to Kafka cluster resources are defined by Schema actions. See: User authorization.
Configuration
Kpow supports Confluent Schema Registry and AWS Glue Schema Registry.
Kpow connects to a Schema registry with environment variables.
Confluent Schema Registry
| Variable | Description |
|---|---|
| SCHEMA_REGISTRY_NAME | UI and logs friendly name for this Schema registry |
| SCHEMA_REGISTRY_URL | The client connection URL for your registry |
| SCHEMA_REGISTRY_AUTH | USER_INFO if basic authentication is configured |
| SCHEMA_REGISTRY_USER | Username if basic authentication is configured |
| SCHEMA_REGISTRY_PASSWORD | Password if basic authentication is configured |
| SCHEMA_REGISTRY_RESOURCE_IDS | Optional, comma separated list of unique ids. Only specify when configuring multiple schema registries. |
| SCHEMA_REGISTRY_OBSERVATION_VERSION | Optional, sets the observation version used to snapshot schema resources. Default=1. See Observation Version. |
Confluent SSL Configuration
The following environment variables can be used to configure Confluent schema registry connections with mutual-TLS:
| Variable | Description |
|---|---|
| SCHEMA_REGISTRY_SSL_KEYSTORE_LOCATION | Location of the keystore file |
| SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD | Password to access the keystore file |
| SCHEMA_REGISTRY_SSL_KEYSTORE_KEY | Specific key to use within the keystore |
| SCHEMA_REGISTRY_SSL_KEY_PASSWORD | Password to access the key within the keystore |
| SCHEMA_REGISTRY_SSL_KEYSTORE_TYPE | Keystore type |
| SCHEMA_REGISTRY_SSL_KEYMANAGER_ALGORITHM | Keymanager algorithm |
| SCHEMA_REGISTRY_SSL_TRUSTSTORE_LOCATION | Location of the truststore file |
| SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD | Password to access the truststore file |
| SCHEMA_REGISTRY_SSL_TRUSTSTORE_TYPE | Truststore type |
| SCHEMA_REGISTRY_SSL_TRUSTMANAGER_ALGORITHM | Trustmanager algorithm |
| SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM | SSL endpoint identification algorithm |
| SCHEMA_REGISTRY_SSL_PROVIDER | SSL provider |
| SCHEMA_REGISTRY_SSL_CIPHER_SUITES | SSL cipher suites |
| SCHEMA_REGISTRY_SSL_PROTOCOL | SSL protocol |
| SCHEMA_REGISTRY_SSL_ENABLED_PROTOCOLS | SSL enabled protocols |
| SCHEMA_REGISTRY_SSL_SECURE_RANDOM_IMPLEMENTATION | SSL secure random implementation |
| SCHEMA_REGISTRY_SSL_KEYSTORE_CERTIFICATE_CHAIN | SSL keystore certificate chain |
| SCHEMA_REGISTRY_SSL_TRUSTSTORE_CERTIFICATES | SSL truststore certificates |
| SCHEMA_REGISTRY_SSL_ENGINE_FACTORY_CLASS | SSL engine factory class |
Confluent Cloud Oauth Configuration
The following environment variables can be used to configure Confluent schema registry with Confluent Cloud via Oauth
See the Confluent Cloud guide to Schema Registry Oauth for more information.
| Variable | Description |
|---|---|
| SCHEMA_REGISTRY_BEARER_AUTH_CREDENTIALS_SOURCE | OAUTHBEARER or SASL_OAUTHBEARER_INHERIT |
| SCHEMA_REGISTRY_BEARER_AUTH_TOKEN | Bearer Auth token |
| SCHEMA_REGISTRY_BEARER_AUTH_ISSUER_ENDPOINT_URL | OpenID provider URL |
| SCHEMA_REGISTRY_BEARER_AUTH_CLIENT_ID | Client ID |
| SCHEMA_REGISTRY_BEARER_AUTH_CLIENT_SECRET | Client secret |
| SCHEMA_REGISTRY_BEARER_AUTH_SCOPE | Scope |
| SCHEMA_REGISTRY_BEARER_AUTH_SCOPE_CLAIM_NAME | Scope claim name |
| SCHEMA_REGISTRY_BEARER_AUTH_SUB_CLAIM_NAME | Scope sub claim name |
| SCHEMA_REGISTRY_BEARER_AUTH_LOGICAL_CLUSTER | LSRC resource ID |
| SCHEMA_REGISTRY_BEARER_AUTH_IDENTITY_POOL_ID | Identity pool ID |
| SCHEMA_REGISTRY_BEARER_AUTH_CACHE_EXPIRY_BUFFER_SECONDS | Cache expiry buffer seconds |
AWS Glue Schema Registry
| Variable | Description |
|---|---|
| SCHEMA_REGISTRY_NAME | UI and logs friendly name for this Schema Registry |
| SCHEMA_REGISTRY_ARN | The ARN of your AWS Glue Schema Registry |
| SCHEMA_REGISTRY_REGION | The Region of your AWS Glue Schema Registry (default: us-east-1) |
AWS Glue authentication
By default, Kpow will use the DefaultAWSCredentialsProviderChain to authenticate with AWS Glue schema registry.
Static credentials
You can authenticate using static credentials by specifying:
SCHEMA_REGISTRY_ACCESS_KEY_ID=xxx
SCHEMA_REGISTRY_SECRET_ACCESS_KEY=xxx
Cross-account AWS Access (STS Assume Role)
If you wish to configure cross-account AWS Schema registry access (where for example Kpow is living in AWS Account A and AWS Glue is living in Account B) you can configure an IAM role+trust policy.
This blog post demonstrates how to configure an IAM role for this exact scenario with example Terraform in the GitHub repo.
You can configure Kpow to assume a role with the following environment variables:
SCHEMA_REGISTRY_STS_ROLE_ARN="arn:aws:iam::ACCOUNT_B_ID:root"
SCHEMA_REGISTRY_STS_SESSION_NAME="session_name"
SCHEMA_REGISTRY_STS_REGION="us-east-2" # (default us-east-1)
Configuring multiple Schema Registries
Kpow supports multiple Schema Registries associated to a single Kafka cluster.
To configure multiple Schema Registries, use the environment variable SCHEMA_REGISTRY_RESOURCE_IDS to define a comma separated list of Schema Registries. Kpow uses the resource ID as a prefix in the environment variable.
Example configuration when configuring two Schema Registries:
SCHEMA_REGISTRY_RESOURCE_IDS=DEV1,QA2
DEV1_SCHEMA_REGISTRY_URL="https://dev1.schema-registry.mycorp.org"
DEV1_SCHEMA_REGISTRY_AUTH="USER_INFO"
DEV1_SCHEMA_REGISTRY_USER=""
DEV1_SCHEMA_REGISTRY_PASSWORD=""
QA2_SCHEMA_REGISTRY_URL="https://qa2.schema-registry.mycorp.org"
QA2_SCHEMA_REGISTRY_AUTH="USER_INFO"
QA2_SCHEMA_REGISTRY_USER=""
QA2_SCHEMA_REGISTRY_PASSWORD=""
In this example we have defined connections to two Schema Registry resources: DEV1 and QA2
Note: The values for SCHEMA_REGISTRY_RESOURCE_IDS must be upper-case like DEV1 and not dev1.
Observation version
To control how Kpow snapshots schemas, use the SCHEMA_REGISTRY_OBSERVATION_VERSION flag to set the observation version.
By default, when SCHEMA_REGISTRY_OBSERVATION_VERSION is set to 1, Kpow follows a two-step process to capture schema metadata. First, it retrieves all schema names with a single query. Then, with the specified parallelism level, it makes two REST calls for each schema: one to fetch metadata and another to fetch compatibility.
This method provides the most context at an aggregate level, but can be resource-intensive for larger schema registries, as the number of REST calls increases proportionally to the number of schemas.
Alternatively, when SCHEMA_REGISTRY_OBSERVATION_VERSION is set to 2 (Confluent only), Kpow uses a single-step process to capture schema metadata. It makes a single REST call to fetch all schemas, including their metadata. This approach can be much more efficient than version 1, but compatibility information is only available at the individual schema level in Kpow's UI.