Configuration
Amazon MSK
Kpow fully supports both Amazon MSK (Managed Streaming for Apache Kafka) cluster types - Provisioned and Serverless. This flexibility allows you to use Kpow seamlessly regardless of how you choose to manage your Kafka infrastructure on AWS.
IAM Access Control
Among the supported authentication mechanisms, Kpow offers first-class support for IAM Access Control, making it easy to connect securely to your MSK clusters using AWS credentials and without needing to manage additional authentication layers such as username-password pairs or certificates.
To enable IAM authentication, configure Kpow with the following connection properties:
SSL_TRUSTSTORE_LOCATION=<PATH_TO_TRUST_STORE_FILE>
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=AWS_MSK_IAM
SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler
These properties ensure Kpow connects to your MSK cluster using IAM-based SASL authentication over a secure SSL connection. The trust store file should contain the Amazon root CA used by your MSK cluster, and it's typically needed to validate the broker certificates. See the AWS documentation for more information, including JAAS config for named profiles.
Example IAM policies for Kpow
As Kpow is designed to manage/monitor Kafka clusters and associated resources, we can give administrative privileges to it. For more fine-grained access control within a Kafka cluster, we can rely on Apache Kafka ACLs, and the Enterprise Edition of Kpow provides robust support for it - see Kpow's ACL management documentation for more details.
Below shows example policies that can be attached.
Option 1: Admin Access to ALL MSK Clusters in the Region/Account
This policy allows listing/describing all clusters and performing any data-plane action (kafka-cluster:*
) on any cluster within the specified region and account.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "kafka-cluster:*",
"Resource": "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/*"
},
{
"Effect": "Allow",
"Action": [
"kafka:ListClusters",
"kafka:DescribeCluster",
"kafka:GetBootstrapBrokers"
],
"Resource": "*"
}
]
}
Option 2: Admin Access to a Specific LIST of MSK Clusters
This policy allows listing/describing all clusters but restricts the powerful kafka-cluster:*
data-plane actions to only the specific clusters listed in the Resource
array.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "kafka-cluster:*",
"Resource": [
"arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-1>/<GUID-1>",
"arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-2>/<GUID-2>"
// Add more cluster ARNs here as needed following the same pattern
// "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-3>/<GUID-3>"
]
},
{
"Effect": "Allow",
"Action": [
"kafka:ListClusters",
"kafka:DescribeCluster",
"kafka:GetBootstrapBrokers"
],
"Resource": "*"
}
]
}
Option 3: Limited Access to a Specific MSK Cluster and Its Resources
This IAM policy grants scoped access to a specific MSK cluster, allowing topic-level operations, data read/write, and consumer group management.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kafka-cluster:*"
],
"Resource": "arn:aws:kafka:<REGION>:cluster/<CLUSTER-NAME>/<GUID>"
},
{
"Effect": "Allow",
"Action": [
"kafka-cluster:*Topic*",
"kafka-cluster:WriteData",
"kafka-cluster:ReadData"
],
"Resource": "arn:aws:kafka:<REGION>:topic/<CLUSTER-NAME>/<GUID>/*"
},
{
"Effect": "Allow",
"Action": [
"kafka-cluster:AlterGroup",
"kafka-cluster:DescribeGroup"
],
"Resource": "arn:aws:<REGION>:group/<CLUSTER-NAME>/<GUID>/*"
}
]
}
Configuration Example
This configuration defines how Kpow connects to an MSK cluster:
ENVIRONMENT_NAME=<environment-name>
BOOTSTRAP=<bootstrap-addresses>
KAFKA_VARIANT=MSK_SERVERLESS
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=AWS_MSK_IAM
SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler
ENVIRONMENT_NAME
: A human-readable name for the Kafka environment shown in the Kpow UI.BOOTSTRAP
: The Kafka bootstrap server URL for an MSK Serverless cluster.KAFKA_VARIANT
: (Optional) Set this toMSK_SERVERLESS
to ensure Kpow creates its internal topics with the constrained topic configuration properties and service limitations specific to MSK Serverless.
Secure communication with the cluster is established using SASL over SSL:
SECURITY_PROTOCOL
: Set toSASL_SSL
to enable encrypted client-server communication.SASL_MECHANISM
: Set toAWS_MSK_IAM
to use AWS IAM for Kafka client authentication.SASL_JAAS_CONFIG
: Specifies the use of theIAMLoginModule
provided by Amazon for secure authentication.SASL_CLIENT_CALLBACK_HANDLER_CLASS
: Points toIAMClientCallbackHandler
, which automates the process of retrieving and refreshing temporary credentials via IAM.
You may also add the default number of partitions to 1 (NUM_PARTITIONS=1
) to overcome Amazon MSK's limitation on maximum number of partitions (120).
Limitations
The following functionality will be impacted due to Amazon MSK's own limitations:
- Kpow's internal audit log topic will only have a retention of 1 day
- Broker disk metrics are not available in Amazon MSK serverless
Amazon MSK Connect
Kpow supports integration with Amazon MSK Connect.
See the Amazon MSK Connect configuration guide for more information.