Amazon MSK

Kpow fully supports both Amazon MSK (Managed Streaming for Apache Kafka) cluster types - Provisioned and Serverless. This flexibility allows you to use Kpow seamlessly regardless of how you choose to manage your Kafka infrastructure on AWS.

IAM Access Control

Among the supported authentication mechanisms, Kpow offers first-class support for IAM Access Control, making it easy to connect securely to your MSK clusters using AWS credentials and without needing to manage additional authentication layers such as username-password pairs or certificates.

To enable IAM authentication, configure Kpow with the following connection properties:

SSL_TRUSTSTORE_LOCATION=<PATH_TO_TRUST_STORE_FILE>
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=AWS_MSK_IAM
SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

These properties ensure Kpow connects to your MSK cluster using IAM-based SASL authentication over a secure SSL connection. The trust store file should contain the Amazon root CA used by your MSK cluster, and it's typically needed to validate the broker certificates. See the AWS documentation for more information, including JAAS config for named profiles.

Example IAM policies for Kpow

As Kpow is designed to manage/monitor Kafka clusters and associated resources, we can give administrative privileges to it. For more fine-grained access control within a Kafka cluster, we can rely on Apache Kafka ACLs, and the Enterprise Edition of Kpow provides robust support for it - see Kpow's ACL management documentation for more details.

Below shows example policies that can be attached.

Option 1: Admin Access to ALL MSK Clusters in the Region/Account

This policy allows listing/describing all clusters and performing any data-plane action (kafka-cluster:*) on any cluster within the specified region and account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kafka-cluster:*",
            "Resource": "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka:ListClusters",
                "kafka:DescribeCluster",
                "kafka:GetBootstrapBrokers"
            ],
            "Resource": "*"
        }
    ]
}

Option 2: Admin Access to a Specific LIST of MSK Clusters

This policy allows listing/describing all clusters but restricts the powerful kafka-cluster:* data-plane actions to only the specific clusters listed in the Resource array.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kafka-cluster:*",
            "Resource": [
                "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-1>/<GUID-1>",
                "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-2>/<GUID-2>"
                // Add more cluster ARNs here as needed following the same pattern
                // "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-3>/<GUID-3>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka:ListClusters",
                "kafka:DescribeCluster",
                "kafka:GetBootstrapBrokers"
            ],
            "Resource": "*"
        }
    ]
}

Option 3: Limited Access to a Specific MSK Cluster and Its Resources

This IAM policy grants scoped access to a specific MSK cluster, allowing topic-level operations, data read/write, and consumer group management.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*"
            ],
            "Resource": "arn:aws:kafka:<REGION>:cluster/<CLUSTER-NAME>/<GUID>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:<REGION>:topic/<CLUSTER-NAME>/<GUID>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:<REGION>:group/<CLUSTER-NAME>/<GUID>/*"
        }
    ]
}

Configuration Example

This configuration defines how Kpow connects to an MSK cluster:

ENVIRONMENT_NAME=<environment-name>
BOOTSTRAP=<bootstrap-addresses>
KAFKA_VARIANT=MSK_SERVERLESS
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=AWS_MSK_IAM
SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

• ENVIRONMENT_NAME: A human-readable name for the Kafka environment shown in the Kpow UI.
• BOOTSTRAP: The Kafka bootstrap server URL for an MSK Serverless cluster.
• KAFKA_VARIANT: (Optional) Set this to MSK_SERVERLESS to ensure Kpow creates its internal topics with the constrained topic configuration properties and service limitations specific to MSK Serverless.

Secure communication with the cluster is established using SASL over SSL:

• SECURITY_PROTOCOL: Set to SASL_SSL to enable encrypted client-server communication.
• SASL_MECHANISM: Set to AWS_MSK_IAM to use AWS IAM for Kafka client authentication.
• SASL_JAAS_CONFIG: Specifies the use of the IAMLoginModule provided by Amazon for secure authentication.
• SASL_CLIENT_CALLBACK_HANDLER_CLASS: Points to IAMClientCallbackHandler, which automates the process of retrieving and refreshing temporary credentials via IAM.

You may also add the default number of partitions to 1 (NUM_PARTITIONS=1) to overcome Amazon MSK's limitation on maximum number of partitions (120).

Limitations

The following functionality will be impacted due to Amazon MSK's own limitations:

• Kpow's internal audit log topic will only have a retention of 1 day
• Broker disk metrics are not available in Amazon MSK serverless

Amazon MSK Connect

Kpow supports integration with Amazon MSK Connect.

See the Amazon MSK Connect configuration guide for more information.