Google Cloud Managed Service for Apache Kafka

Configuration for authenticating to Google Cloud

Authentication to a Kafka cluster can be configured using either the OAUTHBEARER (recommended) or SASL/PLAIN mechanism, as outlined in the Google Cloud documentation.

For the OAUTHBEARER mechanism, simply set your Kpow connection fields appropriately, e.g.

SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=OAUTHBEARER
SASL_LOGIN_CALLBACK_HANDLER_CLASS=com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler
SASL_JAAS_CONFIG=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

For the SASL/PLAIN mechanism, it can be configured as e.g.

SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="EMAIL_ADDRESS" password="PASSWORD_VALUE";

See the Google Cloud documentation for more information, including how to obtain the credentials from a service account key JSON file, or an access token.

Access control with IAM and Kafka ACLs

Managed Service for Apache Kafka uses two levels of access control:

Identity and Access Management (IAM) roles: These roles control who can connect and manage your Managed Service for Apache Kafka cluster using Google Cloud APIs and tools.
- See this Google Cloud documentation for details about predefined IAM roles and permissions associated with Managed Kafka APIs
Apache Kafka ACLs: For more granular control over access to resources within a cluster, such as topics and consumer groups, use Kafka ACLs.
- Kpow provides robust support for Kafka ACLs. See the ACL management documentation for details.

Limitations

Kpow's functionality may be impacted by the limitations of Google Cloud Managed Service for Apache Kafka.

Google Cloud MSK Connect

Google Cloud MSK Connect is currently in Preview. Integration with Kpow will be addressed at a later stage.