Configuration

Kafka resources

Refer to Kpow's Kafka cluster configuration for documentation on how to configure Kafka resources for Factor Platform.

Supported Kafka resources include:

• ksqlDB
• Kafka Connect (Apache Connect, MSK Connect)
• Kafka Streams
• Schema Registry (Confluent, AWS Glue, Apicurio)

Kafka cluster configuration

Factor Platform internal topics

Factor Platform checks for the following internal topics in your primary cluster (the first bootstrap in your configuration) on startup and will attempt to automatically create them if required:

fh.metrics-fh_metric_store_pt1m-changelog
fh_metrics_pt1m
fh.snapshots-fh_snapshot_store-changelog
fh_snapshots
fh_audit_log

You can manually create these topics if you prefer, see Create internal topics.

Once started, Factor Platform creates two internal streaming compute applications:

fh.metrics
fh.snapshots

Minimum required ACLs

At a minimum, Factor Platform must have ACL permission to describe clusters, topics, configuration, and groups.

A basic set of Kafka ACLs that allows Factor Platform to operate provides ALLOW on the following:

Kafka Resource	Kafka ACL	Detail
Cluster	Describe	*
Cluster	DescribeConfigs	*
Cluster	Create	* (if not manually creating Factor Platform topics)
Topic	Describe	*
Topic	DescribeConfigs	*
Topic	Read	* (or Factor Platform topics only)
Topic	Write	* (or Factor Platform topics only)
Group	Describe	*
Group	Read	* (or Factor Platform groups only)

Factor Platform does not read from or write to topics other than internal ones as a part of normal operation.

Feature specific ACLS

The following ACLS are optional and only required if you intend to permit the associated Factor Platform action.

Kafka Resource	Kafka ACL	Required for user action
Cluster	Alter	ACL_EDIT
Cluster	AlterConfigs	BROKER_EDIT
Cluster	Create	TOPIC_CREATE
Topic	AlterConfigs	TOPIC_EDIT
Topic	Create	TOPIC_CREATE
Topic	Delete	TOPIC_DELETE, TRUNCATE_TOPIC
Topic	Read	TOPIC_INSPECT
Topic	Write	TOPIC_PRODUCE
Topic	Read	TOPIC_CLONE_SOURCE
Topic	Write	TOPIC_CLONE_SINK
Group	Read / Delete	GROUP_EDIT

Configuring Kafka ACLS

info

Creating ACLs on a cluster with no existing ACL configuration can cause issues.

Consult your cluster provider documentation first.

For example the Amazon MSK ACL Guide describes extra ACLs required to allow inter-broker replication, and suggests not to set CLUSTER level ACLs.

Create a file containing client configuration for a user who has permissions to create ACLs.

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";

Basic ACL creation script

Use the kafka-acls.sh script provided by Apache Kafka to create the basic set of ACLs described above. This allows Factor Platform to operate plus the ALTER CLUSTER ACL allowing Factor Platform to create and delete ACLs.

./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Describe --cluster '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation DescribeConfigs  --cluster '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Create --cluster '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Write --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Read --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Describe --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation DescribeConfigs --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Describe --group '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Read --group '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:factorplatform --operation Alter --cluster '*'

That set of ACLs can then be listed using kafka-acls.sh.

./kafka-acls.sh -bootstrap-server 127.0.0.1:9092 --command-config client.conf --list

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
 	(principal=User:factorplatform, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:factorplatform, host=*, operation=DESCRIBE, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:
 	(principal=User:factorplatform, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:factorplatform, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:factorplatform, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:factorplatform, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`:
 	(principal=User:factorplatform, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
	(principal=User:factorplatform, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:factorplatform, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:factorplatform, host=*, operation=ALTER, permissionType=ALLOW)

Create Factor Platform topics

info

Note: this step is completely optional as Factor Platform will automaticlly create these topics on first launch. However, you can manually create them if you would prefer.

Using the same client configuration file as above, the following script correctly creates the required internal Factor Platform topics. You can run this script and create these topics on the primary cluster in which you expect Factor Platform to keep data.

./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic fh_audit_log \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config segment.bytes=104857600 \
         --config retention.ms=-1
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic fh_metrics_pt1m \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config segment.bytes=104857600 \
         --config retention.ms=7200000 \
         --config segment.ms=5400000
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic fh_snapshots \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config segment.bytes=104857600 \
         --config retention.ms=7200000 \
         --config message.timestamp.type=LogAppendTime \
         --config segment.ms=5400000
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic fh.metrics-fh_metric_store_pt1m-changelog \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config cleanup.policy=compact,delete \
         --config segment.bytes=52428800 \
         --config retention.ms=5400000 \
         --config message.timestamp.type=CreateTime \
         --config segment.ms=1800000
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic fh.snapshots-fh_snapshot_store-changelog \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config cleanup.policy=compact,delete \
         --config segment.bytes=52428800 \
         --config retention.ms=604800000 \
         --config message.timestamp.type=CreateTime \
         --config segment.ms=604800000

Flink resources

Refer to Flex's Flink cluster configuration for documentation on how to configure Kafka resources for Factor Platform.

Supported deployments include:

• Self-managed
• Ververica Platform

Factor Platform configuration

For now, you can refer to Kpow's environment variable reference for reference on how to configure Factor Platform's web server, as well as authentication, authorization and tenancy capabilities.

tip

Factor Platform is still in early access.

Future work on dynamic configuration will mean that a lot of Platform configuration currently exposed via environment variables will move to persistent configuration stored in PostgreSQL, and configured through either Factor Platform's user interface or API.

PostgreSQL configuration

Factor Platform persists its configuration and state to a PostgreSQL database.

Prerequisites

Factor Platform requires PostgreSQL >=14.

Factor Platform requires no extensions or manual schema setup. All database migrations run automatically on initialization of the product.

Database preparation

Create a database and user for Factor Platform. If you're using a managed service (e.g. AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL), follow your provider's instructions to create the database and user, then skip to PostgreSQL environment variables

For self-managed PostgreSQL you will need to create both a role and database for Factor Platform:

CREATE ROLE factor_platform WITH LOGIN PASSWORD 'secure-password-here';
CREATE DATABASE factor_platform WITH OWNER factor_platform TEMPLATE template0 ENCODING UTF8 LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8';

Ensure the user can connect from the Factor Platform host by configuring pg_hba.conf as appropriate for your network.

Factor Platform configuration

Configure the PostgreSQL connection using the following environment variables:

Variable	Description	Required	Default
POSTGRES_USER	PostgreSQL username	Yes	-
POSTGRES_PASSWORD	PostgreSQL password	Yes	-
POSTGRES_DB_NAME	Database name for Factor Platform	No	factor_platform
POSTGRES_HOST	PostgreSQL server hostname or IP	Yes	-
POSTGRES_PORT	PostgreSQL server port	No	5432
POSTGRES_SSL	Enable SSL for PostgreSQL connection	No	false
POSTGRES_SSLMODE	SSL mode (e.g. disable, prefer, require)	No	prefer
POSTGRES_SSLROOTCERT	Path to SSL root certificate	No	-
POSTGRES_SSLCERT	Path to SSL client certificate	No	-
POSTGRES_SSLKEY	Path to SSL client key	No	-
POSTGRES_AWS_WRAPPER	Whether to enable the aws-advanced-jdbc-wrapper if using RDS with IAM authentication	No	false

Connection notes

Amazon RDS connection notes

Factor Platform supports IAM database authentication for Amazon RDS and Aurora PostgreSQL instances.

Refer to the official AWS documentation to enable IAM authentication and create an IAM policy for Factor Platform.

Setting POSTGRES_AWS_WRAPPER=true configures Factor Platform to use the AWS Advanced JDBC Wrapper with the following defaults:

• failover,efm2 wrapper plugins for cluster failover and host monitoring
• A maximum connection pool lifetime of 10 minutes, ensuring connections are recycled before IAM auth tokens expire

Note: RDS requires SSL connections by default. Set POSTGRES_SSL=true when using the AWS wrapper.

Application name

The connection is established with ApplicationName set to Factor Platform. This can be useful for identifying Factor Platform connections when debugging or monitoring active sessions in PostgreSQL (e.g. via pg_stat_activity).

Database connection over TLS

For connections over a public network, enable TLS by setting the SSL environment variables above. Most managed database providers handle server-side certificate configuration automatically - you'll typically just need to set POSTGRES_SSL=true and POSTGRES_SSLMODE to your desired level (e.g. require or verify-full).

For self-managed PostgreSQL or mutual TLS (mTLS), provide the client certificate paths via POSTGRES_SSLROOTCERT, POSTGRES_SSLCERT, and POSTGRES_SSLKEY.