Running the platform

To get the platform running, you first need to configure your local environment. This involves setting environment variables to select the edition you want to run (Community or Enterprise) and providing the file paths to your licenses. Once these prerequisites are set, you can launch the services using docker compose. You have two primary options: you can start all services (Kpow, Flex, and Pinot) together for a fully integrated experience, or you can run Kpow and Flex independently for more focused use cases. When you are finished, remember to run the corresponding down command to stop and remove the containers, and unset the environment variables to clean up your session.

Prerequisite: Configure Your Environment

Before running any commands, configure your shell environment by uncommenting and setting the following variables in your terminal.

# --- Configuration ---

# Edition (choose one by uncommenting the lines)
# For Enterprise Edition:
# unset KPOW_SUFFIX
# unset FLEX_SUFFIX

# For Community Edition:
# export KPOW_SUFFIX="-ce"
# export FLEX_SUFFIX="-ce"

# Licenses (provide the path to your license .env files)
# export KPOW_LICENSE="/path/to/your/kpow-license.env"
# export FLEX_LICENSE="/path/to/your/flex-license.env"

Run the Stacks

This platform is designed to be modular. You can run each stack individually or combine them to create a complete, integrated environment.

Key Concepts

Before you start, understand these three concepts:

1. Project Isolation (-p <name>): We use the -p flag (e.g., -p kpow) to give each stack a unique project name. This prevents Docker from getting confused if multiple files define a service with the same name (like a database).
2. Stack Profiles (--profile): Files like compose-store.yml and compose-obsv.yml contain multiple, independent services. The --profile flag lets you choose which one to activate (e.g., --profile pinot).
3. Shared Network (USE_EXT): By default, the compose-kpow.yml stack creates a shared Docker network called factorhouse. Other stacks are designed to connect to this network so services can communicate.
   • If you run a stack without the kpow stack, you must tell it to create its own network. This is done by setting USE_EXT=false.

---

Usage Examples

While you can launch all services together, the most common and resource-friendly approach is to launch only the stacks you need.

1. Kafka & Kpow Stack

This is the foundational stack. It creates the shared factorhouse network that other stacks can connect to.

# Start
docker compose -p kpow -f ./compose-kpow.yml up -d

# Stop
docker compose -p kpow -f ./compose-kpow.yml down

2. Flink & Flex Stack

• To run with the Kafka stack (recommended):

  docker compose -p flex -f ./compose-flex.yml up -d

• To run standalone (creates its own network):

  USE_EXT=false docker compose -p flex -f ./compose-flex.yml up -d

• To stop:

  # Note: Use USE_EXT=false here as well if you started it standalone
  docker compose -p flex -f ./compose-flex.yml down

3. Real-Time Data Stores (Pinot, ClickHouse, StarRocks)

These are selected from compose-store.yml via profiles.

💡 You can specify multiple profiles to launch more than one data store at once, e.g., --profile pinot --profile clickhouse.

# Start Apache Pinot (connected to the factorhouse network)
docker compose -p store --profile pinot -f ./compose-store.yml up -d

# Stop Apache Pinot
docker compose -p store --profile pinot -f ./compose-store.yml down

# Start ClickHouse (connected to the factorhouse network)
docker compose -p store --profile clickhouse -f ./compose-store.yml up -d

# Stop ClickHouse
docker compose -p store --profile clickhouse -f ./compose-store.yml down

# Start StarRocks (connected to the factorhouse network)
docker compose -p store --profile starrocks -f ./compose-store.yml up -d

# Stop StarRocks
docker compose -p store --profile starrocks -f ./compose-store.yml down

4. Observability Stacks (Lineage & Telemetry)

These are selected from compose-obsv.yml using profiles.

# Start Marquez for data lineage (connected to the factorhouse network)
docker compose -p obsv --profile lineage -f ./compose-obsv.yml up -d

# Stop Marquez
docker compose -p obsv --profile lineage -f ./compose-obsv.yml down

# Start all observability services (Lineage + Telemetry)
docker compose -p obsv --profile lineage --profile telemetry -f ./compose-obsv.yml up -d

# Stop all observability services
docker compose -p obsv --profile lineage --profile telemetry -f ./compose-obsv.yml down

5. Metadata Stack

This is selected from compose-metadata.yml.

# Start OpenMetadata (connected to the factorhouse network)
docker compose -p metadata --profile omt -f ./compose-metadata.yml up -d

# Stop OpenMetadata
docker compose -p metadata --profile omt -f ./compose-metadata.yml down

Clean Up Your Environment

After stopping the services, run this command to unset the environment variables from your terminal session.

unset KPOW_SUFFIX FLEX_SUFFIX KPOW_LICENSE FLEX_LICENSE