Skip to main content
CommunityTeamEnterprise

Bufstream

What is Bufstream?​

Bufstream is the Kafka-compatible message queue built for the data lakehouse era. It's a drop-in replacement for Apache Kafka®, but instead of requiring expensive machines with large attached disks, Bufstream builds on top of off-the-shelf technologies such as S3 and Postgres to provide a Kafka implementation that is built for 2025.

Quickstart​

We begin by creating a dedicated Docker network named factorhouse, which establishes a private communication channel for the Bufstream and Kpow containers. We then launch a single Bufstream broker and Kpow instance in dedicated containers.

Here's an overview of the containers:

  • Bufstream (bufstream)
    • Image: bufbuild/bufstream:latest
    • Host Ports:
      • 9092: Exposes the Kafka API to the host machine.
    • Configuration:
      • Mode: Runs in inmemory mode, creating a temporary, single-node broker where all data is lost on shutdown.
      • Kafka API: Listens for connections internally on port 9092 and is also mapped to 9092 on the host. It advertises itself as bufstream:9092 within the Docker network.
    • Network: Attached to the factorhouse network, making it reachable by other containers at the hostname bufstream.
  • Kpow (kpow)
    • Image: factorhouse/kpow-ce:latest
    • Host Ports:
      • 3000: Exposes the Kpow web UI to the host machine, accessible at http://localhost:3000.
    • Configuration:
      • ENVIRONMENT_NAME: UI friendly label for this cluster and resources.
      • BOOTSTRAP: Configured to connect to the Bufstream broker at bufstream:9092, using the internal Docker network for communication.
      • SCHEMA_REGISTRY_NAME: UI friendly label for a schema registry.
      • SCHEMA_REGISTRY_URL: Configured to connect to the Confluent-compatible API endpoint of Buf's public demo registry (https://demo.buf.dev/integrations/confluent/bufstream-demo). For details on connecting to a custom Confluent Schema Registry instance, see the official documentation.
      • Licensing: The configuration is loaded from an environment file specified by the $KPOW_LICENSE_FILE shell variable, which is required to run the container.
    • Network: Attached to the factorhouse network, which allows it to resolve and connect to bufstream:9092.
## Create a docker network to be shared
docker network create factorhouse

## Start a Bufstream broker
docker run -d -p 9092:9092 --name bufstream --network factorhouse \
  -e BUFSTREAM_KAFKA_HOST="0.0.0.0" \
  -e BUFSTREAM_KAFKA_PUBLIC_HOST="bufstream" \
  -e BUFSTREAM_KAFKA_PUBLIC_PORT="9092" \
  bufbuild/bufstream:latest serve --inmemory

## Start a Kpow instance
docker run -d -p 3000:3000 --name kpow --network factorhouse \
  -e ENVIRONMENT_NAME="Bufstream" \
  -e BOOTSTRAP="bufstream:9092" \
  -e SCHEMA_REGISTRY_NAME="Buf Registry" \
  -e SCHEMA_REGISTRY_URL="https://demo.buf.dev/integrations/confluent/bufstream-demo" \
  --env-file=$KPOW_LICENSE_FILE \
  factorhouse/kpow-ce:latest

Once the containers are running, navigate to http://localhost:3000 to access the Kpow UI. We will see that Kpow has automatically discovered and connected to the single Bufstream broker and Buf's public demo registry.

Kpow Overview