Rust-native selective replication

Shape the stream
before it lands.

Filter, transform, redact, and route Kafka records between topics and clusters—without standing up Kafka Connect or a general-purpose stream processor.

Run the quickstart View on GitHub

Kafka-first
Redpanda compatibility target
Apache 2.0

pipeline / orders running

source raw-orders

filter $region == 'us'

transform construct(order_id=$order_id)

destination analytics-orders

destination pii-safe-orders

Validated AWS baseline Method, environment, and limits

Median delivery: 106,693 msg/s
Timed runs: 3 × 120 sec
Validation: 16.2M exact/run
Errors / CV: 0 / 0.058%

Control the record, not just the topic

One input. Explicit policy. Purpose-built outputs.

StreamForge applies routing policy while records are in motion. Downstream teams receive the fields and events they need; lower-trust destinations do not receive the data they should not have.

Filter

Select records by payload, key, header, or timestamp before they cross a boundary.

Explore the DSL

Transform

Extract fields, reshape payloads, and hash or remove sensitive values in flight.

Build a pipeline

Route

Fan one source topic out to distinct destinations with policy defined per output.

See examples

First run

Validate the policy before moving data.

The local demo starts Redpanda, validates the checked-in selective replication config, and runs StreamForge against the same file.

Follow the complete quickstart

terminal

$ podman compose -f \
  examples/redpanda/docker-compose.yml up -d

$ cargo run --quiet \
  --bin streamforge-validate -- \
  examples/redpanda/selective-replication.yaml

$ CONFIG_FILE=examples/redpanda/\
selective-replication.yaml cargo run --release \
  --bin streamforge

Choose the operating model

Start as a binary. Grow into Kubernetes.

Standalone

Configuration-driven

Run one binary with YAML-defined sources, destinations, filtering, and transformation.

Configure the binary

Kubernetes

Operator-managed

Install with Helm, declare pipelines as custom resources, and operate through Kubernetes.

Deploy on Kubernetes

A deliberate boundary

Selective replication, not a universal MirrorMaker replacement.

Choose StreamForge for field-level shaping, PII-safe replication, topic fan-out, and analytics or lake delivery.

Choose MirrorMaker 2 when you need active-active replication, consumer-offset synchronization, ACL mirroring, or Kafka Connect integration.

Compare the fit

Documentation

From first record to production operations.

StartQuickstartRun the local selective replication demo. BuildUsage guideDesign filters, transforms, and routes. SecureSecurityConfigure TLS, SASL, and credentials. ObserveObservabilityMonitor metrics and consumer lag. ReasonDelivery guaranteesUnderstand retries, DLQs, and failure modes. ExtendWebAssembly UDFsRun digest-pinned, resource-bounded custom policy. VerifyCompatibilityReview supported broker targets and scope.