Data Processing & Streaming

Why not Kafka + Spark? Broadway builds concurrent, multi-stage data pipelines with back-pressure and batch processing built in. GenStage provides the producer-consumer foundation. Flow adds MapReduce-style parallel processing. Membrane handles multimedia streaming. All running on the BEAM's lightweight processes — no JVM cluster to manage.

Broadway

library

Need concurrent data ingestion pipelines?

Build multi-stage, concurrent data processing pipelines with back-pressure and batch processing

Replaces: Kafka consumers Spark Streaming Flink

GenStage

library

Need back-pressure aware data processing?

Producer-consumer pipeline framework with built-in back-pressure for demand-driven data processing

Replaces: Kafka RabbitMQ consumers

Flow

library

Need to process data in parallel?

Computational parallel flows built on GenStage for bounded and unbounded data processing

Replaces: Apache Spark Hadoop MapReduce

Membrane

library

Need audio/video processing?

Multimedia processing framework for audio, video, and streaming with pipeline architecture

Replaces: FFmpeg wrappers GStreamer Wowza

View all tools

Companion Project

Postgres Is Enough

Simplify your database layer too. A curated directory of PostgreSQL extensions and tools that replace Redis, Elasticsearch, Kafka, and more.