User Experience on mobile might not be great yet, but I'm working on it.

Your first time on this page? Allow me to give some explanations.

Awesome Streaming

a curated list of awesome streaming frameworks, applications, etc

Here you can see meta information about this topic like the time we last updated this page, the original creator of the awesome list and a link to the original GitHub repository.

Last Update: Sept. 25, 2022, 6:08 p.m.

Thank you manuzhang & contributors
View Topic on GitHub:
manuzhang/awesome-streaming

Search for resources by name or description.
Simply type in what you are looking for and the results will be filtered on the fly.

Further filter the resources on this page by type (repository/other resource), number of stars on GitHub and time of last commit in months.

Streaming Engine

Mirror of Apache Apex core

346
179
1y 111d
Apache-2.0

Apache Arrow Ballista Distributed Query Engine

184
25
52d
Apache-2.0

Apache Flink

17.45K
9.78K
10m
Apache-2.0

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter

3.56K
620
10m
Apache-2.0

Mirror of Apache Samza

701
291
11m
Apache-2.0

Apache Spark - A unified analytics engine for large-scale data processing

31.23K
24.69K
10m
Apache-2.0

Mirror of Apache Storm

6.29K
4.08K
10m
n/a

SQL-based streaming analytics platform at scale

1.2K
291
2y 97d
Apache-2.0

Python Stream Processing

6K
513
1y 11m
n/a

Lightweight real-time big data streaming engine over Akka

754
162
12m
Apache-2.0

Distributed Stream and Batch Processing

943
187
1y 16d
n/a

Haskell distributed stream processing with exactly-once semantics

87
7
8y 109d
Apache-2.0

Stream Processing Made Easy

12
0
11m
MIT

A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications

1.17K
101
11m
Apache-2.0

Muppet

126
34
1y 4m
Apache-2.0

Distributed, masterless, high performance, fault tolerant data processing

2.02K
212
3y 27d
EPL-1.0

Mirror of Apache S4

39
17
3y 9m
Apache-2.0

Window-Based Hybrid CPU/GPU Stream Processing Engine

38
11
1y 95d
n/a

Flexible and efficient data processing engine and an evolution of the popular Scramjet Framework based on node.js. Our Transform Hub was designed specifically for data processing and has its own unique algorithms included.

27
2
10m
AGPL-3.0

Spooker is a dynamic framework for processing high volume data streams via processing pipelines

27
5
6y 7m
Apache-2.0

High Throughput Real-time Stream Processing Framework

277
35
5y 5m
Apache-2.0

The core libraries of the teknek stream processing platform

7
1
6y 9m
n/a

Trill is a single-node query processor for temporal or streaming data.

1.16K
121
1y 61d
MIT

Distributed Stream Processing

1.46K
69
1y 5m
Apache-2.0

Multi-core Window-Based Stream Processing Engine

36
6
11m
Apache-2.0

The database built for IoT streaming data storage and real-time stream processing.

451
46
7m
n/a

A lightweight IoT edge analytics software

524
154
10m
Apache-2.0

[C++] - A C++17 Data Stream Processing Parallel Library for Multicores and GPUs

Streaming Library

Mirror of Apache Kafka

20.27K
10.7K
10m
Apache-2.0

.NET Stream Processing Library for Apache Kafka ๐Ÿš€

300
51
24d
MIT

Build highly concurrent, distributed, and resilient message-driven applications on the JVM

11.84K
3.44K
10m
n/a

Daggy - Data Aggregation Utility and C/C++ developer library for data streams catching

123
10
8m
MIT

Fancy stream processing made operationally mundane

3.99K
382
7m
MIT

Compositional, streaming I/O library for Scala

1.97K
514
10m
n/a

Asynchronous, Reactive Programming for Scala and Scala.js.

1.81K
233
10m
Apache-2.0

Simple yet powerful live data computation framework.

27
0
5m
MIT

Python port of Scramjet framework

24
0
5m
MIT

Simple yet powerful live data computation framework. C++ port of Scramjet framework.

2
0
62d
MIT

StreamLine - Streaming Analytics

155
98
11m
Apache-2.0

StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.

2.61K
327
11m
Apache-2.0

A lightweight Reactive Streams Infrastructure Toolkit for Scala.

174
22
4y 101d
MPL-2.0

Real-time stream processing for python

1K
127
10m
n/a

Stream Ops is a fully embeddable data streaming engine and stream processing API for Java.

42
9
2y 11m
n/a

Streaming reactive and dataflow graphs in Python

276
25
1y 34d
Apache-2.0

๐Ÿฆ– Serverless Streaming Framework for Low-latency Edge Computing applications, running atop QUIC protocol, as Metaverse infrastructure, engaging 5G technology.

623
54
10m
Apache-2.0

Streaming Application

A platform for real-time streaming search

99
22
6y 6m
MIT

A scalable, mature and versatile web crawler based on Apache Storm

727
228
10m
Apache-2.0

IoT

Lightweight stream processing engine for IoT

210
40
2y 10m
MIT

Mirror of Apache Edgent (Incubating)

203
140
2y 11m
Apache-2.0

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

266
53
10m
Apache-2.0

DSL

Apache Beam is a unified programming model for Batch and Streaming

5.06K
3.23K
10m
n/a

Experiments in Streaming

59
3
6y 31d
Apache-2.0

Esper Complex Event Processing, Streaming SQL and Event Series Analysis

713
227
11m
n/a

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.

1.45K
217
8m
n/a

Streaming MapReduce with Scalding and Storm

2.11K
260
1y 10m
Apache-2.0

Data Pipeline

Mirror of Apache Kafka

20.27K
10.7K
10m
Apache-2.0

Apache Pulsar - distributed pub-sub messaging system

9.83K
2.47K
10m
Apache-2.0

Mirror of Apache RocketMQ

16.4K
9.2K
7m
Apache-2.0

An extensible distributed system for reliable nearline data streaming at scale

728
107
10m
BSD-2-Clause

LinkedIn's previous generation Kafka to HDFS pipeline.

884
473
2y 30d
n/a

Source-agnostic distributed change data capture system

3.31K
707
2y 4m
Apache-2.0

Mirror of Apache Flume

2.18K
1.46K
10m
Apache-2.0

An intelligent end-to-end data streaming platform that combines connectors, distributed streaming and state processing.

895
67
7m
Apache-2.0

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

284
33
10m
MIT

A high available,high performance distributed messaging system.

1.31K
702
2y 5m
Apache-2.0

NATS Streaming System Server

2.33K
279
11m
Apache-2.0

A realtime distributed messaging platform

20.47K
2.62K
11m
MIT

Redpanda is a streaming data platform for developers. Kafka API compatible, 10x faster, ZooKeeper free, JVM free!

3.58K
223
6m
n/a

Privacy and Security focused Segment-alternative, in Golang and React

2.79K
166
10m
AGPL-3.0

Netflix's distributed Data Pipeline

765
177
6y 9m
Apache-2.0

datacollector-oss

28
33
1y 11d
Apache-2.0

[C++] - a high-performant distributed system by Facebook for streaming and storing sequential data, using a log structure.

Online Machine Learning

Mirror of Apache Samoa (Incubating)

238
108
2y 7m
Apache-2.0

Core Java Sketch Library.

734
188
10m
Apache-2.0

Stream Data Mining Library for Spark Streaming

464
146
1y 7m
Apache-2.0

Python application to setup and run streaming (contextual) bandit experiments.

72
17
2y 109d
MIT

Apache Storm + OpenCV = large scale distributed image and video analysis.

165
60
5y 9m
Apache-2.0

Trident-ML : A realtime online machine learning library

386
90
2y 7m
n/a

Anomaly detection framework @ PayPal

105
34
3y 25d
Apache-2.0

Streaming SQL

High-performance time-series aggregation for PostgreSQL

2.44K
231
1y 51d
Apache-2.0

A streaming / online query processing / analytics engine based on Apache Storm

266
97
5y 4m
Apache-2.0
0
0
5y 77d
Apache-2.0

The database purpose-built for stream processing applications.

4.58K
863
10m
n/a

Stream Processing and Complex Event Processing Engine

1.31K
479
7m
Apache-2.0

[Rust] - A source-available streaming SQL engine for maintaining materialized views on data from message brokers and databases.

Benchmark

A simple storm performance/stress test

78
38
1y 11m
n/a

Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, ...

553
292
1y 114d
Apache-2.0

Automated message queue orchestration for scaled-up benchmarking.

231
38
6y 8m
Apache-2.0

Toolkit

Build highly concurrent, distributed, and resilient message-driven applications on the JVM

11.84K
3.44K
10m
n/a

Event driven concurrent framework for Python

1.87K
165
2y 9m
BSD-3-Clause

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

5.85K
713
7m
Apache-2.0

StreamFlowโ„ข is a stream processing tool designed to help build and monitor processing workflows.

241
66
2y 7m
Apache-2.0

Integration of Samza and Luwak

99
16
7y 10m
n/a

SSE Stream Aggregator

821
257
4y 111d
Apache-2.0

A visual tool to define and run real-time decision algorithms. Brings agility to business teams, liberates developers to focus on technology.

237
62
7m
Apache-2.0

Closed Source

[.NET] a massively scalable, fully managed, real-time, data stream engine provided by Microsoft Azure.

[C++] - a distributed stream processing framework built in C++ on top of Apache.

[C++] - distributed processing framework and streaming machine learning library.

Readings