User Experience on mobile might not be great yet, but I'm working on it.

Your first time on this page? Allow me to give some explanations.

Awesome Streaming

a curated list of awesome streaming frameworks, applications, etc

Here you can see meta information about this topic like the time we last updated this page, the original creator of the awesome list and a link to the original GitHub repository.

Last Update: Oct. 20, 2021, 9:11 a.m.

Thank you manuzhang & contributors
View Topic on GitHub:
manuzhang/awesome-streaming

Search for resources by name or description.
Simply type in what you are looking for and the results will be filtered on the fly.

Further filter the resources on this page by type (repository/other resource), number of stars on GitHub and time of last commit in months.

Streaming Engine

Mirror of Apache Apex core

345
178
4y 36d
Apache-2.0

Apache Arrow DataFusion and Ballista query engines

550
70
4m
Apache-2.0

Apache Flink

15.54K
8.59K
8m
Apache-2.0

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter

3.5K
614
8m
Apache-2.0

Mirror of Apache Samza

675
277
8m
Apache-2.0

Apache Spark - A unified analytics engine for large-scale data processing

28.83K
23.38K
8m
Apache-2.0

Mirror of Apache Storm

6.2K
4.07K
8m
n/a

SQL-based streaming analytics platform at scale

1.17K
281
3y 78d
Apache-2.0

Python Stream Processing

5.32K
438
1y 11d
n/a

Lightweight real-time big data streaming engine over Akka

746
159
1y 10m
Apache-2.0

Distributed Stream and Batch Processing

844
172
8m
n/a

Haskell distributed stream processing with exactly-once semantics

86
7
7y 4m
Apache-2.0

Stream Processing Made Easy

7
0
8m
MIT

A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications

1.09K
92
8m
Apache-2.0

Muppet

126
32
6y 112d
Apache-2.0

Distributed, masterless, high performance, fault tolerant data processing

1.99K
211
2y 51d
EPL-1.0

Mirror of Apache S4

39
17
9y 83d
Apache-2.0

Window-Based Hybrid CPU/GPU Stream Processing Engine

35
10
1y 46d
n/a

Flexible and efficient data processing engine and an evolution of the popular Scramjet Framework based on node.js. Our Transform Hub was designed specifically for data processing and has its own unique algorithms included.

26
2
28d
AGPL-3.0

Spooker is a dynamic framework for processing high volume data streams via processing pipelines

27
5
5y 8m
Apache-2.0

High Throughput Real-time Stream Processing Framework

274
34
4y 6m
Apache-2.0

The core libraries of the teknek stream processing platform

7
1
5y 10m
Apache-2.0

Trill is a single-node query processor for temporal or streaming data.

1.11K
109
8m
MIT

Distributed Stream Processing

1.44K
66
11m
Apache-2.0

Multi-core Window-Based Stream Processing Engine

29
5
9m
Apache-2.0

The database built for IoT streaming data storage and real-time stream processing.

258
21
6m
n/a

A lightweight IoT edge analytics software

396
113
4m
Apache-2.0

Streaming Library

Mirror of Apache Kafka

18.06K
9.61K
8m
Apache-2.0

Build highly concurrent, distributed, and resilient message-driven applications on the JVM

11.38K
3.35K
8m
n/a

Declarative streaming ETL for mundane tasks, written in Go

2.87K
249
7m
MIT

Compositional, streaming I/O library for Scala

1.81K
473
8m
n/a

Asynchronous, Reactive Programming for Scala and Scala.js.

1.76K
228
8m
Apache-2.0

Simple yet powerful live data computation framework

191
15
39d
MIT

StreamLine - Streaming Analytics

151
95
2y 96d
Apache-2.0

StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.

2.46K
303
8m
Apache-2.0

A lightweight Reactive Streams Infrastructure Toolkit for Scala.

175
23
3y 4m
MPL-2.0

Real-time stream processing for python

910
113
9m
n/a

Stream Ops is a fully embeddable data streaming engine and stream processing API for Java.

38
8
2y 1d
n/a

Streaming reactive and dataflow graphs in Python

224
23
9m
Apache-2.0

๐Ÿฆ– Serverless Streaming Framework for Low-latency Edge Computing applications, running atop QUIC protocol, as Metaverse infrastructure, engaging 5G technology.

576
47
15d
Apache-2.0

Streaming Application

A platform for real-time streaming search

98
21
5y 7m
MIT

A scalable, mature and versatile web crawler based on Apache Storm

696
222
8m
Apache-2.0

IoT

Lightweight stream processing engine for IoT

200
33
1y 11m
MIT

Mirror of Apache Edgent (Incubating)

201
135
2y 6m
Apache-2.0

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

204
38
8m
Apache-2.0

DSL

Apache Beam is a unified programming model for Batch and Streaming

4.59K
2.87K
8m
Apache-2.0

Experiments in Streaming

59
3
5y 55d
Apache-2.0

Esper Complex Event Processing, Streaming SQL and Event Series Analysis

675
214
8m
n/a

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.

1.43K
216
10m
n/a

Streaming MapReduce with Scalding and Storm

2.09K
260
2y 8m
Apache-2.0

Data Pipeline

Mirror of Apache Kafka

18.06K
9.61K
8m
Apache-2.0

Apache Pulsar - distributed pub-sub messaging system

7.3K
1.83K
8m
Apache-2.0

An extensible distributed system for reliable nearline data streaming at scale

616
81
1y 49d
BSD-2-Clause

LinkedIn's previous generation Kafka to HDFS pipeline.

877
472
6y 0d
n/a

Source-agnostic distributed change data capture system

3.1K
685
1y 5m
Apache-2.0

Mirror of Apache Flume

2.07K
1.41K
1y 5m
Apache-2.0

Programmable platform for data in motion

519
34
84d
Apache-2.0

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

228
30
1y 33d
MIT

A high available,high performance distributed messaging system.

1.29K
699
4y 9m
Apache-2.0

NATS Streaming System Server

2.21K
272
8m
Apache-2.0

A realtime distributed messaging platform

19.2K
2.49K
8m
MIT

Privacy and Security focused Segment-alternative, in Golang and React

2.28K
111
8m
AGPL-3.0

Netflix's distributed Data Pipeline

752
173
5y 10m
Apache-2.0

datacollector-oss

13
11
5m
Apache-2.0

C++] - a high-performant distributed system by Facebook for streaming and storing sequential data, using a log structure.

Online Machine Learning

Mirror of Apache Samoa (Incubating)

238
105
1y 7m
Apache-2.0

Core Java Sketch Library.

696
185
8m
Apache-2.0

Stream Data Mining Library for Spark Streaming

453
144
12m
Apache-2.0

Python application to setup and run streaming (contextual) bandit experiments.

69
16
1y 4m
MIT

Apache Storm + OpenCV = large scale distributed image and video analysis.

163
60
5y 14d
Apache-2.0

Trident-ML : A realtime online machine learning library

383
90
6y 20d
n/a

Anomaly detection framework @ PayPal

103
30
2y 49d
Apache-2.0

Streaming SQL

High-performance time-series aggregation for PostgreSQL

2.36K
217
2y 5m
Apache-2.0

A streaming / online query processing / analytics engine based on Apache Storm

268
98
4y 5m
Apache-2.0
0
0
4y 101d
Apache-2.0

The event streaming database purpose-built for stream processing applications

4.05K
757
8m
n/a

Stream Processing and Complex Event Processing Engine

1.18K
448
7m
Apache-2.0

[Rust] - A source-available streaming SQL engine for maintaining materialized views on data from message brokers and databases.

Benchmark

42
40
4y 11m
Apache-2.0

A simple storm performance/stress test

78
39
1y 7d
n/a

Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, ...

527
282
1y 7d
Apache-2.0

Automated message queue orchestration for scaled-up benchmarking.

231
36
5y 9m
Apache-2.0

Toolkit

Build highly concurrent, distributed, and resilient message-driven applications on the JVM

11.38K
3.35K
8m
n/a

Event driven concurrent framework for Python

1.86K
161
1y 9m
BSD-3-Clause

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

5.39K
640
7m
Apache-2.0

StreamFlowโ„ข is a stream processing tool designed to help build and monitor processing workflows.

239
65
6y 53d
Apache-2.0

Integration of Samza and Luwak

101
15
6y 11m
n/a

SSE Stream Aggregator

811
250
6y 104d
Apache-2.0

Closed Source

Provides real-time data processing over large, distributed data streams.

NET] a massively scalable, fully managed, real-time, data stream engine provided by Microsoft Azure.

Serverless stream and batch data processing service.

C++] - a distributed stream processing framework built in C++ on top of Apache.

platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)

C++] - distributed processing framework and streaming machine learning library.

framework for building low-latency data-processing applications that is widely used at Google.

Readings