User Experience on mobile might not be great yet, but
I'm working on it.
Your first time on this page? Allow me to give some explanations.
Awesome Streaming
a curated list of awesome streaming frameworks, applications, etc
Here you can see meta information about this topic like the time we last updated this page, the original creator of the awesome list and a link to the original GitHub repository.
Thank you manuzhang & contributors
View Topic on GitHub:
manuzhang/awesome-streaming
Search for resources by name or description.
Simply type in what you are looking for and the results will be filtered on the fly.
Further filter the resources on this page by type (repository/other resource), number of stars on GitHub and time of last commit in months.
Streaming Engine
Mirror of Apache Apex core
Apache Arrow DataFusion and Ballista query engines
Apache Flink
Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
Mirror of Apache Samza
Apache Spark - A unified analytics engine for large-scale data processing
Mirror of Apache Storm
SQL-based streaming analytics platform at scale
Python Stream Processing
Lightweight real-time big data streaming engine over Akka
Distributed Stream and Batch Processing
Haskell distributed stream processing with exactly-once semantics
Stream Processing Made Easy
A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications
Muppet
Distributed, masterless, high performance, fault tolerant data processing
Mirror of Apache S4
Window-Based Hybrid CPU/GPU Stream Processing Engine
Flexible and efficient data processing engine and an evolution of the popular Scramjet Framework based on node.js. Our Transform Hub was designed specifically for data processing and has its own unique algorithms included.
Spooker is a dynamic framework for processing high volume data streams via processing pipelines
High Throughput Real-time Stream Processing Framework
The core libraries of the teknek stream processing platform
Trill is a single-node query processor for temporal or streaming data.
Distributed Stream Processing
Multi-core Window-Based Stream Processing Engine
The database built for IoT streaming data storage and real-time stream processing.
A lightweight IoT edge analytics software
Streaming Library
Mirror of Apache Kafka
Build highly concurrent, distributed, and resilient message-driven applications on the JVM
Declarative streaming ETL for mundane tasks, written in Go
Compositional, streaming I/O library for Scala
Asynchronous, Reactive Programming for Scala and Scala.js.
Simple yet powerful live data computation framework
StreamLine - Streaming Analytics
StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
A lightweight Reactive Streams Infrastructure Toolkit for Scala.
Real-time stream processing for python
Stream Ops is a fully embeddable data streaming engine and stream processing API for Java.
Streaming reactive and dataflow graphs in Python
🦖 Serverless Streaming Framework for Low-latency Edge Computing applications, running atop QUIC protocol, as Metaverse infrastructure, engaging 5G technology.
Streaming Application
A platform for real-time streaming search
A scalable, mature and versatile web crawler based on Apache Storm
IoT
Lightweight stream processing engine for IoT
Mirror of Apache Edgent (Incubating)
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.
DSL
Apache Beam is a unified programming model for Batch and Streaming
Experiments in Streaming
Esper Complex Event Processing, Streaming SQL and Event Series Analysis
Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
Streaming MapReduce with Scalding and Storm
Data Pipeline
Mirror of Apache Kafka
Apache Pulsar - distributed pub-sub messaging system
An extensible distributed system for reliable nearline data streaming at scale
LinkedIn's previous generation Kafka to HDFS pipeline.
Source-agnostic distributed change data capture system
Mirror of Apache Flume
Programmable platform for data in motion
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
A high available,high performance distributed messaging system.
NATS Streaming System Server
A realtime distributed messaging platform
Privacy and Security focused Segment-alternative, in Golang and React
Netflix's distributed Data Pipeline
datacollector-oss
C++] - a high-performant distributed system by Facebook for streaming and storing sequential data, using a log structure.
Online Machine Learning
Mirror of Apache Samoa (Incubating)
Core Java Sketch Library.
Stream Data Mining Library for Spark Streaming
Python application to setup and run streaming (contextual) bandit experiments.
Apache Storm + OpenCV = large scale distributed image and video analysis.
Trident-ML : A realtime online machine learning library
Anomaly detection framework @ PayPal
Streaming SQL
High-performance time-series aggregation for PostgreSQL
A streaming / online query processing / analytics engine based on Apache Storm
The event streaming database purpose-built for stream processing applications
Stream Processing and Complex Event Processing Engine
[Rust] - A source-available streaming SQL engine for maintaining materialized views on data from message brokers and databases.
Benchmark
A simple storm performance/stress test
Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, ...
Automated message queue orchestration for scaled-up benchmarking.
Toolkit
Build highly concurrent, distributed, and resilient message-driven applications on the JVM
Event driven concurrent framework for Python
Efficient reliable UDP unicast, UDP multicast, and IPC message transport
StreamFlow™ is a stream processing tool designed to help build and monitor processing workflows.
Integration of Samza and Luwak
SSE Stream Aggregator
Closed Source
Provides real-time data processing over large, distributed data streams.
NET] a massively scalable, fully managed, real-time, data stream engine provided by Microsoft Azure.
Serverless stream and batch data processing service.
C++] - a distributed stream processing framework built in C++ on top of Apache.
platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)
C++] - distributed processing framework and streaming machine learning library.
framework for building low-latency data-processing applications that is widely used at Google.
Readings
Grokking Streaming Systems helps you unravel what streaming systems are, how they work, and whether they’re right for your business. Written to be tool-agnostic, you’ll be able to apply what you learn no matter which framework you choose.