Awesome Bigdata

An Open Source Machine Learning Framework for Everyone

Apache Superset is a Data Visualization and Data Exploration Platform

scikit-learn: machine learning in Python

Deep Learning for humans

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

Apache Spark - A unified analytics engine for large-scale data processing

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

A library for efficient similarity search and clustering of dense vectors.

TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

Data Apps & Dashboards for Python. No JavaScript Required.

An orchestration platform for the development, production, and observation of data assets.

Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

Easy & Flexible Alerting With ElasticSearch

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

The Open Source Feature Store for AI/ML

The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.

Privacy and Security focused Segment-alternative, in Golang and React

Deploy and manage containers (including Docker) on top of Apache Mesos at scale.

Scribe is a server for aggregating log data streamed in real time from a large number of servers.

Get your data in RAM. Get compute close to data. Enjoy the performance.

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter

🛠 All-in-one web-based IDE specialized for machine learning and data science.

PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Models (CIKM 2021)

chDB is an in-process OLAP SQL Engine 🚀 powered by ClickHouse

Open source framework for processing, monitoring, and alerting on time series data

Karate Club: An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs (CIKM 2020)

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Elassandra = Elasticsearch + Apache Cassandra

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

SQL-based streaming analytics platform at scale

Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster

Fast and reliable message broker built on top of Kafka.

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

Little Ball of Fur - A graph sampling extension library for NetworKit and NetworkX (CIKM 2020)

A distributed system designed to ingest and process time series data

SiriDB is a highly-scalable, robust and super fast time series database. Build from the ground up SiriDB uses a unique mechanism to operate without a global index and allows server resources to be added on the fly. SiriDB's unique query language includes dynamic grouping of time series for easy analysis over large amounts of time series.

Accumulo backed time series database

Sparrow scheduling platform (U.C. Berkeley).

Serverless proxy for Spark cluster

A distributed Spark/Scala implementation of the isolation forest and extended isolation forest algorithms for unsupervised outlier detection, featuring support for scalable training and ONNX export for easy cross-platform inference.

The official implementation of "The Shapley Value of Classifiers in Ensemble Games" (CIKM 2021).