Awesome Bigdata

Apache Spark - A unified analytics engine for large-scale data processing

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Apache Druid: a high performance real-time analytics database.

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

The Open Source Feature Store for AI/ML

Gephi - The Open Graph Viz Platform

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

A Graph Traversal Language (no longer active - see Apache TinkerPop)

An Internet-Scale Database.

Secor is a service implementing Kafka log persistence

A large-scale entity and relation database supporting aggregation of properties

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Fast scalable time series database

🦎 A lightweight, multi-protocol gateway for event-driven applications and AI agents. Expose and govern Kafka, MQTT, APIs, and MCP through one high-performance engine with shared routing, security, schema, and observability.

Bloomberg's distributed RDBMS

HiBench is a big data benchmark suite.

An open source event analytics platform

SQL-based streaming analytics platform at scale

Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.

CPU and GPU-accelerated Machine Learning Library

📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)

Banana for Solr - A Port of Kibana

A distributed system designed to ingest and process time series data

A flexible, partial, out-of-order and real-time typeahead search library

An open-source columnar data format designed for fast & realtime analytic with big data.

No description.

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

Accumulo backed time series database

A Relational Database Backed by Apache Kafka

realtime search/indexing system

Sparrow scheduling platform (U.C. Berkeley).

High Throughput Real-time Stream Processing Framework

Haeinsa is linearly scalable multi-row, multi-table transaction library for HBase

Apache Tephra: Transactions for HBase.

RDF-Centric Map/Reduce Framework and Freebase data conversion tool

No description.

Schedoscope is a scheduling framework for painfree agile development, testing, (re)loading, and monitoring of your datahub, lake, or whatever you choose to call your Hadoop data warehouse these days.

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

Mirror of Apache Slider

Tuple MapReduce for Hadoop: Hadoop API made easy

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

Develop streaming applications for IBM Streams in Python, Java & Scala.

DocId set compression and set operation library