Search awesome repositories

Apache Superset is a Data Visualization and Data Exploration Platform

Learn how to develop, deploy and iterate on production-grade ML applications.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Workflow Engine for Kubernetes

An orchestration platform for the development, production, and observation of data assets.

The Data Engineering Cookbook

Always know what to expect from your data.

Always know what to expect from your data.

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Fancy stream processing made operationally mundane

Open Source Feature Flags, Experimentation, and Product Analytics

The Open Source Feature Store for AI/ML

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

lakeFS - Data version control for your data lake | Git for data

Privacy and Security focused Segment-alternative, in Golang and React

DeepAnalyze is the first agentic LLM for autonomous data science. 🎈你的AI数据分析师，自动分析大量数据，一键生成专业分析报告！

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Maestro: Netflix’s Workflow Orchestrator

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Learn to build your Second Brain AI assistant with LLMs, agents, RAG, fine-tuning, LLMOps and AI systems techniques.

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.