Awesome Opensource Ai

A high-throughput and memory-efficient inference and serving engine for LLMs

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Making large AI models cheaper, faster and more accessible

Cross-platform, customizable ML solutions for live and streaming media.

SGLang is a high-performance serving framework for large language models and multimodal models.

Faster Whisper transcription with CTranslate2

ncnn is a high-performance neural network inference framework optimized for the mobile platform

Nano vLLM

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

Easily fine-tune, evaluate and deploy Qwen, Gemma, or any open weight LLM!

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

Fast inference engine for Transformer models

Pre-trained Deep Learning models and demos (high quality and extremely fast)

Achieve state of the art inference performance with modern accelerators on Kubernetes

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

PyTorch native quantization for training and inference

Vendor-agnostic orchestration for training, inference and agentic workloads across NVIDIA, AMD, TPU, and Tenstorrent on clouds, Kubernetes, and bare metal.

Manages Unified Access to Generative AI Services built on Envoy Gateway

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.