Search awesome repositories

A high-throughput and memory-efficient inference and serving engine for LLMs

Port of OpenAI's Whisper model in C/C++

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Making large AI models cheaper, faster and more accessible

Cross-platform, customizable ML solutions for live and streaming media.

SGLang is a high-performance serving framework for large language models and multimodal models.

Faster Whisper transcription with CTranslate2

ncnn is a high-performance neural network inference framework optimized for the mobile platform

Nano vLLM

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

Easily fine-tune, evaluate and deploy Gemma 4, Qwen3.5, Qwen3.6, gpt-oss, DeepSeek-R1, or any open source LLM / VLM!

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

Fast inference engine for Transformer models

Pre-trained Deep Learning models and demos (high quality and extremely fast)

Achieve state of the art inference performance with modern accelerators on Kubernetes

Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

PyTorch native quantization and sparsity for training and inference

Vendor-agnostic orchestration for training, inference and agentic workloads across NVIDIA, AMD, TPU, and Tenstorrent on clouds, Kubernetes, and bare metal.

Manages Unified Access to Generative AI Services built on Envoy Gateway

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.