Search awesome repositories

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Find secrets with Gitleaks 🔑

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

Official inference library for Mistral models

High-speed Large Language Model Serving for Local Deployment

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

A Datacenter Scale Distributed Inference Serving Framework

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

FlashInfer: Kernel Library for LLM Serving

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

Optimizing inference proxy for LLMs

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.

The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that interact with your data and UI.

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.