chitu

A high-throughput and memory-efficient inference and serving engine for LLMs

A high-performance ML model serving framework, offers dynamic batching and CPU/G

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qw

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs