lorax

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-

A highly optimized LLM inference acceleration engine for Llama and its variants.

JetStream is a throughput and memory optimized engine for LLM inference on XLA d

A high-throughput and memory-efficient inference and serving engine for LLMs