Pinned Loading

1. ModelCloud/GPTQModel ModelCloud/GPTQModel Public

   Production ready LLM model compression/quantization toolkit with accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.

   Python 163 30

2. ModelCloud/Device-SMI ModelCloud/Device-SMI Public

   Self-contained Python lib with zero-dependencies that give you a unified device properties for gpu, cpu, and npu. No more calling separate tools such as nvidia-smi or /proc/cpuinfo and parsing it y…

   Python 9

3. sgl-project/sglang sgl-project/sglang Public

   SGLang is a fast serving framework for large language models and vision language models.

   Python 6.6k 595

4. vllm-project/vllm vllm-project/vllm Public

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python 32.4k 4.9k

5. flashinfer-ai/flashinfer flashinfer-ai/flashinfer Public

   FlashInfer: Kernel Library for LLM Serving

   Cuda 1.6k 160

6. Dao-AILab/flash-attention Dao-AILab/flash-attention Public

   Fast and memory-efficient exact attention

   Python 14.7k 1.4k