Trending repositories for topic llm-inference

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

gpt4all: run open-source LLMs anywhere

本项目旨在分享大模型相关技术原理以及实战经验。

Official inference library for Mistral models

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

The easiest way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Multi-model Inference Graph/Pipelines, LLM/RAG apps, and more!

The open-source serverless GPU container runtime.

FlashInfer: Kernel Library for LLM Serving

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

AICI: Prompts as (Wasm) Programs

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

LLM in Godot

Gradio based tool to run opensource LLM models directly from Huggingface

Sequence Parallel Attention for Long Context LLM Model Training and Inference

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

A Python package for LLM dynamic routing through the Unify REST API.

LLM in Godot

Gradio based tool to run opensource LLM models directly from Huggingface

Run generative AI models in sophgo BM1684X

The open-source serverless GPU container runtime.

Sequence Parallel Attention for Long Context LLM Model Training and Inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

A collection of hand on notebook for LLMs practitioner

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.

开源的智能体项目 支持6种聊天平台 Onebotv11一对多连接 流式信息 agent 对话keyboard气泡生成 支持6种大模型接口(持续增加中) 具有将多种大模型接口转化为带有上下文的通用格式的能力.

An innovative library for efficient LLM inference via low-bit quantization

FlashInfer: Kernel Library for LLM Serving

A multi-platform SwiftUI frontend for running local LLMs with Apple's MLX framework.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

本项目旨在分享大模型相关技术原理以及实战经验。

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

A library to communicate with ChatGPT, Claude, Copilot, Gemini, HuggingChat, and Pi

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

gpt4all: run open-source LLMs anywhere

本项目旨在分享大模型相关技术原理以及实战经验。

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

Official inference library for Mistral models

Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

The easiest way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Multi-model Inference Graph/Pipelines, LLM/RAG apps, and more!

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

FlashInfer: Kernel Library for LLM Serving

Sequence Parallel Attention for Long Context LLM Model Training and Inference

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

Practical Llama 3 inference in Java

LLM in Godot

The open-source serverless GPU container runtime.

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

A Python package for LLM dynamic routing through the Unify REST API.

LLM in Godot

Gradio based tool to run opensource LLM models directly from Huggingface

Sequence Parallel Attention for Long Context LLM Model Training and Inference

Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

开源的智能体项目 支持6种聊天平台 Onebotv11一对多连接 流式信息 agent 对话keyboard气泡生成 支持6种大模型接口(持续增加中) 具有将多种大模型接口转化为带有上下文的通用格式的能力.

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

Run generative AI models in sophgo BM1684X

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

Practical Llama 3 inference in Java

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

The open-source serverless GPU container runtime.

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inferenc...

An innovative library for efficient LLM inference via low-bit quantization

A collection of hand on notebook for LLMs practitioner

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

export llama to onnx

FlashInfer: Kernel Library for LLM Serving

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

gpt4all: run open-source LLMs anywhere

本项目旨在分享大模型相关技术原理以及实战经验。

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

Practical Llama 3 inference in Java

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.

Official inference library for Mistral models

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

The open-source serverless GPU container runtime.

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

The easiest way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Multi-model Inference Graph/Pipelines, LLM/RAG apps, and more!

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

FlashInfer: Kernel Library for LLM Serving

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalab...

Sequence Parallel Attention for Long Context LLM Model Training and Inference

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

LLM in Godot

The open-source serverless GPU container runtime.

Gradio based tool to run opensource LLM models directly from Huggingface

A Python package for LLM dynamic routing through the Unify REST API.

Sequence Parallel Attention for Long Context LLM Model Training and Inference

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inferenc...

Run generative AI models in sophgo BM1684X

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.

开源的智能体项目 支持6种聊天平台 Onebotv11一对多连接 流式信息 agent 对话keyboard气泡生成 支持6种大模型接口(持续增加中) 具有将多种大模型接口转化为带有上下文的通用格式的能力.

Efficient and general syntactical decoding for Large Language Models

Local LLM inference Library

Chat With RTX Python API

Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application

An innovative library for efficient LLM inference via low-bit quantization

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

Official inference library for Mistral models

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Code examples and resources for DBRX, a large language model developed by Databricks

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

AICI: Prompts as (Wasm) Programs

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

LLMs as Copilots for Theorem Proving in Lean

FlashInfer: Kernel Library for LLM Serving

LLMFlows - Simple, Explicit and Transparent LLM Apps

[ICML'24] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

Efficient AI Inference & Serving

The official repo of Aquila2 series proposed by BAAI, including pretrained & chat large language models.

LLM (Large Language Model) FineTuning

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search Engine.

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

gpt4all: run open-source LLMs anywhere

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

Official inference library for Mistral models

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

本项目旨在分享大模型相关技术原理以及实战经验。

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalab...

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Code examples and resources for DBRX, a large language model developed by Databricks

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

AICI: Prompts as (Wasm) Programs

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

The easiest way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Multi-model Inference Graph/Pipelines, LLM/RAG apps, and more!

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

Sparsity-aware deep learning inference runtime for CPUs

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

本项目旨在分享大模型相关技术原理以及实战经验。

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalab...

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

LLMs as Copilots for Theorem Proving in Lean

An innovative library for efficient LLM inference via low-bit quantization

irresponsible innovation. Try now at https://chat.dev/

LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search Engine.

Official inference library for Mistral models

Sequence Parallel Attention for Long Context LLM Model Training and Inference

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inferenc...

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

[ICML'24] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource...

[HPCA'21] SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning

Gradio based tool to run opensource LLM models directly from Huggingface

Minimalist web-searching app with an AI assistant that runs directly from your browser. Uses Web-LLM, Ratchet-ML, Wllama and SearXNG. Demo: https://felladrin-minisearch.hf.space