Trending repositories for topic llm-inference

Graph-Based Editor for LLM Workflows

A programming framework for agentic AI 🤖 PyPi: autogen-agentchat Discord: https://aka.ms/autogen-discord Office Hour: https://aka.ms/autogen-officehour

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

FlashInfer: Kernel Library for LLM Serving

A list of software that allows searching the web with the assistance of AI.

MagicPIG: LSH Sampling for Efficient LLM Generation

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Chrome extension that interacts with content using Groq

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Arch is an intelligent gateway for agents. Engineered with (fast) LLMs for the secure handling, rich observability, and seamless integration of prompts with your APIs - all outside business logic. Bui...

Optimizing inference proxy for LLMs

Run any open-source LLMs, such as Llama, Mistral, as OpenAI compatible API endpoint in the cloud.

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Official inference library for Mistral models

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Efficient LLM Inference over Long Sequences

Chrome extension that interacts with content using Groq

Graph-Based Editor for LLM Workflows

MagicPIG: LSH Sampling for Efficient LLM Generation

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

A list of software that allows searching the web with the assistance of AI.

All the code and materials related to our YouTube channel (TwoSetAI)

The llama model inference lite framework by triton.

Open Source LLMOps tool for AI teams

llm theoretical performance analysis tools and support params, flops, memory and latency analysis.

FlashInfer: Kernel Library for LLM Serving

Efficient LLM Inference over Long Sequences

《大模型项目实战：多领域智能应用开发》配套资源

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes

Arch is an intelligent gateway for agents. Engineered with (fast) LLMs for the secure handling, rich observability, and seamless integration of prompts with your APIs - all outside business logic. Bui...

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

A research project to add some brrrrrr to Burp

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Uses WebLLM, Wllama and SearXNG. Demo: https://felladrin-minisearch.hf.space

Optimizing inference proxy for LLMs

开源的智能体项目 支持6种聊天平台 Onebotv11一对多连接 流式信息 agent 对话keyboard气泡生成 支持10+大模型接口(持续更新) 具有将多种大模型接口转化为带有上下文的通用格式的能力.

Graph-Based Editor for LLM Workflows

A programming framework for agentic AI 🤖 PyPi: autogen-agentchat Discord: https://aka.ms/autogen-discord Office Hour: https://aka.ms/autogen-officehour

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

FlashInfer: Kernel Library for LLM Serving

A list of software that allows searching the web with the assistance of AI.

Run any open-source LLMs, such as Llama, Mistral, as OpenAI compatible API endpoint in the cloud.

Arch is an intelligent gateway for agents. Engineered with (fast) LLMs for the secure handling, rich observability, and seamless integration of prompts with your APIs - all outside business logic. Bui...

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

MagicPIG: LSH Sampling for Efficient LLM Generation

📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉

Optimizing inference proxy for LLMs

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

The llama model inference lite framework by triton.

Chrome extension that interacts with content using Groq

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

Graph-Based Editor for LLM Workflows

Chrome extension that interacts with content using Groq

The llama model inference lite framework by triton.

All the code and materials related to our YouTube channel (TwoSetAI)

A research project to add some brrrrrr to Burp

MagicPIG: LSH Sampling for Efficient LLM Generation

A list of software that allows searching the web with the assistance of AI.

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes

Efficient LLM Inference over Long Sequences

llm theoretical performance analysis tools and support params, flops, memory and latency analysis.

Arch is an intelligent gateway for agents. Engineered with (fast) LLMs for the secure handling, rich observability, and seamless integration of prompts with your APIs - all outside business logic. Bui...

Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.

FlashInfer: Kernel Library for LLM Serving

[NeurIPS 2024] A Novel Rank-Based Metric for Evaluating Large Language Models

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

Pure C++ implementation of several models for real-time chatting on your computer (CPU)

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

A research project to add some brrrrrr to Burp

Chrome extension that interacts with content using Groq

A programming framework for agentic AI 🤖 PyPi: autogen-agentchat Discord: https://aka.ms/autogen-discord Office Hour: https://aka.ms/autogen-officehour

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Graph-Based Editor for LLM Workflows

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Efficient LLM Inference over Long Sequences

Arch is an intelligent gateway for agents. Engineered with (fast) LLMs for the secure handling, rich observability, and seamless integration of prompts with your APIs - all outside business logic. Bui...

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Optimizing inference proxy for LLMs

Run any open-source LLMs, such as Llama, Mistral, as OpenAI compatible API endpoint in the cloud.

📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉

A list of software that allows searching the web with the assistance of AI.

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

FlashInfer: Kernel Library for LLM Serving

Run serverless GPU workloads with fast cold starts on bare-metal servers, anywhere in the world

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

MagicPIG: LSH Sampling for Efficient LLM Generation

Standardized Serverless ML Inference Platform on Kubernetes

Graph-Based Editor for LLM Workflows

The llama model inference lite framework by triton.

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes

A research project to add some brrrrrr to Burp

MagicPIG: LSH Sampling for Efficient LLM Generation

Chrome extension that interacts with content using Groq

《大模型项目实战：多领域智能应用开发》配套资源

Open Source LLMOps tool for AI teams

All the code and materials related to our YouTube channel (TwoSetAI)

A list of software that allows searching the web with the assistance of AI.

Arch is an intelligent gateway for agents. Engineered with (fast) LLMs for the secure handling, rich observability, and seamless integration of prompts with your APIs - all outside business logic. Bui...

[NeurIPS 2024] A Novel Rank-Based Metric for Evaluating Large Language Models

A declarative way to control LLMs.

llm theoretical performance analysis tools and support params, flops, memory and latency analysis.

Run serverless GPU workloads with fast cold starts on bare-metal servers, anywhere in the world

🌱 EcoLogits tracks the energy consumption and environmental footprint of using generative AI models through APIs.

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.

Code and data to evaluate LLMs on the ENEM, the main standardized Brazilian university admission exams.

Code examples and resources for DBRX, a large language model developed by Databricks

Optimizing inference proxy for LLMs

Arch is an intelligent gateway for agents. Engineered with (fast) LLMs for the secure handling, rich observability, and seamless integration of prompts with your APIs - all outside business logic. Bui...

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

Graph-Based Editor for LLM Workflows

Practical Llama 3 inference in Java

A list of software that allows searching the web with the assistance of AI.

Efficient AI Inference & Serving

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

A multi-platform SwiftUI frontend for running local LLMs with Apple's MLX framework.

Efficient LLM Inference over Long Sequences

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

SiLLM simplifies the process of training and running Large Language Models (LLMs) on Apple Silicon by leveraging the MLX framework.

An acceleration library that supports arbitrary bit-width combinatorial quantization operations

The official evaluation suite and dynamic data release for MixEval.

This project collects GPU benchmarks from various cloud providers and compares them to fixed per token costs. Use our tool for efficient LLM GPU selections and cost-effective AI models. LLM provider p...

A programming framework for agentic AI 🤖 PyPi: autogen-agentchat Discord: https://aka.ms/autogen-discord Office Hour: https://aka.ms/autogen-officehour

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Official inference library for Mistral models

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Run any open-source LLMs, such as Llama, Mistral, as OpenAI compatible API endpoint in the cloud.

📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Code examples and resources for DBRX, a large language model developed by Databricks

AICI: Prompts as (Wasm) Programs

Optimizing inference proxy for LLMs

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Tensor parallelism is all you need. Run LLMs on an AI cluster at home using any device. Distribute the workload, divide RAM usage, and increase inference speed.

FlashInfer: Kernel Library for LLM Serving

Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Graph-Based Editor for LLM Workflows

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

An acceleration library that supports arbitrary bit-width combinatorial quantization operations

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

On-device LLM Inference Powered by X-Bit Quantization

Tensor parallelism is all you need. Run LLMs on an AI cluster at home using any device. Distribute the workload, divide RAM usage, and increase inference speed.

No description

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search Engine.

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Uses WebLLM, Wllama and SearXNG. Demo: https://felladrin-minisearch.hf.space

MagicPIG: LSH Sampling for Efficient LLM Generation

An innovative library for efficient LLM inference via low-bit quantization

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource...

LLM (Large Language Model) FineTuning

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inferenc...

It's time for a paradigm shift! The future of software is in plain English ✨

Gradio based tool to run opensource LLM models directly from Huggingface

FlashInfer: Kernel Library for LLM Serving

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.