24 results found Sort:

meta-llama
llama-recipes meta-llama
1.7k
11.8k
unknown
95
Scripts for fine-tuning Meta Llama with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a ...
ai llm vllm llama llama2 python pytorch langchain finetuning machine-learning
Created 2023-07-17
1,526 commits to main branch, last one 2 days ago
xorbitsai
inference xorbitsai
391
4.9k
apache-2.0
42
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any...
llm ggml glm4 qwen vllm gemma llama llama3 chatglm flan-t5 mistral pytorch whisper llamacpp wizardlm inference deployment openai-api machine-learning artificial-intelligence
Created 2023-06-14
944 commits to main branch, last one a day ago
DefTruth
Awesome-LLM-Inference DefTruth
175
2.6k
gpl-3.0
85
📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
llm llms sora vllm deepseek open-sora awesome-llm tensorrt-llm llm-inference flash-attention paged-attention flash-attention-2 flash-attention-3
Created 2023-08-27
390 commits to main branch, last one a day ago
OpenRLHF
OpenRLHF OpenRLHF
206
2.1k
apache-2.0
21
An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & Mixtral)
vllm raylib deepspeed transformers large-language-models reinforcement-learning reinforcement-learning-from-human-feedback
Created 2023-07-30
903 commits to main branch, last one 3 days ago
bricks-cloud
BricksLLM bricks-cloud
59
881
mit
6
🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI...
ai api gpt llm vllm azure docker golang openai privacy rest-api security anthropic postgresql open-source self-hosted ycombinator generative-ai artificial-intelligence
Created 2023-07-18
698 commits to main branch, last one 12 days ago
prometheus-eval
prometheus-eval prometheus-eval
47
762
apache-2.0
3
Evaluate your LLM's response with Prometheus and GPT4 💯
llm gpt4 vllm llmops python litellm evaluation llm-as-a-judge llm-as-evaluator
Created 2024-04-18
203 commits to main branch, last one 19 days ago
varunshenoy
super-json-mode varunshenoy
12
382
unknown
2
Low latency JSON generation using LLMs ⚡️
llm vllm openai huggingface-transformers
Created 2023-11-15
76 commits to main branch, last one 6 months ago
substratusai
kubeai substratusai
31
341
apache-2.0
9
Private Open AI on Kubernetes
ai k8s llm vllm ollama whisper autoscaler kubernetes openai-api vllm-operator faster-whisper ollama-operator inference-operator
Created 2023-10-21
179 commits to main branch, last one a day ago
av
harbor av
14
311
apache-2.0
6
Effortlessly run LLM backends, APIs, frontends, and services with one command.
ai cli llm cmdh tool vllm llama tools docker ollama mistral llamacpp exllamav2 huggingface self-hosted docker-compose open-interpreter text-generation-inference
Created 2024-07-27
265 commits to main branch, last one a day ago
microsoft
vidur microsoft
27
241
mit
7
A large-scale simulation framework for LLM inference
llm vllm inference simulation transformer
Created 2023-11-02
20 commits to main branch, last one about a month ago
ModelTC
llmc ModelTC
27
234
apache-2.0
9
This is the official PyTorch implementation of "LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit".
awq llm tool vllm llama3 quarot smollm minicpm mixtral pruning lightllm benchmark internlm2 omniquant deployment evaluation smoothquant quantization large-language-models post-training-quantization
Created 2024-03-06
255 commits to main branch, last one a day ago
runpod-workers
worker-vllm runpod-workers
87
229
mit
6
The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
llm vllm runpod language-model
Created 2023-07-03
296 commits to main branch, last one 12 days ago
containers
ramalama containers
20
179
mit
11
The goal of ramalama is to make working with AI boring.
ai llms vllm local podman llamacpp containers inference-server
Created 2024-07-24
479 commits to main branch, last one a day ago
jasonacox
TinyLLM jasonacox
14
159
mit
7
Setup and run a local LLM and Chatbot using consumer grade hardware.
llm rag vllm openai chatbot llama-cpp-python large-language-models artificial-intelligence retrieval-augmented-generation
Created 2023-09-12
326 commits to main branch, last one 2 days ago
NetEase-Media
grps NetEase-Media
13
147
apache-2.0
11
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。
vllm torch serving tensorrt tensorflow tensorrt-llm dynamic-batching triton-inference-server
Created 2024-07-04
53 commits to master branch, last one 23 days ago
gotzmann
booster gotzmann
6
137
other
7
Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
gpt llm ggml vllm llama ollama openai chatgpt exllama llamacpp llama-cpp oobabooga
Created 2023-05-04
491 commits to main branch, last one about a month ago
nbasyl
DoRA nbasyl
3
122
other
2
Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
dora lora vllm peft-fine-tuning-llm
This repository has been archived (exclude archived)
Created 2024-03-05
22 commits to main branch, last one 5 months ago
Trainy-ai
llm-atc Trainy-ai
2
85
apache-2.0
3
Fine-tuning and serving LLMs on any cloud
llms vllm llama2 finetuning
This repository has been archived (exclude archived)
Created 2023-07-30
44 commits to main branch, last one 10 months ago
OpenCSGs
llm-inference OpenCSGs
17
69
apache-2.0
13
llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource...
ray vllm deepspeed llama-cpp transformer llm-inference
Created 2024-02-28
116 commits to main branch, last one 4 months ago
wangcx18
llm-vscode-inference-server wangcx18
8
52
apache-2.0
1
An endpoint server for efficiently serving quantized open-source LLMs for code.
llm vllm llm-inference vscode-extension
Created 2023-09-25
3 commits to main branch, last one 11 months ago
asprenger
ray_vllm_inference asprenger
4
49
apache-2.0
2
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
llm ray vllm mlops llmops pytorch inference llm-serving transformer model-serving
Created 2023-10-28
52 commits to master branch, last one 9 months ago
yoziru
nextjs-vllm-ui yoziru
8
46
mit
1
Fully-featured, beautiful web interface for vLLM - built with NextJS.
ai ui vllm webui llm-ui nextjs vllm-ui llm-webui openai-api typescript self-hosted tailwindcss
Created 2024-03-05
129 commits to main branch, last one 2 months ago
gameofdimension
vllm-cn gameofdimension
1
31
unknown
1
演示 vllm 对中文大语言模型的神奇效果
vllm
Created 2023-07-08
6 commits to master branch, last one 10 months ago
phospho-app
fastassert phospho-app
0
28
unknown
3
Dockerized LLM inference server with constrained output (JSON mode), built on top of vLLM and outlines. Faster, cheaper and without rate limits. Compare the quality and latency to your current LLM API...
llm vllm docker outlines llm-inference
Created 2024-02-13
20 commits to main branch, last one 7 months ago