32 results found Sort:

meta-llama
llama-recipes meta-llama
2.2k
15.5k
unknown
195
Scripts for fine-tuning Meta Llama with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a ...
ai llm vllm llama llama2 python pytorch langchain finetuning machine-learning
Created 2023-07-17
1,771 commits to main branch, last one 6 days ago
xorbitsai
inference xorbitsai
460
5.6k
apache-2.0
42
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any...
llm ggml glm4 qwen vllm gemma llama llama3 chatglm flan-t5 mistral pytorch whisper llamacpp wizardlm inference deployment openai-api machine-learning artificial-intelligence
Created 2023-06-14
1,003 commits to main branch, last one 14 hours ago
katanaml
sparrow katanaml
380
3.7k
gpl-3.0
50
Data processing with ML, LLM and Vision LLM
gpt llm rag vllm computer-vision machinelearning nlp-machine-learning huggingface-transformers
Created 2022-01-08
505 commits to main branch, last one a day ago
DefTruth
Awesome-LLM-Inference DefTruth
198
2.9k
gpl-3.0
94
📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉
llm llms sora vllm deepseek open-sora awesome-llm tensorrt-llm llm-inference flash-attention paged-attention flash-attention-2 flash-attention-3
Created 2023-08-27
426 commits to main branch, last one 2 days ago
OpenRLHF
OpenRLHF OpenRLHF
266
2.8k
apache-2.0
24
An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention)
vllm raylib deepspeed transformers large-language-models reinforcement-learning reinforcement-learning-from-human-feedback
Created 2023-07-30
979 commits to main branch, last one 2 days ago
bricks-cloud
BricksLLM bricks-cloud
66
926
mit
7
🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI...
ai api gpt llm vllm azure docker golang openai privacy rest-api security anthropic postgresql open-source self-hosted ycombinator generative-ai artificial-intelligence
Created 2023-07-18
728 commits to main branch, last one 14 days ago
prometheus-eval
prometheus-eval prometheus-eval
49
808
apache-2.0
3
Evaluate your LLM's response with Prometheus and GPT4 💯
llm gpt4 vllm llmops python litellm evaluation llm-as-a-judge llm-as-evaluator
Created 2024-04-18
204 commits to main branch, last one 3 days ago
substratusai
kubeai substratusai
46
560
apache-2.0
10
AI Inference Operator for Kubernetes
ai k8s llm vllm ollama whisper autoscaler kubernetes openai-api vllm-operator faster-whisper ollama-operator inference-operator
Created 2023-10-21
224 commits to main branch, last one 20 hours ago
varunshenoy
super-json-mode varunshenoy
13
386
unknown
2
Low latency JSON generation using LLMs ⚡️
llm vllm openai huggingface-transformers
Created 2023-11-15
76 commits to main branch, last one 8 months ago
ModelTC
llmc ModelTC
37
332
apache-2.0
9
[EMNLP 2024 Industry Track] This is the official PyTorch implementation of "LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit".
awq llm lvlm tool vllm llama3 quarot mixtral pruning lightllm benchmark internlm2 omniquant spinquant deployment evaluation smoothquant quantization large-language-models post-training-quantization
Created 2024-03-06
404 commits to main branch, last one a day ago
microsoft
vidur microsoft
45
285
mit
7
A large-scale simulation framework for LLM inference
llm vllm inference simulation transformer
Created 2023-11-02
23 commits to main branch, last one 14 days ago
containers
ramalama containers
49
284
mit
16
The goal of RamaLama is to make working with AI boring.
ai llms vllm local podman llamacpp containers inference-server
Created 2024-07-24
947 commits to main branch, last one 6 days ago
runpod-workers
worker-vllm runpod-workers
101
247
mit
7
The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
llm vllm runpod language-model
Created 2023-07-03
324 commits to main branch, last one 3 days ago
chtmp223
topicGPT chtmp223
37
233
unknown
5
TopicGPT: A Prompt-Based Framework for Topic Modeling (NAACL'24)
llm nlp vllm openai python topic-modeling
Created 2023-11-02
20 commits to main branch, last one 21 days ago
jasonacox
TinyLLM jasonacox
19
191
mit
9
Setup and run a local LLM and Chatbot using consumer grade hardware.
llm rag vllm openai chatbot llama-cpp-python large-language-models artificial-intelligence retrieval-augmented-generation
Created 2023-09-12
345 commits to main branch, last one 23 days ago
harleyszhang
llm_note harleyszhang
17
188
unknown
5
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes
llm vllm kv-cache llm-inference triton-kernels cuda-programming transformer-models
Created 2024-09-18
166 commits to main branch, last one a day ago
varunvasudeva1
llm-server-docs varunvasudeva1
14
187
unknown
2
Documentation on setting up an LLM server on Debian from scratch, using Ollama/vLLM, Open WebUI, OpenedAI Speech, and ComfyUI.
llm vllm linux debian ollama server comfyui open-webui openedai-speech
Created 2024-03-26
13 commits to main branch, last one about a month ago
NetEase-Media
grps NetEase-Media
13
167
apache-2.0
11
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。
vllm torch serving tensorrt tensorflow tensorrt-llm dynamic-batching triton-inference-server
Created 2024-07-04
58 commits to master branch, last one 19 days ago
gotzmann
booster gotzmann
6
141
other
7
Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
gpt llm ggml vllm llama ollama openai chatgpt exllama llamacpp llama-cpp oobabooga
Created 2023-05-04
491 commits to main branch, last one 3 months ago
dnth
x.infer dnth
12
125
other
3
Framework agnostic computer vision inference. Run 1000+ models by changing only one line of code. Supports models from transformers, timm, ultralytics, vllm, ollama and your custom model.
vllm ollama ultralytics transformers inference-api computer-vision pytorch-image-models
Created 2024-10-10
284 commits to main branch, last one 6 days ago
nbasyl
DoRA nbasyl
4
123
other
2
Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
dora lora vllm peft-fine-tuning-llm
This repository has been archived (exclude archived)
Created 2024-03-05
22 commits to main branch, last one 7 months ago
Trainy-ai
llm-atc Trainy-ai
2
87
apache-2.0
3
Fine-tuning and serving LLMs on any cloud
llms vllm llama2 finetuning
This repository has been archived (exclude archived)
Created 2023-07-30
44 commits to main branch, last one about a year ago
OpenCSGs
llm-inference OpenCSGs
16
70
apache-2.0
12
llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource...
ray vllm deepspeed llama-cpp transformer llm-inference
Created 2024-02-28
116 commits to main branch, last one 6 months ago
yoziru
nextjs-vllm-ui yoziru
13
69
mit
1
Fully-featured, beautiful web interface for vLLM - built with NextJS.
ai ui vllm webui llm-ui nextjs vllm-ui llm-webui openai-api typescript self-hosted tailwindcss
Created 2024-03-05
129 commits to main branch, last one 4 months ago
asprenger
ray_vllm_inference asprenger
4
55
apache-2.0
2
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
llm ray vllm mlops llmops pytorch inference llm-serving transformer model-serving
Created 2023-10-28
52 commits to master branch, last one 11 months ago
wangcx18
llm-vscode-inference-server wangcx18
8
54
apache-2.0
1
An endpoint server for efficiently serving quantized open-source LLMs for code.
llm vllm llm-inference vscode-extension
Created 2023-09-25
3 commits to main branch, last one about a year ago
VectorInstitute
vector-inference VectorInstitute
7
42
mit
7
Efficient LLM inference on Slurm clusters using vLLM.
llm vlm vllm inference llm-inference text-embedding reward-modeling
Created 2024-03-06
217 commits to main branch, last one 4 days ago
llmariner
llmariner llmariner
4
35
apache-2.0
1
Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.
ai ml gpu k8s llm vllm openai operator inference kubernetes autoscaling fine-tuning multi-cluster
Created 2024-03-27
698 commits to main branch, last one 7 days ago
gameofdimension
vllm-cn gameofdimension
1
31
unknown
1
演示 vllm 对中文大语言模型的神奇效果
vllm
Created 2023-07-08
6 commits to master branch, last one about a year ago
InftyAI
llmaz InftyAI
10
31
apache-2.0
4
☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!
llm helm vllm sglang llamacpp kubernetes modelscope huggingface llm-inference text-generation-inference
Created 2023-11-20
250 commits to main branch, last one a day ago