37 results found Sort:

meta-llama
llama-recipes meta-llama
2.3k
15.8k
unknown
203
Scripts for fine-tuning Meta Llama with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a ...
ai llm vllm llama llama2 python pytorch langchain finetuning machine-learning
Created 2023-07-17
1,812 commits to main branch, last one 16 days ago
xorbitsai
inference xorbitsai
481
5.8k
apache-2.0
43
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any...
llm ggml glm4 qwen vllm gemma llama llama3 chatglm flan-t5 mistral pytorch whisper llamacpp wizardlm inference deployment openai-api machine-learning artificial-intelligence
Created 2023-06-14
1,039 commits to main branch, last one 2 days ago
modelscope
ms-swift modelscope
429
4.9k
apache-2.0
23
Use PEFT or Full-parameter to finetune 400+ LLMs (Qwen2.5, Llama3.2, GLM4, Internlm2.5, Yi1.5, Mistral, Baichuan2, DeepSeek, ...) or 150+ MLLMs (Qwen2-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL...
dpo llm sft lora peft vllm agent liger llama qwen2 deploy llama3 lora-ga emu3-gen internvl qwen2-vl minicpm-v modelscope multimodal pre-training
Created 2023-08-01
1,327 commits to main branch, last one 11 hours ago
katanaml
sparrow katanaml
397
4.1k
gpl-3.0
53
Data processing with ML, LLM and Vision LLM
gpt llm rag vllm computer-vision machinelearning nlp-machine-learning huggingface-transformers
Created 2022-01-08
535 commits to main branch, last one 12 days ago
OpenRLHF
OpenRLHF OpenRLHF
331
3.5k
apache-2.0
28
An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT)
vllm raylib openai-o1 transformers large-language-models reinforcement-learning proximal-policy-optimization reinforcement-learning-from-human-feedback
Created 2023-07-30
1,038 commits to main branch, last one a day ago
DefTruth
Awesome-LLM-Inference DefTruth
210
3.1k
gpl-3.0
105
📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉
llm llms sora vllm deepseek open-sora awesome-llm tensorrt-llm llm-inference flash-attention paged-attention flash-attention-2 flash-attention-3
Created 2023-08-27
434 commits to main branch, last one a day ago
bricks-cloud
BricksLLM bricks-cloud
66
955
mit
7
🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI...
ai api gpt llm vllm azure docker golang openai privacy rest-api security anthropic postgresql open-source self-hosted ycombinator generative-ai artificial-intelligence
Created 2023-07-18
737 commits to main branch, last one 3 days ago
prometheus-eval
prometheus-eval prometheus-eval
49
833
apache-2.0
3
Evaluate your LLM's response with Prometheus and GPT4 💯
llm gpt4 vllm llmops python litellm evaluation llm-as-a-judge llm-as-evaluator
Created 2024-04-18
204 commits to main branch, last one about a month ago
substratusai
kubeai substratusai
50
620
apache-2.0
12
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports LLMs, embeddings, and speech-to-text.
ai k8s llm vllm ollama whisper autoscaler kubernetes openai-api vllm-operator faster-whisper ollama-operator inference-operator
Created 2023-10-21
236 commits to main branch, last one a day ago
jakobdylanc
llmcord jakobdylanc
79
407
mit
5
Make Discord your LLM frontend ● Supports any OpenAI compatible API (Ollama, LM Studio, vLLM, OpenRouter, xAI, Mistral, Groq and more)
bot gpt llm xai chat grok groq vllm gpt-4 llama llava gpt-4o llama3 ollama openai chatbot discord mistral frontend oobabooga
Created 2023-05-08
359 commits to main branch, last one 14 hours ago
varunshenoy
super-json-mode varunshenoy
14
388
unknown
3
Low latency JSON generation using LLMs ⚡️
llm vllm openai huggingface-transformers
Created 2023-11-15
76 commits to main branch, last one 10 months ago
ModelTC
llmc ModelTC
43
374
apache-2.0
10
[EMNLP 2024 Industry Track] This is the official PyTorch implementation of "LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit".
awq llm lvlm tool vllm llama3 quarot mixtral pruning lightllm benchmark internlm2 omniquant spinquant deployment evaluation smoothquant quantization large-language-models post-training-quantization
Created 2024-03-06
428 commits to main branch, last one a day ago
harleyszhang
llm_note harleyszhang
30
330
unknown
5
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
llm vllm kv-cache llm-inference triton-kernels cuda-programming transformer-models
Created 2024-09-18
188 commits to main branch, last one 9 hours ago
containers
ramalama containers
55
314
mit
17
The goal of RamaLama is to make working with AI boring.
ai llms vllm local podman llamacpp containers inference-server
Created 2024-07-24
1,003 commits to main branch, last one a day ago
microsoft
vidur microsoft
53
307
mit
7
A large-scale simulation framework for LLM inference
llm vllm inference simulation transformer
Created 2023-11-02
23 commits to main branch, last one about a month ago
runpod-workers
worker-vllm runpod-workers
104
256
mit
7
The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
llm vllm runpod language-model
Created 2023-07-03
332 commits to main branch, last one 5 days ago
chtmp223
topicGPT chtmp223
37
241
unknown
5
TopicGPT: A Prompt-Based Framework for Topic Modeling (NAACL'24)
llm nlp vllm openai python topic-modeling
Created 2023-11-02
20 commits to main branch, last one about a month ago
varunvasudeva1
llm-server-docs varunvasudeva1
19
216
unknown
3
Documentation on setting up an LLM server on Debian from scratch, using Ollama/vLLM, Open WebUI, OpenedAI Speech, and ComfyUI.
llm vllm linux debian ollama server comfyui open-webui openedai-speech
Created 2024-03-26
13 commits to main branch, last one 2 months ago
jasonacox
TinyLLM jasonacox
20
200
mit
10
Setup and run a local LLM and Chatbot using consumer grade hardware.
llm rag vllm openai chatbot llama-cpp-python large-language-models artificial-intelligence retrieval-augmented-generation
Created 2023-09-12
351 commits to main branch, last one a day ago
NetEase-Media
grps NetEase-Media
13
170
apache-2.0
11
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。
vllm torch serving tensorrt tensorflow tensorrt-llm dynamic-batching triton-inference-server
Created 2024-07-04
59 commits to master branch, last one 11 days ago
gotzmann
booster gotzmann
6
145
other
7
Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
gpt llm ggml vllm llama ollama openai chatgpt exllama llamacpp llama-cpp oobabooga
Created 2023-05-04
491 commits to main branch, last one 4 months ago
shell-nlp
gpt_server shell-nlp
13
144
apache-2.0
4
gpt_server是一个用于生产级部署LLMs或Embedding的开源框架。
gpt llm tts vllm llama openai rerank fastchat infinity lmdeploy embedding text-moderation function-calling prompt-injection
Created 2023-12-16
283 commits to main branch, last one 2 days ago
dnth
x.infer dnth
12
130
other
4
Framework agnostic computer vision inference. Run 1000+ models by changing only one line of code. Supports models from transformers, timm, ultralytics, vllm, ollama and your custom model.
vllm ollama ultralytics transformers inference-api computer-vision pytorch-image-models
Created 2024-10-10
284 commits to main branch, last one about a month ago
nbasyl
DoRA nbasyl
4
123
other
2
Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
dora lora vllm peft-fine-tuning-llm
This repository has been archived (exclude archived)
Created 2024-03-05
22 commits to main branch, last one 8 months ago
Trainy-ai
llm-atc Trainy-ai
2
87
apache-2.0
3
Fine-tuning and serving LLMs on any cloud
llms vllm llama2 finetuning
This repository has been archived (exclude archived)
Created 2023-07-30
44 commits to main branch, last one about a year ago
yoziru
nextjs-vllm-ui yoziru
14
80
mit
1
Fully-featured, beautiful web interface for vLLM - built with NextJS.
ai ui vllm webui llm-ui nextjs vllm-ui llm-webui openai-api typescript self-hosted tailwindcss
Created 2024-03-05
129 commits to main branch, last one 5 months ago
OpenCSGs
llm-inference OpenCSGs
16
73
apache-2.0
12
llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource...
ray vllm deepspeed llama-cpp transformer llm-inference
Created 2024-02-28
116 commits to main branch, last one 7 months ago
apconw
sanic-web apconw
14
71
unknown
6
一个轻量级、支持全链路且易于二次开发的大模型应用项目 基于 Dify 、Ollama&Vllm、Sanic 和 Text2SQL 📊 等技术构建的一站式大模型应用开发项目，采用 Vue3、TypeScript 和 Vite 5 打造现代UI。它支持通过 ECharts 📈 实现基于大模型的数据图形化问答，具备处理 CSV 文件 📂 表格问答的能力。同时，能方便对接第三方开源 RAG 系统 ...
ai llm rag chat dify qwen vllm vue3 sanic ollama python bigdata chatgpt echarts text2sql
Created 2024-11-15
133 commits to master branch, last one 3 days ago
asprenger
ray_vllm_inference asprenger
4
56
apache-2.0
2
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
llm ray vllm mlops llmops pytorch inference llm-serving transformer model-serving
Created 2023-10-28
52 commits to master branch, last one about a year ago
wangcx18
llm-vscode-inference-server wangcx18
8
54
apache-2.0
1
An endpoint server for efficiently serving quantized open-source LLMs for code.
llm vllm llm-inference vscode-extension
Created 2023-09-25
3 commits to main branch, last one about a year ago