11 results found Sort:

DefTruth
Awesome-LLM-Inference DefTruth
193
2.9k
gpl-3.0
93
📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
llm llms sora vllm deepseek open-sora awesome-llm tensorrt-llm llm-inference flash-attention paged-attention flash-attention-2 flash-attention-3
Created 2023-08-27
414 commits to main branch, last one 3 days ago
janhq
cortex.cpp janhq
124
2.1k
apache-2.0
17
Local AI API Platform
gguf onnx llamacpp onnxruntime tensorrt-llm
Created 2023-09-11
1,861 commits to dev branch, last one a day ago
collabora
WhisperLive collabora
282
2.1k
mit
30
A nearly-live implementation of OpenAI's Whisper.
obs openai whisper tensorrt dictation translation tensorrt-llm text-to-speech whisper-tensorrt voice-recognition
Created 2023-05-04
450 commits to main branch, last one a day ago
shashikg
WhisperS2T shashikg
32
313
mit
14
An Optimized Speech-to-Text Pipeline for the Whisper Model Supporting Multiple Inference Engine
asr vad whisper tensorrt tensorrt-llm deep-learning speech-to-text speech-recognition voice-activity-detection
Created 2023-12-16
81 commits to main branch, last one 2 months ago
huggingface
optimum-benchmark huggingface
48
257
apache-2.0
5
🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
pytorch openvino benchmark onnxruntime tensorrt-llm neural-compressor text-generation-inference
Created 2023-04-26
692 commits to main branch, last one about a month ago
npuichigo
openai_trtllm npuichigo
27
177
mit
8
OpenAI compatible API for TensorRT LLM triton backend
llm langchain openai-api tensorrt-llm triton-inference-server
Created 2023-11-06
33 commits to main branch, last one 3 months ago
NetEase-Media
grps NetEase-Media
13
166
apache-2.0
11
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。
vllm torch serving tensorrt tensorflow tensorrt-llm dynamic-batching triton-inference-server
Created 2024-07-04
58 commits to master branch, last one 7 days ago
NetEase-Media
grps_trtllm NetEase-Media
3
92
apache-2.0
3
【grps接入trtllm】通过GPRS+TensorRT-LLM+Tokenizers.cpp实现纯C++版高性能OpenAI LLM服务，支持chat和function call模式，支持ai agent，支持分布式多卡推理，支持多模态，支持gradio聊天界面。
llm qwen2 llama3 openai chatglm qwen-vl ai-agent internvl2 llama-index multi-modal tensorrt-llm function-call
Created 2024-08-21
101 commits to master branch, last one 18 days ago
openhackathons-org
End-to-End-LLM openhackathons-org
32
67
apache-2.0
9
This repository is an AI Bootcamp material that consist of a workflow for LLM
llm genai p-tuning tensorrt-llm deep-learning nemo-megatron prompt-tuning nemo-guardrails question-answering natural-language-processing
Created 2022-10-31
48 commits to main branch, last one 3 months ago
vossr
Chat-With-RTX-python-api vossr
10
58
unknown
3
Chat With RTX Python API
llm tensorrt llama2-13b mistral-7b tensorrt-llm chat-with-rtx llm-inference nvidia-chat-with-rtx
Created 2024-02-23
16 commits to master branch, last one 2 months ago
fgblanch
OutlookLLM fgblanch
1
26
other
4
Add-in for new Outlook that adds LLM new features (Composition, Summarizing, Q&A). It uses a local LLM via Nvidia TensorRT-LLM
tensorrt-llm outlook-addin
Created 2024-02-21
26 commits to main branch, last one about a month ago