13 results found Sort:

xlite-dev
Awesome-LLM-Inference xlite-dev
275
3.9k
gpl-3.0
122
📚A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, FlashAttention, PagedAttention, MLA, Parallelism etc.
mla vllm deepseek flash-mla minimax-01 awesome-llm deepseek-r1 deepseek-v3 tensorrt-llm llm-inference flash-attention paged-attention flash-attention-3
Created 2023-08-27
471 commits to main branch, last one 3 days ago
collabora
WhisperLive collabora
355
2.7k
mit
33
A nearly-live implementation of OpenAI's Whisper.
obs openai whisper openvino tensorrt dictation translation tensorrt-llm openvino-intel text-to-speech whisper-tensorrt voice-recognition
Created 2023-05-04
495 commits to main branch, last one 5 days ago
shashikg
WhisperS2T shashikg
49
394
mit
15
An Optimized Speech-to-Text Pipeline for the Whisper Model Supporting Multiple Inference Engine
asr vad whisper tensorrt tensorrt-llm deep-learning speech-to-text speech-recognition voice-activity-detection
Created 2023-12-16
81 commits to main branch, last one 7 months ago
huggingface
optimum-benchmark huggingface
58
295
apache-2.0
4
🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
pytorch openvino benchmark onnxruntime tensorrt-llm neural-compressor text-generation-inference
Created 2023-04-26
713 commits to main branch, last one 2 months ago
coderonion
awesome-cuda-and-hpc coderonion
29
249
unknown
7
🚀🚀🚀 This repository lists some awesome public CUDA, cuda-python, cuBLAS, cuDNN, CUTLASS, TensorRT, TensorRT-LLM, Triton, TVM, MLIR, PTX and High Performance Computing (HPC) projects.
gpu hpc llm ptx tvm vlm blas cuda gemm mlir cudnn llama cublas triton awesome cutlass pytorch deepseek tensorrt tensorrt-llm
Created 2023-02-23
39 commits to main branch, last one a day ago
npuichigo
openai_trtllm npuichigo
28
205
mit
8
OpenAI compatible API for TensorRT LLM triton backend
llm langchain openai-api tensorrt-llm triton-inference-server
Created 2023-11-06
33 commits to main branch, last one 8 months ago
NetEase-Media
grps NetEase-Media
13
157
apache-2.0
9
Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offeri...
vllm torch serving tensorrt tensorflow tensorrt-llm dynamic-batching triton-inference-server
Created 2024-07-04
62 commits to master branch, last one 27 days ago
NetEase-Media
grps_trtllm NetEase-Media
8
130
apache-2.0
4
Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service implemented with GPRS+TensorRT-LLM+Tokenizers.cpp, supporting chat and function call, AI agents, d...
llm phi qwq qwen2 llama3 olmocr openai chatglm ai-agent internvl qwen2-vl janus-pro minicpm-v deepseek-r1 internvideo llama-index multi-modal tensorrt-llm function-call
Created 2024-08-21
158 commits to master branch, last one 3 days ago
openhackathons-org
End-to-End-LLM openhackathons-org
36
84
apache-2.0
8
This repository is an AI Bootcamp material that consist of a workflow for LLM
llm genai p-tuning tensorrt-llm deep-learning nemo-megatron prompt-tuning nemo-guardrails question-answering natural-language-processing
Created 2022-10-31
48 commits to main branch, last one 8 months ago
vossr
Chat-With-RTX-python-api vossr
10
66
unknown
3
Chat With RTX Python API
llm tensorrt llama2-13b mistral-7b tensorrt-llm chat-with-rtx llm-inference nvidia-chat-with-rtx
Created 2024-02-23
18 commits to master branch, last one 4 months ago
guidance-ai
llgtrt guidance-ai
10
49
mit
7
TensorRT-LLM server with Structured Outputs (JSON) built with Rust
cfg json regex guidance openai-api tensorrt-llm structured-generation
Created 2024-10-04
123 commits to main branch, last one 16 days ago
argonne-lcf
LLM-Inference-Bench argonne-lcf
4
39
bsd-3-clause
9
LLM-Inference-Bench
llm vllm llamacpp benchmark deepspeed inference tensorrt-llm
Created 2024-07-29
85 commits to main branch, last one 3 months ago
fgblanch
OutlookLLM fgblanch
2
37
other
4
Add-in for new Outlook that adds LLM new features (Composition, Summarizing, Q&A). It uses a local LLM via Nvidia TensorRT-LLM
tensorrt-llm outlook-addin
Created 2024-02-21
28 commits to main branch, last one 4 months ago