12 results found Sort:

DefTruth
Awesome-LLM-Inference DefTruth
206
3.0k
gpl-3.0
102
📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉
llm llms sora vllm deepseek open-sora awesome-llm tensorrt-llm llm-inference flash-attention paged-attention flash-attention-2 flash-attention-3
Created 2023-08-27
428 commits to main branch, last one 13 days ago
collabora
WhisperLive collabora
298
2.2k
mit
32
A nearly-live implementation of OpenAI's Whisper.
obs openai whisper tensorrt dictation translation tensorrt-llm text-to-speech whisper-tensorrt voice-recognition
Created 2023-05-04
462 commits to main branch, last one 3 days ago
janhq
cortex.cpp janhq
134
2.2k
apache-2.0
21
Local AI API Platform
gguf onnx llamacpp onnxruntime tensorrt-llm
Created 2023-09-11
1,957 commits to dev branch, last one 2 days ago
shashikg
WhisperS2T shashikg
36
332
mit
14
An Optimized Speech-to-Text Pipeline for the Whisper Model Supporting Multiple Inference Engine
asr vad whisper tensorrt tensorrt-llm deep-learning speech-to-text speech-recognition voice-activity-detection
Created 2023-12-16
81 commits to main branch, last one 3 months ago
huggingface
optimum-benchmark huggingface
52
279
apache-2.0
5
🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
pytorch openvino benchmark onnxruntime tensorrt-llm neural-compressor text-generation-inference
Created 2023-04-26
708 commits to main branch, last one 4 days ago
npuichigo
openai_trtllm npuichigo
27
180
mit
8
OpenAI compatible API for TensorRT LLM triton backend
llm langchain openai-api tensorrt-llm triton-inference-server
Created 2023-11-06
33 commits to main branch, last one 4 months ago
NetEase-Media
grps NetEase-Media
13
169
apache-2.0
11
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。
vllm torch serving tensorrt tensorflow tensorrt-llm dynamic-batching triton-inference-server
Created 2024-07-04
58 commits to master branch, last one about a month ago
coderonion
awesome-cuda-triton-hpc coderonion
22
168
unknown
5
🔥🔥🔥 A collection of some awesome public CUDA, cuBLAS, TensorRT, TensorRT-LLM, Triton and High Performance Computing (HPC) projects.
gpu hpc llm blas cuda gemm yolo cudnn llama scipy cublas lapack triton awesome cutlass pytorch openblas tensorrt tensorrt-llm parallel-computing
Created 2023-02-23
14 commits to main branch, last one 6 days ago
NetEase-Media
grps_trtllm NetEase-Media
3
98
apache-2.0
3
【grps接入trtllm】通过GPRS+TensorRT-LLM+Tokenizers.cpp实现纯C++版高性能OpenAI LLM服务，支持chat和function call模式，支持ai agent，支持分布式多卡推理，支持多模态，支持gradio聊天界面。
llm qwen2 llama3 openai chatglm qwen-vl ai-agent qwen2-vl internvl2 llama-index multi-modal tensorrt-llm function-call
Created 2024-08-21
107 commits to master branch, last one a day ago
openhackathons-org
End-to-End-LLM openhackathons-org
32
68
apache-2.0
9
This repository is an AI Bootcamp material that consist of a workflow for LLM
llm genai p-tuning tensorrt-llm deep-learning nemo-megatron prompt-tuning nemo-guardrails question-answering natural-language-processing
Created 2022-10-31
48 commits to main branch, last one 4 months ago
vossr
Chat-With-RTX-python-api vossr
10
59
unknown
4
Chat With RTX Python API
llm tensorrt llama2-13b mistral-7b tensorrt-llm chat-with-rtx llm-inference nvidia-chat-with-rtx
Created 2024-02-23
18 commits to master branch, last one 20 days ago
fgblanch
OutlookLLM fgblanch
2
27
other
4
Add-in for new Outlook that adds LLM new features (Composition, Summarizing, Q&A). It uses a local LLM via Nvidia TensorRT-LLM
tensorrt-llm outlook-addin
Created 2024-02-21
28 commits to main branch, last one 17 days ago