4 results found Sort:

vllm-project
vllm vllm-project
6.6k
43.2k
apache-2.0
357
A high-throughput and memory-efficient inference and serving engine for LLMs
amd gpt hpu llm tpu xpu cuda qwen rocm llama mlops llmops pytorch deepseek trainium inference inferentia llm-serving transformer model-serving
Created 2023-02-09
5,531 commits to main branch, last one 17 hours ago
aphrodite-engine
aphrodite-engine aphrodite-engine
149
1.4k
agpl-3.0
20
Large-scale LLM inference engine
tpu cuda lora rocm intel api-rest inferentia inference-engine machine-learning speculative-decoding
Created 2023-06-23
1,253 commits to main branch, last one 3 days ago
aws-samples
foundation-model-benchmarking-tool aws-samples
41
233
mit-0
8
Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.
g5 g6 p5 g6e p4d llama2 llama3 bedrock deepseek trainium benchmark sagemaker inferentia deepseek-r1 benchmarking generative-ai foundation-models evaluation-metrics
Created 2024-01-09
1,491 commits to main branch, last one 26 days ago
aws-solutions-library-samples
guidance-for-machine-learning-inference-on-aws aws-solutions-library-samples
12
42
mit-0
18
This Guidance demonstrates how to deploy a machine learning inference architecture on Amazon Elastic Kubernetes Service (Amazon EKS). It addresses the basic implementation requirements as well as ways...
ml graviton3 inferentia eks-cluster do-framework mlops-workflow
Created 2021-08-11
108 commits to main branch, last one 5 months ago