2 results found Sort:

vllm-project
vllm vllm-project
4.6k
30.5k
apache-2.0
248
A high-throughput and memory-efficient inference and serving engine for LLMs
amd gpt hpu llm tpu xpu cuda rocm llama mlops llmops pytorch trainium inference inferentia llm-serving transformer model-serving
Created 2023-02-09
3,511 commits to main branch, last one 13 hours ago
aws-samples
foundation-model-benchmarking-tool aws-samples
31
196
mit-0
8
Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.
g5 g6 p5 g6e p4d llama2 llama3 bedrock trainium benchmark sagemaker inferentia benchmarking generative-ai foundation-models evaluation-metrics
Created 2024-01-09
1,213 commits to main branch, last one 19 hours ago