3 results found Sort:

vllm-project
vllm vllm-project
2.8k
20.3k
apache-2.0
194
A high-throughput and memory-efficient inference and serving engine for LLMs
amd gpt llm cuda rocm llama mlops llmops pytorch trainium inference inferentia llm-serving transformer model-serving
Created 2023-02-09
1,444 commits to main branch, last one 13 hours ago
PygmalionAI
aphrodite-engine PygmalionAI
80
660
agpl-3.0
13
PygmalionAI's large-scale inference engine
cuda rocm avx512 api-rest inferentia inference-engine machine-learning
Created 2023-06-23
632 commits to main branch, last one 15 days ago
aws-samples
foundation-model-benchmarking-tool aws-samples
19
132
mit-0
6
Foundation model benchmarking tool. Run any model on Amazon SageMaker and benchmark for performance across instance type and serving stack options.
p4d llama2 llama3 bedrock benchmark sagemaker inferentia benchmarking generative-ai foundation-models
Created 2024-01-09
359 commits to main branch, last one 18 hours ago