43 results found Sort:

vllm-project
vllm vllm-project
2.8k
20.3k
apache-2.0
194
A high-throughput and memory-efficient inference and serving engine for LLMs
amd gpt llm cuda rocm llama mlops llmops pytorch trainium inference inferentia llm-serving transformer model-serving
Created 2023-02-09
1,444 commits to main branch, last one 12 hours ago
bentoml
BentoML bentoml
753
6.7k
apache-2.0
74
The easiest way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Multi-model Inference Graph/Pipelines, LLM/RAG apps, and more!
llm mlops llmops python multimodal llm-serving deep-learning generative-ai llm-inference model-serving ml-engineering machine-learning inference-platform model-inference-service
Created 2019-04-02
3,108 commits to main branch, last one 24 hours ago
ahkarami
Deep-Learning-in-Production ahkarami
683
4.3k
unknown
148
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
flask keras mxnet react caffe2 python pytorch serving rest-api tutorial angularjs production tesnorflow c-plus-plus deep-learning model-serving tensorflow-models deep-neural-networks convert-pytorch-models serving-pytorch-models
Created 2018-05-03
219 commits to master branch, last one 14 days ago
FedML-AI
FedML FedML-AI
769
4.1k
apache-2.0
114
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on a...
mlops edge-ai ai-agent deep-learning model-serving inference-engine machine-learning model-deployment federated-learning on-device-training distributed-training
Created 2020-07-21
12,120 commits to master branch, last one 18 days ago
kserve
kserve kserve
978
3.2k
apache-2.0
63
Standardized Serverless ML Inference Platform on Kubernetes
k8s istio mlops kserve knative pytorch sklearn xgboost kubeflow kubernetes tensorflow service-mesh hacktoberfest model-serving mlops-community machine-learning hacktoberfest2023 kubeflow-pipelines model-interpretability artificial-intelligence
Created 2019-03-27
1,499 commits to master branch, last one 4 days ago
tensorchord
envd tensorchord
156
1.9k
apache-2.0
22
🏕️ Reproducible development environment
mlops docker llmops buildkit hacktoberfest model-serving mlops-workflow developer-tools development-environment
Created 2022-04-11
1,109 commits to main branch, last one 19 days ago
ModelTC
lightllm ModelTC
171
1.9k
apache-2.0
20
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
gpt llm nlp llama deep-learning model-serving openai-triton
Created 2023-07-22
220 commits to main branch, last one 5 days ago
microsoft
aici microsoft
72
1.8k
mit
19
AICI: Prompts as (Wasm) Programs
ai llm rust wasm llmops wasmtime inference llm-serving transformer llm-framework llm-inference model-serving language-model
Created 2023-09-26
1,341 commits to main branch, last one 11 days ago
predibase
lorax predibase
116
1.7k
apache-2.0
29
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
gpt llm lora llama llmops pytorch fine-tuning llm-serving transformers llm-inference model-serving
Created 2023-10-20
706 commits to main branch, last one 2 days ago
mlrun
mlrun mlrun
239
1.3k
apache-2.0
25
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates t...
mlops python workflow kubernetes data-science model-serving mlops-workflow data-engineering machine-learning experiment-tracking
Created 2019-09-01
4,756 commits to development branch, last one a day ago
logicalclocks
hopsworks logicalclocks
143
1.1k
agpl-3.0
32
Hopsworks - Data-Intensive AI platform with a Feature Store
ml aws gcp azure mlops kserve python pyspark hopsworks governance serverless data-science feature-store model-serving machine-learning feature-management feature-engineering
Created 2018-07-26
6,395 commits to master branch, last one 3 days ago
basetenlabs
truss basetenlabs
58
848
mit
11
The simplest way to serve AI/ML models in production
falcon whisper wizardlm packaging easy-to-use open-source inference-api model-serving inference-server machine-learning stable-diffusion artificial-intelligence
Created 2022-07-06
976 commits to main branch, last one 14 hours ago
bentoml
Yatai bentoml
70
770
other
19
Model Deployment at Scale on Kubernetes 🦄️
k8s mlops bentoml kubernetes model-serving machine-learning model-deployment
Created 2021-07-30
639 commits to main branch, last one 23 days ago
mosecorg
mosec mosecorg
48
712
apache-2.0
13
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
cv gpu jax llm tts rust mlops mxnet python pytorch tensorflow llm-serving deep-learning hacktoberfest model-serving nerual-network machine-learning machine-learning-platform
Created 2021-03-13
365 commits to main branch, last one 11 days ago
openvinotoolkit
model_server openvinotoolkit
196
641
apache-2.0
30
A scalable inference server for models optimized with OpenVINO™
ai dag edge cloud serving openvino inference kubernetes deep-learning model-serving machine-learning
Created 2018-09-26
2,338 commits to main branch, last one 23 hours ago
underneathall
pinferencia underneathall
85
558
apache-2.0
41
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
ai nlp python predict pytorch serving inference tensorflow huggingface modelserver data-science paddlepaddle transformers deep-learning model-serving computer-vision inference-server machine-learning model-deployment artificial-intelligence
Created 2022-04-04
55 commits to main branch, last one about a year ago
alibaba
rtp-llm alibaba
29
404
apache-2.0
11
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
gpt llm llama llmops inference llm-serving model-serving
Created 2023-12-27
776 commits to main branch, last one 10 days ago
Lightning-Universe
stable-diffusion-deploy Lightning-Universe
38
393
apache-2.0
19
Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services worki...
model-serving stable-diffusion
This repository has been archived (exclude archived)
Created 2022-08-23
314 commits to main branch, last one 8 months ago
eightBEC
fastapi-ml-skeleton eightBEC
74
351
apache-2.0
6
FastAPI Skeleton App to serve machine learning models production-ready.
python fastapi python3 model-serving machine-learning
Created 2020-02-15
12 commits to main branch, last one 28 days ago
bentoml
OneDiffusion bentoml
24
324
apache-2.0
12
OneDiffusion: Run any Stable Diffusion models and fine-tuned weights with ease
ai lora kubernetes fine-tuning model-serving diffusion-models stable-diffusion
Created 2023-06-12
47 commits to main branch, last one 2 days ago
intel
xFasterTransformer intel
47
247
apache-2.0
11
This repository has no description...
llm qwen xeon intel llama chatglm inference transformer model-serving
Created 2023-06-14
282 commits to main branch, last one 18 hours ago
aniketmaurya
chitra aniketmaurya
37
224
apache-2.0
6
A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.
mlops python fastapi gradcam pytorch tensorflow deep-learning hacktoberfest image-dataset model-serving visualization bounding-boxes image-processing machine-learning model-deployment object-detection model-visualization image-classification model-interpretation
Created 2020-01-23
371 commits to master branch, last one 3 months ago
jozu-ai
kitops jozu-ai
20
201
apache-2.0
9
Tools for easing the handoff between AI/ML and App/SRE teams.
ai ml code gguf mlops devops models pytorch sklearn datasets kubernetes tensorflow mlops-tools devops-tools model-packer model-serving kubernetes-deployment model-interpretability
Created 2024-02-02
386 commits to main branch, last one 14 hours ago
google
JetStream google
17
148
apache-2.0
14
JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).
gpt gpu jax llm tpu gemma llama mlops llama2 llmops pytorch inference transformer llm-inference model-serving large-language-models
Created 2024-03-01
82 commits to main branch, last one 2 days ago
spotify
zoltar spotify
33
138
apache-2.0
25
Common library for serving TensorFlow, XGBoost and scikit-learn models in production.
java xgboost tensorflow model-serving machine-learning
This repository has been archived (exclude archived)
Created 2018-03-12
505 commits to main branch, last one about a year ago
FederatedAI
FATE-Serving FederatedAI
77
134
apache-2.0
29
A scalable, high-performance serving system for federated learning models
monitor inference model-serving model-versioning federated-learning
Created 2019-09-10
155 commits to master branch, last one 6 months ago
allegroai
clearml-serving allegroai
40
128
apache-2.0
11
ClearML - Model-Serving Orchestration and Repository Solution
ai mlops devops triton clearml serving kubernetes serving-ml deep-learning model-serving machine-learning tensorflow-serving serving-pytorch-models triton-inference-server
Created 2021-04-12
138 commits to main branch, last one 2 months ago
alvarobartt
serving-pytorch-models alvarobartt
15
101
mit
1
Serving PyTorch models with TorchServe :fire:
mlops pytorch torchserve pytorch-cnn model-serving serve-pytorch machine-learning model-deployment image-classification
Created 2020-11-01
111 commits to main branch, last one 3 years ago
notAI-tech
fastDeploy notAI-tech
18
93
mit
8
Deploy DL/ ML inference pipelines with minimal extra code.
docker falcon gevent python triton pytorch serving gunicorn websocket tf-serving torchserve http-server deep-learning model-serving triton-server streaming-audio inference-server model-deployment tensorflow-serving triton-inference-server
Created 2020-04-09
458 commits to master branch, last one about a month ago
NimbleBoxAI
nbox NimbleBoxAI
13
88
apache-2.0
4
The official python package for NimbleBox. Exposes all APIs as CLIs and contains modules to make ML 🌸
mlops ml-service mlops-tool ml-platform data-science model-serving mlops-pipeline mlops-workflow practical-mlops machine-learning mlops-automation model-deployment model-management model-monitoring ml-infrastructure
Created 2021-07-29
396 commits to master branch, last one 11 months ago