Search Results - RepositoryStats

4.6k

30.5k

apache-2.0

248

A high-throughput and memory-efficient inference and serving engine for LLMs

amd gpt hpu llm tpu xpu cuda rocm llama mlops llmops pytorch trainium inference inferentia llm-serving transformer model-serving

Created 2023-02-09

3,511 commits to main branch, last one 13 hours ago

BentoML bentoml

792

7.2k

apache-2.0

77

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

llm mlops llmops python multimodal llm-serving ai-inference deep-learning generative-ai llm-inference model-serving ml-engineering machine-learning inference-platform model-inference-service

Created 2019-04-02

3,365 commits to main branch, last one a day ago

Deep-Learning-in-Production ahkarami

687

4.3k

unknown

148

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

Created 2018-05-03

221 commits to master branch, last one 12 days ago

FedML FedML-AI

785

4.2k

apache-2.0

118

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on a...

mlops edge-ai ai-agent deep-learning model-serving inference-engine machine-learning model-deployment federated-learning on-device-training distributed-training

Created 2020-07-21

12,120 commits to master branch, last one 6 months ago

kserve kserve

1.1k

3.6k

apache-2.0

65

Standardized Serverless ML Inference Platform on Kubernetes

k8s genai istio mlops kserve knative pytorch sklearn xgboost kubeflow kubernetes tensorflow service-mesh hacktoberfest llm-inference model-serving machine-learning model-interpretability artificial-intelligence

Created 2019-03-27

1,630 commits to master branch, last one 2 days ago

AI-System-School HuaizhengZhang

310

2.7k

mit

125

🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys...

genai mlsys llmsys ai-infra model-serving model-training large-language-models

Created 2019-01-07

509 commits to master branch, last one 3 months ago

lightllm ModelTC

206

2.6k

apache-2.0

23

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

gpt llm nlp llama deep-learning model-serving openai-triton

Created 2023-07-22

344 commits to main branch, last one 3 days ago

lorax predibase

144

2.2k

apache-2.0

33

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

gpt llm lora llama llmops pytorch fine-tuning llm-serving transformers llm-inference model-serving

Created 2023-10-20

840 commits to main branch, last one 18 hours ago

envd tensorchord

160

2.0k

apache-2.0

23

🏕️ Reproducible development environment

mlops docker llmops buildkit hacktoberfest model-serving mlops-workflow developer-tools development-environment

Created 2022-04-11

1,113 commits to main branch, last one 4 months ago

aici microsoft

78

1.9k

mit

23

AICI: Prompts as (Wasm) Programs

ai llm rust wasm llmops wasmtime inference llm-serving transformer llm-framework llm-inference model-serving language-model

Created 2023-09-26

1,616 commits to main branch, last one 10 days ago

mlrun mlrun

254

1.4k

apache-2.0

28

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates t...

mlops python workflow kubernetes data-science model-serving mlops-workflow data-engineering machine-learning experiment-tracking

Created 2019-09-01

5,566 commits to development branch, last one 21 hours ago

hopsworks logicalclocks

144

1.2k

agpl-3.0

35

Hopsworks - Data-Intensive AI platform with a Feature Store

ml aws gcp azure mlops kserve python pyspark hopsworks governance serverless data-science feature-store model-serving machine-learning feature-management feature-engineering

Created 2018-07-26

6,412 commits to master branch, last one 17 days ago

truss basetenlabs

71

918

mit

19

The simplest way to serve AI/ML models in production

falcon whisper wizardlm packaging easy-to-use open-source inference-api model-serving inference-server machine-learning stable-diffusion artificial-intelligence

Created 2022-07-06

1,266 commits to main branch, last one a day ago

mosec mosecorg

60

792

apache-2.0

12

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

cv gpu jax llm tts rust mlops mxnet python pytorch tensorflow llm-serving deep-learning hacktoberfest model-serving nerual-network machine-learning machine-learning-platform

Created 2021-03-13

390 commits to main branch, last one 2 days ago

Yatai bentoml

69

790

other

19

Model Deployment at Scale on Kubernetes 🦄️

k8s mlops bentoml kubernetes model-serving machine-learning model-deployment

Created 2021-07-30

639 commits to main branch, last one 6 months ago

model_server openvinotoolkit

212

675

apache-2.0

31

A scalable inference server for models optimized with OpenVINO™

ai dag edge cloud serving openvino inference kubernetes deep-learning model-serving machine-learning

Created 2018-09-26

2,610 commits to main branch, last one 22 hours ago

Nanoflow efeslab

26

636

apache-2.0

7

A throughput-oriented high-performance serving framework for LLMs

llm cuda llama2 inference llm-serving model-serving

Created 2024-08-19

43 commits to main branch, last one 2 months ago

pinferencia underneathall

87

564

apache-2.0

41

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

Created 2022-04-04

55 commits to main branch, last one about a year ago

rtp-llm alibaba

50

545

apache-2.0

12

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

gpt llm llama llmops inference llm-serving model-serving

Created 2023-12-27

1,607 commits to main branch, last one about a month ago

kitops jozu-ai

52

502

apache-2.0

13

Securely share and store AI/ML projects as OCI artifacts in your container registry.

Created 2024-02-02

609 commits to main branch, last one 20 hours ago

fastapi-ml-skeleton eightBEC

83

395

apache-2.0

6

FastAPI Skeleton App to serve machine learning models production-ready.

python fastapi python3 model-serving machine-learning

Created 2020-02-15

12 commits to main branch, last one 6 months ago

stable-diffusion-deploy Lightning-Universe

39

392

apache-2.0

19

Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services worki...

model-serving stable-diffusion

This repository has been archived (exclude archived)

Created 2022-08-23

314 commits to main branch, last one about a year ago

xFasterTransformer intel

64

380

apache-2.0

15

This repository has no description...

llm qwen xeon intel llama chatglm inference transformer model-serving

Created 2023-06-14

347 commits to main branch, last one 9 days ago

ServerlessLLM ServerlessLLM

32

352

apache-2.0

13

Serverless LLM Serving for Everyone.

cuda pytorch model-serving model-as-a-service serverless-inference large-language-models huggingface-transformers

Created 2024-01-23

93 commits to main branch, last one 2 days ago

BentoDiffusion bentoml

25

337

apache-2.0

12

BentoDiffusion: A collection of diffusion models served with BentoML

ai lora kubernetes fine-tuning model-serving diffusion-models stable-diffusion

Created 2023-06-12

56 commits to main branch, last one 20 days ago

JetStream AI-Hypercomputer

31

235

apache-2.0

18

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

gpt gpu jax llm tpu gemma llama mlops llama2 llmops pytorch inference transformer llm-inference model-serving large-language-models

Created 2024-03-01

123 commits to main branch, last one 2 days ago

chitra aniketmaurya

37

224

apache-2.0

6

A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.

mlops python fastapi gradcam pytorch tensorflow deep-learning hacktoberfest image-dataset model-serving visualization bounding-boxes image-processing machine-learning model-deployment object-detection model-visualization image-classification model-interpretation

Created 2020-01-23

372 commits to master branch, last one 5 months ago

zoltar spotify

33

139

apache-2.0

24

Common library for serving TensorFlow, XGBoost and scikit-learn models in production.

java xgboost tensorflow model-serving machine-learning

This repository has been archived (exclude archived)

Created 2018-03-12

505 commits to main branch, last one about a year ago

FATE-Serving FederatedAI

77

139

apache-2.0

30

A scalable, high-performance serving system for federated learning models

monitor inference model-serving model-versioning federated-learning

Created 2019-09-10

157 commits to master branch, last one 4 months ago

clearml-serving allegroai

40

137

apache-2.0

11

ClearML - Model-Serving Orchestration and Repository Solution

ai mlops devops triton clearml serving kubernetes serving-ml deep-learning model-serving machine-learning tensorflow-serving serving-pytorch-models triton-inference-server

Created 2021-04-12

140 commits to main branch, last one 4 months ago