15 results found Sort:

hpcaitech
ColossalAI hpcaitech
4.5k
40.8k
apache-2.0
393
Making large AI models cheaper, faster and more accessible
ai hpc big-model inference large-scale deep-learning data-parallelism foundation-models model-parallelism pipeline-parallelism distributed-computing heterogeneous-training
Created 2021-10-28
3,809 commits to main branch, last one 8 hours ago
deepspeedai
DeepSpeed deepspeedai
4.3k
38.0k
apache-2.0
351
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
gpu zero pytorch inference compression deep-learning data-parallelism machine-learning model-parallelism billion-parameters mixture-of-experts trillion-parameters pipeline-parallelism
Created 2020-01-23
2,754 commits to master branch, last one 17 hours ago
bigscience-workshop
petals bigscience-workshop
553
9.6k
mit
100
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
gpt nlp bloom llama falcon chatbot guanaco mixtral pytorch transformer deep-learning language-models neural-networks machine-learning pretrained-models tensor-parallelism distributed-systems volunteer-computing pipeline-parallelism large-language-models
Created 2022-06-12
522 commits to main branch, last one 7 months ago
kakaobrain
torchgpipe kakaobrain
99
836
bsd-3-clause
32
A GPipe implementation in PyTorch
gpipe pytorch parallelism checkpointing deep-learning model-parallelism pipeline-parallelism
Created 2019-05-10
350 commits to master branch, last one 4 years ago
PaddlePaddle
PaddleFleetX PaddlePaddle
164
465
apache-2.0
22
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
cloud elastic benchmark fleet-api lightning large-scale paddlecloud pretraining paddlepaddle data-parallelism model-parallelism pipeline-parallelism distributed-algorithm unsupervised-learning self-supervised-learning
Created 2018-12-12
440 commits to develop branch, last one about a year ago
Coobiw
MPP-LLaVA Coobiw
23
437
unknown
6
Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train...
mllm qwen deepspeed fine-tuning pretraining model-parallel pipeline-parallelism video-language-model video-large-language-models multimodal-large-language-models
Created 2023-10-24
135 commits to master branch, last one about a month ago
Oneflow-Inc
libai Oneflow-Inc
56
401
apache-2.0
41
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training
nlp oneflow large-scale transformer deep-learning data-parallelism model-parallelism vision-transformer distributed-training pipeline-parallelism self-supervised-learning
Created 2021-10-25
358 commits to main branch, last one 5 months ago
InternLM
InternEvo InternLM
64
379
apache-2.0
10
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
910b gemma llava zero3 llama3 pytorch internlm internlm2 multi-modal llm-training llm-framework ring-attention flash-attention deepspeed-ulysses tensor-parallelism transformers-models pipeline-parallelism sequence-parallelism
Created 2024-01-16
511 commits to develop branch, last one about a month ago
alibaba
EasyParallelLibrary alibaba
49
267
apache-2.0
12
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
gpu deep-learning data-parallelism memory-efficient model-parallelism distributed-training pipeline-parallelism
Created 2022-02-23
21 commits to main branch, last one 2 years ago
Shenggan
awesome-distributed-ml Shenggan
27
231
unknown
10
A curated list of awesome projects and papers for distributed training or inference
deep-learning machine-learning model-parallelism distributed-systems pipeline-parallelism high-performance-computing
Created 2022-04-22
33 commits to main branch, last one 6 months ago
torchpipe
torchpipe torchpipe
13
160
apache-2.0
5
Serving Inside Pytorch
ray serve pytorch serving tensorrt inference torch2trt deployment llm-serving pipeline-parallelism triton-inference-server
Created 2023-10-24
348 commits to v0 branch, last one about a month ago
ai-decentralized
BloomBee ai-decentralized
13
88
apache-2.0
10
Decentralized LLMs fine-tuning and inference with offloading
llama pytorch deep-learning machine-learning tensor-parallelism distributed-systems pipeline-parallelism
Created 2025-01-23
49 commits to main branch, last one about a month ago
xrsrke
pipegoose xrsrke
18
81
mit
3
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
moe zero-1 megatron megatron-lm transformers 3d-parallelism data-parallelism model-parallelism mixture-of-experts tensor-parallelism pipeline-parallelism sequence-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling
Created 2023-06-14
554 commits to main branch, last one about a year ago
ParCIS
Chimera ParCIS
8
62
gpl-3.0
1
Chimera: bidirectional pipeline parallelism for efficiently training large-scale models.
transformers pipeline-parallelism distributed-deep-learning
Created 2021-05-30
19 commits to main branch, last one 29 days ago
saareliad
FTPipe saareliad
7
41
unknown
3
FTPipe and related pipeline model parallelism research.
t5 nlp fine-tuning deep-neural-networks distributed-training pipeline-parallelism
Created 2020-06-29
2,372 commits to master branch, last one about a year ago