13 results found Sort:

hpcaitech
ColossalAI hpcaitech
4.5k
40.8k
apache-2.0
392
Making large AI models cheaper, faster and more accessible
ai hpc big-model inference large-scale deep-learning data-parallelism foundation-models model-parallelism pipeline-parallelism distributed-computing heterogeneous-training
Created 2021-10-28
3,809 commits to main branch, last one 5 days ago
deepspeedai
DeepSpeed deepspeedai
4.3k
38.1k
apache-2.0
352
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
gpu zero pytorch inference compression deep-learning data-parallelism machine-learning model-parallelism billion-parameters mixture-of-experts trillion-parameters pipeline-parallelism
Created 2020-01-23
2,759 commits to master branch, last one 2 days ago
kakaobrain
torchgpipe kakaobrain
99
836
bsd-3-clause
32
A GPipe implementation in PyTorch
gpipe pytorch parallelism checkpointing deep-learning model-parallelism pipeline-parallelism
Created 2019-05-10
350 commits to master branch, last one 4 years ago
PaddlePaddle
PaddleFleetX PaddlePaddle
164
465
apache-2.0
22
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
cloud elastic benchmark fleet-api lightning large-scale paddlecloud pretraining paddlepaddle data-parallelism model-parallelism pipeline-parallelism distributed-algorithm unsupervised-learning self-supervised-learning
Created 2018-12-12
440 commits to develop branch, last one about a year ago
Oneflow-Inc
libai Oneflow-Inc
56
402
apache-2.0
41
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training
nlp oneflow large-scale transformer deep-learning data-parallelism model-parallelism vision-transformer distributed-training pipeline-parallelism self-supervised-learning
Created 2021-10-25
358 commits to main branch, last one 5 months ago
kaiyuyue
torchshard kaiyuyue
15
298
apache-2.0
10
Slicing a PyTorch Tensor Into Parallel Shards
pytorch model-parallelism tensor-parallelism
Created 2021-04-27
9 commits to main branch, last one 3 years ago
alibaba
EasyParallelLibrary alibaba
49
267
apache-2.0
12
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
gpu deep-learning data-parallelism memory-efficient model-parallelism distributed-training pipeline-parallelism
Created 2022-02-23
21 commits to main branch, last one 2 years ago
Shenggan
awesome-distributed-ml Shenggan
27
231
unknown
10
A curated list of awesome projects and papers for distributed training or inference
deep-learning machine-learning model-parallelism distributed-systems pipeline-parallelism high-performance-computing
Created 2022-04-22
33 commits to main branch, last one 6 months ago
xrsrke
pipegoose xrsrke
18
82
mit
3
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
moe zero-1 megatron megatron-lm transformers 3d-parallelism data-parallelism model-parallelism mixture-of-experts tensor-parallelism pipeline-parallelism sequence-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling
Created 2023-06-14
554 commits to main branch, last one about a year ago
tanyuqian
redco tanyuqian
8
65
apache-2.0
4
NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference
jax ppo maml gemma llama mlsys fedavg seq2seq flan-t5-xxl meta-learning mixed-precision diffusion-models image-captioning stable-diffusion model-parallelism federated-learning differential-privacy distributed-training large-language-models reinforcement-learning
Created 2022-12-19
674 commits to master branch, last one 4 months ago
hkproj
pytorch-transformer-distributed hkproj
28
64
unknown
2
Distributed training (multi-node) of a Transformer model
pytorch tutorial deep-learning data-parallelism machine-learning model-parallelism distributed-training gradient-accumulation collective-communication distributed-data-parallel
Created 2023-12-08
87 commits to main branch, last one about a year ago
NERSC
sc23-dl-tutorial NERSC
9
43
unknown
21
SC23 Deep Learning at Scale Tutorial Material
deep-learning ai-for-science data-parallelism model-parallelism vision-transformers
Created 2023-08-28
85 commits to main branch, last one about a year ago
vdutts7
dnn-distributed vdutts7
12
42
unknown
2
Distributed training of DNNs • C++/MPI Proxies (GPT-2, GPT-3, CosmoFlow, DLRM)
dnn mpi model-parallelism deep-neural-networks distributed-deep-learning message-passing-interface
Created 2024-02-22
17 commits to main branch, last one 2 years ago