6 results found Sort:

bigscience-workshop
petals bigscience-workshop
545
9.5k
mit
99
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
gpt nlp bloom llama falcon chatbot guanaco mixtral pytorch transformer deep-learning language-models neural-networks machine-learning pretrained-models tensor-parallelism distributed-systems volunteer-computing pipeline-parallelism large-language-models
Created 2022-06-12
522 commits to main branch, last one 7 months ago
InternLM
InternEvo InternLM
63
368
apache-2.0
10
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
910b gemma llava zero3 llama3 pytorch internlm internlm2 multi-modal llm-training llm-framework ring-attention flash-attention deepspeed-ulysses tensor-parallelism transformers-models pipeline-parallelism sequence-parallelism
Created 2024-01-16
511 commits to develop branch, last one 9 days ago
kaiyuyue
torchshard kaiyuyue
15
298
apache-2.0
10
Slicing a PyTorch Tensor Into Parallel Shards
pytorch model-parallelism tensor-parallelism
Created 2021-04-27
9 commits to main branch, last one 3 years ago
ai-decentralized
BloomBee ai-decentralized
12
87
apache-2.0
10
Decentralized LLMs fine-tuning and inference with offloading
llama pytorch deep-learning machine-learning tensor-parallelism distributed-systems pipeline-parallelism
Created 2025-01-23
49 commits to main branch, last one 28 days ago
xrsrke
pipegoose xrsrke
18
81
mit
3
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
moe zero-1 megatron megatron-lm transformers 3d-parallelism data-parallelism model-parallelism mixture-of-experts tensor-parallelism pipeline-parallelism sequence-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling
Created 2023-06-14
554 commits to main branch, last one about a year ago
aniquetahir
JORA aniquetahir
1
33
other
1
JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
jax lora machine-learning tensor-parallelism large-language-models
Created 2024-03-14
45 commits to master branch, last one 11 months ago