5 results found Sort:

bigscience-workshop
petals bigscience-workshop
525
9.2k
mit
95
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
gpt nlp bloom llama falcon chatbot guanaco mixtral pytorch transformer deep-learning language-models neural-networks machine-learning pretrained-models tensor-parallelism distributed-systems volunteer-computing pipeline-parallelism large-language-models
Created 2022-06-12
522 commits to main branch, last one 2 months ago
InternLM
InternEvo InternLM
52
310
apache-2.0
10
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
910b gemma llava zero3 llama3 pytorch internlm internlm2 multi-modal llm-training llm-framework ring-attention flash-attention deepspeed-ulysses tensor-parallelism transformers-models pipeline-parallelism sequence-parallelism
Created 2024-01-16
479 commits to develop branch, last one 3 days ago
kaiyuyue
torchshard kaiyuyue
15
296
apache-2.0
11
Slicing a PyTorch Tensor Into Parallel Shards
pytorch model-parallelism tensor-parallelism
Created 2021-04-27
9 commits to main branch, last one 3 years ago
xrsrke
pipegoose xrsrke
18
80
mit
4
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
moe zero-1 megatron megatron-lm transformers 3d-parallelism data-parallelism model-parallelism mixture-of-experts tensor-parallelism pipeline-parallelism sequence-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling
Created 2023-06-14
554 commits to main branch, last one 11 months ago
aniquetahir
JORA aniquetahir
1
29
other
2
JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
jax lora machine-learning tensor-parallelism large-language-models
Created 2024-03-14
45 commits to master branch, last one 6 months ago