Search Results - RepositoryStats

2 results found Sort:

317

apache-2.0

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

910b gemma llava zero3 llama3 pytorch internlm internlm2 multi-modal llm-training llm-framework ring-attention flash-attention deepspeed-ulysses tensor-parallelism transformers-models pipeline-parallelism sequence-parallelism

Created 2024-01-16

493 commits to develop branch, last one 4 days ago

mit

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

moe zero-1 megatron megatron-lm transformers 3d-parallelism data-parallelism model-parallelism mixture-of-experts tensor-parallelism pipeline-parallelism sequence-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling

Created 2023-06-14

554 commits to main branch, last one about a year ago