3 results found Sort:

feifeibear
long-context-attention feifeibear
33
454
apache-2.0
5
USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
pytorch llm-training llm-inference ring-attention deepspeed-ulysses attention-is-all-you-need
Created 2024-03-27
239 commits to main branch, last one 4 days ago
InternLM
InternEvo InternLM
63
368
apache-2.0
10
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
910b gemma llava zero3 llama3 pytorch internlm internlm2 multi-modal llm-training llm-framework ring-attention flash-attention deepspeed-ulysses tensor-parallelism transformers-models pipeline-parallelism sequence-parallelism
Created 2024-01-16
511 commits to develop branch, last one 10 days ago
DAMO-NLP-SG
Inf-CLIP DAMO-NLP-SG
11
237
apache-2.0
5
[CVPR 2025] The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A super memory-efficiency CLIP training scheme.
clip ring-attention flash-attention memory-efficient infinite-batch-size contrastive-learning
Created 2024-10-16
30 commits to main branch, last one 2 months ago