Search Results - RepositoryStats

4.2k

35.9k

apache-2.0

346

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

gpu zero pytorch inference compression deep-learning data-parallelism machine-learning model-parallelism billion-parameters mixture-of-experts trillion-parameters pipeline-parallelism

Created 2020-01-23

2,625 commits to master branch, last one a day ago

mixtral-offloading dvmazur

227

2.3k

mit

29

Run Mixtral-8x7B models in Colab or consumer desktops

llm pytorch offloading google-colab quantization deep-learning colab-notebook language-model mixture-of-experts

Created 2023-12-15

86 commits to master branch, last one 11 months ago

hivemind learning-at-home

170

2.1k

mit

56

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

dht asyncio pytorch hivemind deep-learning neural-networks machine-learning mixture-of-experts distributed-systems volunteer-computing distributed-training asynchronous-programming

Created 2020-02-27

583 commits to master branch, last one about a month ago

MoE-LLaVA PKU-YuanGroup

127

2.0k

apache-2.0

24

Mixture-of-Experts for Large Vision-Language Models

moe multi-modal mixture-of-experts large-vision-language-model

Created 2023-12-14

228 commits to main branch, last one 18 days ago

optillm codelion

139

1.8k

apache-2.0

22

Optimizing inference proxy for LLMs

Created 2024-08-22

267 commits to main branch, last one 22 days ago

mixture-of-experts davidmrau

105

1.0k

gpl-3.0

6

PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538

moe pytorch re-implementation mixture-of-experts sparsely-gated-mixture-of-experts

Created 2019-07-19

30 commits to master branch, last one 8 months ago

Aria rhymes-ai

76

917

apache-2.0

19

Codebase for Aria - an Open Multimodal Native MoE

multimodal mixture-of-experts vision-and-language

Created 2024-09-29

204 commits to main branch, last one 3 days ago

llama-moe pjlab-sys4nlp

47

896

apache-2.0

8

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

llm moe llama expert-partition mixture-of-experts continual-pre-training

Created 2023-07-24

212 commits to main branch, last one 5 months ago

Tutel microsoft

94

740

mit

15

Tutel MoE: An Optimized Mixture-of-Experts Implementation

moe nlp pytorch transformer mixture-of-experts

Created 2021-08-06

182 commits to main branch, last one 29 days ago

smt SMTorg

213

706

bsd-3-clause

29

Surrogate Modeling Toolbox

sampling derivative multi-fidelity machine-learning surrogate-models mixture-of-experts predictive-modeling

Created 2016-11-08

1,568 commits to master branch, last one a day ago

keras-mmoe drawbridge

222

697

mit

15

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)

keras kdd2018 tensorflow data-science deep-learning machine-learning mixture-of-experts multi-task-learning deep-neural-networks

Created 2018-09-10

22 commits to master branch, last one 3 years ago

mixture-of-experts lucidrains

50

658

mit

6

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

transformer deep-learning mixture-of-experts artificial-intelligence

Created 2020-07-13

33 commits to master branch, last one about a year ago

makeMoE AviSoori1x

62

602

mit

7

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

llm pytorch deep-learning neural-networks mixture-of-experts large-language-models pytorch-implementation

Created 2024-01-22

116 commits to main branch, last one about a month ago

Chinese-Mixtral ymcui

43

589

apache-2.0

15

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

32k 64k llm moe nlp mixtral mixture-of-experts large-language-models

Created 2024-01-11

31 commits to main branch, last one 7 months ago

mergoo Leeroo-AI

26

420

lgpl-3.0

5

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

llm nlp lora merge fine-tuning multi-model open-source transformers generative-ai mixture-of-experts mixture-of-adapters large-language-models artificial-intelligence

Created 2024-04-08

43 commits to main branch, last one 3 months ago

Generalizable-Mixture-of-Experts Luodian

35

298

mit

15

GMoE could be the next backbone model for many kinds of generalization task.

pytorch deep-learning mixture-of-experts domain-generalization pytorch-implementation

Created 2022-05-28

28 commits to main branch, last one about a year ago

st-moe-pytorch lucidrains

25

294

mit

7

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

deep-learning mixture-of-experts artificial-intelligence conditional-computation

Created 2023-03-26

68 commits to main branch, last one 6 months ago

soft-moe-pytorch lucidrains

8

249

mit

12

Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch

transformers deep-learning mixture-of-experts artificial-intelligence

Created 2023-08-04

28 commits to main branch, last one 8 months ago

inferflow inferflow

24

236

mit

8

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

moe qwen bloom gemma phi-2 falcon llama2 m2m100 minicpm mistral mixtral deepseek internlm llamacpp baichuan2 llm-inference mixture-of-experts model-quantization multi-gpu-inference

Created 2023-12-26

116 commits to main branch, last one 9 months ago