Search Results - RepositoryStats

4.1k

35.5k

apache-2.0

346

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

gpu zero pytorch inference compression deep-learning data-parallelism machine-learning model-parallelism billion-parameters mixture-of-experts trillion-parameters pipeline-parallelism

Created 2020-01-23

2,573 commits to master branch, last one a day ago

mixtral-offloading dvmazur

226

2.3k

mit

29

Run Mixtral-8x7B models in Colab or consumer desktops

llm pytorch offloading google-colab quantization deep-learning colab-notebook language-model mixture-of-experts

Created 2023-12-15

86 commits to master branch, last one 10 months ago

hivemind learning-at-home

169

2.1k

mit

56

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

dht asyncio pytorch hivemind deep-learning neural-networks machine-learning mixture-of-experts distributed-systems volunteer-computing distributed-training asynchronous-programming

Created 2020-02-27

583 commits to master branch, last one 16 days ago

MoE-LLaVA PKU-YuanGroup

128

2.0k

apache-2.0

24

Mixture-of-Experts for Large Vision-Language Models

moe multi-modal mixture-of-experts large-vision-language-model

Created 2023-12-14

226 commits to main branch, last one 6 months ago

optillm codelion

127

1.6k

apache-2.0

22

Optimizing inference proxy for LLMs

Created 2024-08-22

243 commits to main branch, last one 5 days ago

mixture-of-experts davidmrau

104

983

gpl-3.0

5

PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538

moe pytorch re-implementation mixture-of-experts sparsely-gated-mixture-of-experts

Created 2019-07-19

30 commits to master branch, last one 7 months ago

llama-moe pjlab-sys4nlp

46

883

apache-2.0

8

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

llm moe llama expert-partition mixture-of-experts continual-pre-training

Created 2023-07-24

212 commits to main branch, last one 4 months ago

Aria rhymes-ai

70

840

apache-2.0

18

Codebase for Aria - an Open Multimodal Native MoE

multimodal mixture-of-experts vision-and-language

Created 2024-09-29

185 commits to main branch, last one a day ago

Tutel microsoft

93

735

mit

15

Tutel MoE: An Optimized Mixture-of-Experts Implementation

moe nlp pytorch transformer mixture-of-experts

Created 2021-08-06

181 commits to main branch, last one 3 days ago

keras-mmoe drawbridge

221

694

mit

15

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)

keras kdd2018 tensorflow data-science deep-learning machine-learning mixture-of-experts multi-task-learning deep-neural-networks

Created 2018-09-10

22 commits to master branch, last one 3 years ago

smt SMTorg

208

690

bsd-3-clause

29

Surrogate Modeling Toolbox

sampling derivative multi-fidelity machine-learning surrogate-models mixture-of-experts predictive-modeling

Created 2016-11-08

1,550 commits to master branch, last one 6 days ago

mixture-of-experts lucidrains

49

642

mit

6

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

transformer deep-learning mixture-of-experts artificial-intelligence

Created 2020-07-13

33 commits to master branch, last one about a year ago

makeMoE AviSoori1x

62

594

mit

7

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

llm pytorch deep-learning neural-networks mixture-of-experts large-language-models pytorch-implementation

Created 2024-01-22

116 commits to main branch, last one 21 days ago

Chinese-Mixtral ymcui

43

584

apache-2.0

15

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

32k 64k llm moe nlp mixtral mixture-of-experts large-language-models

Created 2024-01-11

31 commits to main branch, last one 6 months ago

mergoo Leeroo-AI

25

408

lgpl-3.0

4

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

llm nlp lora merge fine-tuning multi-model open-source transformers generative-ai mixture-of-experts mixture-of-adapters large-language-models artificial-intelligence

Created 2024-04-08

43 commits to main branch, last one 2 months ago

Generalizable-Mixture-of-Experts Luodian

35

296

mit

15

GMoE could be the next backbone model for many kinds of generalization task.

pytorch deep-learning mixture-of-experts domain-generalization pytorch-implementation

Created 2022-05-28

28 commits to main branch, last one about a year ago

st-moe-pytorch lucidrains

25

293

mit

7

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

deep-learning mixture-of-experts artificial-intelligence conditional-computation

Created 2023-03-26

68 commits to main branch, last one 5 months ago

soft-moe-pytorch lucidrains

9

247

mit

12

Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch

transformers deep-learning mixture-of-experts artificial-intelligence

Created 2023-08-04

28 commits to main branch, last one 7 months ago

inferflow inferflow

24

236

mit

8

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

moe qwen bloom gemma phi-2 falcon llama2 m2m100 minicpm mistral mixtral deepseek internlm llamacpp baichuan2 llm-inference mixture-of-experts model-quantization multi-gpu-inference

Created 2023-12-26

116 commits to main branch, last one 8 months ago