Search Results - RepositoryStats

3.9k

33.4k

apache-2.0

338

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

gpu zero pytorch inference compression deep-learning data-parallelism machine-learning model-parallelism billion-parameters mixture-of-experts trillion-parameters pipeline-parallelism

Created 2020-01-23

2,332 commits to master branch, last one a day ago

mixtral-offloading dvmazur

223

2.3k

mit

30

Run Mixtral-8x7B models in Colab or consumer desktops

llm pytorch offloading google-colab quantization deep-learning colab-notebook language-model mixture-of-experts

Created 2023-12-15

86 commits to master branch, last one 5 months ago

hivemind learning-at-home

141

1.9k

mit

57

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

dht asyncio pytorch hivemind deep-learning neural-networks machine-learning mixture-of-experts distributed-systems volunteer-computing distributed-training asynchronous-programming

Created 2020-02-27

575 commits to master branch, last one a day ago

MoE-LLaVA PKU-YuanGroup

105

1.8k

apache-2.0

21

Mixture-of-Experts for Large Vision-Language Models

moe multi-modal mixture-of-experts large-vision-language-model

Created 2023-12-14

226 commits to main branch, last one 29 days ago

mixture-of-experts davidmrau

92

881

gpl-3.0

4

PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538

moe pytorch re-implementation mixture-of-experts sparsely-gated-mixture-of-experts

Created 2019-07-19

30 commits to master branch, last one about a month ago

llama-moe pjlab-sys4nlp

37

766

apache-2.0

8

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

llm moe llama expert-partition mixture-of-experts continual-pre-training

Created 2023-07-24

211 commits to main branch, last one 3 months ago

keras-mmoe drawbridge

216

680

mit

15

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)

keras kdd2018 tensorflow data-science deep-learning machine-learning mixture-of-experts multi-task-learning deep-neural-networks

Created 2018-09-10

22 commits to master branch, last one 2 years ago

tutel microsoft

84

677

mit

14

Tutel MoE: An Optimized Mixture-of-Experts Implementation

moe nlp pytorch transformer mixture-of-experts

Created 2021-08-06

173 commits to main branch, last one 7 days ago

smt SMTorg

196

636

bsd-3-clause

29

Surrogate Modeling Toolbox

sampling derivative multi-fidelity machine-learning surrogate-models mixture-of-experts predictive-modeling

Created 2016-11-08

1,489 commits to master branch, last one 15 days ago

mixture-of-experts lucidrains

45

562

mit

5

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

transformer deep-learning mixture-of-experts artificial-intelligence

Created 2020-07-13

33 commits to master branch, last one 9 months ago

makeMoE AviSoori1x

48

542

mit

7

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

llm pytorch deep-learning neural-networks mixture-of-experts large-language-models pytorch-implementation

Created 2024-01-22

115 commits to main branch, last one 2 months ago

Chinese-Mixtral ymcui

40

540

apache-2.0

15

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

32k 64k llm moe nlp mixtral mixture-of-experts large-language-models

Created 2024-01-11

31 commits to main branch, last one about a month ago

mergoo Leeroo-AI

19

347

lgpl-3.0

4

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

llm nlp lora merge fine-tuning multi-model open-source transformers generative-ai mixture-of-experts mixture-of-adapters large-language-models artificial-intelligence

Created 2024-04-08

41 commits to main branch, last one 12 days ago

Generalizable-Mixture-of-Experts Luodian

35

286

mit

15

GMoE could be the next backbone model for many kinds of generalization task.

pytorch deep-learning mixture-of-experts domain-generalization pytorch-implementation

Created 2022-05-28

28 commits to main branch, last one about a year ago

st-moe-pytorch lucidrains

23

242

mit

5

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

deep-learning mixture-of-experts artificial-intelligence conditional-computation

Created 2023-03-26

67 commits to main branch, last one 9 days ago

inferflow inferflow

22

229

mit

7

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

moe qwen bloom gemma phi-2 falcon llama2 m2m100 minicpm mistral mixtral deepseek internlm llamacpp baichuan2 llm-inference mixture-of-experts model-quantization multi-gpu-inference

Created 2023-12-26

116 commits to main branch, last one 3 months ago

soft-moe-pytorch lucidrains

8

212

mit

10

Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch

transformers deep-learning mixture-of-experts artificial-intelligence

Created 2023-08-04

28 commits to main branch, last one about a month ago

fiddler efeslab

16

148

apache-2.0

8

Fast Inference of MoE Models with CPU-GPU Orchestration

llm mixtral-8x7b llm-inference local-inference mixture-of-experts

Created 2024-02-05

49 commits to main branch, last one about a month ago

awesome-adaptive-computation koayon

7

107

apache-2.0

4

A curated reading list of research in Adaptive Computation, Dynamic Compute & Mixture of Experts (MoE).

nlp pytorch tensorflow transformers computer-vision machine-learning mixture-of-experts adaptive-computation

Created 2023-08-17

123 commits to main branch, last one 22 days ago

mixture-of-attention lucidrains

3

99

mit

8

Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts

deep-learning routed-attention mixture-of-experts attention-mechanisms artificial-intelligence

Created 2023-04-21

41 commits to main branch, last one 11 months ago

RealCompo YangLing0818

3

97

unknown

4

RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models

layout-to-image diffusion-models mixture-of-experts text-to-image-diffusion

Created 2024-02-20

15 commits to main branch, last one 18 days ago

MOELoRA-peft liuqidong07

10

95

mit

3

[SIGIR'24] The official implementation code of MOELoRA.

peft chatglm multi-task mixture-of-experts multitask-learning low-rank-adaptation peft-fine-tuning-llm large-language-models parameter-efficient-fine-tuning

Created 2023-10-19

20 commits to master branch, last one 17 days ago

pipegoose xrsrke

17

75

mit

4

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

moe zero-1 megatron megatron-lm transformers 3d-parallelism data-parallelism model-parallelism mixture-of-experts tensor-parallelism pipeline-parallelism sequence-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling

Created 2023-06-14

554 commits to main branch, last one 6 months ago

seemoredetails eduardzamfir

1

68

unknown

5

Repository for "See More Details: Efficient Image Super-Resolution by Experts Mining", ICML 2024

icml-2024 efficiency computer-vision low-level-vision super-resolution mixture-of-experts low-rank-matrix-decomposition

Created 2024-02-05

30 commits to main branch, last one 12 days ago

MoE-Infinity TorchMoE

5

63

apache-2.0

8

PyTorch library for cost-effective, fast and easy serving of MoE models.

pytorch huggingface inference-engine mixture-of-experts large-language-models

Created 2024-01-22

15 commits to main branch, last one 11 days ago

soft-mixture-of-experts fkodom

4

60

mit

3

PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)

pytorch transformer deep-learning mixture-of-experts vision-transformer

Created 2023-08-07

21 commits to main branch, last one 8 months ago

MC-SMoE UNITES-Lab

7

58

mit

4

[ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"

merging efficiency mixture-of-experts

Created 2023-09-30

24 commits to main branch, last one 7 days ago

leeroo_orchestrator Leeroo-AI

4

48

mit

5

The implementation of "Leeroo Orchestrator: Elevating LLMs Performance Through Model Integration"

ai llm nlp cost gpt4 efficiency opensource performance mixture-of-experts large-language-models orchestration-of-experts

Created 2024-01-23

19 commits to main branch, last one about a month ago

egobox relf

1

48

apache-2.0

3

Efficient global optimization toolbox in Rust: bayesian optimization, mixture of gaussian processes, sampling methods

surrogate-models gaussian-processes mixture-of-experts global-optimization latin-hypercube-sampling

Created 2020-08-27

490 commits to master branch, last one 6 days ago

Neural-Implicit-Dict VITA-Group

1

31

mit

9

[ICML 2022] "Neural Implicit Dictionary via Mixture-of-Expert Training" by Peihao Wang, Zhiwen Fan, Tianlong Chen, Zhangyang Wang

mixture-of-experts dictionary-learning neural-implicit-representations

Created 2022-07-08

11 commits to main branch, last one 5 months ago