Search Results - RepositoryStats

3.9k

33.6k

apache-2.0

339

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

gpu zero pytorch inference compression deep-learning data-parallelism machine-learning model-parallelism billion-parameters mixture-of-experts trillion-parameters pipeline-parallelism

Created 2020-01-23

2,352 commits to master branch, last one 11 hours ago

mixtral-offloading dvmazur

224

2.3k

mit

30

Run Mixtral-8x7B models in Colab or consumer desktops

llm pytorch offloading google-colab quantization deep-learning colab-notebook language-model mixture-of-experts

Created 2023-12-15

86 commits to master branch, last one 5 months ago

hivemind learning-at-home

142

1.9k

mit

57

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

dht asyncio pytorch hivemind deep-learning neural-networks machine-learning mixture-of-experts distributed-systems volunteer-computing distributed-training asynchronous-programming

Created 2020-02-27

575 commits to master branch, last one 14 days ago

MoE-LLaVA PKU-YuanGroup

108

1.8k

apache-2.0

21

Mixture-of-Experts for Large Vision-Language Models

moe multi-modal mixture-of-experts large-vision-language-model

Created 2023-12-14

226 commits to main branch, last one about a month ago

mixture-of-experts davidmrau

93

889

gpl-3.0

4

PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538

moe pytorch re-implementation mixture-of-experts sparsely-gated-mixture-of-experts

Created 2019-07-19

30 commits to master branch, last one 2 months ago

llama-moe pjlab-sys4nlp

41

780

apache-2.0

8

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

llm moe llama expert-partition mixture-of-experts continual-pre-training

Created 2023-07-24

212 commits to main branch, last one 2 days ago

tutel microsoft

84

683

mit

15

Tutel MoE: An Optimized Mixture-of-Experts Implementation

moe nlp pytorch transformer mixture-of-experts

Created 2021-08-06

173 commits to main branch, last one 20 days ago

keras-mmoe drawbridge

217

679

mit

15

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)

keras kdd2018 tensorflow data-science deep-learning machine-learning mixture-of-experts multi-task-learning deep-neural-networks

Created 2018-09-10

22 commits to master branch, last one 2 years ago

smt SMTorg

196

641

bsd-3-clause

29

Surrogate Modeling Toolbox

sampling derivative multi-fidelity machine-learning surrogate-models mixture-of-experts predictive-modeling

Created 2016-11-08

1,499 commits to master branch, last one 19 hours ago

mixture-of-experts lucidrains

46

570

mit

5

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

transformer deep-learning mixture-of-experts artificial-intelligence

Created 2020-07-13

33 commits to master branch, last one 9 months ago

makeMoE AviSoori1x

49

547

mit

7

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

llm pytorch deep-learning neural-networks mixture-of-experts large-language-models pytorch-implementation

Created 2024-01-22

115 commits to main branch, last one 2 months ago

Chinese-Mixtral ymcui

40

546

apache-2.0

15

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

32k 64k llm moe nlp mixtral mixture-of-experts large-language-models

Created 2024-01-11

31 commits to main branch, last one about a month ago

mergoo Leeroo-AI

19

355

lgpl-3.0

4

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

llm nlp lora merge fine-tuning multi-model open-source transformers generative-ai mixture-of-experts mixture-of-adapters large-language-models artificial-intelligence

Created 2024-04-08

41 commits to main branch, last one 25 days ago

Generalizable-Mixture-of-Experts Luodian

35

286

mit

15

GMoE could be the next backbone model for many kinds of generalization task.

pytorch deep-learning mixture-of-experts domain-generalization pytorch-implementation

Created 2022-05-28

28 commits to main branch, last one about a year ago

st-moe-pytorch lucidrains

23

248

mit

6

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

deep-learning mixture-of-experts artificial-intelligence conditional-computation

Created 2023-03-26

68 commits to main branch, last one 10 days ago

inferflow inferflow

22

229

mit

7

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

moe qwen bloom gemma phi-2 falcon llama2 m2m100 minicpm mistral mixtral deepseek internlm llamacpp baichuan2 llm-inference mixture-of-experts model-quantization multi-gpu-inference

Created 2023-12-26

116 commits to main branch, last one 3 months ago

soft-moe-pytorch lucidrains

8

218

mit

10

Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch

transformers deep-learning mixture-of-experts artificial-intelligence

Created 2023-08-04

28 commits to main branch, last one 2 months ago

fiddler efeslab

16

151

apache-2.0

8

Fast Inference of MoE Models with CPU-GPU Orchestration

llm mixtral-8x7b llm-inference local-inference mixture-of-experts

Created 2024-02-05

49 commits to main branch, last one about a month ago

awesome-adaptive-computation koayon

7

108

apache-2.0

4

A curated reading list of research in Adaptive Computation, Dynamic Compute & Mixture of Experts (MoE).

nlp pytorch tensorflow transformers computer-vision machine-learning mixture-of-experts adaptive-computation

Created 2023-08-17

125 commits to main branch, last one 3 days ago

MOELoRA-peft liuqidong07

10

102

mit

3

[SIGIR'24] The official implementation code of MOELoRA.

peft chatglm multi-task mixture-of-experts multitask-learning low-rank-adaptation peft-fine-tuning-llm large-language-models parameter-efficient-fine-tuning

Created 2023-10-19

20 commits to master branch, last one about a month ago

mixture-of-attention lucidrains

3

100

mit

8

Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts

deep-learning routed-attention mixture-of-experts attention-mechanisms artificial-intelligence

Created 2023-04-21

41 commits to main branch, last one 11 months ago

RealCompo YangLing0818

3

97

unknown

4

RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models

layout-to-image diffusion-models mixture-of-experts text-to-image-diffusion

Created 2024-02-20

15 commits to main branch, last one about a month ago

pipegoose xrsrke

17

75

mit

4

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

moe zero-1 megatron megatron-lm transformers 3d-parallelism data-parallelism model-parallelism mixture-of-experts tensor-parallelism pipeline-parallelism sequence-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling

Created 2023-06-14

554 commits to main branch, last one 6 months ago

seemoredetails eduardzamfir

1

73

unknown

5

Repository for "See More Details: Efficient Image Super-Resolution by Experts Mining", ICML 2024

icml-2024 efficiency computer-vision low-level-vision super-resolution mixture-of-experts low-rank-matrix-decomposition

Created 2024-02-05

30 commits to main branch, last one 25 days ago

MoEL HLTCHKUST

14

72

mit

6

MoEL: Mixture of Empathetic Listeners

chatbot empathy transformer dialogue-systems mixture-of-experts dialogue-generation transformer-pytorch

Created 2019-08-15

15 commits to master branch, last one 3 years ago

MoE-Infinity TorchMoE

5

69

apache-2.0

8

PyTorch library for cost-effective, fast and easy serving of MoE models.

pytorch huggingface inference-engine mixture-of-experts large-language-models

Created 2024-01-22

15 commits to main branch, last one 24 days ago

MC-SMoE UNITES-Lab

8

60

mit

4

[ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"

merging efficiency mixture-of-experts

Created 2023-09-30

24 commits to main branch, last one 20 days ago

soft-mixture-of-experts fkodom

4

60

mit

3

PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)

pytorch transformer deep-learning mixture-of-experts vision-transformer

Created 2023-08-07

21 commits to main branch, last one 8 months ago

egobox relf

1

48

apache-2.0

3

Efficient global optimization toolbox in Rust: bayesian optimization, mixture of gaussian processes, sampling methods

surrogate-models gaussian-processes mixture-of-experts global-optimization latin-hypercube-sampling

Created 2020-08-27

493 commits to master branch, last one 2 days ago

leeroo_orchestrator Leeroo-AI

4

48

mit

5

The implementation of "Leeroo Orchestrator: Elevating LLMs Performance Through Model Integration"

ai llm nlp cost gpt4 efficiency opensource performance mixture-of-experts large-language-models orchestration-of-experts

Created 2024-01-23

19 commits to main branch, last one about a month ago