Search Results - RepositoryStats

211

2.1k

apache-2.0

28

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

rag 4-bits gaudi3 habana chatbot chatpdf llm-cpu autoround retrieval neural-chat streamingllm llm-inference neural-chat-7b large-language-model speculative-decoding intel-optimized-llamacpp

This repository has been archived (exclude archived)

Created 2022-11-11

2,126 commits to main branch, last one about a month ago

aphrodite-engine PygmalionAI

126

1.1k

agpl-3.0

14

Large-scale LLM inference engine

tpu cuda lora rocm intel api-rest inferentia inference-engine machine-learning speculative-decoding

Created 2023-06-23

825 commits to main branch, last one 23 hours ago

EAGLE SafeAILab

83

826

apache-2.0

13

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)

llm-inference speculative-decoding large-language-models

Created 2023-12-07

276 commits to main branch, last one 4 days ago

Sequoia Infini-AI-Lab

34

316

unknown

5

scalable and robust tree-based speculative decoding algorithm

llm inference efficiency speculative-decoding

Created 2024-02-29

77 commits to main branch, last one 3 months ago

LayerSkip facebookresearch

16

230

other

6

Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024

llm early-exit layer-drop optimization transformers speculative-decoding

Created 2024-02-26

26 commits to main branch, last one 24 days ago

TriForce Infini-AI-Lab

12

230

unknown

1

[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

llm inference efficiency acceleration long-context llm-inference speculative-decoding

Created 2024-04-04

17 commits to main branch, last one 2 months ago

REST FasterDecoding

11

176

apache-2.0

5

REST: Retrieval-Based Speculative Decoding, NAACL 2024

retrieval llm-inference speculative-decoding

Created 2023-11-15

9 commits to main branch, last one a day ago

BigLittleDecoder kssteven418

10

86

apache-2.0

6

[NeurIPS'23] Speculative Decoding with Big Little Decoder

llm decoding fast-inference efficient-inference speculative-decoding speculative-execution

Created 2023-02-10

11,217 commits to main branch, last one 9 months ago

SpecDec hemingkx

0

33

unknown

2

Codes for our paper "Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation" (EMNLP 2023 Findings)

non-autoregressive speculative-decoding

Created 2022-03-31

41 commits to main branch, last one 11 months ago

Speculative-Decoding romsto

0

26

mit

2

Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.

llm llm-inference fast-inference llm-optimization speculative-decoding

Created 2024-04-22

24 commits to main branch, last one 24 days ago

SWIFT hemingkx

1

26

apache-2.0

3

SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

speculative-decoding

Created 2024-10-09

10 commits to main branch, last one about a month ago