Search Results - RepositoryStats

211

2.1k

apache-2.0

28

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

rag 4-bits gaudi3 habana chatbot chatpdf llm-cpu autoround retrieval neural-chat streamingllm llm-inference neural-chat-7b large-language-model speculative-decoding intel-optimized-llamacpp

This repository has been archived (exclude archived)

Created 2022-11-11

2,126 commits to main branch, last one 2 months ago

aphrodite-engine PygmalionAI

130

1.2k

agpl-3.0

14

Large-scale LLM inference engine

tpu cuda lora rocm intel api-rest inferentia inference-engine machine-learning speculative-decoding

Created 2023-06-23

911 commits to main branch, last one 5 hours ago

EAGLE SafeAILab

91

863

apache-2.0

13

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)

llm-inference speculative-decoding large-language-models

Created 2023-12-07

278 commits to main branch, last one 26 days ago

Sequoia Infini-AI-Lab

37

322

unknown

5

scalable and robust tree-based speculative decoding algorithm

llm inference efficiency speculative-decoding

Created 2024-02-29

77 commits to main branch, last one 4 months ago

LayerSkip facebookresearch

18

244

other

6

Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024

llm early-exit layer-drop optimization transformers speculative-decoding

Created 2024-02-26

32 commits to main branch, last one 10 days ago

TriForce Infini-AI-Lab

13

236

unknown

1

[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

llm inference efficiency acceleration long-context llm-inference speculative-decoding

Created 2024-04-04

17 commits to main branch, last one 3 months ago

REST FasterDecoding

11

182

apache-2.0

6

REST: Retrieval-Based Speculative Decoding, NAACL 2024

retrieval llm-inference speculative-decoding

Created 2023-11-15

11 commits to main branch, last one 23 days ago

BigLittleDecoder kssteven418

10

87

apache-2.0

6

[NeurIPS'23] Speculative Decoding with Big Little Decoder

llm decoding fast-inference efficient-inference speculative-decoding speculative-execution

Created 2023-02-10

11,217 commits to main branch, last one 10 months ago

SpecDec hemingkx

1

34

unknown

2

Codes for our paper "Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation" (EMNLP 2023 Findings)

non-autoregressive speculative-decoding

Created 2022-03-31

41 commits to main branch, last one about a year ago

Speculative-Decoding romsto

1

31

mit

2

Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.

llm llm-inference fast-inference llm-optimization speculative-decoding

Created 2024-04-22

26 commits to main branch, last one 19 days ago

SWIFT hemingkx

1

30

apache-2.0

3

SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

speculative-decoding

Created 2024-10-09

11 commits to main branch, last one 16 days ago