DOCSEARCHbyAlgolia
Submit Docs

Speculative Decoding Implementations

repository·main·Indexed 43 days ago

https://github.com/lucidrains/speculative-decoding

Implementations and research explorations of speculative decoding techniques to accelerate inference for large language models and semantic decoders, including early exit schemes, batched speculative decoding, and speculative sampling using a prophet transformer idea.

Tokens
105
Snippets
0
Records
1
Agent score
5%

What's inside Speculative Decoding

  1. Overview of Speculative Decoding explorations

    main
    This repository contains explorations into recent techniques surrounding speculative decoding, a method used to accelerate the inference of large models. The project includes implementations of various strategies such as early exit schemes, batched speculative decoding, and speculative sampling using a prophet transformer idea. The primary goal is to speed up the text-to-semantic decoder in the Spear-TTS project.