SSSD: Simply-Scalable Speculative Decoding

Autor:	Marzollo, Michele, Zhuang, Jiawei, Roemer, Niklas, Müller, Lorenz K., Cavigelli, Lukas
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language Computer Science - Artificial Intelligence Computer Science - Machine Learning I.2.7
Druh dokumentu:	Working Paper
Popis:	Over the past year, Speculative Decoding has gained popularity as a technique for accelerating Large Language Model inference. While several methods have been introduced, most struggle to deliver satisfactory performance at batch sizes typical for data centers ($\geq 8$) and often involve significant deployment complexities. In this work, we offer a theoretical explanation of how Speculative Decoding can be effectively utilized with larger batch sizes. We also introduce a method that integrates seamlessly into existing systems without additional training or the complexity of deploying a small LLM. In a continuous batching setting, we achieve a 4x increase in throughput without any latency impact for short context generation, and a 1.7-2x improvement in both latency and throughput for longer contexts. Comment: 14 pages, 7 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2411.05894 Zobrazit plný text záznamu View this record from Arxiv