Inference acceleration for large language models using 'stairs' assisted greedy generation

Autor:	Grigaliūnas, Domas, Lukoševičius, Mantas
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language Computer Science - Machine Learning 68T07 68T50 68T05 I.2.6 I.2.7
Druh dokumentu:	Working Paper
Popis:	Large Language Models (LLMs) with billions of parameters are known for their impressive predicting capabilities but require lots of resources to run. With their massive rise in popularity, even a small reduction in required resources could have an impact on environment. On the other hand, smaller models require fewer resources but may sacrifice accuracy. In this work, we are proposing an implementation of ``stairs'' assisted greedy generation. It is a modified assisted generation methodology that makes use of a smaller model's fast generation, large model's batch prediction, and "stairs" validation in order to achieve a speed up in prediction generation. Results show between 9.58 and 17.24 percent inference time reduction compared to a stand-alone large LLM prediction in a text generation task without a loss in accuracy. Comment: Accepted at the 29th International Conference on Information Society and University Studies (IVUS 2024)
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2407.19947 Zobrazit plný text záznamu View this record from Arxiv