Flash normalization: fast RMSNorm for LLMs

Autor:	Graef, Nils, Clapp, Matthew, Wasielewski, Andrew
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	RMSNorm is used by many LLMs such as Llama, Mistral, and OpenELM. This paper details FlashNorm, which is an exact but faster implementation of RMSNorm followed by linear layers. See https://huggingface.co/open-machine/FlashNorm for code and more transformer tricks. Comment: 7 pages, 8 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2407.09577 Zobrazit plný text záznamu View this record from Arxiv