Arbitrary-Length Generalization for Addition in a Tiny Transformer

Autor:	Patriota, Alexandre Galvao
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning Statistics - Applications Statistics - Machine Learning
Druh dokumentu:	Working Paper
Popis:	This paper introduces a novel training methodology that enables a Transformer model to generalize the addition of two-digit numbers to numbers with unseen lengths of digits. The proposed approach employs an autoregressive generation technique, processing from right to left, which mimics a common manual method for adding large numbers. To the best of my knowledge, this methodology has not been previously explored in the literature. All results are reproducible, and the corresponding R code is available at github.com/AGPatriota/ALGA-R/. Comment: Testing-Digits.R output with 50-digit numbers (8 pages, 1 figure)
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2406.00075 Zobrazit plný text záznamu View this record from Arxiv