Benchmarking quantized LLaMa-based models on the Brazilian Secondary School Exam

Autor:	Santos, Matheus L. O., Campelo, Cláudio E. C.
Rok vydání:	2023
Předmět:	Computer Science - Artificial Intelligence Computer Science - Computation and Language I.2.7 I.2.0
Druh dokumentu:	Working Paper
DOI:	10.21528/CBIC2023-177
Popis:	Although Large Language Models (LLMs) represent a revolution in the way we interact with computers, allowing the construction of complex questions and the ability to reason over a sequence of statements, their use is restricted due to the need for dedicated hardware for execution. In this study, we evaluate the performance of LLMs based on the 7 and 13 billion LLaMA models, subjected to a quantization process and run on home hardware. The models considered were Alpaca, Koala, and Vicuna. To evaluate the effectiveness of these models, we developed a database containing 1,006 questions from the ENEM (Brazilian National Secondary School Exam). Our analysis revealed that the best performing models achieved an accuracy of approximately 46% for the original texts of the Portuguese questions and 49% on their English translations. In addition, we evaluated the computational efficiency of the models by measuring the time required for execution. On average, the 7 and 13 billion LLMs took approximately 20 and 50 seconds, respectively, to process the queries on a machine equipped with an AMD Ryzen 5 3600x processor Comment: 8 pages, 6 figures, 4 tables
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2309.12071 Zobrazit plný text záznamu View this record from Arxiv