Accelerating AI Performance using Anderson Extrapolation on GPUs

Autor:	Dajani, Saleem Abdul Fattah Ahmed Al, Keyes, David E.
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning Computer Science - Artificial Intelligence Computer Science - Performance Mathematics - Numerical Analysis
Zdroj:	Neural Information Processing Systems (NeurIPS). Machine Learning with New Compute Paradigms (MLNCP) Workshop, October 2024
Druh dokumentu:	Working Paper
Popis:	We present a novel approach for accelerating AI performance by leveraging Anderson extrapolation, a vector-to-vector mapping technique based on a window of historical iterations. By identifying the crossover point where a mixing penalty is incurred, the method focuses on reducing iterations to convergence, with fewer more compute-intensive but generally cacheable iterations, balancing speed and memory usage with accuracy and algorithmic stability, respectively. We demonstrate significant improvements, in both training and inference, motivated by scalability and efficiency extensions to the realm of high-performance computing (HPC). Comment: 6 pages, 6 figures, 1 table, Accepted by NeurIPS 2024 Workshop MLNCP https://openreview.net/forum?id=wkP2ZFRn9e
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2410.19460 Zobrazit plný text záznamu View this record from Arxiv