Computational Complexity of Sub-Linear Convergent Algorithms

Autor:	AlQuabeh, Hilal, AlBreiki, Farha, Azizov, Dilshod
Rok vydání:	2022
Předmět:	Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Optimizing machine learning algorithms that are used to solve the objective function has been of great interest. Several approaches to optimize common algorithms, such as gradient descent and stochastic gradient descent, were explored. One of these approaches is reducing the gradient variance through adaptive sampling to solve large-scale optimization's empirical risk minimization (ERM) problems. In this paper, we will explore how starting with a small sample and then geometrically increasing it and using the solution of the previous sample ERM to compute the new ERM. This will solve ERM problems with first-order optimization algorithms of sublinear convergence but with lower computational complexity. This paper starts with theoretical proof of the approach, followed by two experiments comparing the gradient descent with the adaptive sampling of the gradient descent and ADAM with adaptive sampling ADAM on different datasets. Comment: 8 Pages
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2209.14558 Zobrazit plný text záznamu View this record from Arxiv