A Fast Saddle-Point Dynamical System Approach to Robust Deep Learning
| Autor: | Umesh Vaidya, Soumik Sarkar, Keivan Ebrahimi, Yasaman Esfandiari, Aditya Balu, Nicola Elia |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning 0209 industrial biotechnology Mathematical optimization Optimization problem Databases Factual Computer science Cognitive Neuroscience Machine Learning (stat.ML) 02 engineering and technology Dynamical system Pattern Recognition Automated Machine Learning (cs.LG) Deep Learning 020901 industrial engineering & automation Statistics - Machine Learning Artificial Intelligence Robustness (computer science) Saddle point 0202 electrical engineering electronic engineering information engineering business.industry Deep learning Robust optimization Function (mathematics) Maximization Norm (mathematics) 020201 artificial intelligence & image processing Neural Networks Computer Artificial intelligence business Algorithms |
| DOI: | 10.48550/arxiv.1910.08623 |
| Popis: | Recent focus on robustness to adversarial attacks for deep neural networks produced a large variety of algorithms for training robust models. Most of the effective algorithms involve solving the min–max optimization problem for training robust models (min step) under worst-case attacks (max step). However, they often suffer from high computational cost from running several inner maximization iterations (to find an optimal attack) inside every outer minimization iteration. Therefore, it becomes difficult to readily apply such algorithms for moderate to large size real world data sets. To alleviate this, we explore the effectiveness of iterative descent–ascent algorithms where the maximization and minimization steps are executed in an alternate fashion to simultaneously obtain the worst-case attack and the corresponding robust model. Specifically, we propose a novel discrete-time dynamical system-based algorithm that aims to find the saddle point of a min–max optimization problem in the presence of uncertainties. Under the assumptions that the cost function is convex and uncertainties enter concavely in the robust learning problem, we analytically show that our algorithm converges asymptotically to the robust optimal solution under a general adversarial budget constraints as induced by l p norm, for 1 ≤ p ≤ ∞ . Based on our proposed analysis, we devise a fast robust training algorithm for deep neural networks. Although such training involves highly non-convex robust optimization problems, empirical results show that the algorithm can achieve significant robustness compared to other state-of-the-art robust models on benchmark data sets. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|