Direct Error-Driven Learning for Deep Neural Networks With Applications to Big Data
| Autor: | R. Krishnan, V. A. Samaranayake, Sarangapani Jagannathan |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Vanishing gradient problem
Artificial neural network Noise measurement Computer Networks and Communications Generalization business.industry Computer science Big data 02 engineering and technology Generalization error Measure (mathematics) Computer Science Applications Noise Artificial Intelligence 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing business Algorithm Software |
| Zdroj: | INNS Conference on Big Data |
| ISSN: | 2162-2388 2162-237X |
| DOI: | 10.1109/tnnls.2019.2920964 |
| Popis: | In this brief, heterogeneity and noise in big data are shown to increase the generalization error for a traditional learning regime utilized for deep neural networks (deep NNs). To reduce this error, while overcoming the issue of vanishing gradients, a direct error-driven learning (EDL) scheme is proposed. First, to reduce the impact of heterogeneity and data noise, the concept of a neighborhood is introduced. Using this neighborhood, an approximation of generalization error is obtained and an overall error, comprised of learning and the approximate generalization errors, is defined. A novel NN weight-tuning law is obtained through a layer-wise performance measure enabling the direct use of overall error for learning. Additional constraints are introduced into the layer-wise performance measure to guide and improve the learning process in the presence of noisy dimensions. The proposed direct EDL scheme effectively addresses the issue of heterogeneity and noise while mitigating vanishing gradients and noisy dimensions. A comprehensive simulation study is presented where the proposed approach is shown to mitigate the vanishing gradient problem while improving generalization by 6%. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|