Neural Network Learning Using Entropy Cycle
| Autor: | Sevki S. Erdogan, Khue Hiang Chan, Geok See Ng, Harcharan Singh |
|---|---|
| Rok vydání: | 2000 |
| Předmět: |
Hidden node problem
business.industry Neural network learning Sigmoid function Information theory Network topology Human-Computer Interaction Cross entropy Artificial Intelligence Hardware and Architecture Entropy (information theory) Artificial intelligence Extreme value theory business Algorithm Software Information Systems Mathematics |
| Zdroj: | Knowledge and Information Systems. 2:53-72 |
| ISSN: | 0219-3116 0219-1377 |
| DOI: | 10.1007/s101150050003 |
| Popis: | In this paper, an additional entropy penalty term is used to steer the direction of the hidden node's activation in the process of learning. A state with minimum entropy means that most nodes are operating in the non-linear zones (i.e. saturation zones) near the extreme ends of the Sigmoid curve. As the training proceeds, redundant hidden nodes' activations are pushed towards their extreme value corresponding to a low entropy state with maximum information, while some relevant nodes remain active in the linear zone. As training progresses, more nodes get into saturation zones. The early creation of such nodes may impair generalization performance. To prevent the network from being driven into saturation before it can really learn, an entropy cycle is proposed in this paper to dampen the creation of such inactive nodes in the early stage of training. At the end of training, these inactive nodes can then be eliminated without affecting the performance of the original network. The concept has been successfully applied for pruning in two classification problems. The experiments indicate that redundant nodes are pruned resulting in optimal network topologies. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full text from SpringerLink