Searching for activation functions using a self-adaptive evolutionary algorithm
| Autor: | Andrew Nader, Danielle Azar |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Artificial neural network
Computer science business.industry Deep learning Evolutionary algorithm Context (language use) Genetic programming Monotonic function 0102 computer and information sciences 02 engineering and technology Sigmoid function Machine learning computer.software_genre 01 natural sciences Evolutionary computation 010201 computation theory & mathematics 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Artificial intelligence business computer |
| Zdroj: | GECCO Companion |
| Popis: | The introduction of the ReLU function in neural network architectures yielded substantial improvements over sigmoidal activation functions and allowed for the training of deep networks. Ever since, the search for new activation functions in neural networks has been an active research topic. However, to the best of our knowledge, the design of new activation functions has mostly been done by hand. In this work, we propose the use of a self-adaptive evolutionary algorithm that searches for new activation functions using a genetic programming approach, and we compare the performance of the obtained activation functions to ReLU. We also analyze the shape of the obtained activations to see if they have any common traits such as monotonicity or piece-wise linearity, and we study the effects of the self-adaptation to see which operators perform well in the context of a search for new activation functions. We perform a thorough experimental study on datasets of different sizes and types, using different types of neural network architectures. We report favorable results obtained from the mean and standard deviation of the performance metrics over multiple runs. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|