Competing Mutual Information Constraints with Stochastic Competition-Based Activations for Learning Diversified Representations
| Autor: | Panousis, Konstantinos, Antoniadis, Anastasios, Chatzis, Sotirios |
|---|---|
| Rok vydání: | 2022 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Artificial intelligence Information theory Stochastic systems Classification (of information) Statistics - Machine Learning Engineering and Technology Machine Learning (stat.ML) General Medicine Network layers Electrical Engineering - Electronic Engineering - Information Engineering Machine Learning (cs.LG) |
| Zdroj: | Proceedings of the AAAI Conference on Artificial Intelligence. 36:7931-7940 |
| ISSN: | 2374-3468 2159-5399 |
| DOI: | 10.1609/aaai.v36i7.20763 |
| Popis: | This work aims to address the long-established problem of learning diversified representations. To this end, we combine information-theoretic arguments with stochastic competition-based activations, namely Stochastic Local Winner-Takes-All (LWTA) units. In this context, we ditch the conventional deep architectures commonly used in Representation Learning, that rely on non-linear activations; instead, we replace them with sets of locally and stochastically competing linear units. In this setting, each network layer yields sparse outputs, determined by the outcome of the competition between units that are organized into blocks of competitors. We adopt stochastic arguments for the competition mechanism, which perform posterior sampling to determine the winner of each block. We further endow the considered networks with the ability to infer the sub-part of the network that is essential for modeling the data at hand; we impose appropriate stick-breaking priors to this end. To further enrich the information of the emerging representations, we resort to information-theoretic principles, namely the Information Competing Process (ICP). Then, all the components are tied together under the stochastic Variational Bayes framework for inference. We perform a thorough experimental investigation for our approach using benchmark datasets on image classification. As we experimentally show, the resulting networks yield significant discriminative representation learning abilities. In addition, the introduced paradigm allows for a principled investigation mechanism of the emerging intermediate network representations. Comment: Accepted AAAI-22 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|