Globally Injective ReLU Networks
| Autor: | Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmanic, Maarten de Hoop |
|---|---|
| Přispěvatelé: | Department of Mathematics and Statistics, Inverse Problems, Doctoral Programme in Mathematics and Statistics, Matti Lassas / Principal Investigator |
| Jazyk: | angličtina |
| Rok vydání: | 2022 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning random neural networks Statistics - Machine Learning universal approximation deep learning Machine Learning (stat.ML) bilipschitz networks injective generative models 113 Computer and information sciences Machine Learning (cs.LG) |
| Zdroj: | Michael Puthawala University of Helsinki |
| Popis: | Injectivity plays an important role in generative models where it enables inference; in inverse problems and compressed sensing with generative priors it is a precursor to well posedness. We establish sharp characterizations of injectivity of fully-connected and convolutional ReLU layers and networks. First, through a layerwise analysis, we show that an expansivity factor of two is necessary and sufficient for injectivity by constructing appropriate weight matrices. We show that global injectivity with iid Gaussian matrices, a commonly used tractable model, requires larger expansivity between 3.4 and 10.5. We also characterize the stability of inverting an injective network via worst-case Lipschitz constants of the inverse. We then use arguments from differential topology to study injectivity of deep networks and prove that any Lipschitz map can be approximated by an injective ReLU network. Finally, using an argument based on random projections, we show that an end-to-end -- rather than layerwise -- doubling of the dimension suffices for injectivity. Our results establish a theoretical basis for the study of nonlinear inverse and inference problems using neural networks. 48 pages, 18 figures, submitted to JMLR |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|