Robust and Resource-Efficient Identification of Two Hidden Layer Neural Networks
Autor: | Timo Klock, Massimo Fornasier, Michael Rauchensteiner |
---|---|
Rok vydání: | 2021 |
Předmět: |
General Mathematics
Matrix norm 010103 numerical & computational mathematics 02 engineering and technology Type (model theory) 01 natural sciences Minimax approximation algorithm ddc Article Deep neural networks Active sampling Exact identifiability Deparametrization Frames Nonconvex optimization on matrix spaces 65D15 68T07 90C26 Combinatorics Computational Mathematics Matrix (mathematics) Diagonal matrix Homogeneous space 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing 0101 mathematics Algebraic number Unit (ring theory) Analysis Mathematics |
Zdroj: | Constructive Approximation. 55:475-536 |
ISSN: | 1432-0940 0176-4276 |
DOI: | 10.1007/s00365-021-09550-5 |
Popis: | We address the structure identification and the uniform approximation of two fully nonlinear layer neural networks of the type$$f(x)=1^T h(B^T g(A^T x))$$f(x)=1Th(BTg(ATx))on$$\mathbb R^d$$Rd, where$$g=(g_1,\dots , g_{m_0})$$g=(g1,⋯,gm0),$$h=(h_1,\dots , h_{m_1})$$h=(h1,⋯,hm1),$$A=(a_1|\dots |a_{m_0}) \in \mathbb R^{d \times m_0}$$A=(a1|⋯|am0)∈Rd×m0and$$B=(b_1|\dots |b_{m_1}) \in \mathbb R^{m_0 \times m_1}$$B=(b1|⋯|bm1)∈Rm0×m1, from a small number of query samples. The solution of the case of two hidden layers presented in this paper is crucial as it can be further generalized to deeper neural networks. We approach the problem by sampling actively finite difference approximations to Hessians of the network. Gathering several approximate Hessians allows reliably to approximate the matrix subspace$$\mathcal W$$Wspanned by symmetric tensors$$a_1 \otimes a_1,\dots ,a_{m_0}\otimes a_{m_0}$$a1⊗a1,⋯,am0⊗am0formed by weights of the first layer together with the entangled symmetric tensors$$v_1 \otimes v_1 ,\dots ,v_{m_1}\otimes v_{m_1}$$v1⊗v1,⋯,vm1⊗vm1, formed by suitable combinations of the weights of the first and second layer as$$v_\ell =A G_0 b_\ell /\Vert A G_0 b_\ell \Vert _2$$vℓ=AG0bℓ/‖AG0bℓ‖2,$$\ell \in [m_1]$$ℓ∈[m1], for a diagonal matrix$$G_0$$G0depending on the activation functions of the first layer. The identification of the 1-rank symmetric tensors within$$\mathcal W$$Wis then performed by the solution of a robust nonlinear program, maximizing the spectral norm of the competitors constrained over the unit Frobenius sphere. We provide guarantees of stable recovery under a posteriori verifiable conditions. Once the 1-rank symmetric tensors$$\{a_i \otimes a_i, i\in [m_0]\}\cup \{v_\ell \otimes v_\ell , \ell \in [m_1] \}$${ai⊗ai,i∈[m0]}∪{vℓ⊗vℓ,ℓ∈[m1]}are computed, we address their correct attribution to the first or second layer ($$a_i$$ai’s are attributed to the first layer). The attribution to the layers is currently based on a semi-heuristic reasoning, but it shows clear potential of reliable execution. Having the correct attribution of the$$a_i,v_\ell $$ai,vℓto the respective layers and the consequent de-parametrization of the network, by using a suitably adapted gradient descent iteration, it is possible to estimate, up to intrinsic symmetries, the shifts of the activations functions of the first layer and compute exactly the matrix$$G_0$$G0. Eventually, from the vectors$$v_\ell =A G_0 b_\ell /\Vert A G_0 b_\ell \Vert _2$$vℓ=AG0bℓ/‖AG0bℓ‖2’s and$$a_i$$ai’s one can disentangle the weights$$b_\ell $$bℓ’s, by simple algebraic manipulations. Our method of identification of the weights of the network is fully constructive, with quantifiable sample complexity and therefore contributes to dwindle the black box nature of the network training phase. We corroborate our theoretical results by extensive numerical experiments, which confirm the effectiveness and feasibility of the proposed algorithmic pipeline. |
Databáze: | OpenAIRE |
Externí odkaz: |