Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers

Autor:	Grishina, Ekaterina, Gorbunov, Mikhail, Rakhuba, Maxim
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in CNNs. Existing methods for computing the norm either tend to overestimate it or their performance may deteriorate quickly with increasing the input and kernel sizes. In this paper, we demonstrate that the tensor version of the spectral norm of a four-dimensional convolution kernel, up to a constant factor, serves as an upper bound for the spectral norm of the Jacobian matrix associated with the convolution operation. This new upper bound is independent of the input image resolution, differentiable and can be efficiently calculated during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures. Comment: ECCV 2024
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2409.11859 Zobrazit plný text záznamu View this record from Arxiv