Zobrazeno 1 - 10
of 32
pro vyhledávání: '"Gusak, Julia"'
Autor:
Cherniuk, Daria, Abukhovich, Stanislav, Phan, Anh-Huy, Oseledets, Ivan, Cichocki, Andrzej, Gusak, Julia
Tensor decomposition of convolutional and fully-connected layers is an effective way to reduce parameters and FLOP in neural networks. Due to memory and power consumption limitations of mobile or embedded devices, the quantization step is usually nec
Externí odkaz:
http://arxiv.org/abs/2308.04595
We propose Rockmate to control the memory requirements when training PyTorch DNN models. Rockmate is an automatic tool that starts from the model code and generates an equivalent model, using a predefined amount of memory for activations, at the cost
Externí odkaz:
http://arxiv.org/abs/2307.01236
Large-scale transformer models have shown remarkable performance in language modelling tasks. However, such models feature billions of parameters, leading to difficulties in their deployment and prohibitive training costs from scratch. To reduce the
Externí odkaz:
http://arxiv.org/abs/2306.02697
Autor:
Gusak, Julia, Cherniuk, Daria, Shilova, Alena, Katrutsa, Alexander, Bershatsky, Daniel, Zhao, Xunyi, Eyraud-Dubois, Lionel, Shlyazhko, Oleg, Dimitrov, Denis, Oseledets, Ivan, Beaumont, Olivier
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU device or can be trained using only a small per-GPU batch size. This sur
Externí odkaz:
http://arxiv.org/abs/2202.10435
Autor:
Novikov, Georgii, Bershatsky, Daniel, Gusak, Julia, Shonenkov, Alex, Dimitrov, Denis, Oseledets, Ivan
Memory footprint is one of the main limiting factors for large neural network training. In backpropagation, one needs to store the input to each operation in the computational graph. Every modern neural network model has quite a few pointwise nonline
Externí odkaz:
http://arxiv.org/abs/2202.00441
Autor:
Bershatsky, Daniel, Mikhalev, Aleksandr, Katrutsa, Alexandr, Gusak, Julia, Merkulov, Daniil, Oseledets, Ivan
In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to perform backpropagation through linear layers. Since the gradients of
Externí odkaz:
http://arxiv.org/abs/2201.13195
A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss function. However, such an approach is tailored for a specific discretization
Externí odkaz:
http://arxiv.org/abs/2103.08561
Autor:
Phan, Anh-Huy, Sobolev, Konstantin, Sozykin, Konstantin, Ermilov, Dmitry, Gusak, Julia, Tichavsky, Petr, Glukhov, Valeriy, Oseledets, Ivan, Cichocki, Andrzej
Most state of the art deep neural networks are overparameterized and exhibit a high computational cost. A straightforward approach to this problem is to replace convolutional kernels with its low-rank tensor approximations, whereas the Canonical Poly
Externí odkaz:
http://arxiv.org/abs/2008.05441
Autor:
Gusak, Julia, Markeeva, Larisa, Daulbaev, Talgat, Katrutsa, Alexandr, Cichocki, Andrzej, Oseledets, Ivan
Normalization is an important and vastly investigated technique in deep learning. However, its role for Ordinary Differential Equation based networks (neural ODEs) is still poorly understood. This paper investigates how different normalization techni
Externí odkaz:
http://arxiv.org/abs/2004.09222
Autor:
Daulbaev, Talgat, Katrutsa, Alexandr, Markeeva, Larisa, Gusak, Julia, Cichocki, Andrzej, Oseledets, Ivan
We propose a simple interpolation-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method (known in the literature as "adjoint method") to train neural ODEs on classification, dens
Externí odkaz:
http://arxiv.org/abs/2003.05271