Zobrazeno 1 - 3
of 3
pro vyhledávání: '"Mikhalchuk, Matvey"'
Autor:
Razzhigaev, Anton, Mikhalchuk, Matvey, Goncharova, Elizaveta, Gerasimenko, Nikolai, Oseledets, Ivan, Dimitrov, Denis, Kuznetsov, Andrey
This paper reveals a novel linear characteristic exclusive to transformer decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We analyze embedding transformations between sequential layers, uncovering a near-perfect linear relations
Externí odkaz:
http://arxiv.org/abs/2405.12250
Autor:
Goncharova, Elizaveta, Razzhigaev, Anton, Mikhalchuk, Matvey, Kurkin, Maxim, Abdullaeva, Irina, Skripkin, Matvey, Oseledets, Ivan, Dimitrov, Denis, Kuznetsov, Andrey
Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and adapters for visual moda
Externí odkaz:
http://arxiv.org/abs/2404.06212
Autor:
Razzhigaev, Anton, Mikhalchuk, Matvey, Goncharova, Elizaveta, Oseledets, Ivan, Dimitrov, Denis, Kuznetsov, Andrey
In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encoders and decoders. Our findings reveal that the anisotropy profile in tr
Externí odkaz:
http://arxiv.org/abs/2311.05928