Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve

Autor:	Daras, Giannis, Raoof, Negin, Gkalitsiou, Zoi, Dimakis, Alexandros G.
Rok vydání:	2022
Předmět:	Computer Science - Machine Learning Computer Science - Artificial Intelligence Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise compared to equivalent monolingual ones. We provide a theoretical justification for this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. Our analysis connects robustness to spectral properties of the learned representation and proves that multitasking leads to higher robustness for diverse task vectors. We open-source our code and models: https://github.com/giannisdaras/multilingual_robustness Comment: Accepted at NeurIPS 2022. 22 pages, 11 Figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2210.11618 Zobrazit plný text záznamu View this record from Arxiv