Unsupervised pretraining transfers well across languages

Autor: Pierre-Emmanuel Mazaré, Emmanuel Dupoux, Morgane Riviere, Armand Joulin
Přispěvatelé: Facebook AI Research [Paris] (FAIR), Facebook, Apprentissage machine et développement cognitif (CoML), Inria de Paris, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Laboratoire de sciences cognitives et psycholinguistique (LSCP), Département d'Etudes Cognitives - ENS Paris (DEC), École normale supérieure - Paris (ENS Paris), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École normale supérieure - Paris (ENS Paris), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École des hautes études en sciences sociales (EHESS)-Centre National de la Recherche Scientifique (CNRS)-Département d'Etudes Cognitives - ENS Paris (DEC), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École des hautes études en sciences sociales (EHESS)-Centre National de la Recherche Scientifique (CNRS), Laboratoire de sciences cognitives et psycholinguistique (LSCP), CIFAR program in Learning in Machines & Brains CIFAR LMB program, ANR-19-P3IA-0001,PRAIRIE,PaRis Artificial Intelligence Research InstitutE(2019), ANR-17-EURE-0017,FrontCog,Frontières en cognition(2017), ANR-10-IDEX-0001,PSL,Paris Sciences et Lettres(2010), École normale supérieure - Paris (ENS-PSL), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École normale supérieure - Paris (ENS-PSL), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École des hautes études en sciences sociales (EHESS)-Centre National de la Recherche Scientifique (CNRS)-Inria de Paris, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)
Jazyk: angličtina
Rok vydání: 2020
Předmět:
FOS: Computer and information sciences
Sound (cs.SD)
Computer Science - Machine Learning
Computer science
InformationSystems_INFORMATIONSTORAGEANDRETRIEVAL
02 engineering and technology
010501 environmental sciences
computer.software_genre
01 natural sciences
[INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL]
Computer Science - Sound
Machine Learning (cs.LG)
Transcription (linguistics)
[INFO.INFO-LG]Computer Science [cs]/Machine Learning [cs.LG]
Audio and Speech Processing (eess.AS)
FOS: Electrical engineering
electronic engineering
information engineering
0202 electrical engineering
electronic engineering
information engineering
0105 earth and related environmental sciences
Computer Science - Computation and Language
business.industry
Low resources
020206 networking & telecommunications
ComputingMethodologies_PATTERNRECOGNITION
Unsupervised pretraining
[INFO.INFO-SD]Computer Science [cs]/Sound [cs.SD]
Artificial intelligence
business
Computation and Language (cs.CL)
computer
Natural language processing
Cross-lingual
Electrical Engineering and Systems Science - Audio and Speech Processing
Zdroj: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing
ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing, May 2020, Barcelona / Virtual, Spain. pp.7414-7418, ⟨10.1109/ICASSP40776.2020.9054548⟩
ICASSP
Popis: Cross-lingual and multi-lingual training of Automatic Speech Recognition (ASR) has been extensively investigated in the supervised setting. This assumes the existence of a parallel corpus of speech and orthographic transcriptions. Recently, contrastive predictive coding (CPC) algorithms have been proposed to pretrain ASR systems with unlabelled data. In this work, we investigate whether unsupervised pretraining transfers well across languages. We show that a slight modification of the CPC pretraining extracts features that transfer well to other languages, being on par or even outperforming supervised pretraining. This shows the potential of unsupervised methods for languages with few linguistic resources.
6 pages. Accepted at ICASSP 2020. However the 2 pages of supplementary materials will appear only in the arxiv version
Databáze: OpenAIRE
Externí odkaz: