YFACC: A Yor\`ub\'a speech-image dataset for cross-lingual keyword localisation through visual grounding

Autor:	Olaleye, Kayode, Oneata, Dan, Kamper, Herman
Rok vydání:	2022
Předmět:	Computer Science - Computation and Language Electrical Engineering and Systems Science - Audio and Speech Processing
Druh dokumentu:	Working Paper
Popis:	Visually grounded speech (VGS) models are trained on images paired with unlabelled spoken captions. Such models could be used to build speech systems in settings where it is impossible to get labelled data, e.g. for documenting unwritten languages. However, most VGS studies are in English or other high-resource languages. This paper attempts to address this shortcoming. We collect and release a new single-speaker dataset of audio captions for 6k Flickr images in Yor\`ub\'a -- a real low-resource language spoken in Nigeria. We train an attention-based VGS model where images are automatically tagged with English visual labels and paired with Yor\`ub\'a utterances. This enables cross-lingual keyword localisation: a written English query is detected and located in Yor\`ub\'a speech. To quantify the effect of the smaller dataset, we compare to English systems trained on similar and more data. We hope that this new dataset will stimulate research in the use of VGS models for real low-resource languages. Comment: Accepted to IEEE SLT 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2210.04600 Zobrazit plný text záznamu View this record from Arxiv