Exploring Dual Encoder Architectures for Question Answering

Autor:	Dong, Zhe, Ni, Jianmo, Bikel, Daniel M., Alfonseca, Enrique, Wang, Yuan, Qu, Chen, Zitouni, Imed
Rok vydání:	2022
Předmět:	Computer Science - Computation and Language Computer Science - Information Retrieval Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Dual encoders have been used for question-answering (QA) and information retrieval (IR) tasks with good results. Previous research focuses on two major types of dual encoders, Siamese Dual Encoder (SDE), with parameters shared across two encoders, and Asymmetric Dual Encoder (ADE), with two distinctly parameterized encoders. In this work, we explore different ways in which the dual encoder can be structured, and evaluate how these differences can affect their efficacy in terms of QA retrieval tasks. By evaluating on MS MARCO, open domain NQ and the MultiReQA benchmarks, we show that SDE performs significantly better than ADE. We further propose three different improved versions of ADEs by sharing or freezing parts of the architectures between two encoder towers. We find that sharing parameters in projection layers would enable ADEs to perform competitively with or outperform SDEs. We further explore and explain why parameter sharing in projection layer significantly improves the efficacy of the dual encoders, by directly probing the embedding spaces of the two encoder towers with t-SNE algorithm. Comment: Published in EMNLP 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2204.07120 Zobrazit plný text záznamu View this record from Arxiv