Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models
| Autor: | Dinh Phung, Gholamreza Haffari, Thuy-Trang Vu |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
Domain adaptation Computer Science - Computation and Language Computer science business.industry 02 engineering and technology 010501 environmental sciences computer.software_genre Machine learning 01 natural sciences Masking (Electronic Health Record) Named-entity recognition 0202 electrical engineering electronic engineering information engineering Embedding 020201 artificial intelligence & image processing Language model Artificial intelligence F1 score business computer Computation and Language (cs.CL) 0105 earth and related environmental sciences |
| Zdroj: | EMNLP (1) |
| Popis: | Recent work has shown the importance of adaptation of broad-coverage contextualised embedding models on the domain of the target task of interest. Current self-supervised adaptation methods are simplistic, as the training signal comes from a small percentage of \emph{randomly} masked-out tokens. In this paper, we show that careful masking strategies can bridge the knowledge gap of masked language models (MLMs) about the domains more effectively by allocating self-supervision where it is needed. Furthermore, we propose an effective training strategy by adversarially masking out those tokens which are harder to reconstruct by the underlying MLM. The adversarial objective leads to a challenging combinatorial optimisation problem over \emph{subsets} of tokens, which we tackle efficiently through relaxation to a variational lowerbound and dynamic programming. On six unsupervised domain adaptation tasks involving named entity recognition, our method strongly outperforms the random masking strategy and achieves up to +1.64 F1 score improvements. EMNLP2020 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|