Attention Word Embedding
Autor: | Richard G. Baraniuk, Shashank Sonkar, Andrew E. Waters |
---|---|
Rok vydání: | 2020 |
Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Word embedding Computer science Initialization Context (language use) 02 engineering and technology 010501 environmental sciences computer.software_genre 01 natural sciences Machine Learning (cs.LG) Similarity (psychology) 0202 electrical engineering electronic engineering information engineering Word2vec 0105 earth and related environmental sciences Computer Science - Computation and Language business.industry Embedding 020201 artificial intelligence & image processing Artificial intelligence business Computation and Language (cs.CL) computer Natural language processing Word (computer architecture) Sentence |
Zdroj: | COLING |
Popis: | Word embedding models learn semantically rich vector representations of words and are widely used to initialize natural processing language (NLP) models. The popular continuous bag-of-words (CBOW) model of word2vec learns a vector embedding by masking a given word in a sentence and then using the other words as a context to predict it. A limitation of CBOW is that it equally weights the context words when making a prediction, which is inefficient, since some words have higher predictive value than others. We tackle this inefficiency by introducing the Attention Word Embedding (AWE) model, which integrates the attention mechanism into the CBOW model. We also propose AWE-S, which incorporates subword information. We demonstrate that AWE and AWE-S outperform the state-of-the-art word embedding models both on a variety of word similarity datasets and when used for initialization of NLP models. |
Databáze: | OpenAIRE |
Externí odkaz: |