Zobrazeno 1 - 10
of 106
pro vyhledávání: '"de Jong, Michiel"'
Autor:
Zemlyanskiy, Yury, de Jong, Michiel, Vilnis, Luke, Ontañón, Santiago, Cohen, William W., Sanghai, Sumit, Ainslie, Joshua
Retrieval augmentation is a powerful but expensive method to make language models more knowledgeable about the world. Memory-based methods like LUMEN pre-compute token representations for retrieved passages to drastically speed up inference. However,
Externí odkaz:
http://arxiv.org/abs/2308.14903
Autor:
de Jong, Michiel, Zemlyanskiy, Yury, FitzGerald, Nicholas, Sanghai, Sumit, Cohen, William W., Ainslie, Joshua
Memory-augmentation is a powerful approach for efficiently incorporating external information into language models, but leads to reduced performance relative to retrieving text. Recent work introduced LUMEN, a memory-retrieval hybrid that partially p
Externí odkaz:
http://arxiv.org/abs/2306.10231
Autor:
Ainslie, Joshua, Lee-Thorp, James, de Jong, Michiel, Zemlyanskiy, Yury, Lebrón, Federico, Sanghai, Sumit
Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We
Externí odkaz:
http://arxiv.org/abs/2305.13245
Autor:
Ainslie, Joshua, Lei, Tao, de Jong, Michiel, Ontañón, Santiago, Brahma, Siddhartha, Zemlyanskiy, Yury, Uthus, David, Guo, Mandy, Lee-Thorp, James, Tay, Yi, Sung, Yun-Hsuan, Sanghai, Sumit
Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but also from applying feedforward and projection layers to every token. H
Externí odkaz:
http://arxiv.org/abs/2303.09752
Autor:
de Jong, Michiel, Zemlyanskiy, Yury, FitzGerald, Nicholas, Ainslie, Joshua, Sanghai, Sumit, Sha, Fei, Cohen, William
Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks. However, they are also expensive, due to the need to encode a large number of retrieved passages. Some
Externí odkaz:
http://arxiv.org/abs/2301.10448
Autor:
de Jong, Michiel, Zemlyanskiy, Yury, Ainslie, Joshua, FitzGerald, Nicholas, Sanghai, Sumit, Sha, Fei, Cohen, William
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks. However, the architecture used for FiD was chosen by making minimal modifications to a standard T5 model, w
Externí odkaz:
http://arxiv.org/abs/2212.08153
Autor:
Zemlyanskiy, Yury, de Jong, Michiel, Ainslie, Joshua, Pasupat, Panupong, Shaw, Peter, Qiu, Linlu, Sanghai, Sumit, Sha, Fei
A common recent approach to semantic parsing augments sequence-to-sequence models by retrieving and appending a set of training samples, called exemplars. The effectiveness of this recipe is limited by the ability to retrieve informative exemplars th
Externí odkaz:
http://arxiv.org/abs/2209.14899
Autor:
Chen, Wenhu, Cohen, William W., De Jong, Michiel, Gupta, Nitish, Presta, Alessandro, Verga, Pat, Wieting, John
In this position paper, we propose a new approach to generating a type of knowledge base (KB) from text, based on question generation and entity linking. We argue that the proposed type of KB has many of the key advantages of a traditional symbolic K
Externí odkaz:
http://arxiv.org/abs/2207.00630
Retrieval augmented language models have recently become the standard for knowledge intensive tasks. Rather than relying purely on latent semantics within the parameters of large neural models, these methods enlist a semi-parametric memory to encode
Externí odkaz:
http://arxiv.org/abs/2204.04581
Natural language understanding tasks such as open-domain question answering often require retrieving and assimilating factual information from multiple sources. We propose to address this problem by integrating a semi-parametric representation of a l
Externí odkaz:
http://arxiv.org/abs/2110.06176