Zobrazeno 1 - 10
of 539
pro vyhledávání: '"Riquelme Carlos"'
Autor:
Pinnaparaju, Nikhil, Adithyan, Reshinth, Phung, Duy, Tow, Jonathan, Baicoianu, James, Datta, Ashish, Zhuravinskyi, Maksym, Mahan, Dakota, Bellagente, Marco, Riquelme, Carlos, Cooper, Nathan
We introduce Stable Code, the first in our new-generation of code language models series, which serves as a general-purpose base code language model targeting code completion, reasoning, math, and other software engineering-based tasks. Additionally,
Externí odkaz:
http://arxiv.org/abs/2404.01226
Autor:
Bellagente, Marco, Tow, Jonathan, Mahan, Dakota, Phung, Duy, Zhuravinskyi, Maksym, Adithyan, Reshinth, Baicoianu, James, Brooks, Ben, Cooper, Nathan, Datta, Ashish, Lee, Meng, Mostaque, Emad, Pieler, Michael, Pinnaparju, Nikhil, Rocha, Paulo, Saini, Harry, Teufel, Hannah, Zanichelli, Niccolo, Riquelme, Carlos
We introduce StableLM 2 1.6B, the first in a new generation of our language model series. In this technical report, we present in detail the data and training procedure leading to the base and instruction-tuned versions of StableLM 2 1.6B. The weight
Externí odkaz:
http://arxiv.org/abs/2402.17834
Mixture-of-Experts (MoE) models are a promising way to scale up model capacity without significantly increasing computational cost. A key component of MoEs is the router, which decides which subset of parameters (experts) process which feature embedd
Externí odkaz:
http://arxiv.org/abs/2401.15969
We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i.e., "foundation models"), in both vision and language domains. In this setting, we identify the first scaling law describing the relati
Externí odkaz:
http://arxiv.org/abs/2309.08520
Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the
Externí odkaz:
http://arxiv.org/abs/2308.00951
Autor:
Dehghani, Mostafa, Djolonga, Josip, Mustafa, Basil, Padlewski, Piotr, Heek, Jonathan, Gilmer, Justin, Steiner, Andreas, Caron, Mathilde, Geirhos, Robert, Alabdulmohsin, Ibrahim, Jenatton, Rodolphe, Beyer, Lucas, Tschannen, Michael, Arnab, Anurag, Wang, Xiao, Riquelme, Carlos, Minderer, Matthias, Puigcerver, Joan, Evci, Utku, Kumar, Manoj, van Steenkiste, Sjoerd, Elsayed, Gamaleldin F., Mahendran, Aravindh, Yu, Fisher, Oliver, Avital, Huot, Fantine, Bastings, Jasmijn, Collier, Mark Patrick, Gritsenko, Alexey, Birodkar, Vighnesh, Vasconcelos, Cristina, Tay, Yi, Mensink, Thomas, Kolesnikov, Alexander, Pavetić, Filip, Tran, Dustin, Kipf, Thomas, Lučić, Mario, Zhai, Xiaohua, Keysers, Daniel, Harmsen, Jeremiah, Houlsby, Neil
The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image an
Externí odkaz:
http://arxiv.org/abs/2302.05442
Autor:
Puigcerver, Joan, Jenatton, Rodolphe, Riquelme, Carlos, Awasthi, Pranjal, Bhojanapalli, Srinadh
Adversarial robustness is a key desirable property of neural networks. It has been empirically shown to be affected by their sizes, with larger networks being typically more robust. Recently, Bubeck and Sellke proved a lower bound on the Lipschitz co
Externí odkaz:
http://arxiv.org/abs/2210.10253
Autor:
Chen, Xi, Wang, Xiao, Changpinyo, Soravit, Piergiovanni, AJ, Padlewski, Piotr, Salz, Daniel, Goodman, Sebastian, Grycner, Adam, Mustafa, Basil, Beyer, Lucas, Kolesnikov, Alexander, Puigcerver, Joan, Ding, Nan, Rong, Keran, Akbari, Hassan, Mishra, Gaurav, Xue, Linting, Thapliyal, Ashish, Bradbury, James, Kuo, Weicheng, Seyedhosseini, Mojtaba, Jia, Chao, Ayan, Burcu Karagol, Riquelme, Carlos, Steiner, Andreas, Angelova, Anelia, Zhai, Xiaohua, Houlsby, Neil, Soricut, Radu
Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates
Externí odkaz:
http://arxiv.org/abs/2209.06794
Large sparsely-activated models have obtained excellent performance in multiple domains. However, such models are typically trained on a single modality at a time. We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of
Externí odkaz:
http://arxiv.org/abs/2206.02770
Autor:
Renggli, Cedric, Pinto, André Susano, Houlsby, Neil, Mustafa, Basil, Puigcerver, Joan, Riquelme, Carlos
Transformers are widely applied to solve natural language understanding and computer vision tasks. While scaling up these architectures leads to improved performance, it often comes at the expense of much higher computational costs. In order for larg
Externí odkaz:
http://arxiv.org/abs/2202.12015