Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers

Autor: Han, Seungwook, Song, Jinyeop, Gore, Jeff, Agrawal, Pulkit

Humans distill complex experiences into fundamental abstractions that enable rapid learning and adaptation. Similarly, autoregressive transformers exhibit adaptive learning through in-context learning (ICL), which begs the question of how. In this pa

Externí odkaz: http://arxiv.org/abs/2412.12276

A Resource Model For Neural Scaling Law

Autor: Song, Jinyeop, Liu, Ziming, Tegmark, Max, Gore, Jeff

Neural scaling laws characterize how model performance improves as the model size scales up. Inspired by empirical observations, we introduce a resource model of neural scaling. A task is usually composite hence can be decomposed into many subtasks,

Externí odkaz: http://arxiv.org/abs/2402.05164

Tento výsledek nelze pro nepřihlášené uživatele zobrazit.
K zobrazení výsledku je třeba se přihlásit.

Tento výsledek nelze pro nepřihlášené uživatele zobrazit.
K zobrazení výsledku je třeba se přihlásit.

MethylGPT: a foundation model for the DNA methylome.

Autor: Ying K; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA.; T. H. Chan School of Public Health, Harvard University, Boston, MA, USA., Song J; Department of Physics, MIT, Cambridge, MA, USA., Cui H; Peter Munk Cardiac Centre, University Health Network, Toronto, ON, Canada.; Department of Computer Science, University of Toronto, Toronto, ON, Canada.; Vector Institute, Toronto, ON, Canada., Zhang Y; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA., Li S; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA., Chen X; Department of Computer Science, University of Toronto, Toronto, ON, Canada.; Vector Institute, Toronto, ON, Canada., Liu H; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA., Eames A; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA., McCartney DL; Centre for Genomic and Experimental Medicine, Institute of Genetics and Molecular Medicine, University of Edinburgh, Edinburgh, Scotland, United Kingdom., Marioni RE; Centre for Genomic and Experimental Medicine, Institute of Genetics and Molecular Medicine, University of Edinburgh, Edinburgh, Scotland, United Kingdom., Poganik JR; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA., Moqri M; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA., Wang B; Department of Computer Science, University of Toronto, Toronto, ON, Canada.; Vector Institute, Toronto, ON, Canada.; Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, ON, Canada., Gladyshev VN; Division of Genetics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA.

Publikováno v: BioRxiv : the preprint server for biology [bioRxiv] 2024 Nov 04. Date of Electronic Publication: 2024 Nov 04.