Multimodal neural networks better explain multivoxel patterns in the hippocampus
| Autor: | Bhavin Choksi, Milad Mozafari, Rufin VanRullen, Leila Reddy |
|---|---|
| Přispěvatelé: | Reddy, Leila, Centre de recherche cerveau et cognition (CERCO UMR5549), Université Toulouse III - Paul Sabatier (UT3), Université Fédérale Toulouse Midi-Pyrénées-Université Fédérale Toulouse Midi-Pyrénées-Centre National de la Recherche Scientifique (CNRS)-Toulouse Mind & Brain Institut (TMBI), Université Toulouse - Jean Jaurès (UT2J)-Université Toulouse III - Paul Sabatier (UT3), Université Fédérale Toulouse Midi-Pyrénées-Université Fédérale Toulouse Midi-Pyrénées-Université Toulouse - Jean Jaurès (UT2J)-Université Toulouse III - Paul Sabatier (UT3), Université Fédérale Toulouse Midi-Pyrénées, MEthodes et ingénierie des Langues, des Ontologies et du DIscours (IRIT-MELODI), Institut de recherche en informatique de Toulouse (IRIT), Université Toulouse Capitole (UT Capitole), Université Fédérale Toulouse Midi-Pyrénées-Centre National de la Recherche Scientifique (CNRS)-Institut National Polytechnique (Toulouse) (Toulouse INP), Université Fédérale Toulouse Midi-Pyrénées-Toulouse Mind & Brain Institut (TMBI), Université Fédérale Toulouse Midi-Pyrénées-Université Fédérale Toulouse Midi-Pyrénées-Université Toulouse III - Paul Sabatier (UT3), Université Fédérale Toulouse Midi-Pyrénées-Université Toulouse Capitole (UT Capitole), Artificial and Natural Intelligence Toulouse Institute (ANITI), Centre de recherche cerveau et cognition (CERCO), Institut des sciences du cerveau de Toulouse. (ISCT), Université Fédérale Toulouse Midi-Pyrénées-Université Fédérale Toulouse Midi-Pyrénées-CHU Toulouse [Toulouse]-Institut National de la Santé et de la Recherche Médicale (INSERM)-Centre National de la Recherche Scientifique (CNRS)-Université Toulouse - Jean Jaurès (UT2J)-Université Toulouse III - Paul Sabatier (UT3), Université Fédérale Toulouse Midi-Pyrénées-Université Fédérale Toulouse Midi-Pyrénées-CHU Toulouse [Toulouse]-Institut National de la Santé et de la Recherche Médicale (INSERM)-Centre National de la Recherche Scientifique (CNRS)-Centre National de la Recherche Scientifique (CNRS), Université Toulouse 1 Capitole (UT1), Université Fédérale Toulouse Midi-Pyrénées-Université Toulouse 1 Capitole (UT1) |
| Rok vydání: | 2021 |
| Předmět: |
FOS: Computer and information sciences
concept cells Computer Science - Machine Learning genetic structures Cognitive Neuroscience education [INFO] Computer Science [cs] Hippocampus Machine Learning (cs.LG) neuroscience [SCCO]Cognitive science Artificial Intelligence Humans [INFO]Computer Science [cs] Neural and Evolutionary Computing (cs.NE) multimodal networks Neurons fMRI Computer Science - Neural and Evolutionary Computing deep learning [SCCO] Cognitive science Magnetic Resonance Imaging nervous system Quantitative Biology - Neurons and Cognition FOS: Biological sciences Neurons and Cognition (q-bio.NC) Neural Networks Computer |
| Zdroj: | Neural Information Processing Systems (NeurIPS) conference: 3rd Workshop on Shared Visual Representations in Human and Machine Intelligence (SVRHM 2021) Neural Information Processing Systems (NeurIPS) conference: 3rd Workshop on Shared Visual Representations in Human and Machine Intelligence (SVRHM 2021), Dec 2021, Virtual Conference, United States |
| ISSN: | 1879-2782 |
| Popis: | The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et at., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus. Comment: Oral at SVRHM Workshop (NeurIPS 2021) |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect