InversionView: A General-Purpose Method for Reading Information from Neural Activations

Autor:	Huang, Xinting, Panwar, Madhur, Goyal, Navin, Hahn, Michael
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning Computer Science - Artificial Intelligence Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information is embodied by the subset of inputs that give rise to similar activations. We propose InversionView, which allows us to practically inspect this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding of the algorithms implemented by transformer models. We present four case studies where we investigate models ranging from small transformers to GPT-2. In these studies, we show that InversionView can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information. Comment: NeurIPS 2024; ICML 2024 Mechanistic Interpretability Workshop oral
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2405.17653 Zobrazit plný text záznamu View this record from Arxiv