Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains

Autor:	Clymer, Joshua, Baker, Garrett, Subramani, Rohan, Wang, Sam
Rok vydání:	2023
Předmět:	Computer Science - Artificial Intelligence Computer Science - Computation and Language Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	As AI systems become more intelligent and their behavior becomes more challenging to assess, they may learn to game the flaws of human feedback instead of genuinely striving to follow instructions; however, this risk can be mitigated by controlling how LLMs generalize human feedback to situations where it is unreliable. To better understand how reward models generalize, we craft 69 distribution shifts spanning 8 categories. We find that reward models do not learn to evaluate `instruction-following' by default and instead favor personas that resemble internet text. Techniques for interpreting reward models' internal representations achieve better generalization than standard fine-tuning, but still frequently fail to distinguish instruction-following from conflated behaviors. We consolidate the 15 most challenging distribution shifts into the GENeralization analogIES (GENIES) benchmark, which we hope will enable progress toward controlling reward model generalization. Comment: Code: https://github.com/Joshuaclymer/GENIES Website: https://joshuaclymer.github.io/generalization-analogies-website/
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2311.07723 Zobrazit plný text záznamu View this record from Arxiv