Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges

Autor:	Spinner, Thilo, Kehlbeck, Rebecca, Sevastjanova, Rita, Stähle, Tobias, Keim, Daniel A., Deussen, Oliver, Spitz, Andreas, El-Assady, Mennatallah
Rok vydání:	2023
Předmět:	Computer Science - Computation and Language Computer Science - Artificial Intelligence Computer Science - Human-Computer Interaction H.5.2 I.2.7
Druh dokumentu:	Working Paper
Popis:	The growing popularity of generative language models has amplified interest in interactive methods to guide model outputs. Prompt refinement is considered one of the most effective means to influence output among these methods. We identify several challenges associated with prompting large language models, categorized into data- and model-specific, linguistic, and socio-linguistic challenges. A comprehensive examination of model outputs, including runner-up candidates and their corresponding probabilities, is needed to address these issues. The beam search tree, the prevalent algorithm to sample model outputs, can inherently supply this information. Consequently, we introduce an interactive visual method for investigating the beam search tree, facilitating analysis of the decisions made by the model during generation. We quantitatively show the value of exposing the beam search tree and present five detailed analysis scenarios addressing the identified challenges. Our methodology validates existing results and offers additional insights. Comment: 9 pages paper, 2 pages references, 7 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2310.11252 Zobrazit plný text záznamu View this record from Arxiv