'What's important here?': Opportunities and Challenges of Using LLMs in Retrieving Information from Web Interfaces

Autor:	Huq, Faria, Bigham, Jeffrey P., Martelaro, Nikolas
Rok vydání:	2023
Předmět:	Computer Science - Computation and Language Computer Science - Information Retrieval
Druh dokumentu:	Working Paper
Popis:	Large language models (LLMs) that have been trained on a corpus that includes large amount of code exhibit a remarkable ability to understand HTML code. As web interfaces are primarily constructed using HTML, we design an in-depth study to see how LLMs can be used to retrieve and locate important elements for a user given query (i.e. task description) in a web interface. In contrast with prior works, which primarily focused on autonomous web navigation, we decompose the problem as an even atomic operation - Can LLMs identify the important information in the web page for a user given query? This decomposition enables us to scrutinize the current capabilities of LLMs and uncover the opportunities and challenges they present. Our empirical experiments show that while LLMs exhibit a reasonable level of performance in retrieving important UI elements, there is still a substantial room for improvement. We hope our investigation will inspire follow-up works in overcoming the current challenges in this domain. Comment: Accepted to NeurIPS 2023 R0-FoMo Workshop
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2312.06147 Zobrazit plný text záznamu View this record from Arxiv