DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries
| Autor: | Mishra, Manit, Braham, Abderrahman, Marsom, Charles, Chung, Bryan, Griffin, Gavin, Sidnerlikar, Dakshesh, Sarin, Chatanya, Rajaram, Arjun |
|---|---|
| Rok vydání: | 2024 |
| Předmět: | |
| Druh dokumentu: | Working Paper |
| DOI: | 10.1109/ICAIC60265.2024.10433803 |
| Popis: | Conventional processes for analyzing datasets and extracting meaningful information are often time-consuming and laborious. Previous work has identified manual, repetitive coding and data collection as major obstacles that hinder data scientists from undertaking more nuanced labor and high-level projects. To combat this, we evaluated OpenAI's GPT-3.5 as a "Language Data Scientist" (LDS) that can extrapolate key findings, including correlations and basic information, from a given dataset. The model was tested on a diverse set of benchmark datasets to evaluate its performance across multiple standards, including data science code-generation based tasks involving libraries such as NumPy, Pandas, Scikit-Learn, and TensorFlow, and was broadly successful in correctly answering a given data science query related to the benchmark dataset. The LDS used various novel prompt engineering techniques to effectively answer a given question, including Chain-of-Thought reinforcement and SayCan prompt engineering. Our findings demonstrate great potential for leveraging Large Language Models for low-level, zero-shot data analysis. Comment: 5 pages, Submitted to International Conference on AI in Cybersecurity |
| Databáze: | arXiv |
| Externí odkaz: |
|