Vamsa: Automated Provenance Tracking in Data Science Scripts
| Autor: | Mohammad Hossein Namaki, Ashvin Agrawal, Fotis Psallidas, Yinghui Wu, Yiwen Zhu, Avrilia Floratou, Subru Krishnan, Markus Weimer |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Computer science Latency (audio) Context (language use) Machine Learning (stat.ML) Python (programming language) computer.software_genre Data science Machine Learning (cs.LG) Range (mathematics) Computer Science - Distributed Parallel and Cluster Computing Statistics - Machine Learning Scripting language Code (cryptography) Data Corruption Distributed Parallel and Cluster Computing (cs.DC) Precision and recall computer computer.programming_language |
| Zdroj: | KDD |
| DOI: | 10.48550/arxiv.2001.01861 |
| Popis: | There has recently been a lot of ongoing research in the areas of fairness, bias and explainability of machine learning (ML) models due to the self-evident or regulatory requirements of various ML applications. We make the following observation: All of these approaches require a robust understanding of the relationship between ML models and the data used to train them. In this work, we introduce the ML provenance tracking problem: the fundamental idea is to automatically track which columns in a dataset have been used to derive the features/labels of an ML model. We discuss the challenges in capturing such information in the context of Python, the most common language used by data scientists. We then present Vamsa, a modular system that extracts provenance from Python scripts without requiring any changes to the users' code. Using 26K real data science scripts, we verify the effectiveness of Vamsa in terms of coverage, and performance. We also evaluate Vamsa's accuracy on a smaller subset of manually labeled data. Our analysis shows that Vamsa's precision and recall range from 90.4% to 99.1% and its latency is in the order of milliseconds for average size scripts. Drawing from our experience in deploying ML models in production, we also present an example in which Vamsa helps automatically identify models that are affected by data corruption issues. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|