Skluma: An Extensible Metadata Extraction Pipeline for Disorganized Data
| Autor: | Kyle Chard, Ian Foster, Galen Harrison, Tyler J. Skluzacek, Ryan Chard, Paul Beckman, Rohan Kumar |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
021110 strategic
defence & security studies Service (systems architecture) Information retrieval Computer science business.industry 0211 other engineering and technologies Context (language use) Cloud computing 02 engineering and technology JSON Pipeline (software) Metadata Workflow 020204 information systems Server Data_FILES 0202 electrical engineering electronic engineering information engineering business computer computer.programming_language |
| Zdroj: | eScience |
| DOI: | 10.1109/escience.2018.00040 |
| Popis: | To mitigate the effects of high-velocity data expansion and to automate the organization of filesystems and data repositories, we have developed Skluma-a system that automatically processes a target filesystem or repository, extracts content-and context-based metadata, and organizes extracted metadata for subsequent use. Skluma is able to extract diverse metadata, including aggregate values derived from embedded structured data; named entities and latent topics buried within free-text documents; and content encoded in images. Skluma implements an overarching probabilistic pipeline to extract increasingly specific metadata from files. It applies machine learning methods to determine file types, dynamically prioritizes and then executes a suite of metadata extractors, and explores contextual metadata based on relationships among files. The derived metadata, represented in JSON, describes probabilistic knowledge of each file that may be subsequently used for discovery or organization. Skluma's architecture enables it to be deployed both locally and used as an on-demand, cloud-hosted service to create and execute dynamic extraction workflows on massive numbers of files. It is modular and extensible-allowing users to contribute their own specialized metadata extractors. Thus far we have tested Skluma on local filesystems, remote FTP-accessible servers, and publicly-accessible Globus endpoints. We have demonstrated its efficacy by applying it to a scientific environmental data repository of more than 500,000 files. We show that we can extract metadata from those files with modest cloud costs in a few hours. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|