EXPERIENCE
| Autor: | Ciro D'Urso |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
Information Systems and Management
Database business.industry Computer science 02 engineering and technology Machine learning computer.software_genre Masking (Electronic Health Record) Data point 020204 information systems Data quality Outlier 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Anomaly detection Artificial intelligence Overall performance Data mining Coordinate space business Transformation algorithm computer Information Systems |
| Zdroj: | Journal of Data and Information Quality. 7:1-22 |
| ISSN: | 1936-1963 1936-1955 |
| DOI: | 10.1145/2950109 |
| Popis: | Enterprise's archives are inevitably affected by the presence of data quality problems (also called glitches). This article proposes the application of a new method to analyze the quality of datasets stored in the tables of a database, with no knowledge of the semantics of the data and without the need to define repositories of rules. The proposed method is based on proper revisions of different approaches for outlier detection that are combined to boost overall performance and accuracy. A novel transformation algorithm is conceived that treats the items in database tables as data points in real coordinate space of n dimensions, so that fields containing dates and fields containing text are processed to calculate distances between those data points. The implementation of an iterative approach ensures that global and local outliers are discovered even if they are subject, primarily in datasets with multiple outliers or clusters of outliers, to masking and swamping effects. The application of the method to a set of archives, some of which have been studied extensively in the literature, provides very promising experimental results and outperforms the application of a single other technique. Finally, a list of future research directions is highlighted. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|