Applying maching learning to data cleansing
| Autor: | Marjanović, Vitomir |
|---|---|
| Přispěvatelé: | Vrdoljak, Boris |
| Jazyk: | chorvatština |
| Rok vydání: | 2021 |
| Předmět: |
čišćenje podataka
pandas mice TEHNIČKE ZNANOSTI. Računarstvo TEHNIČKE ZNANOSTI. Elektrotehnika imputation dedupe imputacija klasifikacija anomaly detection strojno učenje data cleansing python machine learning classification TECHNICAL SCIENCES. Electrical Engineering TECHNICAL SCIENCES. Computing regresija scikit-learn regression uklanjanje duplikata detekcija anomalija duplicates removal |
| Popis: | Cilj istraživanja provedenog ovim diplomskim radom je ispitati razne metode strojnog učenja pri postupku čišćenja podataka u svrhu unapređenja daljnjeg procesa analize podataka. Postupak čišćenja podijeljen je u tri koraka – uklanjanje dupliciranih vrijednosti, detekcija i uklanjanje anomalija te umetanje vrijednosti koje nedostaju. Za svaki od navedenih koraka ispitane su implementacije odabranih algoritama u programskim jezicima Python i R te je u konačnici napravljena njihova usporedba. Prilikom analize odabranih metoda, korišteni su skupovi podataka s karakteristikama koje odgovaraju scenarijima koje se ispituju kako bi rješenje bilo što uočljivije i kvalitetnije. Po završetku usporedbe odabranih metoda, prikazani su zaključci te potencijalni problemi prilikom implementacije navedenih metoda i samog postupka čišćenja u cijelosti. The aim of the conducted research as a part of this master's thesis was to invesigate different machine learning methods in data cleansing in order to improve the following dana analysis process. Data cleansing process was divided in three steps – removing duplicated values, anomaly detection and anomaly removal and missing values imputation. For each of the named steps a selected set of algorithms implemented in Python and R programming languages were tested, and the testing results were provided and compaired to summarize it. In the process, the datasets for the testing were selected to have as many characteristics that describe the tested problem as possible so the output results are clearly visible and representative. By the end of the method comparison, some important conclusions were made, but also some of the underlying problems were pointed out when implementing the tested methods and conducting a data cleansing process as a whole. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|