Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting

Autor: J. Prasanna Kumar, P. Govindarajulu
Jazyk: angličtina
Rok vydání: 2013
Předmět:
Zdroj: International Journal of Computational Intelligence Systems, Vol 6, Iss 1 (2013)
Druh dokumentu: article
ISSN: 18756891
1875-6883
DOI: 10.1080/18756891.2013.752657
Popis: Duplicate and near-duplicate web pages are the chief concerns for web search engines. In reality, they incur enormous space to store the indexes, ultimately slowing down and increasing the cost of serving results. A variety of techniques have been developed to identify pairs of web pages that are “similar” to each other. The problem of finding near-duplicate web pages has been a subject of research in the database and web-search communities for some years. In order to identify the near duplicate web pages, we make use of sentence level features along with fingerprinting method. When a large number of web documents are in consideration for the detection of web pages, then at first, we use K-mode clustering and subsequently sentence feature and fingerprint comparison is used. Using these steps, we exactly identify the near duplicate web pages in an efficient manner. The experimentation is carried out on the web page collections and the results ensured the efficiency of the proposed approach in detecting the near duplicate web pages.
Databáze: Directory of Open Access Journals
Externí odkaz: