Communication-Efficient Jaccard Similarity for High-Performance Distributed Genome Comparisons
Autor: | Harun Mustafa, Maciej Besta, Gunnar Rätsch, Torsten Hoefler, Raghavendra Kanakagiri, Edgar Solomonik, Mikhail Karasikov |
---|---|
Jazyk: | angličtina |
Rok vydání: | 2019 |
Předmět: |
FOS: Computer and information sciences
0301 basic medicine Jaccard index Computer science 010103 numerical & computational mathematics computer.software_genre 01 natural sciences Computational Engineering Finance and Science (cs.CE) Set (abstract data type) 03 medical and health sciences Similarity (network science) Synchronization (computer science) Quantitative Biology - Genomics 0101 mathematics Computer Science - Computational Engineering Finance and Science Sparse matrix Genomics (q-bio.GN) Computer Science - Performance Performance (cs.PF) 030104 developmental biology Computer Science - Distributed Parallel and Cluster Computing Distributed algorithm FOS: Biological sciences Distributed Parallel and Cluster Computing (cs.DC) Data mining computer |
Zdroj: | IPDPS |
Popis: | The Jaccard similarity index is an important measure of the overlap of two sets, widely used in machine learning, computational genomics, information retrieval, and many other areas. We design and implement SimilarityAtScale, the first communication-efficient distributed algorithm for computing the Jaccard similarity among pairs of large datasets. Our algorithm provides an efficient encoding of this problem into a multiplication of sparse matrices. Both the encoding and sparse matrix product are performed in a way that minimizes data movement in terms of communication and synchronization costs. We apply our algorithm to obtain similarity among all pairs of a set of large samples of genomes. This task is a key part of modern metagenomics analysis and an evergrowing need due to the increasing availability of high-throughput DNA sequencing data. The resulting scheme is the first to enable accurate Jaccard distance derivations for massive datasets, using large-scale distributed-memory systems. We package our routines in a tool, called GenomeAtScale, that combines the proposed algorithm with tools for processing input sequences. Our evaluation on real data illustrates that one can use GenomeAtScale to effectively employ tens of thousands of processors to reach new frontiers in large-scale genomic and metagenomic analysis. While GenomeAtScale can be used to foster DNA research, the more general underlying SimilarityAtScale algorithm may be used for high-performance distributed similarity computations in other data analytics application domains. |
Databáze: | OpenAIRE |
Externí odkaz: |