Analysis-ready VCF at Biobank scale using Zarr.

Autor:	Czech E; Related Sciences and Lincoln, Lincoln, New Zealand., Millar TR; The New Zealand Institute for Plant & Food Research Ltd, Lincoln, New Zealand.; Department of Biochemistry, School of Biomedical Sciences, University of Otago, Dunedin, New Zealand., White T; Tom White Consulting Ltd., Li Ka Shing Centre for Health Information and Discovery, University of Oxford, UK., Jeffery B; Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, University of Oxford, UK., Miles A; Wellcome Sanger Institute, McGill University, Montreal, QC, Canada., Tallman S; Genomics England, McGill University, Montreal, QC, Canada., Wojdyla R; Related Sciences and Lincoln, Lincoln, New Zealand., Zabad S; School of Computer Science, McGill University, Montreal, QC, Canada., Hammerbacher J; Related Sciences and Lincoln, Lincoln, New Zealand., Kelleher J; Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, University of Oxford, UK.
Jazyk:	angličtina
Zdroj:	BioRxiv : the preprint server for biology [bioRxiv] 2024 Jun 12. Date of Electronic Publication: 2024 Jun 12.
DOI:	10.1101/2024.06.11.598241
Abstrakt:	Background: Variant Call Format (VCF) is the standard file format for interchanging genetic variation data and associated quality control metrics. The usual row-wise encoding of the VCF data model (either as text or packed binary) emphasises efficient retrieval of all data for a given variant, but accessing data on a field or sample basis is inefficient. Biobank scale datasets currently available consist of hundreds of thousands of whole genomes and hundreds of terabytes of compressed VCF. Row-wise data storage is fundamentally unsuitable and a more scalable approach is needed. Results: We present the VCF Zarr specification, an encoding of the VCF data model using Zarr which makes retrieving subsets of the data much more efficient. Zarr is a cloud-native format for storing multi-dimensional data, widely used in scientific computing. We show how this format is far more efficient than standard VCF based approaches, and competitive with specialised methods for storing genotype data in terms of compression ratios and calculation performance. We demonstrate the VCF Zarr format (and the vcf2zarr conversion utility) on a subset of the Genomics England aggV2 dataset comprising 78,195 samples and 59,880,903 variants, with a 5X reduction in storage and greater than 300X reduction in CPU usage in some representative benchmarks. Conclusions: Large row-encoded VCF files are a major bottleneck for current research, and storing and processing these files incurs a substantial cost. The VCF Zarr specification, building on widely-used, open-source technologies has the potential to greatly reduce these costs, and may enable a diverse ecosystem of next-generation tools for analysing genetic variation data directly from cloud-based object stores.
Databáze:	MEDLINE
Externí odkaz:	Zobrazit plný text záznamu