aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3.
Autor: | García-Ruiz S; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK., Reynolds RH; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK., Grant-Peters M; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK., Gustavsson EK; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK., Fairbrother-Browne A; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; Department of Medical and Molecular Genetics, School of Basic and Medical Biosciences, King's College London, London, UK.; Department of Neurodegenerative Disease, Queen Square Institute of Neurology, UCL, London, UK., Chen Z; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; Department of Neurodegenerative Disease, Queen Square Institute of Neurology, UCL, London, UK., Brenton JW; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK., Ryten M; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK.; NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK. |
---|---|
Jazyk: | angličtina |
Zdroj: | GigaByte (Hong Kong, China) [GigaByte] 2023 Aug 23; Vol. 2023, pp. gigabyte87. Date of Electronic Publication: 2023 Aug 23 (Print Publication: 2023). |
DOI: | 10.46471/gigabyte.87 |
Abstrakt: | Amazon Simple Storage Service (Amazon S3) is a widely used platform for storing large biomedical datasets. Unintended data alterations can occur during data writing and transmission, altering the original content and generating unexpected results. However, no open-source and easy-to-use tool exists to verify end-to-end data integrity. Here, we present aws-s3-integrity-check , a user-friendly, lightweight, and reliable bash tool to verify the integrity of a dataset stored in an Amazon S3 bucket. Using this tool, we only needed ∼114 min to verify the integrity of 1,045 records ranging between 5 bytes and 10 gigabytes and occupying ∼935 gigabytes of the Amazon S3 cloud. Our aws-s3-integrity-check tool also provides file-by-file on-screen and log-file-based information about the status of each integrity check. To our knowledge, this tool is the only open-source one that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage quickly, reliably, and efficiently. The tool is freely available for download and use at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check. Competing Interests: The authors declare that they have no competing interests. (© The Author(s) 2023.) |
Databáze: | MEDLINE |
Externí odkaz: |