RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure

Autor:	Zhao, Mark, Choudhary, Dhruv, Tyagi, Devashish, Somani, Ajay, Kaplan, Max, Lin, Sung-Han, Pumma, Sarunya, Park, Jongsoo, Basant, Aarti, Agarwal, Niket, Wu, Carole-Jean, Kozyrakis, Christos
Rok vydání:	2022
Předmět:	Computer Science - Machine Learning Computer Science - Distributed Parallel and Cluster Computing Computer Science - Information Retrieval Computer Science - Performance
Druh dokumentu:	Working Paper
Popis:	We present RecD (Recommendation Deduplication), a suite of end-to-end infrastructure optimizations across the Deep Learning Recommendation Model (DLRM) training pipeline. RecD addresses immense storage, preprocessing, and training overheads caused by feature duplication inherent in industry-scale DLRM training datasets. Feature duplication arises because DLRM datasets are generated from interactions. While each user session can generate multiple training samples, many features' values do not change across these samples. We demonstrate how RecD exploits this property, end-to-end, across a deployed training pipeline. RecD optimizes data generation pipelines to decrease dataset storage and preprocessing resource demands and to maximize duplication within a training batch. RecD introduces a new tensor format, InverseKeyedJaggedTensors (IKJTs), to deduplicate feature values in each batch. We show how DLRM model architectures can leverage IKJTs to drastically increase training throughput. RecD improves the training and preprocessing throughput and storage efficiency by up to 2.48x, 1.79x, and 3.71x, respectively, in an industry-scale DLRM training system. Comment: Published in the Proceedings of the Sixth Conference on Machine Learning and Systems (MLSys 2023)
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2211.05239 Zobrazit plný text záznamu View this record from Arxiv