Výsledky vyhledávání

Akademický článek

Systematic Analysis of Low-Precision Training in Deep Neural Networks: Factors Influencing Matrix Computations

Autor: Ao Shen, Zhiquan Lai, Lizhi Zhang

Publikováno v: Applied Sciences, Vol 14, Iss 21, p 10025 (2024)

As Deep Neural Networks (DNNs) continue to increase in complexity, the computational demands of their training have become a significant bottleneck. Low-precision training has emerged as a crucial strategy, wherein full-precision values are quantized

Externí odkaz: https://doaj.org/article/f4ef1ca0076647f3a20a60d4f1f249d1

Zobrazit plný text záznamu

Merak: An Efficient Distributed DNN Training Framework With Automated 3D Parallelism for Giant Foundation Models

Autor: Zhiquan Lai, Shengwei Li, Xudong Tang, Keshi Ge, Weijie Liu, Yabo Duan, Linbo Qiao, Dongsheng Li

Publikováno v: IEEE Transactions on Parallel and Distributed Systems. 34:1466-1478

Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the training pr

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::0a2ffcf75d20114e0b298d41511b1662
https://doi.org/10.1109/tpds.2023.3247001

Zobrazit plný text záznamu

Hierarchical Adaptive Pooling by Capturing High-Order Dependency for Graph Representation Learning

Autor: Ning Liu, Songlei Jian, Dongsheng Li, Yiming Zhang, Zhiquan Lai, Hongzuo Xu

Publikováno v: IEEE Transactions on Knowledge and Data Engineering. 35:3952-3965

Graph neural networks (GNN) have been proven to be mature enough for handling graph-structured data on node-level graph representation learning tasks. However, the graph pooling technique for learning expressive graph-level representation is critical

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::13dca15be2e19331741d73372634c23c
https://doi.org/10.1109/tkde.2021.3133646

Zobrazit plný text záznamu

Compressed Collective Sparse-Sketch for Distributed Data-Parallel Training of Deep Learning Models

Autor: Keshi Ge, Kai Lu, Yongquan Fu, Xiaoge Deng, Zhiquan Lai, Dongsheng Li

Publikováno v: IEEE Journal on Selected Areas in Communications. 41:941-963

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_________::46809de78a5d41c251ed401e0cf4ba4f
https://doi.org/10.1109/jsac.2023.3242733

Zobrazit plný text záznamu

Accelerate distributed deep learning with cluster-aware sketch quantization

Autor: Keshi Ge, Yiming Zhang, Yongquan Fu, Zhiquan Lai, Xiaoge Deng, Dongsheng Li

Publikováno v: Science China Information Sciences. 66

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_________::f397c1fd58719411dbd4ab046eaa49fa
https://doi.org/10.1007/s11432-021-3532-8

Zobrazit plný text záznamu

Increasing momentum-like factors: A method for reducing training errors on multiple GPUs

Autor: Zhiquan Lai, Zhaoning Zhang, Zhigang Kan, Dongsheng Li, Lujia Yin, Yu Tang, Linbo Qiao

Publikováno v: Tsinghua Science and Technology. 27:114-126

In distributed training, increasing batch size can improve parallelism, but it can also bring many difficulties to the training process and cause training errors. In this work, we investigate the occurrence of training errors in theory and train ResN

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_________::3b4646f5ddcfcd3ce62f8fd71b3ad11e
https://doi.org/10.26599/tst.2020.9010023

Zobrazit plný text záznamu

Vyhledávací nástroje:

Upřesnit hledání