Výsledky vyhledávání - "Cao, Zongyan"

Report

HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

Autor: Zhang, Shiwei, Diao, Lansong, Wu, Chuan, Cao, Zongyan, Wang, Siyu, Lin, Wei

Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning

Externí odkaz: http://arxiv.org/abs/2401.05965

Zobrazit plný text záznamu

Report

Ada-Grouper: Accelerating Pipeline Parallelism in Preempted Network by Adaptive Group-Scheduling for Micro-Batches

Autor: Wang, Siyu, Cao, Zongyan, Si, Chang, Diao, Lansong, Wang, Jiamang, Lin, Wei

Pipeline parallelism has been demonstrated to be a remarkable approach to improve throughput for training deep neural networks with billions of parameters over heterogeneous clusters. The 1F1B scheduling plan is a widely adopted strategy for memory a

Externí odkaz: http://arxiv.org/abs/2303.01675

Zobrazit plný text záznamu

Report

Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platform

Autor: Zhang, Shiwei, Diao, Lansong, Wang, Siyu, Cao, Zongyan, Gu, Yiliang, Si, Chang, Shi, Ziji, Zheng, Zhen, Wu, Chuan, Lin, Wei

We present Rhino, a system for accelerating tensor programs with automatic parallelization on AI platform for real production environment. It transforms a tensor program written for a single device into an equivalent distributed program that is capab

Externí odkaz: http://arxiv.org/abs/2302.08141

Zobrazit plný text záznamu

Report

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

Autor: Fan, Shiqing, Rong, Yi, Meng, Chen, Cao, Zongyan, Wang, Siyu, Zheng, Zhen, Wu, Chuan, Long, Guoping, Yang, Jun, Xia, Lixue, Diao, Lansong, Liu, Xiaoyong, Lin, Wei

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are

Externí odkaz: http://arxiv.org/abs/2007.01045

Zobrazit plný text záznamu

Report

PHoToNs--A Parallel Heterogeneous & Threads oriented code for cosmological N-body simulation

Autor: Wang, Qiao, Cao, Zongyan, Gao, Liang, Chi, Xuebin, Meng, Chen, Wang, Jie, Wang, Long

We introduce a new code for cosmological simulations, PHoToNs, which has features on performing massive cosmological simulations on heterogeneous high performance Computer (HPC) and threads oriented programming. PHoToNs adopts a hybrid scheme to comp

Externí odkaz: http://arxiv.org/abs/1802.09824

Zobrazit plný text záznamu