Výsledky vyhledávání

Report

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models

Autor: Cheng, Jialiang, Gao, Ning, Yue, Yun, Ye, Zhiling, Jiang, Jiadi, Sha, Jian

Distributed training methods are crucial for large language models (LLMs). However, existing distributed training methods often suffer from communication bottlenecks, stragglers, and limited elasticity. Local SGD methods have been proposed to address

Externí odkaz: http://arxiv.org/abs/2412.07210

Zobrazit plný text záznamu

Report

Couler: Unified Machine Learning Workflow Optimization in Cloud

Autor: Wang, Xiaoda, Tang, Yuan, Guo, Tengda, Sang, Bo, Wu, Jingji, Sha, Jian, Zhang, Ke, Qian, Jiang, Tang, Mingjie

Machine Learning (ML) has become ubiquitous, fueling data-driven applications across various organizations. Contrary to the traditional perception of ML in research, ML workflows can be complex, resource-intensive, and time-consuming. Expanding an ML

Externí odkaz: http://arxiv.org/abs/2403.07608

Zobrazit plný text záznamu

Report

mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

Autor: Ye, Zhengmao, Li, Dengchun, Hu, Zetao, Lan, Tingfeng, Sha, Jian, Zhang, Sicong, Duan, Lei, Zuo, Jie, Lu, Hui, Zhou, Yuanchun, Tang, Mingjie

Transformer-based, pre-trained large language models (LLMs) have demonstrated outstanding performance across diverse domains, particularly in the emerging {\em pretrain-then-finetune} paradigm. Low-Rank Adaptation (LoRA), a parameter-efficient fine-t

Externí odkaz: http://arxiv.org/abs/2312.02515

Zobrazit plný text záznamu

Report

DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud

Autor: Wang, Qinlong, Lan, Tingfeng, Tang, Yinghao, Huang, Ziling, Du, Yiheng, Zhang, Haitao, Sha, Jian, Lu, Hui, Zhou, Yuanchun, Zhang, Ke, Tang, Mingjie

Deep learning recommendation models (DLRM) rely on large embedding tables to manage categorical sparse features. Expanding such embedding tables can significantly enhance model performance, but at the cost of increased GPU/CPU/memory usage. Meanwhile

Externí odkaz: http://arxiv.org/abs/2304.01468

Zobrazit plný text záznamu