Výsledky vyhledávání

Report

ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference

Autor: He, Xin, Zhang, Shunkang, Wang, Yuxin, Yin, Haiyan, Zeng, Zihao, Shi, Shaohuai, Tang, Zhenheng, Chu, Xiaowen, Tsang, Ivor, Soon, Ong Yew

Sparse Mixture of Experts (MoE) models, while outperforming dense Large Language Models (LLMs) in terms of performance, face significant deployment challenges during inference due to their high memory demands. Existing offloading techniques, which in

Externí odkaz: http://arxiv.org/abs/2410.17954

Zobrazit plný text záznamu

Report

MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection

Autor: Lin, Bokai, Zeng, Zihao, Xiao, Zipeng, Kou, Siqi, Hou, Tianqi, Gao, Xiaofeng, Zhang, Hao, Deng, Zhijie

KV cache has become a de facto technique for the inference of large language models (LLMs), where tensors of shape (layer number, head number, sequence length, feature dimension) are introduced to cache historical information for self-attention. As t

Externí odkaz: http://arxiv.org/abs/2410.14731

Zobrazit plný text záznamu

Report

In-context KV-Cache Eviction for LLMs via Attention-Gate

Autor: Zeng, Zihao, Lin, Bokai, Hou, Tianqi, Zhang, Hao, Deng, Zhijie

The KV-Cache technique has become the standard for the inference of large language models (LLMs). It caches states of self-attention to avoid recomputation. Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference syste

Externí odkaz: http://arxiv.org/abs/2410.12876

Zobrazit plný text záznamu

Report

AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models

Autor: Zeng, Zihao, Miao, Yibo, Gao, Hongcheng, Zhang, Hao, Deng, Zhijie

Mixture of experts (MoE) has become the standard for constructing production-level large language models (LLMs) due to its promise to boost model capacity without causing significant overheads. Nevertheless, existing MoE methods usually enforce a con

Externí odkaz: http://arxiv.org/abs/2406.13233

Zobrazit plný text záznamu