Masa: Responsive Multi-DNN Inference on the Edge
| Autor: | Robert Birke, Bart Cox, Jeroen Galjaard, Lydia Y. Chen, Amirmasoud Ghiassi |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Předmět: |
Ubiquitous computing
Edge device Computer science Distributed computing Response time Image processing 02 engineering and technology 010501 environmental sciences 01 natural sciences Scheduling (computing) Middleware 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Enhanced Data Rates for GSM Evolution Quality of experience 0105 earth and related environmental sciences |
| Zdroj: | PerCom |
| Popis: | Deep neural networks (DNNs) are becoming the core components of many applications running on edge devices, especially for real time image-based analysis. Increasingly, multi-faced knowledge is extracted via executing multiple DNNs inference models, e.g., identifying objects, faces, and genders from images. The response times of multi-DNN highly affect users’ quality of experience and safety as well. Different DNNs exhibit diversified resource requirements and execution patterns across layers and networks, which may easily exceed the available device memory and riskily degrade the responsiveness. In this paper, we design and implement Masa, a responsive memory-aware multi-DNN execution framework, an on-device middleware featuring on modeling inter- and intra-network dependency and leveraging complimentary memory usage of each layer. Masa can consistently ensure the average response time when deterministically and stochastically executing multiple DNN-based image analyses. We extensively evaluate Masa on three configurations of Raspberry Pi and a large set of popular DNN models triggered by different generation patterns of images. Our evaluation results show that Masa can achieve lower average response times by up to 90% on devices with small memory, i.e., 512 MB to 1 GB, compared to the state of the art multi-DNN scheduling solutions. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|