ESS MS-G3D: extension and supplement shift MS-G3D network for the assessment of severe mental retardation

Autor: Quan Liu, Mincheng Cai, Dujuan Liu, Simeng Ma, Qianhong Zhang, Dan Xiang, Lihua Yao, Zhongchun Liu, Jun Yang
Jazyk: angličtina
Rok vydání: 2023
Předmět:
Zdroj: Complex & Intelligent Systems, Vol 10, Iss 2, Pp 2401-2419 (2023)
Druh dokumentu: article
ISSN: 2199-4536
2198-6053
DOI: 10.1007/s40747-023-01275-1
Popis: Abstract Automated mental retardation (MR) assessment is potential for improving the diagnostic efficiency and objectivity in clinical practice. Based on the researches on abnormal behavior characteristics of patients with MR, we propose an extension and supplement shift multi-scale G3D (ESS MS-G3D) network for video-based assessment of MR. Specifically, all videos are collected from clinical diagnostic scenarios and the skeleton sequence of human body is extracted from videos through an advanced pose estimation model. To solve the shortcomings of existing behavior characteristic learning methods, we present: (1) three G3D styles, enable the network to have different input forms; (2) two G3D graphs and two extension graphs, redefine and extend the graph structure of spatial–temporal nodes; (3) two learnable parameters, realize adaptive adjustment of graph structure; (4) a shift layer, enable the network to learn global features. Finally, we construct a three-branch model ESS MS-STGC, which can capture the discriminative spatial–temporal features and explore the co-occurrence relationship between spatial and temporal domains. Experiments in clinical video data set show that our proposed model has good performance in MR assessment and is superior to the existing vision-based methods. In two-classification task, our model with joint stream achieves the highest accuracy of $$94.63\%$$ 94.63 % in validation set and $$89.13\%$$ 89.13 % in test set. The results are further improved to $$96.52\%$$ 96.52 % and $$93.22\%$$ 93.22 % , respectively, by utilizing multi-stream fusion strategy. In four-classification task, our model obtains Top1 accuracy of $$78.84\%$$ 78.84 % and Top2 accuracy of $$91.34\%$$ 91.34 % in test set. The proposed method provides a new idea for clinical mental retardation assessment.
Databáze: Directory of Open Access Journals
Externí odkaz: