Multimodal 3D American sign language recognition for static alphabet and numbers using hand joints and shape coding

Autor: Hossein Ebrahimnezhad, Khadijeh Mahdikhanlou
Rok vydání: 2020
Předmět:
Zdroj: Multimedia Tools and Applications. 79:22235-22259
ISSN: 1573-7721
1380-7501
DOI: 10.1007/s11042-020-08982-8
Popis: American sign language recognition is still a research focus in computer vision community. Recently, most researches mainly extract low-level features for hand gesture recognition. These approaches perform poorly on recognizing gestures posed like a fist. In this paper, we propose a novel multimodal framework for sign language recognition system which exploits the Leap Motion Controller (LMC) and a webcam. We compute two sets of features. The first set is the angles at hand joints acquired by the LMC sensor. When, hand poses like a fist, the positions of the thumb joints captured by the LMC are not very precise. So, we should incorporate the second set of features extracted from the hand shape contour provided by a webcam. In this paper, we introduce a new mid-level feature, called Contour Segment Code (CSC), to represent hand shape contour. The proposed shape representation, first, extracts meaningful landmarks from the hand shape contour. CSC then encodes different segments of the hand contour into a code based on the shape landmarks. The extracted landmarks precisely determine the hand direction. The proposed method is tested by creating a very challenging dataset composed of 64,000 samples. Our experiments study the performance of the LMC and characteristics of CSC in different scenarios. The experimental results demonstrate the privileged performance of the proposed method against the systems which use depth images.
Databáze: OpenAIRE
Externí odkaz: