| Popis: |
There is increasing evidence that hand gestures and speech synchronize their activity on multiple dimensions and time scales. For example, gesture’s kinematic peaks (e.g., maximum speed) are coupled to prosodic markers in speech. Such coupling operates on very short timescales at the level of syllables (200 ms), and therefore requires high resolution estimation of gesture kinematics and speech acoustics. High-resolution speech analysis is common for gesture studies given its classic ties with (psycho)linguistics. However, the field has lagged behind in the objective study of gesture kinematics (e.g., compared to research on action). Often, kinematic peaks in gesture are measured by eye, where a “moment of maximum effort” is determined by several raters. In the current paper, we provide a tutorial on more objective and time-effective methods to quantify temporal properties of gesture kinematics, where we focus on common challenges and possible solutions that come with the complexities of studying multimodal language. We further introduce and compare, using an actual gesture dataset (392 gesture events), the performance of two video-based motion-tracking methods (deep learning vs. pixel change) against a high-performance wired motion-tracking system (Polhemus Liberty). We show that videography methods perform well in the temporal estimation of kinematic peaks, and thus provide a cheap alternative to expensive motion-tracking systems. We hope that the current paper incites gesture researchers to embark on the widespread objective study of gesture kinematics and its relation to speech. |
