Automatic Generation of Spatial Tactile Effects by Analyzing Cross-modality Features of a Video
| Autor: | Sean Follmer, Lawrence H. Kim, Kai Zhang, Yipeng Guo |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Artificial neural network
Computer science business.industry Cross modality 05 social sciences 020207 software engineering 02 engineering and technology Object (computer science) Pipeline (software) Tactile stimuli User experience design Synchronization (computer science) 0202 electrical engineering electronic engineering information engineering 0501 psychology and cognitive sciences Computer vision Artificial intelligence business 050107 human factors Haptic technology |
| Zdroj: | SUI |
| DOI: | 10.1145/3385959.3418459 |
| Popis: | Tactile effects can enhance user experience of multimedia content. However, generating appropriate tactile stimuli without any human intervention remains a challenge. While visual or audio information has been used to automatically generate tactile effects, utilizing cross-modal information may further improve the spatiotemporal synchronization and user experience of the tactile effects. In this paper, we present a pipeline for automatic generation of vibrotactile effects through the extraction of both the visual and audio features from a video. Two neural network models are used to extract the diegetic audio content, and localize a sounding object in the scene. These models are then used to determine the spatial distribution and the intensity of the tactile effects. To evaluate the performance of our method, we conducted a user study to compare the videos with tactile effects generated by our method to both the original videos without any tactile stimuli and videos with tactile effects generated based on visual features only. The study results demonstrate that our cross-modal method creates tactile effects with better spatiotemporal synchronization than the existing visual-based method and provides a more immersive user experience. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|