Multimodal speech driven facial shape animation using deep neural networks
| Autor: | Sasan Asadiabadi, Engin Erzin, Rizwan Sadiq |
|---|---|
| Přispěvatelé: | Erzin, Engin (ORCID 0000-0002-2715-2368 & YÖK ID 34503), Sadiq, Rizwan, Asadiabadi, Sasan, Graduate School of Sciences and Engineering, Department of Electrical and Electronics Engineering |
| Jazyk: | angličtina |
| Rok vydání: | 2018 |
| Předmět: |
Scale (ratio)
Mean squared error business.industry Computer science Speech recognition Deep learning 020207 software engineering 02 engineering and technology Animation Variation (game tree) Engineering electrical and electronic Grid Speech driven animations Deep neural network (DNN) Active shape models (ASM) 030507 speech-language pathology & audiology 03 medical and health sciences Face (geometry) 0202 electrical engineering electronic engineering information engineering Feature (machine learning) Artificial intelligence 0305 other medical science business |
| Zdroj: | 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) APSIPA |
| Popis: | In this paper we present a deep learning multimodal approach for speech driven generation of face animations. Training a speaker independent model, capable of generating different emotions of the speaker, is crucial for realistic animations. Unlike the previous approaches which either use acoustic features or phoneme label features to estimate the facial movements, we utilize both modalities to generate natural looking speaker independent lip animations synchronized with affective speech. A phoneme-based model qualifies generation of speaker independent animation, whereas an acoustic feature-based model enables capturing affective variation during the animation generation. We show that our multimodal approach not only performs significantly better on affective data, but improves performance over neutral data as well. We evaluate the proposed multimodal speech-driven animation model using two large scale datasets, GRID and SAVEE, by reporting the mean squared error (MSE) over various network structures. NA |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|