Audio- and Gaze-driven Facial Animation of Codec Avatars
Autor: | Juergen Gall, Shugao Ma, Alexander Richard, Yaser Sheikh, Colin Lea, Fernando De la Torre |
---|---|
Rok vydání: | 2020 |
Předmět: |
FOS: Computer and information sciences
business.industry Computer science Computer Vision and Pattern Recognition (cs.CV) Computer Science - Computer Vision and Pattern Recognition ComputingMethodologies_IMAGEPROCESSINGANDCOMPUTERVISION 020207 software engineering 02 engineering and technology Virtual reality Gaze Motion (physics) Human–computer interaction Encoding (memory) Face (geometry) 0202 electrical engineering electronic engineering information engineering Eye tracking Codec 020201 artificial intelligence & image processing Artificial intelligence business Computer facial animation |
Zdroj: | WACV |
DOI: | 10.48550/arxiv.2008.05023 |
Popis: | Codec Avatars are a recent class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D (i.e., for virtual reality), and are almost indistinguishable from video [28]. In this paper we describe the first approach to animate these parametric models in real-time which could be deployed on commodity virtual reality hardware using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals such as laughter and excitement solely from la-tent cues in our lossy input signals. To this end we collected over 5 hours of high frame rate 3D face scans across three participants including traditional neutral speech as well as expressive and conversational speech. We investigate a multimodal fusion approach that dynamically identifies which sensor encoding should animate which parts of the face at any time. See the supplemental video which demonstrates our ability to generate full face motion far beyond the typically neutral lip articulations seen in competing work: https://research.fb.com/videos/audio-and-gaze-driven-facial-animation-of-codec-avatars/ |
Databáze: | OpenAIRE |
Externí odkaz: |