ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis

Autor: Mughal, Muhammad Hamza, Dabral, Rishabh, Habibie, Ikhsanul, Donatelli, Lucia, Habermann, Marc, Theobalt, Christian

Gestures play a key role in human communication. Recent methods for co-speech gesture generation, while managing to generate beat-aligned motions, struggle generating gestures that are semantically aligned with the utterance. Compared to beat gesture

Externí odkaz: http://arxiv.org/abs/2403.17936

Learning Speech-driven 3D Conversational Gestures from Video

Autor: Habibie, Ikhsanul, Xu, Weipeng, Mehta, Dushyant, Liu, Lingjie, Seidel, Hans-Peter, Pons-Moll, Gerard, Elgharib, Mohamed, Theobalt, Christian

We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input. Our algorithm uses a CNN architectu

Externí odkaz: http://arxiv.org/abs/2102.06837

Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

Autor: Zhou, Yuxiao, Habermann, Marc, Xu, Weipeng, Habibie, Ikhsanul, Theobalt, Christian, Xu, Feng

We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy. This is enabled by a new learning based architecture designed such that it can make use of all the

Externí odkaz: http://arxiv.org/abs/2003.09572

Imitator: Personalized Speech-driven 3D Facial Animation

Autor: Thambiraja, Balamurugan, Habibie, Ikhsanul, Aliakbarian, Sadegh, Cosker, Darren, Theobalt, Christian, Thies, Justus

Speech-driven 3D facial animation has been widely explored, with applications in gaming, character animation, virtual reality, and telepresence systems. State-of-the-art methods deform the face topology of the target actor to sync the input audio wit

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::c891599e1fe187d50bec999764b4680d

A Motion Matching-Based Framework for Controllable Gesture Synthesis from Speech

Autor: Habibie, Ikhsanul, Elgharib, Mohamed, Sarkar, Kripasindhu, Abdullah, Ahsan, Nyatsanga, Simbarashe, Neff, Michael, Theobalt, Christian

Publikováno v: Proceedings SIGGRAPH 2022 Conference Papers Proceedings
Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::9da7c7caf83b22a5d858c98188dccab0
https://hdl.handle.net/21.11116/0000-000B-7B9F-921.11116/0000-000B-7BA1-5