Audio-Assisted Image Inpainting for Talking Faces
| Autor: | Gerasimos Potamianos, Edmilson da Silva Morais, Samuel Thomas, Alexandros Koumparoulis |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Artificial neural network
business.industry Computer science Inpainting 020207 software engineering 02 engineering and technology Object (computer science) Image (mathematics) Task (computing) Video editing Face (geometry) 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Computer vision Artificial intelligence business Fusion mechanism |
| Zdroj: | ICASSP |
| DOI: | 10.1109/icassp40776.2020.9053184 |
| Popis: | The goal of our work is to complete missing areas of images of talking faces, exploiting information from both the visual and audio modalities. Existing image inpainting methods rely solely on visual content that doesn’t always provide sufficient information for the task. To counter this, we propose a neural network that employs an encoder-decoder architecture with a bimodal fusion mechanism, thus taking into account both visual and audio content. Our proposed method demonstrates consistently superior performance over a baseline visual-only model, reaching for example up to 17% relative improvement in mean absolute error. The presented model is applicable to practical video editing tasks, such as object and overlay-text removal from talking faces, where existing lip and face generation works are not applicable as they require clean input. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|