Face-Cap: Image Captioning Using Facial Expression Analysis
| Autor: | Len Hamey, Mark Dras, Omid Mohamad Nezami, Peter Anderson |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Closed captioning
Facial expression Computer science business.industry Deep learning Sentiment analysis 02 engineering and technology 010501 environmental sciences computer.software_genre 01 natural sciences Image (mathematics) Face (geometry) 0202 electrical engineering electronic engineering information engineering Code (cryptography) 020201 artificial intelligence & image processing Artificial intelligence business computer Natural language processing Natural language 0105 earth and related environmental sciences |
| Zdroj: | Machine Learning and Knowledge Discovery in Databases ISBN: 9783030109240 ECML/PKDD (1) |
| DOI: | 10.1007/978-3-030-10925-7_14 |
| Popis: | Image captioning is the process of generating a natural language description of an image. Most current image captioning models, however, do not take into account the emotional aspect of an image, which is very relevant to activities and interpersonal relationships represented therein. Towards developing a model that can produce human-like captions incorporating these, we use facial expression features extracted from images including human faces, with the aim of improving the descriptive ability of the model. In this work, we present two variants of our Face-Cap model, which embed facial expression features in different ways, to generate image captions. Using all standard evaluation metrics, our Face-Cap models outperform a state-of-the-art baseline model for generating image captions when applied to an image caption dataset extracted from the standard Flickr 30 K dataset, consisting of around 11 K images containing faces. An analysis of the captions finds that, perhaps surprisingly, the improvement in caption quality appears to come not from the addition of adjectives linked to emotional aspects of the images, but from more variety in the actions described in the captions. Code related to this paper is available at: https://github.com/omidmn/Face-Cap. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|