Tell as You Imagine: Sentence Imageability-Aware Image Captioning
| Autor: | Yasutomo Kawanishi, Keisuke Doman, Marc A. Kastner, Daisuke Deguchi, Hiroshi Murase, Ichiro Ide, Kazuki Umemura, Takatsugu Hirayama |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Closed captioning
InformationSystems_INFORMATIONINTERFACESANDPRESENTATION(e.g. HCI) Computer science business.industry media_common.quotation_subject InformationSystems_INFORMATIONSTORAGEANDRETRIEVAL ComputingMethodologies_IMAGEPROCESSINGANDCOMPUTERVISION computer.software_genre Psycholinguistics Task (project management) Image (mathematics) Perception ComputingMethodologies_DOCUMENTANDTEXTPROCESSING ComputingMilieux_COMPUTERSANDSOCIETY Artificial intelligence business computer Natural language processing Sentence media_common |
| Zdroj: | MultiMedia Modeling ISBN: 9783030678340 MMM (2) |
| DOI: | 10.1007/978-3-030-67835-7_6 |
| Popis: | Image captioning as a multimedia task is advancing in terms of performance in generating captions for general purposes. However, it remains difficult to tailor generated captions to different applications. In this paper, we propose a sentence imageability-aware image captioning method to generate captions tailoring to various applications. Sentence imageability describes how easily the caption can be mentally imagined. This concept is applied to the captioning model to obtain a better understanding of the perception of a generated caption. First, we extend an existing image caption dataset by augmenting its captions’ diversity. Then, a sentence imageability score for each augmented caption is calculated. A modified image captioning model is trained using this extended dataset to generate captions tailoring to a specified imageability score. Experiments showed promising results in generating imageability-aware captions. Especially, results from a subjective experiment showed that the perception of the generated captions correlates with the specified score. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|