VIIDA and InViDe: computational approaches for generating and evaluating inclusive image paragraphs for the visually impaired.

Autor:	Fernandes DL; Department of Informatics, Universidade Federal de Viçosa - UFV, Viçosa, Brazil., Ribeiro MHF; Department of Informatics, Universidade Federal de Viçosa - UFV, Viçosa, Brazil., Silva MM; Department of Informatics, Universidade Federal de Viçosa - UFV, Viçosa, Brazil., Cerqueira FR; Department of Informatics, Universidade Federal de Viçosa - UFV, Viçosa, Brazil.; Department of Production Engineering, Universidade Federal Fluminense - UFF, Petrópolis, Brazil.
Jazyk:	angličtina
Zdroj:	Disability and rehabilitation. Assistive technology [Disabil Rehabil Assist Technol] 2024 Dec 11, pp. 1-26. Date of Electronic Publication: 2024 Dec 11.
DOI:	10.1080/17483107.2024.2437567
Abstrakt:	Background: Existing image description methods when used as Assistive Technologies often fall short in meeting the needs of blind or low vision (BLV) individuals. They tend to either compress all visual elements into brief captions, create disjointed sentences for each image region, or provide extensive descriptions. Purpose: To address these limitations, we introduce VIIDA, a procedure aimed at the Visually Impaired which implements an Image Description Approach, focusing on webinar scenes. We also propose InViDe, an Inclusive Visual Description metric, a novel approach for evaluating image descriptions targeting BLV people. Methods: We reviewed existing methods and developed VIIDA by integrating a multimodal Visual Question Answering model with Natural Language Processing (NLP) filters. A scene graph-based algorithm was then applied to structure final paragraphs. By employing NLP tools, InViDe conducts a multicriteria analysis based on accessibility standards and guidelines. Results: Experiments statistically demonstrate that VIIDA generates descriptions closely aligned with image content as well as human-written linguistic features, and that suit BLV needs. InViDe offers valuable insights into the behaviour of the compared methods - among them, state-of-the-art methods based on Large Language Models - across diverse criteria. Conclusion: VIIDA and InViDe emerge as efficient Assistive Technologies, combining Artificial Intelligence models and computational/mathematical techniques to generate and evaluate image descriptions for the visually impaired with low computational costs. This work is anticipated to inspire further research and application development in the domain of Assistive Technologies. Our codes are publicly available at: https://github.com/daniellf/VIIDA-and-InViDe.
Databáze:	MEDLINE
Externí odkaz:	Zobrazit plný text záznamu