Zobrazeno 1 - 10
of 21
pro vyhledávání: '"Swearngin, Amanda"'
With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and underst
Externí odkaz:
http://arxiv.org/abs/2410.09006
UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we intro
Externí odkaz:
http://arxiv.org/abs/2409.13900
Accessibility is crucial for inclusive app usability, yet developers often struggle to identify and fix app accessibility issues due to a lack of awareness, expertise, and inadequate tools. Current accessibility testing tools can identify accessibili
Externí odkaz:
http://arxiv.org/abs/2408.03827
User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-learned model, UIClip, for assessing the design quality and visual rel
Externí odkaz:
http://arxiv.org/abs/2404.12500
Autor:
You, Keen, Zhang, Haotian, Schoop, Eldon, Weers, Floris, Swearngin, Amanda, Nichols, Jeffrey, Yang, Yinfei, Gan, Zhe
Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present
Externí odkaz:
http://arxiv.org/abs/2404.05719
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating pair
Externí odkaz:
http://arxiv.org/abs/2310.04869
Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amo
Externí odkaz:
http://arxiv.org/abs/2310.02424
Autor:
Swearngin, Amanda, Wu, Jason, Zhang, Xiaoyi, Gomez, Esteban, Coughenour, Jen, Stukenborg, Rachel, Garg, Bhavya, Hughes, Greg, Hilliard, Adriana, Bigham, Jeffrey P., Nichols, Jeffrey
Many apps have basic accessibility issues, like missing labels or low contrast. Automated tools can help app developers catch basic issues, but can be laborious or require writing dedicated tests. We propose a system, motivated by a collaborative pro
Externí odkaz:
http://arxiv.org/abs/2310.00091
Autor:
Wu, Jason, Krosnick, Rebecca, Schoop, Eldon, Swearngin, Amanda, Bigham, Jeffrey P., Nichols, Jeffrey
Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human crowd
Externí odkaz:
http://arxiv.org/abs/2308.08726
Understanding user interface (UI) functionality is a useful yet challenging task for both machines and people. In this paper, we investigate a machine learning approach for screen correspondence, which allows reasoning about UIs by mapping their elem
Externí odkaz:
http://arxiv.org/abs/2301.08372