Quantifying the Limits of Segment Anything Model: Analyzing Challenges in Segmenting Tree-Like and Low-Contrast Structures

Autor:	Zhang, Yixin, Konz, Nicholas, Kramer, Kevin, Mazurowski, Maciej A.
Rok vydání:	2024
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Machine Learning Electrical Engineering and Systems Science - Image and Video Processing
Druh dokumentu:	Working Paper
Popis:	Segment Anything Model (SAM) has shown impressive performance in interactive and zero-shot segmentation across diverse domains, suggesting that they have learned a general concept of "objects" from their large-scale training. However, we observed that SAM struggles with certain types of objects, particularly those featuring dense, tree-like structures and low textural contrast from their surroundings. These failure modes are critical for understanding its limitations in real-world use. In order to systematically examine this issue, we propose metrics to quantify two key object characteristics: tree-likeness and textural separability. Through extensive controlled synthetic experiments and testing on real datasets, we demonstrate that SAM's performance is noticeably correlated with these factors. We link these behaviors under the concept of "textural confusion", where SAM misinterprets local structure as global texture, leading to over-segmentation, or struggles to differentiate objects from similarly textured backgrounds. These findings offer the first quantitative framework to model SAM's challenges, providing valuable insights into its limitations and guiding future improvements for vision foundation models. Comment: Code: https://github.com/mazurowski-lab/SAM-TexturalConfusion-Metrics
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2412.04243 Zobrazit plný text záznamu View this record from Arxiv