| Autor: |
Wang, Peiqi, Wells, William M., Berkowitz, Seth, Horng, Steven, Golland, Polina |
| Rok vydání: |
2022 |
| Předmět: |
|
| Druh dokumentu: |
Working Paper |
| DOI: |
10.1007/978-3-031-34048-2_35 |
| Popis: |
Image-text multimodal representation learning aligns data across modalities and enables important medical applications, e.g., image classification, visual grounding, and cross-modal retrieval. In this work, we establish a connection between multimodal representation learning and multiple instance learning. Based on this connection, we propose a generic framework for constructing permutation-invariant score functions with many existing multimodal representation learning approaches as special cases. Furthermore, we use the framework to derive a novel contrastive learning approach and demonstrate that our method achieves state-of-the-art results in several downstream tasks. |
| Databáze: |
arXiv |
| Externí odkaz: |
|
