Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022
| Autor: | Ben-Avraham, Elad, Herzig, Roei, Mangalam, Karttikeya, Bar, Amir, Rohrbach, Anna, Karlinsky, Leonid, Darrell, Trevor, Globerson, Amir |
|---|---|
| Rok vydání: | 2022 |
| Předmět: | |
| Druh dokumentu: | Working Paper |
| Popis: | This technical report describes the SViT approach for the Ego4D Point of No Return (PNR) Temporal Localization Challenge. We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a "Frame-Clip Consistency" loss, which ensures the flow of structured information between images and videos. SViT obtains strong performance on the challenge test set with 0.656 absolute temporal localization error. Comment: Ego4D CVPR22 Object State Localization challenge. arXiv admin note: substantial text overlap with arXiv:2206.06346 |
| Databáze: | arXiv |
| Externí odkaz: |
|