TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Papers/Context-Aware Video Instance Segmentation

Context-Aware Video Instance Segmentation

Seunghun Lee, Jiwan Seo, Kiljoon Han, Minwoo Choi, Sunghoon Im

2024-07-03Panoptic SegmentationVideo Panoptic SegmentationSegmentationSemantic SegmentationInstance SegmentationVideo Instance Segmentation
PaperPDFCode(official)

Abstract

In this paper, we introduce the Context-Aware Video Instance Segmentation (CAVIS), a novel framework designed to enhance instance association by integrating contextual information adjacent to each object. To efficiently extract and leverage this information, we propose the Context-Aware Instance Tracker (CAIT), which merges contextual data surrounding the instances with the core instance features to improve tracking accuracy. Additionally, we introduce the Prototypical Cross-frame Contrastive (PCC) loss, which ensures consistency in object-level features across frames, thereby significantly enhancing instance matching accuracy. CAVIS demonstrates superior performance over state-of-the-art methods on all benchmark datasets in video instance segmentation (VIS) and video panoptic segmentation (VPS). Notably, our method excels on the OVIS dataset, which is known for its particularly challenging videos.

Results

TaskDatasetMetricValueModel
Semantic SegmentationVIPSegSTQ56.1CAVIS(VIT-L)
Semantic SegmentationVIPSegVPQ58.5CAVIS(VIT-L)
Video Instance SegmentationYouTube-VIS 2021AP5087.3CAVIS(VIT-L, Offline)
Video Instance SegmentationYouTube-VIS 2021AP7573.2CAVIS(VIT-L, Offline)
Video Instance SegmentationYouTube-VIS 2021AR149.7CAVIS(VIT-L, Offline)
Video Instance SegmentationYouTube-VIS 2021AR1070.3CAVIS(VIT-L, Offline)
Video Instance SegmentationYouTube-VIS 2021mask AP65.3CAVIS(VIT-L, Offline)
Video Instance SegmentationYouTube-VIS validationAP5089.3CAVIS(ViT-L, Online)
Video Instance SegmentationYouTube-VIS validationAP7576.2CAVIS(ViT-L, Online)
Video Instance SegmentationYouTube-VIS validationAR158.3CAVIS(ViT-L, Online)
Video Instance SegmentationYouTube-VIS validationAR1073.6CAVIS(ViT-L, Online)
Video Instance SegmentationYouTube-VIS validationmask AP68.9CAVIS(ViT-L, Online)
Video Instance SegmentationOVIS validationAP5082.6CAVIS(VIT-L, Offline)
Video Instance SegmentationOVIS validationAP7563.5CAVIS(VIT-L, Offline)
Video Instance SegmentationOVIS validationAR121.2CAVIS(VIT-L, Offline)
Video Instance SegmentationOVIS validationAR1061.8CAVIS(VIT-L, Offline)
Video Instance SegmentationOVIS validationmask AP57.1CAVIS(VIT-L, Offline)
Video Instance SegmentationYoutube-VIS 2022 ValidationmAP_L48.6CAVIS (VIT-L)
10-shot image generationVIPSegSTQ56.1CAVIS(VIT-L)
10-shot image generationVIPSegVPQ58.5CAVIS(VIT-L)
Panoptic SegmentationVIPSegSTQ56.1CAVIS(VIT-L)
Panoptic SegmentationVIPSegVPQ58.5CAVIS(VIT-L)

Related Papers

SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction2025-07-21Deep Learning-Based Fetal Lung Segmentation from Diffusion-weighted MRI Images and Lung Maturity Evaluation for Fetal Growth Restriction2025-07-17DiffOSeg: Omni Medical Image Segmentation via Multi-Expert Collaboration Diffusion Model2025-07-17From Variability To Accuracy: Conditional Bernoulli Diffusion Models with Consensus-Driven Correction for Thin Structure Segmentation2025-07-17Unleashing Vision Foundation Models for Coronary Artery Segmentation: Parallel ViT-CNN Encoding and Variational Fusion2025-07-17SCORE: Scene Context Matters in Open-Vocabulary Remote Sensing Instance Segmentation2025-07-17Unified Medical Image Segmentation with State Space Modeling Snake2025-07-17A Privacy-Preserving Semantic-Segmentation Method Using Domain-Adaptation Technique2025-07-17