Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, Jean-Philippe Thiran
Learning dense visual representations without labels is an arduous task and more so from scene-centric data. We propose to tackle this challenging problem by proposing a Cross-view consistency objective with an Online Clustering mechanism (CrOC) to discover and segment the semantics of the views. In the absence of hand-crafted priors, the resulting method is more generalizable and does not require a cumbersome pre-processing step. More importantly, the clustering algorithm conjointly operates on the features of both views, thereby elegantly bypassing the issue of content not represented in both views and the ambiguous matching of objects from one crop to the other. We demonstrate excellent performance on linear and unsupervised segmentation transfer tasks on various datasets and similarly for video object segmentation. Our code and pre-trained models are publicly available at https://github.com/stegmuel/CrOC.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | COCO-Stuff-27 | Clustering [mIoU] | 21.9 | CrOC (ViT-S/16, COCO+) |
| Unsupervised Semantic Segmentation | COCO-Stuff-27 | Clustering [mIoU] | 21.9 | CrOC (ViT-S/16, COCO+) |
| 10-shot image generation | COCO-Stuff-27 | Clustering [mIoU] | 21.9 | CrOC (ViT-S/16, COCO+) |