TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Papers/Segmentation from Natural Language Expressions

Segmentation from Natural Language Expressions

Ronghang Hu, Marcus Rohrbach, Trevor Darrell

2016-03-20Referring Expression SegmentationSegmentationSemantic Segmentation
PaperPDFCodeCode(official)CodeCode

Abstract

In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefined set of semantic classes, as e.g., the phrase "two men sitting on the right bench" requires segmenting only the two people on the right bench and no one standing or sitting on another bench. Previous approaches suitable for this task were limited to a fixed set of categories and/or rectangular regions. To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information. In our model, a recurrent LSTM network is used to encode the referential expression into a vector representation, and a fully convolutional network is used to a extract a spatial feature map from the image and output a spatial response map for the target object. We demonstrate on a benchmark dataset that our model can produce quality segmentation output from the natural language expression, and outperforms baseline methods by a large margin.

Results

TaskDatasetMetricValueModel
Instance SegmentationA2D SentencesAP0.132Hu et al.
Instance SegmentationA2D SentencesIoU mean0.35Hu et al.
Instance SegmentationA2D SentencesIoU overall0.474Hu et al.
Instance SegmentationA2D SentencesPrecision@0.50.348Hu et al.
Instance SegmentationA2D SentencesPrecision@0.60.236Hu et al.
Instance SegmentationA2D SentencesPrecision@0.70.133Hu et al.
Instance SegmentationA2D SentencesPrecision@0.80.033Hu et al.
Instance SegmentationJ-HMDBAP0.178Hu et al.
Instance SegmentationJ-HMDBIoU mean0.528Hu et al.
Instance SegmentationJ-HMDBIoU overall0.546Hu et al.
Instance SegmentationJ-HMDBPrecision@0.50.633Hu et al.
Instance SegmentationJ-HMDBPrecision@0.60.35Hu et al.
Instance SegmentationJ-HMDBPrecision@0.70.085Hu et al.
Instance SegmentationJ-HMDBPrecision@0.80.002Hu et al.
Referring Expression SegmentationA2D SentencesAP0.132Hu et al.
Referring Expression SegmentationA2D SentencesIoU mean0.35Hu et al.
Referring Expression SegmentationA2D SentencesIoU overall0.474Hu et al.
Referring Expression SegmentationA2D SentencesPrecision@0.50.348Hu et al.
Referring Expression SegmentationA2D SentencesPrecision@0.60.236Hu et al.
Referring Expression SegmentationA2D SentencesPrecision@0.70.133Hu et al.
Referring Expression SegmentationA2D SentencesPrecision@0.80.033Hu et al.
Referring Expression SegmentationJ-HMDBAP0.178Hu et al.
Referring Expression SegmentationJ-HMDBIoU mean0.528Hu et al.
Referring Expression SegmentationJ-HMDBIoU overall0.546Hu et al.
Referring Expression SegmentationJ-HMDBPrecision@0.50.633Hu et al.
Referring Expression SegmentationJ-HMDBPrecision@0.60.35Hu et al.
Referring Expression SegmentationJ-HMDBPrecision@0.70.085Hu et al.
Referring Expression SegmentationJ-HMDBPrecision@0.80.002Hu et al.

Related Papers

SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction2025-07-21Deep Learning-Based Fetal Lung Segmentation from Diffusion-weighted MRI Images and Lung Maturity Evaluation for Fetal Growth Restriction2025-07-17DiffOSeg: Omni Medical Image Segmentation via Multi-Expert Collaboration Diffusion Model2025-07-17From Variability To Accuracy: Conditional Bernoulli Diffusion Models with Consensus-Driven Correction for Thin Structure Segmentation2025-07-17Unleashing Vision Foundation Models for Coronary Artery Segmentation: Parallel ViT-CNN Encoding and Variational Fusion2025-07-17SCORE: Scene Context Matters in Open-Vocabulary Remote Sensing Instance Segmentation2025-07-17Unified Medical Image Segmentation with State Space Modeling Snake2025-07-17A Privacy-Preserving Semantic-Segmentation Method Using Domain-Adaptation Technique2025-07-17