FG-OVD

Benchmark Suite Description for PapersWithCode

Fine-Grained Open-Vocabulary Detection (FG-OVD) Benchmark Suite
The FG-OVD benchmark suite evaluates the ability of open-vocabulary object detectors to discern fine-grained object properties such as color, material, pattern, and transparency. This suite introduces dynamic vocabularies for each object, consisting of one positive caption and several challenging negative captions, crafted using attribute substitution at varying difficulty levels.

Key features include:

• Difficulty-Based Benchmarks: Trivial, Easy, Medium, and Hard benchmarks challenge detectors with progressively harder negative examples.
• Attribute-Based Benchmarks: Focused evaluation of specific attributes, such as color or material, with negative captions differing only in the targeted attribute.
• Metrics: Mean Average Precision (mAP) and Median Rank are used to measure both localization accuracy and fine-grained caption assignment performance.

The suite provides a comprehensive analysis of state-of-the-art models, highlighting their strengths and limitations in fine-grained object recognition.