Dong Huo, Jian Wang, Yiming Qian, Yee-Hong Yang
This paper proposes a new glass segmentation method utilizing paired RGB and thermal images. Due to the large difference between the transmission property of visible light and that of the thermal energy through the glass where most glass is transparent to the visible light but opaque to thermal energy, glass regions of a scene are made more distinguishable with a pair of RGB and thermal images than solely with an RGB image. To exploit such a unique property, we propose a neural network architecture that effectively combines an RGB-thermal image pair with a new multi-modal fusion module based on attention, and integrate CNN and transformer to extract local features and non-local dependencies, respectively. As well, we have collected a new dataset containing 5551 RGB-thermal image pairs with ground-truth segmentation annotations. The qualitative and quantitative evaluations demonstrate the effectiveness of the proposed approach on fusing RGB and thermal data for glass segmentation. Our code and data are available at https://github.com/Dong-Huo/RGB-T-Glass-Segmentation.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | RGB-T-Glass-Segmentation | MAE | 0.024 | RGB-T-Glass-Segmentation |
| Scene Segmentation | RGB-T-Glass-Segmentation | MAE | 0.024 | RGB-T-Glass-Segmentation |
| 2D Object Detection | RGB-T-Glass-Segmentation | MAE | 0.024 | RGB-T-Glass-Segmentation |
| 10-shot image generation | RGB-T-Glass-Segmentation | MAE | 0.024 | RGB-T-Glass-Segmentation |