Mohammad Faiyaz Khan, S. M. Sadiq-Ur-Rahman Shifath, Md. Saiful Islam
Image Captioning is an arduous task of producing syntactically and semantically correct textual descriptions of an image in natural language with context related to the image. Existing notable pieces of research in Bengali Image Captioning (BIC) are based on encoder-decoder architecture. This paper presents an end-to-end image captioning system utilizing a multimodal architecture by combining a one-dimensional convolutional neural network (CNN) to encode sequence information with a pre-trained ResNet-50 model image encoder for extracting region-based visual features. We investigate our approach's performance on the BanglaLekhaImageCaptions dataset using the existing evaluation metrics and perform a human evaluation for qualitative analysis. Experiments show that our approach's language encoder captures the fine-grained information in the caption, and combined with the image features, it generates accurate and diversified caption. Our work outperforms all the existing BIC works and achieves a new state-of-the-art (SOTA) performance by scoring 0.651 on BLUE-1, 0.572 on CIDEr, 0.297 on METEOR, 0.434 on ROUGE, and 0.357 on SPICE.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Image Captioning | BanglaLekhaImageCaptions | BLEU-1 | 65.1 | CNN + 1D CNN |
| Image Captioning | BanglaLekhaImageCaptions | BLEU-2 | 42.6 | CNN + 1D CNN |
| Image Captioning | BanglaLekhaImageCaptions | BLEU-3 | 27.8 | CNN + 1D CNN |
| Image Captioning | BanglaLekhaImageCaptions | BLEU-4 | 17.5 | CNN + 1D CNN |
| Image Captioning | BanglaLekhaImageCaptions | CIDEr | 57.2 | CNN + 1D CNN |
| Image Captioning | BanglaLekhaImageCaptions | METEOR | 29.7 | CNN + 1D CNN |
| Image Captioning | BanglaLekhaImageCaptions | ROUGE-L | 43.4 | CNN + 1D CNN |
| Image Captioning | BanglaLekhaImageCaptions | SPICE | 35.7 | CNN + 1D CNN |