Zhen Wang, Xu Shan, Xiangxie Zhang, Jie Yang
Current news datasets merely focus on text features on the news and rarely leverage the feature of images, excluding numerous essential features for news classification. In this paper, we propose a new dataset, N24News, which is generated from New York Times with 24 categories and contains both text and image information in each news. We use a multitask multimodal method and the experimental results show multimodal news classification performs better than text-only news classification. Depending on the length of the text, the classification accuracy can be increased by up to 8.11%. Our research reveals the relationship between the performance of a multimodal classifier and its sub-classifiers, and also the possible improvements when applying multimodal in news classification. N24News is shown to have great potential to prompt the multimodal news studies.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Cross-Lingual | N15News | Accuracy | 0.9249 | Multimodal(ViT+BERT, Input: Image + Body) |
| Cross-Lingual | N15News | Accuracy | 0.9203 | BERT (Input: Body) |
| Cross-Lingual | N15News | Accuracy | 0.861 | Multimodal(ViT+BERT, Input: Image + Abstract) |
| Cross-Lingual | N15News | Accuracy | 0.8471 | BERT (Input: Abstract) |
| Cross-Lingual | N15News | Accuracy | 0.8202 | Multimodal(ViT+BERT, Input: Image + Headline) - Dot |
| Cross-Lingual | N15News | Accuracy | 0.7951 | Multimodal(ViT+BERT, Input: Image + Caption) - Concatenate |
| Cross-Lingual | N15News | Accuracy | 0.7792 | BERT (Input: Caption) |
| Cross-Lingual | N15News | Accuracy | 0.7727 | BERT (Input: Headline) |
| Cross-Lingual | N15News | Accuracy | 0.6065 | ViT (Input: Image) |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.9249 | Multimodal(ViT+BERT, Input: Image + Body) |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.9203 | BERT (Input: Body) |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.861 | Multimodal(ViT+BERT, Input: Image + Abstract) |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.8471 | BERT (Input: Abstract) |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.8202 | Multimodal(ViT+BERT, Input: Image + Headline) - Dot |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.7951 | Multimodal(ViT+BERT, Input: Image + Caption) - Concatenate |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.7792 | BERT (Input: Caption) |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.7727 | BERT (Input: Headline) |
| Cross-Lingual Document Classification | N15News | Accuracy | 0.6065 | ViT (Input: Image) |