Tornike Tsereteli, Yavuz Selim Kartal, Simone Paolo Ponzetto, Andrea Zielinski, Kai Eckert, Philipp Mayr
In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Cross-Lingual | SV-Ident | mAP@10 | 18.93 | sentence-transformers/distiluse-base-multilingual-cased-v1 |
| Cross-Lingual | SV-Ident | mAP@10 | 13.59 | Sentence-T5 |
| Cross-Lingual | SV-Ident | mAP@10 | 11.27 | SPARTA |
| Cross-Lingual | SV-Ident | mAP@10 | 9.43 | BM25 |
| Text Classification | SV-Ident | F1 | 66.1 | SsciBERT |
| Text Classification | SV-Ident | F1 | 60.17 | Sentence-T5 |
| Classification | SV-Ident | F1 | 66.1 | SsciBERT |
| Classification | SV-Ident | F1 | 60.17 | Sentence-T5 |
| Cross-Lingual Entity Linking | SV-Ident | mAP@10 | 18.93 | sentence-transformers/distiluse-base-multilingual-cased-v1 |
| Cross-Lingual Entity Linking | SV-Ident | mAP@10 | 13.59 | Sentence-T5 |
| Cross-Lingual Entity Linking | SV-Ident | mAP@10 | 11.27 | SPARTA |
| Cross-Lingual Entity Linking | SV-Ident | mAP@10 | 9.43 | BM25 |