Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Šuppa, Hila Gonen, Joseph Marvin Imperial, Börje F. Karlsson, Peiqin Lin, Nikola Ljubešić, LJ Miranda, Barbara Plank, Arij Riabi, Yuval Pinter
We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. The overarching goal of UNER is to provide high-quality, cross-lingually consistent annotations to facilitate and standardize multilingual NER research. UNER v1 contains 18 datasets annotated with named entities in a cross-lingual consistent schema across 12 diverse languages. In this paper, we detail the dataset creation and composition of UNER; we also provide initial modeling baselines on both in-language and cross-lingual learning settings. We release the data, code, and fitted models to the public.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Named Entity Recognition (NER) | UNER v1 (Croatian) | F1 (micro) | 93.6 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (Portuguese) | F1 (micro) | 90.4 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (English) | F1 (micro) | 86 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 - PUD (Swedish) | F1 (micro) | 82.2 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 - PUD (English) | F1 (micro) | 80.1 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (Chinese) | F1 (micro) | 89.5 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (Serbian) | F1 (micro) | 94.7 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (Danish) | F1 (micro) | 82.7 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (Swedish) | F1 (micro) | 88.3 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 - PUD (Chinese) | F1 (micro) | 87.1 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (Chinese Simplified) | F1 (micro) | 89.4 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 (Slovak) | F1 (micro) | 85.5 | UNER XML-R |
| Named Entity Recognition (NER) | UNER v1 - PUD (Portuguese) | F1 (micro) | 88.8 | UNER XML-R |
| Cross-Lingual | UNER v1 - PUD (German) | F1 (micro) | 78.9 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Serbian) | F1 (micro) | 95.2 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 - PUD (English) | F1 (micro) | 79.5 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Danish) | F1 (micro) | 83 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 - PUD (Swedish) | F1 (micro) | 85.3 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Cebuano) | F1 (micro) | 69.6 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Croatian) | F1 (micro) | 90.9 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Tagalog U) | F1 (micro) | 63.8 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 - PUD (Portuguese) | F1 (micro) | 85.1 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 - PUD (Russian) | F1 (micro) | 70.6 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Slovak) | F1 (micro) | 81.6 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Tagalog T) | F1 (micro) | 91.3 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Chinese Simplified) | F1 (micro) | 87.7 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Portuguese) | F1 (micro) | 82.3 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (English) | F1 (micro) | 82.8 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Chinese) | F1 (micro) | 88.2 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 (Swedish) | F1 (micro) | 88.2 | UNER XML-R (all) |
| Cross-Lingual | UNER v1 - PUD (Chinese) | F1 (micro) | 86 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 - PUD (German) | F1 (micro) | 78.9 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Serbian) | F1 (micro) | 95.2 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 - PUD (English) | F1 (micro) | 79.5 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Danish) | F1 (micro) | 83 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 - PUD (Swedish) | F1 (micro) | 85.3 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Cebuano) | F1 (micro) | 69.6 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Croatian) | F1 (micro) | 90.9 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Tagalog U) | F1 (micro) | 63.8 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 - PUD (Portuguese) | F1 (micro) | 85.1 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 - PUD (Russian) | F1 (micro) | 70.6 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Slovak) | F1 (micro) | 81.6 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Tagalog T) | F1 (micro) | 91.3 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Chinese Simplified) | F1 (micro) | 87.7 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Portuguese) | F1 (micro) | 82.3 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (English) | F1 (micro) | 82.8 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Chinese) | F1 (micro) | 88.2 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 (Swedish) | F1 (micro) | 88.2 | UNER XML-R (all) |
| Cross-Lingual Transfer | UNER v1 - PUD (Chinese) | F1 (micro) | 86 | UNER XML-R (all) |