Multilingual Dataset for Training and Evaluating Diacritics Restoration Systems

TextsCC BY-NC-SA 4.0Introduced 2018-05-01

The dataset contains training and evaluation data for 12 languages:

  • Vietnamese
  • Romanian
  • Latvian
  • Czech
  • Polish
  • Slovak
  • Irish
  • Hungarian
  • French
  • Turkish
  • Spanish
  • Croatian

For each language, one training, one development and one testing set acquired from Wikipedia articles is provided. Moreover, each language dataset contains (substantially larger) training set collected from (general) Web texts. All sets, except for Wikipedia and Web training sets that can contain similar sentences, are disjoint. Data are segmented into sentences which are further word tokenized.