Tilde MODEL Corpus – Multilingual Open Data for European Languages

The lack of language resources is one of the biggest obstacles to the development of language technology in Europe. In an effort to overcome this obstacle, Tilde made a commitment to create new multilingual corpora for European languages – particularly the smaller languages that need them most – and make them openly available to researchers and developers. We identified and collected multilingual open data sets in multiple languages and several key domains. In addition, collected resources has been cleaned, aligned, and formatted into a corpora standard TMX format useable for developing new Language technology products and services.

In Tilde MODEL Corpus we submit over 10M segments of multilingual open data for publication on the META-SHARE repository, maintained by the Multilingual Europe Technology Alliance, and on the EU Open Data Portal. These open data sets will be available to technology developers, researchers, localization companies, and machine translation providers. The corpora will provide a crucial resource for boosting the quality of MT engines, including the new breed of MT systems built with neural networks.

The data has been collected from sites allowing free use and reuse of its content, as well as from Public Sector web sites.

The activities have been undertaken as part of the ODINE Open Data Incubator for Europe, which aims to support the next generation of digital businesses and fast-track the development of new products and services.

This document lists all the Tilde MODEL corpora and allows download of each language pair in TMX format.

License: CC-BY - Creative Commons with Attribution

Please cite the following article if you use any part of the corpus in your own work: Roberts Rozis, Raivis Skadins, 2017, Tilde MODEL - Multilingual Open Data for EU Languages. Proceedings of the 21th Nordic Conference of Computational Linguistics NODALIDA 2017.

Tilde MODEL - EESC

Tilde MODEL - EESC is a multilingual corpus compiled from document texts of European Economic and Social Committee document portal. Source: http://dm.eesc.europa.eu/

Languages: 24
Language pairs: 69

bg cs da de el en es et fi fr hr hu is it lt lv mt nl pl pt ro sk sl sv
de 1.2M 1.4M 2.4M 2.4M 2.9M 2.4M 1.3M 2.2M 2.8M 226K 1.2M 209 2.4M 1.3M 1.3M 1.2M 2.4M 1.5M 2.4M 1.2M 1.4M 1.3M 2.2M
en 1.2M 1.3M 2.4M 2.9M 2.4M 2.5M 1.3M 2.1M 3.1M 250K 1.2M 220 2.6M 1.4M 1.4M 1.2M 2.4M 1.5M 2.5M 1.2M 1.5M 1.3M 2.3M
fr 1.2M 1.4M 2.4M 2.8M 2.5M 3.1M 2.6M 1.3M 2.1M 249K 1.2M 207 2.6M 1.4M 1.4M 1.2M 2.4M 1.5M 2.5M 1.2M 1.5M 1.3M 2.3M

Tilde MODEL - RAPID

Tilde MODEL - RAPID multilingual parallel corpus is compiled from all press releases of Press Release Database of European Commission released between 1975 and end of 2016 as available from http://europa.eu/rapid/.

Languages: 25
Language pairs: 72

bg cs da de el en es et fi fr hr hu is it lt lv mt nl no pl pt ro sk sl sv
de 177K 209K 499K 484K 1.0M 529K 188K 373K 983K 41K 194K 179 528K 177K 187K 186K 522K 572 212K 475K 175K 200K 198K 398K
en 199K 243K 537K 1.0M 558K 684K 227K 435K 1.5M 50K 236K 195 673K 213K 233K 231K 620K 727 265K 594K 217K 239K 241K 476K
fr 195K 234K 513K 983K 545K 1.5M 677K 208K 397K 49K 230K 211 661K 211K 223K 222K 605K 710 258K 593K 212K 233K 234K 462K

Tilde MODEL - ECB

Tilde MODEL - ECB multilingual parallel corpus is compiled from the multilingual pages of European Central Bank web site http://ebc.europa.eu/.

Languages: 23
Language pairs: 232

cs da de el en es et fi fr hr hu it lt lv mt nl pl pt ro sk sl sv
bg 3K 3K 3K 455 3K 3K 3K 2K 4K 2K 3K 3K 3K 3K 4K 4K 3K 4K 4K 3K 4K 4K
cs 4K 3K 4K 3K 4K 3K 3K 4K 2K 4K 3K 4K 4K 3K 4K 4K 4K 3K 4K 4K 4K
da 4K 5K 4K 5K 3K 4K 5K 2K 4K 5K 4K 4K 4K 5K 4K 5K 3K 4K 3K 5K
de 4K 4K 4K 3K 4K 4K 1K 2K 4K 3K 3K 3K 5K 3K 4K 2K 3K 3K 4K
el 5K 6K 4K 4K 6K 2K 4K 6K 4K 4K 5K 6K 4K 6K 4K 5K 5K 6K
en 5K 3K 4K 5K 1K 3K 5K 3K 4K 4K 5K 3K 5K 3K 4K 4K 5K
es 4K 5K 7K 2K 4K 7K 4K 4K 5K 7K 4K 6K 4K 5K 5K 6K
et 4K 4K 2K 4K 4K 4K 4K 4K 4K 4K 4K 3K 4K 4K 4K
fi 4K 1K 3K 4K 3K 3K 3K 5K 3K 4K 2K 3K 3K 5K
fr 2K 4K 6K 4K 5K 5K 7K 4K 7K 4K 5K 5K 6K
hr 2K 2K 2K 2K 2K 2K 2K 2K 2K 2K 2K 2K
hu 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
it 4K 4K 5K 7K 4K 6K 4K 4K 5K 6K
lt 4K 4K 4K 4K 4K 3K 4K 4K 4K
lv 4K 5K 4K 5K 4K 4K 4K 4K
mt 5K 4K 5K 4K 4K 5K 5K
nl 4K 6K 4K 5K 5K 6K
pl 4K 4K 4K 4K 4K
pt 4K 5K 5K 6K
ro 4K 4K 4K
sk 5K 5K
sl 5K

Tilde MODEL - EMA

Tilde MODEL - EMA is a corpus compiled from texts of European Medicines Agency document portal as available in http://www.ema.europa.eu/ at the end of 2016.

Languages: 25
Language pairs: 72

bg cs da de el en es et fi fr hr hu is it lt lv mt nl no pl pt ro sk sl sv
de 490K 473K 463K 408K 348K 463K 470K 464K 471K 416K 473K 399K 467K 465K 473K 494K 464K 331K 391K 469K 465K 467K 452K 456K
en 509K 495K 482K 348K 427K 493K 495K 481K 496K 443K 492K 420K 497K 488K 500K 526K 488K 347K 411K 506K 497K 496K 479K 484K
fr 531K 519K 499K 471K 451K 496K 519K 507K 495K 464K 515K 440K 524K 508K 519K 550K 514K 364K 433K 527K 520K 518K 502K 506K

Tilde MODEL - World Bank

Tilde MODEL - World Bank is a corpus compiled from texts of World Bank as available in http://www.worldbank.org/ in 2017.

Languages: 12
Language pairs: 11

es fr hr pl pt ro ru sq sr tr uk
en 54K 57K 2K 207 13K 6K 26K 3K 2K 2K 2K

Tilde MODEL - AirBaltic.com Travel Destinations

Tilde MODEL - AirBaltic.com Travel Destinations is a multilingual parallel corpus compiled from description texts of AirBaltic.com travel destinations as available in https://www.airbaltic.com/en/destinations/ in 2017.

Languages: 7
Language pairs: 21

en et fi lt lv ru
de 839 718 775 672 644 677
en 1K 828 1K 1K 1K
et 864 922 924 967
fi 792 767 780
lt 902 878
lv 880

Tilde MODEL - LiveRiga.com

Tilde MODEL - LiveRiga.com is a multilingual parallel corpus compiled from Riga tourist attractions description texts of http://liveriga.com/ web site in 2017.

Languages: 4
Language pairs: 6

en lv ru
de 839 644 677
en 1K 1K
lv 880

Tilde MODEL - Lithuanian National Philharmonic Society

Tilde MODEL - Lithuanian National Philharmonic Society is a parallel corpus compiled from texts of Lithuanian National Philharmonic Society web site http://www.filharmonija.lt/ in 2017.

Languages: 2
Language pairs: 1

lt
en 9K

Tilde MODEL - mupa.hu

Tilde MODEL - mupa.hu is a parallel corpus from texts of Müpa Budapest - web site of Hungarian national culture house and concert venue https://www.mupa.hu/en/ compiled in spring of 2017.

Languages: 2
Language pairs: 1

hu
en 16.5K

Tilde MODEL - fold.lv

Tilde MODEL - fold.lv is a parallel corpus from texts of fold.lv portal http://www.fold.lv/en/ of the best of Latvian and foreign creative industries as compiled in spring of 2017.

Languages: 2
Language pairs: 1

lv
en 10.7K

Tilde MODEL - czechtourism.com

Tilde MODEL - czechtourism.com is a multilingual parallel corpus from texts of http://czechtourism.com/ portal compiled in spring of 2017.

Languages: 8
Language pairs: 28

en es fr it pl pt ru
de 7K 6K 5K 6K 784 6K 6K
en 7K 7K 7K 866 7K 7K
es 7K 7K 845 7K 7K
fr 7K 797 7K 7K
it 814 7K 7K
pl 764 1K
pt 7K