The lack of language resources is one of the biggest obstacles to the development of language technology in Europe. In an effort to overcome this obstacle, Tilde made a commitment to create new multilingual corpora for European languages – particularly the smaller languages that need them most – and make them openly available to researchers and developers. We identified and collected multilingual open data sets in multiple languages and several key domains. In addition, collected resources has been cleaned, aligned, and formatted into a corpora standard TMX format useable for developing new Language technology products and services.
In Tilde MODEL Corpus we submit over 10M segments of multilingual open data for publication on the META-SHARE repository, maintained by the Multilingual Europe Technology Alliance, and on the EU Open Data Portal. These open data sets will be available to technology developers, researchers, localization companies, and machine translation providers. The corpora will provide a crucial resource for boosting the quality of MT engines, including the new breed of MT systems built with neural networks.
The data has been collected from sites allowing free use and reuse of its content, as well as from Public Sector web sites.
The activities have been undertaken as part of the ODINE Open Data Incubator for Europe, which aims to support the next generation of digital businesses and fast-track the development of new products and services.
This document lists all the Tilde MODEL corpora and allows download of each language pair in TMX format.
License: CC-BY - Creative Commons with Attribution
Please cite the following article if you use any part of the corpus in your own work: Roberts Rozis, Raivis Skadins, 2017, Tilde MODEL - Multilingual Open Data for EU Languages. Proceedings of the 21th Nordic Conference of Computational Linguistics NODALIDA 2017.
Tilde MODEL - EESC is a multilingual corpus compiled from document texts of European Economic and Social Committee document portal. Source: http://dm.eesc.europa.eu/
Languages: 24
Language pairs: 69
| bg | cs | da | de | el | en | es | et | fi | fr | hr | hu | is | it | lt | lv | mt | nl | pl | pt | ro | sk | sl | sv | |
| de | 1.2M | 1.4M | 2.4M | 2.4M | 2.9M | 2.4M | 1.3M | 2.2M | 2.8M | 226K | 1.2M | 209 | 2.4M | 1.3M | 1.3M | 1.2M | 2.4M | 1.5M | 2.4M | 1.2M | 1.4M | 1.3M | 2.2M | |
| en | 1.2M | 1.3M | 2.4M | 2.9M | 2.4M | 2.5M | 1.3M | 2.1M | 3.1M | 250K | 1.2M | 220 | 2.6M | 1.4M | 1.4M | 1.2M | 2.4M | 1.5M | 2.5M | 1.2M | 1.5M | 1.3M | 2.3M | |
| fr | 1.2M | 1.4M | 2.4M | 2.8M | 2.5M | 3.1M | 2.6M | 1.3M | 2.1M | 249K | 1.2M | 207 | 2.6M | 1.4M | 1.4M | 1.2M | 2.4M | 1.5M | 2.5M | 1.2M | 1.5M | 1.3M | 2.3M |
Tilde MODEL - RAPID multilingual parallel corpus is compiled from all press releases of Press Release Database of European Commission released between 1975 and end of 2016 as available from http://europa.eu/rapid/.
Languages: 25
Language pairs: 72
| bg | cs | da | de | el | en | es | et | fi | fr | hr | hu | is | it | lt | lv | mt | nl | no | pl | pt | ro | sk | sl | sv | |
| de | 177K | 209K | 499K | 484K | 1.0M | 529K | 188K | 373K | 983K | 41K | 194K | 179 | 528K | 177K | 187K | 186K | 522K | 572 | 212K | 475K | 175K | 200K | 198K | 398K | |
| en | 199K | 263K | 537K | 940K | 558K | 684K | 227K | 435K | 1.5M | 50K | 236K | 195 | 673K | 213K | 233K | 231K | 620K | 727 | 278K | 594K | 217K | 239K | 241K | 476K | |
| fr | 195K | 234K | 513K | 983K | 545K | 1.5M | 677K | 208K | 397K | 49K | 230K | 211 | 661K | 211K | 223K | 222K | 605K | 710 | 258K | 593K | 212K | 233K | 234K | 462K |
Tilde MODEL - ECB multilingual parallel corpus is compiled from the multilingual pages of European Central Bank web site http://ebc.europa.eu/.
Languages: 23
Language pairs: 232
| cs | da | de | el | en | es | et | fi | fr | hr | hu | it | lt | lv | mt | nl | pl | pt | ro | sk | sl | sv | |
| bg | 3K | 3K | 3K | 455 | 3K | 3K | 3K | 2K | 4K | 2K | 3K | 3K | 3K | 3K | 4K | 4K | 3K | 4K | 4K | 3K | 4K | 4K |
| cs | 4K | 3K | 4K | 3K | 4K | 3K | 3K | 4K | 2K | 4K | 3K | 4K | 4K | 3K | 4K | 4K | 4K | 3K | 4K | 4K | 4K | |
| da | 4K | 5K | 4K | 5K | 3K | 4K | 5K | 2K | 4K | 5K | 4K | 4K | 4K | 5K | 4K | 5K | 3K | 4K | 3K | 5K | ||
| de | 4K | 4K | 4K | 3K | 4K | 4K | 1K | 2K | 4K | 3K | 3K | 3K | 5K | 3K | 4K | 2K | 3K | 3K | 4K | |||
| el | 5K | 6K | 4K | 4K | 6K | 2K | 4K | 6K | 4K | 4K | 5K | 6K | 4K | 6K | 4K | 5K | 5K | 6K | ||||
| en | 5K | 3K | 4K | 5K | 1K | 3K | 5K | 3K | 4K | 4K | 5K | 3K | 5K | 3K | 4K | 4K | 5K | |||||
| es | 4K | 5K | 7K | 2K | 4K | 7K | 4K | 4K | 5K | 7K | 4K | 6K | 4K | 5K | 5K | 6K | ||||||
| et | 4K | 4K | 2K | 4K | 4K | 4K | 4K | 4K | 4K | 4K | 4K | 3K | 4K | 4K | 4K | |||||||
| fi | 4K | 1K | 3K | 4K | 3K | 3K | 3K | 5K | 3K | 4K | 2K | 3K | 3K | 5K | ||||||||
| fr | 2K | 4K | 6K | 4K | 5K | 5K | 7K | 4K | 7K | 4K | 5K | 5K | 6K | |||||||||
| hr | 2K | 2K | 2K | 2K | 2K | 2K | 2K | 2K | 2K | 2K | 2K | 2K | ||||||||||
| hu | 4K | 4K | 4K | 4K | 4K | 4K | 4K | 4K | 4K | 4K | 4K | |||||||||||
| it | 4K | 4K | 5K | 7K | 4K | 6K | 4K | 4K | 5K | 6K | ||||||||||||
| lt | 4K | 4K | 4K | 4K | 4K | 3K | 4K | 4K | 4K | |||||||||||||
| lv | 4K | 5K | 4K | 5K | 4K | 4K | 4K | 4K | ||||||||||||||
| mt | 5K | 4K | 5K | 4K | 4K | 5K | 5K | |||||||||||||||
| nl | 4K | 6K | 4K | 5K | 5K | 6K | ||||||||||||||||
| pl | 4K | 4K | 4K | 4K | 4K | |||||||||||||||||
| pt | 4K | 5K | 5K | 6K | ||||||||||||||||||
| ro | 4K | 4K | 4K | |||||||||||||||||||
| sk | 5K | 5K | ||||||||||||||||||||
| sl | 5K |
Tilde MODEL - EMA is a corpus compiled from texts of European Medicines Agency document portal as available in http://www.ema.europa.eu/ at the end of 2016.
Languages: 25
Language pairs: 72
| bg | cs | da | de | el | en | es | et | fi | fr | hr | hu | is | it | lt | lv | mt | nl | no | pl | pt | ro | sk | sl | sv | |
| de | 490K | 473K | 463K | 408K | 348K | 463K | 470K | 464K | 471K | 416K | 473K | 399K | 467K | 465K | 473K | 494K | 464K | 331K | 391K | 469K | 465K | 467K | 452K | 456K | |
| en | 509K | 495K | 482K | 348K | 427K | 493K | 495K | 481K | 496K | 443K | 492K | 420K | 497K | 488K | 500K | 526K | 488K | 347K | 411K | 506K | 497K | 496K | 479K | 484K | |
| fr | 531K | 519K | 499K | 471K | 451K | 496K | 519K | 507K | 495K | 464K | 515K | 440K | 524K | 508K | 519K | 550K | 514K | 364K | 433K | 527K | 520K | 518K | 502K | 506K |
Tilde MODEL - World Bank is a corpus compiled from texts of World Bank as available in http://www.worldbank.org/ in 2017.
Languages: 12
Language pairs: 11
| es | fr | hr | pl | pt | ro | ru | sq | sr | tr | uk | |
| en | 54K | 57K | 2K | 207 | 13K | 6K | 26K | 3K | 2K | 2K | 2K |
Tilde MODEL - AirBaltic.com Travel Destinations is a multilingual parallel corpus compiled from description texts of AirBaltic.com travel destinations as available in https://www.airbaltic.com/en/destinations/ in 2017.
Languages: 7
Language pairs: 21
| en | et | fi | lt | lv | ru | |
| de | 839 | 718 | 775 | 672 | 644 | 677 |
| en | 1K | 828 | 1K | 1K | 1K | |
| et | 864 | 922 | 924 | 967 | ||
| fi | 792 | 767 | 780 | |||
| lt | 902 | 878 | ||||
| lv | 880 |
Tilde MODEL - LiveRiga.com is a multilingual parallel corpus compiled from Riga tourist attractions description texts of http://liveriga.com/ web site in 2017.
Languages: 4
Language pairs: 6
| en | lv | ru | |
| de | 839 | 644 | 677 |
| en | 1K | 1K | |
| lv | 880 |
Tilde MODEL - Lithuanian National Philharmonic Society is a parallel corpus compiled from texts of Lithuanian National Philharmonic Society web site http://www.filharmonija.lt/ in 2017.
Languages: 2
Language pairs: 1
| lt | |
| en | 9K |
Tilde MODEL - mupa.hu is a parallel corpus from texts of Müpa Budapest - web site of Hungarian national culture house and concert venue https://www.mupa.hu/en/ compiled in spring of 2017.
Languages: 2
Language pairs: 1
| hu | |
| en | 16.5K |
Tilde MODEL - fold.lv is a parallel corpus from texts of fold.lv portal http://www.fold.lv/en/ of the best of Latvian and foreign creative industries as compiled in spring of 2017.
Languages: 2
Language pairs: 1
| lv | |
| en | 10.7K |
Tilde MODEL - czechtourism.com is a multilingual parallel corpus from texts of http://czechtourism.com/ portal compiled in spring of 2017.
Languages: 8
Language pairs: 28
| en | es | fr | it | pl | pt | ru | |
| de | 7K | 6K | 5K | 6K | 784 | 6K | 6K |
| en | 7K | 7K | 7K | 866 | 7K | 7K | |
| es | 7K | 7K | 845 | 7K | 7K | ||
| fr | 7K | 797 | 7K | 7K | |||
| it | 814 | 7K | 7K | ||||
| pl | 764 | 1K | |||||
| pt | 7K |
