Europeana.eu aggregates metadata describing more than 55 million cultural heritage objects from libraries, museums, archives and audiovisual archives across Europe. Quality (and particularly multilingual quality) of metadata is crucial for enabling search and data re-use in digital libraries. Capturing multilingual aspects of the data requires us to take into account the full lifecycle of data aggregation including data enhancement processes such as data enrichment. Multilinguality cannot be captured as one measure, but needs to be considered as an intersection of several measures, bringing challenges when interpreting and visualising the results. The paper presents an approach for capturing multilinguality as part of data quality dimensions, namely completeness, consistency and accessibility. We describe the measures de ned and implemented, and provide initial interpretations of the results.
Europeana.eu6 is Europe's digital platform for cultural heritage. It aggregates metadata describing more than 55 million cultural heritage objects from a wide variety of institutions (libraries, museums, archives and audiovisual archives) across Europe.
The need for high-quality metadata is particularly motivated by its impact on search, the overall Europeana user experience, and data re-use in other contexts such as the creative industries, education and research. One of the key goals of Europeana is to enable users to nd the cultural heritage objects that are relevant to their information needs irrespective of their national or institutional origin and the material's metadata language.
As highlighted in the White Paper on Best Practices for Multilingual Access
to Digital Libraries [
In the Data Quality Committee (DQC), a body formed at Europeana's
initiative, we specify functional requirements that de ne the purpose of the metadata
and guide data-quality evaluation { following a core principle for metadata
assessment (e.g. [
In the next section, we present data quality frameworks, dimensions and criteria that are commonly referred to in the context of data quality measurement. Section 3 describes how multilingual data is presented in Europeana's data model and the functional requirements that are the basis of our assessment. The data quality dimensions we use will also be presented. In section 4, we describe the implementation of the di erent measures. Section 5 describes rst results and measures that were taken to improve data along the di erent quality dimensions. We conclude this paper with an outline of future work. 2
Addressing data quality requires the identi cation of the data features that need
to be improved and this is closely linked to the purpose the metadata is serving.
Libraries have always highlighted that bibliographic metadata enables users to
nd material, to identify an item and to select and obtain an entity [
Bruce and Hillmann [
rative study of four collections, which included metadata from several institutions. Not surprisingly, they found greater variances for collections aggregated from multiple sources as well as inconsistencies in the encoding scheme { a result also experienced by Europeana.
Designing an information quality framework, Stvilia et al. [
Zaveri et al. [
Mader et al. [
It becomes apparent that while multilinguality is considered in some works,
it is usually not treated as a separate quality dimension, but rather as part of
other criteria or dimensions existing at quite di erent levels in di erent quality
models. To the best of the authors' knowledge, the only resource which measured
multilingual features in metadata is [
Firstly, to determine the multilingual degree of metadata across several quality dimensions we have to understand the di erent ways multilingual information is expressed in Europeana's data model. Secondly, multilingual information is at the core of the functional requirements that constitute the services Europeana is providing for an international audience. Thirdly, these requirements and the structure of multilingual data inform the criteria and metrics that enable us to measure multilinguality across several metadata quality dimensions. 3.1
Multilinguality in Europeana's metadata has two perspectives: concerning the language of the object itself, and the language of the metadata that describes this object. First, the described cultural object, insofar as it is textual, audiovisual or in any other way a linguistic artefact, has a language. The data providers are urged to indicate the language of the object in the dc:language eld in the Europeana Data Model (EDM) in this way: <dc:language>de</dc:language>. This information is then used to populate a language facet allowing users to lter result-sets by language of objects. The language information is essential for users who want to use objects in their preferred language. Second, the language of metadata is essential for retrieving items and determining their relevance. Metadata descriptions are textual and therefore have a language. Each value in the metadata elds can be provided with a language tag (or language attribute). Ideally, the language is known and indicated by this tag for every literal in each eld. If several language tags in di erent languages exist, the multilingual value can be considered to be higher. For instance, consider as an example this data provided by an institution: <#example> a ore:Proxy ; # data from provider dc:subject \Ballet", # literal dc:subject \Opera"@en # literal with language tag
The rst dc:subject statement is without language information, whereas the second tells us that the literal is in English. Europeana assesses data in particular elds to enrich it automatically with controlled and multilingual vocabularies as de ned in the Europeana Semantic Enrichment Framework.8 As shown in the following example, the deferencing of the link (i.e., retrieving all the multilingual data attached to concepts de ned in a linked data service) allows Europeana to add the language variants for this particular keyword to its search index. <#example> a ore:Proxy ; edm:europeanaProxy true ; # enrichment by Euroepana with multilingual vocabulary dc:subject <http://data.europeana.eu/concept/base/264> <http://data.europeana.eu/concept/base/264> a skos:Concept .
# language variants are added to index skos:prefLabel "Ballett"@no, "Ballett"@de, "Bale"@pt,
"Baletas"@lt, "Balet"@hr, "Balets"@lv
The record now has more multilingual information than at the time of ingestion into Europeana. The di erent language versions from multilingual vocabularies are likely to be translation variants. 8 https://docs.google.com/document/d/1JvjrWMTpMIH7WnuieNqcT0zpJAXUPo6x4uMBj1pEx0Y/ edit 3.2
In order to better understand the impact of multilinguality on the overall quality of the data, we have identi ed the multilingual aspects of several functional requirements relevant to Europeana. Note that these requirements are also crucial to better communicating the results of our measures to our data providers, and hence to motivating improvement of the data.
Cross-language recall: When a user wants to search for a document or documents relevant to her needs, and to do so irrespective of the language of the documents and/or their metadata, multilingual metadata is a necessity. It is therefore required that all searchable freetext metadata elements9 be consistently tagged with their language.10 Language based facets: In a language-based facet requirement a clear distinction needs to be made regarding whether the language refers to the language of the metadata or the digital representation of the work. The method of measuring the multilingual saturation is di erent in both cases. While the language of the metadata will be measured against the presence of language tags, the language of the content will be captured in the dc:language eld in EDM. The existence of multilingual metadata records in Europeana demands that language-tagging occur at the level of the metadata element rather than the record as a whole. Entity-based facets (people, places, concepts, periods): while users will be primarily searching for entities in their own language (the city of \Paris"), they will need to access results in di erent languages (Parijs, Parigi, ...). To support this scenario we will rely on the translated labels provided by data providers as part of their data or on multilingual vocabularies to which the data can be linked. In both cases measuring multilinguality is crucial to identify language gaps.
It is also important to determine the impact of work ows contributing to the increase of multilingual labels (such as machine learning and natural language processing techniques for language detection, automatic tagging, or semantic enrichment) in the data in order to de ne the measures but also to interpret the results. 3.3
For measuring multilinguality, we look at three quality dimensions: completeness, consistency and accessibility. Each of these dimensions assesses multilinguality from a di erent perspective. The results re ect how well the functional requirements work. 9 https://docs.google.com/spreadsheets/d/1f6vbeo4mt0stl0yAbVCyp_ -5bzBBCq4Dy6p7xhMcjSI/ 10 for instance, in accordance with the ISO 639-1 or 632-2 (https://www.loc. gov/standards/iso639-2/php/code_list.php) list, the IANA language tag registry (http://www.iana.org/assignments/language-subtag-registry/ language-subtag-registry), or the Languages Name Authority List (NAL) (https://open-data.europa.eu/en/data/dataset/language) Completeness. Completeness is a basic quality measure, expressing the number (fraction) of elds present in a dataset, and identifying non-empty values in a record or (sub-)collection. For a xed set of elds completeness is thus straightforward to measure, and can be expressed as the absolute number or fraction of the elds present and not empty. However, the measure becomes non-trivial when data is represented using a data model with optional elds (that may e.g., only be applicable for certain types of objects), or with certain elds for which the cardinality is unlimited (e.g., allowing zero to many subjects or keywords). These characteristics apply to EDM. In such cases the measure becomes unbounded, and a few elds with high cardinality may outweigh or swamp other elds.
In the context of measuring multilingual completeness, the metric is two-fold. First, the concept of completeness can be applied to measuring the presence of elds with language tags or, second, the presence of the dc:language eld. Any measure of multilinguality must be seen in relation to the results of measuring completeness. Only elds both present and non-empty can be said to have or lack language tags and translations. A record which is 80% complete can still reach 100% multilingual completeness if all present and non-empty elds have a language tag. The completeness a ects all three functional requirements as they all work better the more multilingual information exists in a given set of elds. Consistency. Consistency describes the formal conformity to set standards and eld rules as well as the logical coherence of the metadata. With regard to multilinguality, the dimension assesses the variety of language values in the dc:language eld. The goal is to de ne a standard for language notations and normalize the eld in this regard (see section 5.2). The consistency measure is mainly relevant for the language based facet. The better normalization works, the better the user can lter objects by their language.
Accessibility. Accessibility describes the degree to which multilingual information is present in the data, and allows us to understand how easy or hard it is for users with di erent language backgrounds to access information. So far, Europeana has little knowledge about the distribution of linguistic information in its metadata { especially within single records. To quantify the multilingual degree of data and measure cross-lingual accessibility, the language tag is crucial. Resulting metrics can be scaled to the eld, record and collection levels. In practical terms, the accessibility measure serve to gauge cross-language recall and entity-based facet performance.
To summarize: with regard to multilinguality, we identi ed the following dimensions and quality criteria (Table 1 3.3). 4
Implementation requires a good understanding of EDM in order to not exclude
relevant data. The source data go through several levels of data aggregation
before being aggregated into Europeana. Data processes can take place at each
of theses levels and a ect the nal data output. EDM allows us to distinguish
between values provided by the data provider(s) from the information added
by Europeana (for instance by semantic enrichment), doing so by leveraging the
proxy mechanism from the Object Re-use and Exchange (ORE) model. It enables
the representation of resources in the context of aggregations o ering di erent
views on the same resource [
The di erent metrics for the assessment of multilinguality in metadata are implemented in the Data Quality Assurance framework.11 The measurement has four phases. (1) data collection and preparation: the EDM records were collected via Europeana's OAI-PMH service12, transformed to JSON where each record was stored in a separate line, and stored in Hadoop Distributed File System13 (2) record-level measurement: the Java applications14 measuring di erent features of the records run as Apache Spark jobs, allowing them to scale readily. The process generates CSV les which record the results of the measurements such as the number of eld instances, or complex multilingual metrics. (3) statistical analysis: the CSV les are analyzed using statistical methods implemented in R and Scala. The purpose of this phase is to calculate statistical tendencies on the dataset level and create graphical representations (histograms, boxplots). The results are stored in JSON and PNG les. (4) user interface: interactive HTML and SVG representations of the results such as tables, heath maps, and spider charts. We use PHP, jQuery, d3.js and highchart.js to generate them. 11 http://144.76.218.178/europeana-qa/multilinguality.php?id=all 12 http://labs.europeana.eu/api/oai-pmh-introduction. Our client library: https://github.com/pkiraly/europeana-oai-pmh-client/. 13 Snapshot created at the end of 2015, consisting of more than 46 million records (392
GB disk space): http://hdl.handle.net/21.11101/0000-0001-781F-7. 14 Source code and binaries: http://pkiraly.github.io/about/#source-codes.
The status of these metrics with regard to interpreting their results and thus deriving strategies for metadata improvement di ers considerably. Whereas the measures for the quality dimensions of completeness and accessibility need more re nement, as well as more discussion of appropriate display of the results, the consistency metric has already served to drive a successful project for normalizing languages. These various developments and their outcomes are reported in this section. 5.1
The completeness measure indicated that 904 (out of 3548) collections have no value in the dc:language eld, which shows the eld is missing. On a record level, 58,03% of the records have a dc:language eld.15 Another pattern one can detect in the data relates to the misuse of elds. For example, the metric \cardinality" allowed us to identify collections that have metadata elds with more than 3 instances of dc:language. In one example, as many as 153 language values were found associated with this eld, owing to duplication of the language tag. One recommendation to avoid this in future would be to check the eld for duplicated entries and reduce them to a single language tag. 5.2
Measuring consistency in the dc:language eld shows that values are currently quite heterogeneous in the Europeana dataset. dc:language values are predominantly normalized in ISO-639-1 or ISO-639-3, but, in contrast, values nevertheless sometimes occur in natural language sentences that cannot be processed automatically. We also nd language ISO codes without their reference to the ISO standard in use, or references to languages by their name. Additionally, over 400 di erent language variants are present in the eld. This table presents some general statistics about the presence of ISO-639 codes in the values of dc:language in the Europeana dataset:
Total values in the Europeana dataset 33,070,941 Total values already normalized (ISO-639-1, 2 letter codes) 23,634,661 Total values already normalized (ISO-639-3, three letter codes) 4,831,534 The metric helps us to design further language normalization rules which in turn can be used to further improve the results of the quality measures. Any visualisation implementation of the quality measures would also bene t from such normalisation. We have developed a language-normalization operation 15 http://144.76.218.178/europeana-qa/frequency.php on the Europeana dataset which itself comprises a range of operations - i.e., cleaning, normalization and enrichment of data. The following exemplify some cases of the di erent types of operations:
Input value
Normalization output (ISO 639-1) \English" \eng" \English and Latin" Greek; Latin \en" \en" \en", \la" \el", \la"
The output of the normalization algorithm is a value from any of several authoritative language vocabulary. The algorithms work based on a core vocabulary, the Languages Name Authority List (NAL) published in the European Union Open Data Portal16. All normalization operations are internally performed using the core vocabulary and the alignements to other vocabularies it contains 17. The nal output can be given in accordance with any of the aligned vocabularies and can be con gured to use URIs from the NAL vocabulary or any of the aligned ISO code-sets. 5.3
As noted earlier, our approach to measuring multilingual saturation in metadata allows us not only to measure the data's quality as it is provided by contributing institutions, but also provides us with insight into the e ectiveness of Europeana's data enhancement processes, such as semantic enrichment. The numbers provided as part of the implementation of the quality measures are either not enough or sometimes too complex to interpret the results and draw conclusions. At the time of writing, we are in the process of re ning the saturation score analyzing rst results of the calculations. Visualisation is therefore a necessary tool to supplement the measure and guide the data providers in their data analysis. 6
In this paper, we presented our approach to assessing the multilingual quality of data in the context of Europeana. This approach takes into account the functional requirements identi ed for digital libraries and translates them into actionable measures. These measures look at the data dimensions of completeness, consistency, and accessibility. Finally, we also present the initial outcome of our quality measures, such as the de nition and implementation of normalisation rules for languages. The numbers provided as part of the quality measures 16 https://open-data.europa.eu/en/data/dataset/language 17 NAL is aligned with ISO 639-1 (currently in use at Europeana), ISO 639-2b, ISO 639-2t and ISO 639-3, providing human-readable labels in all European languages still need to be re ned. Future work on re ning visualisation reports will help us both to interpret the numbers and to adjust our measurements. For instance, in order to get a comprehensive view of the quality of a data, the di erent metrics will need to be presented together (e.g. multilinguality on top of completeness) so that the interrelation between the di erent metrics is made visible.