The addition of controlled terms from linked open datasets and vocabularies to metadata can increase the discoverability and accessibility of digital collections. However, the task of semantic enrichment requires a lot of efort and resources that cultural heritage organizations often lack. State-of-the-art AI technologies can be employed to analyse textual metadata and match it with external semantic resources. Depending on the data characteristics and the objective of the enrichment, diferent approaches may need to be combined to achieve high-quality results. What is more, human inspection and validation of the automatic annotations should be an integral part of the overall enrichment methodology. In the current paper, we present a methodology and supporting digital platform, which combines a suite of automatic annotation tools with human validation for the enrichment of cultural heritage metadata within the European data space for cultural heritage. The methodology and platform have been applied and evaluated on a set of datasets on crafts heritage, leading to the publication of more than 133K enriched records to the Europeana platform. A statistical analysis of the achieved results is performed, which allows us to draw some interesting insights as to the appropriateness of annotation approaches in diferent contexts. The process also led to the creation of an openly available annotated dataset, which can be useful for the in-domain adaptation of ML-based enrichment tools.
Semantic enrichment is the process of adding new semantics to unstructured data, such as free text, so that machines can make sense of it and build connections to it. In the case of the metadata that describes Cultural Heritage (CH) items, unstructured data comes in the form of free text that details several aspects of the item, for example its main characteristics, its location, creator, etc. Through the process of semantic enrichment, those textual descriptions are analyzed and augmented with controlled terms from Linked Open datasets, such as Wikidata1 and Geonames2, or controlled vocabularies, such as the Getty Art & Architecture Thesaurus3 (AAT). Those terms represent concepts and attributes (e.g. “costume”, “Renaissance”, colors), named entities, such as persons, locations, and organisations, or chronological periods. For example, the strings “Leonardo da Vinci” and “da Vinci, Leonardo” can be both linked to the Wikidata term representing the Italian Renaissance polymath. This additional piece of information associated with a CH resource is commonly referred to as an annotation, which links the CH object with some URI (Unique Reference Identifier) derived from vocabularies or open data sources.
Semantic enrichment adds meaning and context to digital collections and makes them more
easily discoverable. Given its importance, it has been a main concern and focus of eforts by
the Europeana digital library4 as well as individual data aggregators and providers. Firstly,
linked data makes the meaning of textual metadata unambiguous [
Semantic enrichment is a labour-intensive process, which requires efort and resources that CH institutions often lack. State-of-the-art AI technologies can be employed to automate the time-consuming and often mundane process of manual metadata enrichment. Natural language processing (NLP) tools can be used to analyse textual metadata and detect and classify concepts or named entities mentioned in unstructured text. Machine Learning (ML) approaches are extensively used for the task of disambiguation, which is responsible for deciding if the reference to ‘Leonardo da Vinci’ in the text refers to the Italian polymath or to the battleship. However, the accuracy of the automatic results highly hinges on the specific task at hand visa-vis the algorithm applied. For example, short textual descriptions, which are common in CH metadata, lack context and thus ML algorithms trained on Wikipedia articles may result in many incorrect matches. For similar reasons, they may often miss domain-specific matches that are relevant in the specific CH context. What’s more, even if the automatically detected links are correct, they may be considered undesirable for a certain case study. For example, linking metadata records with terms representing colours may be important for a fashion collection, but it may be undesirable for describing a manuscript that happens to mention a certain colour.
As a result, depending on a number of factors, such as the text characteristics (e.g. its length
and language), the vocabulary that we wish to link it to, and the type of entities to detect (e.g. do
we wish to identify a broad variety of concepts or to limit ourselves to certain domain-specific
terms?), a diferent combination of tools and steps is required to achieve the best possible
results for each specific task. For example, for certain tasks with a well-defined restricted context,
a simple lemmatisation and string matching approach may be more appropriate than complex
ML-based algorithms. Besides the need for flexibility in combining and experimenting with
different approaches and tools, another crucial aspect that needs to be considered is the need to
make human inspection and validation an integral part of the end-to-end semantic enrichment
workflow [
To address the aforementioned challenges, in this paper, we define, implement, and test a methodology and associated digital platform, called SAGE5, which combines automatic annotation tools with human validation for the enrichment of CH items at scale. SAGE is an open source tool6 that streamlines and facilitates the whole workflow of semantic enrichment, from data import and the automatic production of semantic annotations to human validation and data publication. The platform has been configured to serve the needs of the cultural sector and supports seamless interoperability with the common European data space for CH7 and in particular with Europeana.
The methodology and platform have been applied to enrich the metadata records from datasets on various aspects of crafts heritage (from furniture to jewelry and costumes to clocks) coming from 8 diferent CH organisations, including the Fashion Museum Antwerp, the Netherlands Institute for Sound and Vision, the Open University of the Netherlands, the Greek National Documentation Centre, the Museum of Arts and Crafts in Zagreb, the Palais Galliera and Mobilier National in Paris, and the Textile Museum of Prato. The rest of the paper is structured as follows. After discussing related work, we present the steps of the methodology to semantic enrichment that we followed along with the technical architecture and the supporting SAGE platform, the evaluation performed and the results achieved. Finally, we conclude the paper with some general lessons learned.
State-of-the-art Natural Language Processing and Machine Learning technologies have been
extensively used in the CH domain to analyze unstructured text and extract structured
information from it. To achieve automated subject indexing, Annif [22] is an open-source multilingual
toolkit by the National Library of Finland that automatically assigns documents with subjects
from a controlled vocabulary. In [
Among information retrieval approaches, there have been several attempts to apply Named
Entity Recognition (NER) as well as Disambiguation (NED) in the CH and digital humanities
sectors, considering diferent types of data. In [
It should also be noted that NERD tools are trained on generic corpora [
As the uptake of AI tools is expanding, there is increasing need for validation and
moderation by humans to overcome the errors of the machine and achieve higher quality results [
Overall, our work distinguishes itself from previous work on semantic enrichment mainly in that it is based on a generic data management approach, which allows the combination of various annotation tools with flexible parameterisation capabilities (such as the definition of string matching and filtering rules); in that it includes human validation as an integral part of its workflow; and that it supports integrations with other CH-specific data representations and platforms, making it readily reusable in the CH data space. It should be noted that the integration with external annotation tools and CH-related platforms is loosely coupled, via interactions with the APIs (Application Programming Interface) and SPARQL endpoints exposed by the third-party components.
The methodology we followed for the semantic enrichment of CH metadata consists of the following high-level steps: 1. A: Data aggregation and requirements analysis
The first step concerns the preparatory tasks of aggregating the data and specifying the requirements for the the enrichment (e.g. which metadata fields to analyse, which vocabularies to link to etc). 2. B: Automatic metadata enrichment
The second step involves the automatic analysis of the textual metadata, with the aim to derive useful annotations in line with the identified requirements. 3. C: Human validation
Humans are solicited to review and validate the automatically generated annotations as well as to manually add new annotations, that the automatic algorithm has not been able to detect. 4. D: Filtering and data publication
The outcomes of the human validation are analysed to establish appropriate thresholds for filtering and the filtered annotations are embedded as enrichments to the metadata records. The enriched metadata records are ultimately published to the Europeana platform.
Figure 1 provides an overview of the main digital components that support the above methodology.
Publish enriched
datasets
New records
Data Aggregation, Mapping, and
Publication
MINT is a metadata management tool8 that is part of the data space for CH and is used by several aggregators to prepare and publish their data to Europeana. It acts as the link between
Source records
EDM records Filtered annotations
Automatic annotations
Feedback Semantic analysis and enrichment
Human Validation
Vocabularies
Knowledge
Bases API
Human intelligence
SAGE and Europeana and supports steps A and D of the aforementioned methodology, serving
the following purposes: (i) aggregate the metadata records from the data providers and for
mapping them to the Europeana Data Model (EDM) [
SAGE is a web-based platform for generating, enriching, validating, publishing, and searching RDF data. In the context of our methodology, it is responsible for the core steps B and C. The RDF data can be produced from heterogeneous data sources and data formats using the D2RML mapping language [5], and enriched using annotators that wrap web-based or other third party services. The enrichments can then be manually validated, and finally, the entire data can be published in an RDF store and indexed. The SAGE platform has been configured to facilitate the semantic enrichment of CH metadata. In this respect, it ofers a suite of already set-up annotators, i.e. parameterisable enrichment templates, that are connected with relevant in-domain vocabularies and knowledge bases. It also facilitates the direct import/publication of metadata from/to platforms of the European data space for CH, including Europeana and MINT, making use of established APIs and formats .
A dataset is annotated per property, i.e. the user can select from the schema preview a property that links entities to values, and execute an annotator on the values of that property. An annotator in SAGE is a mediator that retrieves all desired values from the triple store where the dataset content is published, generates the appropriate calls to the web or other service, and transforms the results to the RDF annotation specification. As in the case of datasets, the results of an annotator execution are Terse RDF Triple Language11 files stored in the file system of SAGE. In the framework of the data space of CH, annotations are also expressed in a JSONLD equivalent representation model12, which bases on the W3C’s Web Annotation Model13 supported by Europeana. The annotation model is generic enough to accommodate for various enrichment types (e.g. annotations resulting from automatic translation tools, from image analysis etc) and provides sufÏcient provenance information, including information about the annotations’ confidence scores and the validation feedback provided by humans. For metadata records that are compatible with EDM, the annotations are ultimately embedded in the metadata in line with the EDM extension that instructs the representation of metadata statements resulting from semantic enrichment14. This way, the enrichments can be appropriately 9https://pro.europeana.eu/files/Europeana_Professional/Share_your_data/Technical_requirements/EDM_profiles/ EDM_provenance_profile_external_202111.pdf 10https://www.europeana.eu/en/apis 11https://www.w3.org/TR/turtle/ 12https://docs.google.com/document/d/1Cq1Qqx0ji7Vw8iwLVis1CfpYKtv-72ojkcvjnQzrKjs/edit?usp=sharing 13https://www.w3.org/TR/annotation-model/ 14https://pro.europeana.eu/files/Europeana_Professional/Share_your_data/Technical_requirements/EDM_profile s/EDM_provenance_profile_external_202111.pdf handled and presented to the end-user by the Europeana platform.
SAGE supports three main types of annotators, which can be parameterised with respect to diferent aspects (e.g. vocabulary, language, preprocessing functions etc) to serve diferent case studies: • Thesaurus annotators: They link texts to URIs from thesauri that can be imported to the platform by performing smart string matching on the thesaurus labels using lemmatizers (such as the ones provided by the Stanza library15) and other functions to produce improved results (e.g. apply dedicated Regex rules). They are appropriate for application both on generic textual fields and on focused short fields. By selecting thesauri that represent concepts referring to specific domains (e.g. fashion), it is more likely that the extracted terms are relevant to the object in question. Moreover, such annotators can perform massive enrichments in a very short time compared to the other annotators, since they rely on locally stored data. Figure 3 provides an overview of how a Thesaurus Annotator works on a specific example. • Generic NERD annotators: They employ pre-trained NERD tools to detect named entities and link them to respective entities from Wikidata. SAGE supports two diferent pipelines for generic NERD. The first pipeline makes use of the AIDA tool [ 18] for entity detection and disambiguation. The second pipeline makes use of the spaCy library16 for performing the NER part for diferent languages, i.e. for recognising entities and their string boundaries within a sentence, and then of the multilingual mGENRE model [6] for the disambiguation stage and for linking with a URI from Wikidata. Such annotators can be used as they are, with minimal or no configurations and are appropriate for general-purpose enrichments. They conduct disambiguation by using the context contained in longer texts (e.g. description), since they are trained on textual corpora such as Wikipedia articles. At the same time, this process is more likely than the other annotators to link with terms that are too generic or irrelevant in the context of a specific case study, while it is hard to infer with sufÏcient accuracy the type of the extracted entity and its relation to the object in question (e.g. whether it represents the item’s creator, a place of display etc). As a result, in practice, they often produce more accurate results when applied in fields with pre-specified focused semantics. • SPARQL Annotators: SPARQL annotators communicate with external knowledge bases (such as Wikidata and Geonames) through SPARQL endpoints. Thus, they are the best ift when dealing with large knowledge bases that cannot be downloaded locally.They can be applied on focused fields that refer to a single entity. The values of such fields often follow certain patterns (e.g. “surname, name”, “city/region/country” etc) and, thus, pre-processing with Regex is key to the success of the method, so that a normal form of the entity name can be extracted. An example of a query that matches Wikidata entities with the occupation of a painter is presented in Figure 2. 15https://stanfordnlp.github.io/stanza/ 16https://spacy.io/
Human validation was conducted via a dedicated environment provided by SAGE (see Figure 4). Humans are invited to inspect the automatic annotations produced by the AI tools and accept or reject them. Moreover, they can add missed annotations, i.e. relevant annotations that the automatic algorithm failed to identify. During the validation of the results of the semantic analysis, validators are also able to edit the predefined target metadata field in which the URI will end up. It should be noted that SAGE groups together annotations repeated across many records in a dataset and flags annotations referring to URIs that are already included in the metadata. In total, 14 CH professionals with specialized knowledge about the considered collections participated in the validation process, with two to three validators per collection. Participants were instructed to accept or reject annotations based on what they consider as desirable for inclusion in the final metadata. That is, they evaluated not only whether an annotation is a correct match but also in terms of relevance (e.g. matches with the term ”human” may be considered too generic) .
The appropriate size and characteristics of the sample to be validated depend on the available resources that can be invested in the validation process and the nature of the use case. What is considered a ”sufÏcient” amount hinges on many factors, including the total number of automatically produced annotations, their characteristics (e.g. what metadata fields they refer to, their granularity, etc), the characteristics of the automated algorithm that produced them (e.g. its accuracy, the reliability of the automatic confidence scores it assigned to them), the number of participants and the amount of time they can devote to the task. The following criteria were used to guide the selection of the annotations sample to be validated, so as to ensure representativeness across various parameters: • Inspect annotations that appear in a high number of records and thus will have a high impact. • Ensure a balanced representation of metadata fields, including fields with varying semantics and expected text length. • Take into consideration automatic confidence levels assigned by automatic algorithms, if available: inspect annotations with a rather low confidence score but also a sufÏcient number of annotations with a rather high one.
Validation feedback was analysed with the aim of establishing thresholds for annotations that are considered acceptable for publication. To this end, the following metrics have been calculated per dataset, per Annotator, and per analysed metadata field: • Precision considering only unique annotations, that is, unique triples of field textual values, matched sub-string, and identified URI. • Precision considering all annotations, that is, without grouping together identical field textual values (in other words, counting all times the same annotation, defined as a triple, may appear in diferent items).
In both cases, precision was calculated as /( + ) , where = and = . Precision was used as a threshold for filtering out not-reviewed annotations on a field or Annotator basis. What is considered a sufÏciently high precision depends on the requirements of each case study and the expectations of the data provider.
For the use cases we considered, most human experts did not focus on the manual insertion of new annotations and the few manually added annotations we collected do not allow us to sufÏciently estimate false negatives and thus compute recall and the F-score. It should also be noted that for publication to Europeana, a threshold based on precision is considered the most appropriate metric to be used17.
Human judgments can also be used as a means to assess the trustworthiness of the automatic
confidence scores assigned by the AI algorithms. For example, if humans tend to accept all
sample annotations above a certain score, then we may conclude that all annotations above that
score can be regarded as acceptable. In this vein, we explored whether there is a correlation
between the automatic confidence scores, when available, and human judgments. We therefore
plotted the logistic regression between the two variables considering the following metrics:
• The − [
The aforementioned methodology and supporting tools have been applied to metadata records describing crafts heritage items as mentioned in Section 1. The analysed metadata comes in the following languages: Dutch, Italian, French, Greek, English and Croatian. In total, the SAGE annotators were applied on 216, 115 metadata records, giving rise to 915, 472 total annotations and 549, 402 unique annotations. It should be noted that numbers calculated based on unique annotations are considered more reliable since numbers that count in item impact are skewed towards textual values that are repeated in multiple items. In total, 12 experts from 8 CH organisations took part in the validation campaigns. Overall, 30, 910 unique annotations referring to 17https://pro.europeana.eu/post/methodology-for-validating-enrichments more than 15K records were reviewed via SAGE (i.e. 5.6% of the automatically produced unique annotations), with the sample being selected following the criteria outlined in Section 3.2. Of those annotations, 23, 426 were accepted and 7, 474 were rejected.
The overall precision, defined as the number of all accepted automatic annotations produced by SAGE over the number of reviewed automatic annotations, is 0.76, considering unique annotations. If all annotations are counted in, then the overall precision is 0.82. Precision varied largely depending on the analysed metadata field, the type of annotator that was used, and the datasets that were analysed. Table 1 provides an overview of the results achieved by diferent annotators. The minimum and maximum precision reported in the table refer to a per metadata ifeld level.
The choice of the vocabulary used by the thesaurus annotators depended on the respective dataset characteristics and providers’ objectives. The following vocabularies were used: the Europeana fashion thesaurus18; AAT; the EUScreen vocabulary on audiovisual heritage19; and a SKOS vocabulary on Greek crafts heritage 20. Thesaurus annotators were applied to both longer (e.g. dc:description, dc:title) and shorter fields (e.g. dc:format, dc:type), often after case-appropriate regex pre-processing, giving rise to generally satisfactory results in both cases. SPARQL queries on Wikidata were used to retrieve creators for the dc:creator and locations for the dc:spatial fields. Although in most cases it did not produce a high number of annotations, it scored a high precision. mGENRE and AIDA were applied to dc:description and dc:title fields as well as shorter fields (including dc:creator, dc:spatial, and dc:rights). They both produced similar results, performing well for short fields but poorly for longer ones. In the latter case, they both struggled with disambiguation between multiple candidate entities and, even when producing matches that were in principle correct, those were often too generic and considered irrelevant by validators.
For annotators for which an automatic score was produced, we also attempted to plot the
logistic regression between the automatic score and the human judgments. However, no
correlation was found between the two variables and therefore automatic scores were not used
as factors in the filtering rules. A possible explanation for this is that in the case of thesauri
annotators, scores are usually quite high for all annotations: they reflect the string diference
(1-Levenshtein distance [
Annotations have been filtered by discarding all annotations rejected by humans, while including all explicitly accepted ones. (considering a majority vote). For non-reviewed annotations, a threshold based on precision between 0.75 and 0.8 (considering unique annotations) was considered acceptable by data providers. In total, 549.460 have been regarded acceptable, leading to the enrichment of 133.405 out of 216.115 analysed records. All enriched records have been published to Europeana. Enrichments have been indexed to become searchable and are visible as part of the item view via distinct tags, thus contributing to making the respective items more discoverable, contextual, and multilingual. Figure 5 shows an example of how automatic annotations look like on the Europeana platform.
Although domain adaptation is beyond the scope of the current case study, the dataset that
resulted from the validation process can be valuable for the training and fine-tuning of NERD
tools in the field of CH. To this end, a curated selection of annotated metadata enriched and
validated via SAGE has been made openly available21 under a CC0 license, so that it can be
freely reused as data amenable for computational purposes. The dataset includes more than
10K unique annotations (pairs of analysed textual values and URIs). The in-domain adaptation
of NERD tools so that they can more efectively deal with the particular characteristics of CH
metadata [
In the current paper, we present a generic and reusable methodology and supporting digital platform that combines automatic annotators with human expertise in order to enrich them with terms from various linked data sources. The methodology has been applied and evaluated on a case study involving crafts heritage datasets, leading to measurable improvements in the quality of metadata and enhancing the discoverability and usability of the respective resources on Europeana. Building on the practical experience we gained, the current case study allows us to draw some lessons learned, which can prove useful for interested stakeholders who may wish to follow a similar process to enrich their datasets.
Before proceeding to the actual enrichment, it is crucial to scrutinise the data to be analysed, gain a deep understanding of its characteristics and define feasible and meaningful enrichment objectives. One should define the expected benefit of possible enrichments and how they will bring value to the collection. In this respect, one should ask questions such as: What kind of concepts are useful to detect (e.g. persons, locations, domain-specific concepts etc)? Which metadata fields contain relevant information (e.g. descriptions make frequent references to techniques and materials used)? In what languages are the metadata? It should also be noted that the quality of the original metadata afects the quality of the automatic enrichment. If the text contains many typos or is misaligned with the intended semantics of the respective metadata field, then the outputs of the automatic enrichment tools will be less accurate. This step is also crucial for detecting patterns in data that can be exploited in order to produce annotations.
The next step involves the selection and set-up of the semantic annotators that are most appropriate for the specific use case, considering the advantages and disadvantages of each approach as presented in Section 3.1. The selection of knowledge bases and vocabularies that have the case-appropriate granularity and coverage is crucial. Generally, the more focused the automatic enrichment, considering the terminology used (e.g. link with a domain-specific vocabulary versus general-purpose NERD) and the metadata property that is parsed (e.g. topicspecific fields such as dc:creator versus longer ones such as dc:description), the less the risk of producing too many irrelevant or too generic enrichments and the more accurate the resolution of disambiguation. One should opt for knowledge bases that are accessible on the Web via an open license, well-documented, and compliant with Linked Data best practices. Their multilingual coverage (also in relation to the language of your metadata) is also an important aspect that should be taken into consideration.
After the production of the automatic annotations, the validation process should be carefully 21See https://github.com/ails-lab/ai4culture-datasets for the actual dataset and the process that was used for the data curation. organised. The background of the validators is crucial: some tasks may require expert skills (e.g., knowledge of a particular language, domain expertise etc.), while others can be performed by appealing to a general audience. In the former case, it is wiser to keep the validation process closed within a team of experts, while in the latter, organizing an open crowdsourcing campaign will mobilize more people and thus speed up the process. The selection of the sample to be validated is crucial: it does not need to be large but it should be well-balanced, following the criteria outlined in Section 3.2.
The final step involves the filtering of the automatic validation in light of the acquired human feedback. For annotations reviewed by humans, majority vote can typically be used to determine acceptability. Depending on the annotation type, additional criteria might be enforced (e.g. for public validation campaigns where untrustworthy feedback is suspected, we may require that an annotation is reviewed by multiple users). Automatic annotations that have not been reviewed by humans or lack a reliable confidence score should be filtered using automatic evaluation metrics. The appropriate metrics depend on the nature of the task, but precision is a typical one when correctness is at high stake. Thresholds should be established depending on what is considered acceptable given the specific use case requirements.
The work is co-funded by the European Union, under the projects “CRAFTED: Enrich and promote traditional and contemporary crafts” and “AI4Culture: An AI platform for the cultural heritage data space”. We would like to thank all partners of the CRAFTED project, and particularly Panagiotis Tzortzis, for their valuable contributions to this work.
A. Chortaras and G. Stamou. “D2RML: Integrating Heterogeneous Data and Web Services into Custom RDF Graphs”. In: Workshop on Linked Data on the Web co-located with The Web Conference. Vol. 2073. CEUR Workshop Proceedings. 2018.
N. De Cao, L. Wu, K. Popat, M. Artetxe, N. Goyal, M. Plekhanov, L. Zettlemoyer, N. Cancedda, S. Riedel, and F. Petroni. “Multilingual Autoregressive Entity Linking”. In: Transactions of the Association for Computational Linguistics 10 (2022), pp. 274–290.