Much of people's understanding of their cultural heritage is facilitated by the curation and preservation of community-generated digital content (CGDC): archival collections that were created for, with and by local communities. However, communities employ their own conventions in storing and publishing their content. Given this and the fact that semantic information tends to be buried within textual descriptions, CGDC archives are currently siloed and obscured, thus making it dificult for end-users (e.g., members of the public and researchers) to search for fine-grained information (e.g., “Where did Alfred Edward Julian work?”). In this paper, we propose to represent the information within CGDC archives in the form of knowledge graphs. To enable the construction of such knowledge graphs at scale, we developed a zero-shot approach for relation extraction, which we cast as a natural language inference (NLI) problem. Specifically, for each of the 20 relation types drawn from Wikidata that we have identified as relevant to CGDC, we created a premise-hypothesis pair that is presented to an NLI model that determines whether entailment (and thus the relation type) holds. The premise is a sentence from the natural language description and the hypothesis is automatically generated using a template based on each of the relation types. We present the results of comparing and combining three diferent transformer-based models that were already fine-tuned for the NLI task, namely, DeBERTa, BART and T5.
Cultural heritage is preserved and passed on to future generations through archival collections and their
digitalisation. While enormous eforts (e.g., The National Archives of the UK, Europeana) have been
put into making such collections available to the public, a significant part of people’s cultural heritage
is represented only in community-generated digital content (CGDC): digital-born archive collections
developed for, with and by communities [
These communities tend to follow their own conventions to store and represent their CGDC
collections. As a result, most of the rich semantic information contained within these collections remain
buried within textual metadata, such as the titles and descriptions of CGDC items. The description
of a photograph of a local landmark, for instance, might mention the person or organisation who
was responsible for building that landmark—information that is potentially of interest to researchers
or members of the public and yet obscured within text. In this work, we propose to transform the
information within the textual metadata of CGDC collections into a knowledge graph, in order to make
CGDC more searchable and queryable. To this end, we investigated zero-shot approaches to relation
extraction (RE) as a means for automatically curating such knowledge graphs, thus eliminating the
need for any training data. Specifically, we cast RE as a natural language inference (NLI) task, allowing
us to take advantage of three pre-trained transformer-based language models: DeBERTa [
To support our experimentation with transformer-based RE models, we constructed a dataset of CGDC textual metadata where relationships between named entities were manually annotated.
The work of Benkhedda et al. [
In our work, we utilised the CGDC dataset that was annotated with named entities (and their
corresponding identifiers in Wikidata) constructed by Benkhedda et al. [
We then designed a simple relation annotation scheme, where we defined a relation as consisting of a head and tail entity, which are linked according to a relation type. For each relation type, our scheme specifies the possible entity types for each of the head and tail entities. For example, for the notable_work relation type, there are three possible head-tail entity combinations: Person-Miscellaneous (applies when a person created a miscellaneous entity such as a work of art), Organisation-Miscellaneous (if the creator is an organisation) and Person-Location (applies when a person is known for building or designing a location such as a church). Appendix A provides all the valid head-tail entity type combinations for each CGDC relation type.
The brat tool [
Cohen’s Kappa [
The RE task can be defined as follows: given a sentence and a pair of entities 1 and 2 contained within that sentence, as well as a set of classes , an RE model should identify one class ∈ that best describes the relationship between 1 and 2 based on . In this work, we are concerned with 20 relation types that are relevant to CGDC. However, apart from these, we also include an additional class that we refer to as None Of The Above or NOTA, to which pairs of entities that are not related according to any of our 20 CGDC types belong. Thus, there are a total of 21 classes in .
A number of transformer-based models that were already fine-tuned for the NLI task were employed to extract relations between entities in a zero-shot manner. Each of these models is described below.
DeBERTa (which stands for Decoding-Enhanced BERT with disentangled Attention) improves upon
the encoder-only BERT model by employing a disentangled attention mechanism [
BART is an encoder-decoder model that combines a BERT-like encoder and an auto-regressive
decoder similar to GPT. It has demonstrated satisfactory performance on both text generation and
comprehension tasks (such as text classification) [
T5 stands for Text-to-Text Transfer Transformer [
Importantly, we investigated an ensemble model, henceforth referred to as Ensemble, that is a combination of the above three models. The prediction of this ensemble model was determined by taking the majority vote amongst the constituent transformer-based models.
As mentioned above, we cast RE as an NLI task, whereby two sentences are provided to a model as input: a premise and a hypothesis. If based on the premise, the hypothesis is true, then the model should detect that an entailment relation holds between the two sentences. Otherwise, there is no entailment relation between the two sentences. To frame RE as an NLI task, an input sentence is considered to be the premise, while a hypothesis is automatically generated by populating a sentence template—that is predefined for a particular relation type—with input entities 1 and 2. The premise-hypothesis pair is then presented to an NLI model. If the model detects entailment, then we say that the relation type
(corresponding to the template) holds between the input entities. In Table 1, we provide examples of premise-hypothesis pairs, in which the hypothesis was generated based on a template. The complete set of templates is provided as part of our codebase.6
In preparation for applying the NLI models in a zero-shot manner, we firstly segment each document (in the evaluation dataset) into individual sentences. For each of these input sentences, all possible pairs of entities (contained within a sentence) are created. For every entity pair, a hypothesis is generated for each relation type. Each of these generated hypotheses is then paired up with the premise (the input sentence). Finally, our NLI models take every premise-hypothesis pair as an input sample that is then classified as being characterised by entailment or not. If entailment is not detected (with a probability of at least 0.40 in the case of the BART and DeBERTa models) for any of the relation types, we assign the NOTA label to the input sample. In cases where entailment is detected by the model for more than one class (relation type), we simply take the class with the highest probability value, as provided by the BART and DeBERTa models. As the T5 model does not output any probability values, we implemented post-processing rules that specify which relation types should take precedence in case of ties.
In this section, we report the results of applying our chosen NLI models to the RE task in the zero-shot manner described in the preceding section. As previously mentioned, entity pairs were exhaustively created based on our evaluation data (the 237 CGDC documents) to form the input samples. This resulted in 5938 entity pairs that the NLI models need to classify according to relation type (or the lack thereof, in which case the NOTA class applies). Out of those, 5435 (or 92%) belong to the NOTA class.
In Table 2, we report the performance obtained by each of our models in terms of the standard metrics of precision, recall and F1-score. To avoid skewing the results towards the over-represented NOTA class, we report the performance separately for the 20 CGDC relation types, and the NOTA class. Weighted macro-averaging was employed in reporting combined performance for the 20 CGDC relation types.
With respect to the 20 CGDC relation types, the Ensemble model obtained the best weighted
macroaveraged F1-score of 0.487. Considering the IAA of 0.48-0.49 that we obtained (see Section 2.2), the
model comes close to upper-bound performance. Meanwhile, the models obtained F1-scores as high as
0.914 (for T5) on the NOTA class. This is particularly impressive considering that it is well-known that
handling NOTA cases is a challenging task [
We manually examined some cases where the models made incorrect predictions. For instance, in the sentence “She was born on 10th March 1833 at Market Rasen, Lincolnshire, the eldest daughter of Henry Albert Browne and Frances Margaret Nicholson”, no relation (NOTA) holds between the entities “Frances Margaret Nicholson” and “Henry Albert Browne”, according to the gold standard annotations. Both T5 6Our code and annotations (for the 237 in CGDC documents in our dataset) are available at https://github.com/ OurHeritageOurStories/cgdc_re. and BART predicted the spouse label for this entity pair (whereas DeBERTa successfully predicted NOTA). One can, however, argue that even the predictions by T5 and BART are not necessarily incorrect. The interpretability of such results are rather subjective and can be considered as either correct or incorrect based on people’s opinion or the intended use of the models. Overall, it was observed that the models tended to detect or infer implicit relations that human annotators might otherwise miss.
Taking the predictions of the Ensemble model over the 237 documents in our dataset, we populated a knowledge graph (KG) whereby vertices represent named entities and edges represent any relations detected between them. Here, only entities that were manually assigned Wikidata IDs (as part of gold standard entity linking annotations) were included, to ensure that entities in the resulting KG are normalised.
To create the knowledge graph, we utilised the Neo4j framework7, a graph database management tool. Using the Cypher query language, we were able to query the resulting knowledge graph. Figure 1 shows a visualisation of the results of an example query for a use case where the relationships of a particular person of interest (King Edward) with other entities have been retrieved.
In this paper, we demonstrate how a zero-shot approach based on NLI can be employed to extract entity relations in CGDC textual metadata. Our findings show that, based on evaluation using a dataset of 237 CGDC documents, an ensemble of three diferent transformer-based models (BART, DeBERTa and T5) obtains the best weighted macro-averaged F1-score for 20 CGDC relation types. The knowledge graph that was constructed based on automatically extracted relations provides a means for searching for information that is otherwise buried within CGDC textual metadata. Our future work will focus on expanding the CGDC dataset to include more relation-annotated documents that will allow for more robust evaluation, including experimentation and comparison with closed-sourced large language models.
The relation types relevant to CGDC, their equivalent human-readable label, their Wikidata identifiers (linked to a page with their definitions and synonyms) and the types of head and tail entities involved. affiliation depicts inception shows began to exist on work_location P569 P570 P276 P706 P463 P106 P2541 P2652 P19 P551 P3342 P26