Heritage institutions hold rich information on cultural heritage objects involving contextual information about people, places, times, and events. This information is usually kept in institutional silos, where domain researchers often work with data across institutions. Linking entities among diferent institutions can enrich these data sources and, in turn, aid domain research. The aggregated version of data can be further used to infer insightful knowledge that can excel in one of the time-consuming tasks of the domain, which is provenance research. This research will first focus on entity linking across institutions to construct a Knowledge Graph representing both structured metadata of objects and the collector's biography. This work aims to use this newly formed Knowledge Graph to find interesting patterns to scale-up provenance research and analyse the efect of adding such information to the current data source. Experiments with the diferent modalities of data and pattern mining techniques will reveal to which extent this data enrichment places a role in finding useful knowledge for the heritage objects' provenance research.
Knowledge acquisition and knowledge representation are active research fields within computer
science, and countless formalisms have been developed to deal with various types of knowledge
(see [
One of the fundamental problems in the cultural heritage domain is having limited information
about objects’ biographies. In today’s world, it is not enough to represent a heritage object
only from the institutional perspective; rather, the representation should be more informative
about its past or original purpose. Therefore, it is common among museum professionals to
retrieve information about a particular object’s biography through provenance research. From
this research, professionals try to establish a connection between objects and their past based
on historical patterns, literature studies, or evidence from the past [
When hundreds of thousands of objects in museum premises need further provenance
information, it is counter-productive for provenance researchers to search through individual
data sources and come up with links for individual objects. On the other side, due to the recent
collaborative commitments and initiatives, more and more such resources are being available
online with an ambition of sharing collection on the open web [
Concatenating data from multiple sources into a single Knowledge Graph allows
understanding nodes association better and may help identify new knowledge through mining patterns
from existing data. For example, analysing associations among diferent entities, perhaps among
collectors, their participation in historical events, and object acquisition trends may reveal new
information. Moreover, the recent development of graph embedding techniques ([
However, while dealing with colonial heritage objects’ datasets, the incompleteness in object’s data, ambiguous mention in attribute values and subjective views placed in metadata creation make it a challenging problem for data linking and new knowledge discovery. This work investigates the challenges of entity linking on historical data sources to enrich heritage objects KG. In this graph, knowledge discovery techniques will be adopted to infer usable, new knowledge for provenance research. Experimenting with diferent pattern mining techniques and accessing acceptability will determine the practical approach for finding useful domain knowledge.
Incompleteness, ambiguity and subjective views in metadata should be considered when dealing with colonial heritage objects. In addition, historical events play a part in categorising or listing information around collected objects metadata. This work identifies three issues with the current dataset, which are below.
Incompleteness Object information in the museum database starts from its acquisition date. While traditional provenance research requires considerable human resources, it also emphasises imposing institutional bias in prioritising objects. Which object to choose for further research has always been a bureaucratic choice; therefore, some objects are information-rich, whereas others sufer from a lack of information, resulting in a skewed dataset.
Ambiguity As with other historical data, dealing with museum databases of diferent institutes also comes with the challenge of inconsistent mention of the same entity. For example, someone named “Lars Erikson” in one database may be listed as “L. Erikson” or just “Eriksson” in another database. In addition, the date and location attribute value sometimes does not always have a precise specification. This inherent ambiguity makes it a challenging problem to tackle such data.
Subjective view Heritage object metadata is a purposeful creation of museum histories, and
perspectives [
This research will be done as part of the project “Pressing Matter” which investigates ownership, value and the question of colonial heritage in museums. This research aims to provide a solution that helps the domain researcher prioritise their investigation by bringing large-scale information closer in a manageable manner.
This section will explain the research context and the relevant works from the literature. The topics mentioned here are relevant to the research questions mentioned in section 3 and position the proposed work in diferent topic areas.
Knowledge organisation systems have a long history in the museum world, where they have been employed in metadata descriptions to arrange objects and increase findability. In addition, scholarly eforts have been made to create authoritative data to describe a particular group of objects, resulting in taxonomies, vocabularies, and thesaurus. The advent of Linked Data technologies ofers an opportunity for the institutional data silos to enter the realm of the World Wide Web [12].
In the spirit of the Linked Data vision, several data standards and collective commitments have
been made to allow cross-institutional heritage data linking. Europeana Data Model (EDM)1,
CIDOC-CRM2 and Union List of Artist Names (ULAN) and Thesaurus of Geographic Names
(TGN)3 are just some of the examples [
While significant efort has been made to develop a suitable data model for cultural heritage data, it remains less explored whether such data can be further used for knowledge discovery [15]. Furthermore, due to diferent practices in curating metadata descriptions, meta-tags vary significantly across diferent institutions, even when describing the same object. It creates a 1https://pro.europeana.eu 2https://cidoc-crm.org 3https://www.getty.edu/research/tools/vocabularies/ significant challenge when linking and analysing diferent object collections from multiple institutions, which further needs attention from the Linked Data community.
According to the Getty Research Institute (GRI), A complete provenance provides a documented history that can be used to verify ownership, assign the work to a known artist, and establish the legitimacy of the work of art4. This quest for object biography is often based on sources, i.e., digital collection register of museum system(s), public and private archives, online search, literature review, object research, experts’ input, etc.; where the significant amount of these sources are already digitised.
In the context of Pressing Matter, provenance research on the colonial object is approached by untangling the complex acquisition history through the collector’s biography. This approach is adopted based on Actor-network-theory, where objects are not only singular entities but in relational configuration with other actors (people, place, time, event) that inform their biographies. The project identifies four overlapping collecting modes through which colonial objects have entered museums and will use them as a frame to guide understanding of the colonial object’s changing ownership, values, and potentiality.
The problem of Entity Linking can be divided into two parts: surface form extraction and named entity disambiguation (NED). Surface form extraction relates to identifying entities from a continuous span of text. The goal of the NED task is to link a named entity to ground truth entities in a knowledge base [16]. A plethora of automated entity disambiguation techniques have been proposed ranging from rule-based approaches to node-embedding-based approaches (current state-of-the-art for entity disambiguation [17] and [18]). Given that the notion of identity can change under diferent contexts, the task at hand always influences how entity disambiguation algorithms are built.
Entity Linking on Knowledge Graph of historical data comes with a significant challenge, i.e., attributes may have value approximation (e.g., approximate date), lack of naming standard between datasets, attribute that look similar may not mean the same thing, error-prone attribute values etc. Baas et al. mentioned some of these problems in their work and projected on other task-focused Entity Linking approaches in digital humanities [19]. While Baas et al. [19] considered neighbourhood information-based embedding to cluster similar nodes, the other literatures on Entity Linking in the context of digital humanities primarily used deterministic rules based on context to tackle such tasks; therefore not applicable when the task or entity type changes.
Though the nature of the data in [19] is similar to our current context for Entity Linking, the data setup is entirely diferent. In the mentioned work, the entity considered for linking has a substantial amount of property value on them, whereas in our case, there is little to no information on the target entity, i.e., collectors. In our primary dataset from museum, it is common to have more information on related objects and events rather than on the collector 4https://www.getty.edu/research/tools/provenance/ instances. Therefore, it is a matter of investigation if existing approaches of entity linking are still suitable in the given context.
The proposed work can best be placed in Knowledge Discovery in Database (KDD). According to Fayyad et al., KDD is the nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data [20]. KDD entails a number of phases, including data preparation, pattern search, knowledge evaluation, and refining. Finding intelligible patterns that might be perceived as helpful or interesting knowledge is a priority for KDD. Patterns are statements which describe interesting relationships among a subset of the analysed data, typically resulting from a data mining process (classification, cluster-mining, association rules mining and so on).
Background knowledge extraction using linked data is an example of graph data mining [21] where data patterns are explained with linked data. Another example of graph mining is using background knowledge to traverse through the network to find new information [ 22]. In digital humanities, both approaches are interesting, given that there is an abundance of both data and experts’ knowledge. For this reason, this work will explore a mixed-method approach where data mining techniques and experts’ background knowledge will be incorporated together to ifnd new knowledge to aid provenance research.
Heritage object metadata contains complex contextual information about the object, making KG an efective representational means. Nevertheless, it remains an open question how much they support relational learning models in the cultural heritage domain [15], which are known to provide high scalability and accuracy among other domains. Lately, a small body of work ([23], [24]) is emerging on using machine learning or data mining techniques for humanities problem-solving, which had a deep tradition of scepticism towards quantitative and empirical techniques among humanists for long. Nonetheless, a radical departure from previous methods of humanistic inquiry has been proving to bring new perspectives and scalable solutions for domain researchers. Given the lack of related work and possible usefulness in the domain, it is worth investigating contemporary pattern mining techniques in the current problem setting.
This work will investigate if data enrichment of colonial heritage objects has the potential to provide aid in scaling up provenance research. It will take a case-based approach to explore the hypothesis that the collector’s biography information might shed more light on the acquisition modes for heritage objects. This will be concluded by answering the three following research questions (RQ).
RQ1 How to link entities from multiple heritage institutions where ambiguous mention of entity is present? To extend heritage objects’ structured metadata with information on actor, time, place and events, we need to enrich data from multiple sources. However, linking data from historical datasets comes with the challenge of entity disambiguation. This work will investigate if we can use the existing state-of-the-art approach or if there is a need to develop a new algorithm.
RQ2 Which patterns exist in colonial heritage objects’ data in a museum that is actionable and useful for provenance research? There are several pattern mining techniques available for finding interesting patterns from structured graph data. It is yet to determine which data mining technique to apply in the given task, which modality of data to consider while avoiding institutional bias and most importantly, which patterns are useful for provenance research.
RQ3 What is the efect in the result of the dominant pattern from the Knowledge Graph when it is populated with collectors’ biography information? This study will examine if collectors’ biographies and modes of object acquisition can reveal useful patterns. It will also examine how these patterns align with the current classification scheme. Finally, the results will be used to guide a revised classification scheme based on the context of objects’ acquisition modes.
The overall approach of this research is based on entity linking, pattern search, pattern evaluation and assessing usefulness. Having defined the research questions, we will now address the research approaches in more detail:
RQ1 One primary task for data enrichment from diferent sources is to match entities across diferent data sets. To address the challenge of Entity Linking where ambiguous and duplicate mention of Entity exists, this work will begin its investigation from the naive string matching technique and then move towards intelligent entity matching based on vector space embedding([17], and [18]) and analyze to what extent they apply to cases where limited to no text around the Entity is available. Moreover, this research will explore whether contextual information from a Knowledge Graph can be used to guide an automated system to find possible matches across data sources.
RQ2 This work will explore data mining techniques and the explainability of the found patterns. It will also examine diferent data modalities to avoid potential biases in the dataset. In parallel, this research will assess which patterns are interesting for domain experts and useful for their provenance research.
There will be two parallel research works to answer the RQ2. One will explore the suitability of pattern mining techniques considering explainability in domain use. The other will explore which data modality to consider, i.e., KG only with objects’ structured metadata, KG with extracted information from a text description, KG including literal node values etc. As an input, these two works will consider the entire object collection from the museum while experimenting with the diferent modalities of these collections and will produce actionable patterns as an output. The pattern validity will be measured based on user satisfaction.
RQ3 This research question will explore how much the newly found dominant patterns from the resultant Knowledge Graph from RQ1 difer from the found patterns from RQ2. The previous research question tries to find the dominant pattern based on the data sources only from one museum. The current research question will explore the efect of the data enrichment in pattern-finding when the data graph is enriched with collectors’ biography. Thus, it will experiment with two diferent versions of graph data. Emphasis will be given to the clustering methods to determine whether the objects can be grouped based on acquisition modes.
In the following section, we will discuss how to evaluate the results.
RQ1 The Entity Linking algorithms will be evaluated based on the experts’ assessment. A small statistically significant portion of the automated established link by each algorithm will be randomly chosen for further assessment by a group of domain experts. The result will be communicated as precision and recall of the system based on their findings overlapping with experts.
RQ2 User studies will be designed to understand the acceptability of found pattern. Designing user experiments is also a part of the second research under research question RQ2. To find efective evaluation technique, the result of the pattern mining algorithms will be presented using a diferent method of visualization and explainability. The design of evaluation techniques and experiments with the diferent modalities of data will be conducted in parallel so that we can report the usefulness of found patterns and can also use the found patterns to understand the efectiveness of chosen evaluation.
RQ3 For the comparison purpose, RQ3 will also use the established evaluation method for the research question RQ2. The user evaluation will report the found pattern’s usefulness to the provenance researchers. In addition, the comparison study with research output from the previous study will examine the efectiveness of using collectors’ biography with object metadata.
Given that this work is in the first year of the PhD program, this section will project on the dataset considered for the first year’s research work. This work considers object meta-data from the National Museum of World Culture (NMwV) as the primary data source. Each object is represented as a node and further connected with related information nodes, i.e., geographic location, title, material descriptions, collector’s name etc. The NMvW has published two versions of linked data. In the first version, artefacts are placed in the centre of the model and directly connected to the data5. The second version uses an even-centric approach (i.e., CIDOC-CRM) adopted by the LinkedArt6 community for describing artefacts using related events.
For the second dataset for Entity Linking, this work considers data from Museum Bronbeek7 which contains biography information from military personnel and their involvement in diferent historic events (e.g., wars, expedition). From the unstructured retrieved data from the database (i.e., text description), Named Entity Recognition and Extraction tools will be used to enrich the primary data source. However, how many collectors we can spot from automated linking and which version of NMvW data is more appropriate for entity disambiguation is still a matter of investigation.
A small fraction of object collectors in the chosen data set happen to have their wiki-data identifier, which can be used as a gold standard for Entity Linking performance metric. From 5https://collectie.wereldculturen.nl/thesaurus/#/query/ 6https://linked.art 7https://www.defensie.nl/onderwerpen/bronbeek/over-bronbeek an initial study, it has been observed that the number of collectors with a wiki-data identifier is very small; therefore, to conclude on the success rate of entity disambiguation algorithms in the given context, we still need the human evaluation of the output.
As an initial case study and to reporting purpose in a practical use-case, this work will investigate a small set of NMvW material objects (consists of 4242 objects) from Aceh, Indonesia, collected between 1873 and 1942; which has been researched under the PPROCE8 project. From this research, dedicated links have been established between military personnel who has been stationed in Aceh war camps and objects’ collection. This subset will be used for reporting the performance of adopted entity disambiguation algorithms. The complete experiment pipeline to answer the question, whether the existing state-of-the-art approaches is suitable for the given context or if there is a need for developing a new algorithm, is given in Figure-1.
This work plan shows how we intend to answer the question: To what extent does Knowledge Graph constructed from heritage object’s metadata and further enriched with collector’s biography information has the potential to scale-up objects’ provenance research for museum experts. While exploring the three focused research questions, this work aims to answer this overarching question.
The first year’s work will be focused on constructing a dataset by linking two data sources. The remaining years will focus on making this new and existing dataset usable for provenance research. This work will contribute to the field of digital humanities by providing tools or techniques to excel in provenance research. This paper can be seen as a proposal for a hy8https://www.niod.nl/en/projects/pilotproject-provenance-research-objects-colonial-era-pproce pothesis generation tool for the domain experts rather than a concluding outcome for domain understanding.
This research takes a bottom-up approach instead of the traditional process mining approach for object provenance research. Given many objects with missing provenance information, we choose such a data-driven approach where research hypotheses can be generated based on data patterns. Though it is believed that both top-down and bottom-up approaches can benefit our findings in the given context, this work will explore the bottom-up approach primarily to manage the volume of work.
This work is conducted under the Pressing Matter project (https://pressingmatter.nl/) which is financed by the Dutch National Science Agenda (NWA) and coordinated from the Vrije Universiteit Amsterdam. I would like to thank my supervisors, Dr Victor De Boer, Prof. Dr Jacco van Ossenbruggen and Prof. Dr. Susan Legene for their guidance and feedback.