Knowledge Graphs (KGs) are a popular way to structure and represent knowledge in a machine-readable way. While KGs serve as the foundation for many applications, the automatic construction of these KGs from texts is a challenging task where Open Information Extraction techniques are prominently leveraged. In this paper, we focus on generating a domain-specific knowledge graph based on art-historic texts from a digitized text collection. We describe the combined use and adaption of existing open information extraction methods to build an art-historic KG that can facilitate data exploration for domain experts. We discuss the challenges that were faced at each step and present detailed error analysis to identify the limitations of existing methods when working with domain-specific corpora.
Knowledge Graphs (KGs) have gained considerable popularity in both academia and industry.
They are employed to represent information in a structured format after extraction from large
collections of heterogeneous, diverse, and unstructured documents [
There have been several eforts towards automatic construction of general purpose knowledge
graphs from the Web based on machine learning techniques [
KG for Art. The art and cultural heritage domain provides a plethora of opportunities for
knowledge graph applications. An art knowledge graph can enable art historians, as well as
interested users, to explore interesting information that is hidden in large volumes of text in a
structured manner. With a large variety of diverse information sources and manifold application
scenarios, the (automated) construction of task-specific and domain-specific knowledge graphs
becomes even more crucial for this domain. In contrast to general purpose KGs, a KG for
the art domain could comprise a specific set of entity types, such as artworks, galleries, as
well as relevant relations, such as influenced_by, part_of_movement etc., depending on the
specific task and on the specific text collection. The important entities and relations might also
difer across diferent document types, such as auction catalogues, exhibition catalogues, or art
magazines. On one hand, a general purpose, art-oriented ontology may not be well-suited and
comprehensive enough for specific data collections. On the other hand, designing a custom
ontology for the diferent art corpora would be a challenging and expensive task due to the
need for significant domain expertise. In the past, several attempts have been made at creating
KGs for art and related domains [
We make the following contributions in this paper: (i) Construct a domain-specific knowledge graph based on a collection of digitized art-historic documents. (ii) Describe the process of automated construction of the KG with Open IE techniques. (iii) Analyze and discuss the challenges and limitations for the adaptation of Open IE tools to domain-specific datasets.
With the availability of digitized cultural data, several previous works have proposed KGs for
art-related datasets [
Open IE approaches extract triples directly from text, without an explicit ontology or schema
behind the extraction process. Several works have been proposed in the past. TextRunner [
The construction of domain-specific KGs has been the subject of investigation in previous
works for various domains, e.g. software engineering [
In this section, we describe our underlying art-historic dataset as well as the steps employed for the automated extraction of information (in form of triples) to construct an art-historic knowledge graph. Fig. 1 shows an overview of this process.
For this work, we are working with a large collection of recently digitized art-historical texts provided by our project partners. This collection consists of a variety of heterogeneous documents including auction catalogs, exhibition catalogs, art books, etc. that contain semi-structured as well as unstructured texts describing artists, artworks, exhibitions and so on. Art historians regularly study these data collections for art-historical analysis. Therefore, a systematic representation of this data in the form of a KG would be a valuable resource for them to explore this data swiftly and eficiently. The whole collection is quite large ( ≈ 1TB of data), in order to restrict the size of the dataset for a proof-of-concept of our KG construction process, a subset of this dataset pertaining to information about the artist Picasso was chosen. The decision of choosing an artist-oriented subset of the collection enabled us to better understand the context and evaluate the triples that were obtained throughout the process of KG construction. The data was filtered by querying the document collection using the keyword query ‘Picasso’, resulting in 224,469 entries (where each entry corresponds to a page of the original digitized corpus) containing the term ‘Picasso’. Due to the filtering, each entry is an independent document, in the sense that the neighboring entries do not always represent the correct context. This led to some of the entries in our dataset containing incomplete sentences at the beginning or the end of a page. One such example is an entry starting with ‘to say47—Picasso never belittled his work, until . . . ’ where the tokens ‘to say’ belong to a sentence which started in a diferent entry, that might no longer be a part of the dataset under consideration. It is important to note that in the same example we can see more noise, e.g., numbers are mixed in between words in the digitized version of the text. This noise in the dataset was introduced by the optical character recognition (OCR) process during the digitization of the documents (performed in a prior step by the data providers). In general, the dataset contains full sentences, such as ‘Matisse’s return to the study of ancient and Renaissance sculpture is significant in itself.’ , as well as short description phrases, ifgure captions or footnotes such as ‘G. Bloch, Pablo Picasso, Bern, 1972, vol. III, p.142’.
As a first step, it was interesting to inspect if the named entities present in the corpus could be
easily identified. A dictionary-based approach to find the named entities would identify the
mentions with a high precision, but at the cost of very low recall by ignoring many potentially
interesting entities to be discovered in the corpus. Therefore, we chose to follow a machine
learning approach to named entity recognition (NER). Generic NER tools work very well for
the common entity types, such as person, location, organization and so on, though fine-grained
or domain-specific entities are harder to identify [
After obtaining informative sentences from the previous step, we employed Open IE tools to
extract the triples from them. It is important to note that while there are some art-related
ontologies proposed in previous works such as Arco [
To this end, we ran the Stanford CoreNLP OpenIE annotator [
Once the triples were extracted, the entity linking component of the Stanford CoreNLP pipeline [
One of the main challenges when constructing a KG through Open IE techniques, is that of
canonicalization. Multiple surface forms of the same entity or relation might be observed in the
triples extracted with Open IE techniques in the form of noun phrases or verb phrases that need
to be identified and tagged to a single semantic entity or relation in the KG. Since the triples
extracted from our dataset via Open IE method comprised many noisy phrases, as well as new
entities, such as titles of artworks, that may not be available for mapping in existing databases,
entity linking techniques would not sufice in this case. Diferent from entity linking (that can
only link entities already present in external KGs), canonicalization is able to perform clustering
for the entities and relations that may not be present in existing KGs, by labelling them as OOV
(out of vocabulary) instances. In this work, we chose to perform canonicalization with the help
of CESI [
Representative Selection. An important step in the CESI approach is the assignment of representatives for the clusters obtained for the noun and relation phrases. This is decided by calculating a weighted mean of all the cluster members’ embeddings in terms of their frequency of occurrence. The phrase closest to this mean is selected as the representative. However, this technique did not work well for our domain-specific and noisy dataset and many undesirable errors were noticed. For example, an entity cluster obtained from CESI was: Olga_Khokhlova, olga, khokhlova, picasso. Since Picasso is the most frequent entity in the dataset, it was chosen as representative by CESI, but this is clearly wrong since Picasso and Olga are diferent entities. There were several other errors observed, e.g., all days of the week were clustered together in one cluster. This could be a result of the embedding and contexts of the days of the week to be quite similar, hence their vectors would end up together in the vector space. In other cases, the color blue occasionally showed up in a cluster of phrases related to color red, certain dates got clustered and certain related but not interchangeable words got clustered (kill vs murder vs shot). In some cases, the first name was being replaced by the incorrect full name (not every david is david johnson). To mitigate the above discussed errors, we had to perform manual vetting of the clusters for verification and selection of the correct cluster representatives which took around 2-3 person hours. During this process, certain clusters, where the entities were diferent, were removed (such as the cluster with days of the week). After this, the entities and relations were canonicalized as per their chosen cluster representatives leading to a total of 35,305 unique entities and 33,448 unique relations in the final KG 5.
Since a schema or ontology was not employed to extract the triples from text, the entities in
our KG do not have any entity types implicitly assigned to them. Therefore, we attempted
to identify the types of as many entities in our graph as possible. With the help of NER, we
assigned the types to the entities that were recognized in the triples. A total of 14,960 entities
were typed with this technique to generic types such as Person, Product, ORG, LOC, GPE,
NORP and Work_Of_Art, as well as numeric types such as Date, Time and Ordinal. Note
that Work_of_Art is quite a broad category that includes artworks but also movies, books and
various other art forms. Since artworks such as paintings and sculptures are one of the most
important entities in our art-historic KG, it is worthwhile to identify the mention and type of
these entities. However, generic NER process is neither equipped nor optimized to correctly
identify such mentions. Thus, we additionally applied dictionary-based matching. This was
done by compiling a large gazetteer of artwork titles by querying Wikidata with the help of the
Wikidata Query Service6 for the names of paintings and sculptures, retrieving approximately
15,000 artwork titles. In addition, we augmented our dictionary with the names of the artwork
entities from the ArtGraph dataset [
6https://query.wikidata.org/ Person type by NER, such as la_donna_gravida, portrait_of_mary_cassatt and st._paul_in_prison. Similar to artworks, we attempted to additionally identify the names of artists in our triples. While NER could only tag entities as Person, we used a dictionary of artist names from Wikidata to identify 656 unique artist entities in our data. These included names of artists such as Piet Mondrian, Edvard Munch and Rembrandt.
However, the process of entity typing described above is only able to identify and tag around half of the entities in our KG. Several domain and corpus-specific challenges acted as bottlenecks during this process. For example, even after filtering, some triples extracted from Open IE contained either subject or object noun phrases that were generic and did not correspond to any named entity. Examples of such phrases include essay, anthology, periodical, or album that are present in triples such as ⟨album, be_shown_in, Paris⟩. Without designing a custom ontology for this corpus, such entities cannot be hoped to be correctly typed.
The categorization of the relations in the KG is a particularly complicated task due to the wide variety of relations extracted from the Open IE process. Few of the most frequent relations in the KG are will, be_in, have, show, paint, work etc. We estimated that the types of the entities could be utilized to find patterns and link the most popular edges in the KG to the relations in existing graphs such as Wikidata or ArtGraph. However, preliminary analysis led to some interesting observations. Firstly, we noted the presence of multiple relations between pairs of entities in the KG. For example, Picasso and June are connected by various relations such as will_be, work and take_trip_in that were extracted from diferent contexts in the corpus and represent separate meaningful facts. Furthermore, in general, there are several diferent types of semantic relations between the popular entity types in our KG. For instance, two entities of the type artist are connected by several relations including work, meet, know_well, be_with, friend_of and be_admirer_of. While this variety indicates that a large number of interesting facts have been derived by Open IE in the absence of a fixed and limiting schema, normalizing the relations to improve the quality of the KG is a dificult task that is part of the ongoing and future work.
The statistics of the KG generated from the steps as described in the previous section are shown in Table 1.
After obtaining the refined set of triples for the first version of the art-historic KG, we performed
a preliminary analysis of the graph to derive useful insights with the help of the NetworkX7
package. To understand the graph structure, the number of disconnected components of
the graph was measured before and after the canonicalization step. It was noticed that the
number of disconnected components was reduced to around 1,500 (down from 2,500) after
clustering with CESI. This indicates that canonicalization of entities and relations improved
the quality of the knowledge graph by removing unnecessary disconnected parts that were
created through redundant triples. Additionally, we also performed node centrality on the graph
using eigenvector centrality [
Due to the lack of any gold standard for direct comparison, the evaluation of the resulting
KG proved challenging. While an absolute measure of the coverage of any KG is a non-trivial
task due to the open world assumption [
MATCH p=(:Artist)-[r:exhibitat]->() RETURN p) Jacques Villon
exhibitat Georges Braque exh
ibitat salon_des_ independants photo_secession
_gsallery exhibitat
Wassily Kandinsky annual_ exhibition (b) Picasso involved in various Art schools. (corresponding query:
MATCH p=(s)-[r:involvedin]->() WHERE s.name="Pablo Picasso" RETURN p)
subset of Wikidata [
From the obtained Wikidata subset, 100 triples were randomly selected that related to information about Picasso as well as about museums that owned his works. Upon careful manual inspection (independently by three annotators) and resolution of conflicts with discussions, it was measured that the facts represented in 43% of these triples were also present in our KG as a direct match or in a diferent form with the same meaning. Notably, our KG was missing information about the museums that own Picasso’s works, this is because our underlying corpus is also lacking comprehensive information on this topic. Therefore, triples relating to museums from Wikidata could not be matched. Additionally, we checked how many of our entities and entity pairs are written in exactly the same way as in the Wikidata graph. Overall, around 12% of entities and 10% of entity pairs in our graph have exact matches in Wikidata. These preliminary results are promising and point towards the need for a domain-oriented construction process for further improvement of the art-historic KG. In particular, the precision of the triples in art-historic KG is more important to the users and therefore, factual verification for the triples that were extracted from our dataset but are not found in Wikidata needs to be conducted by enlisting the help of domain experts.
Taking cue from related work [
Due to the source corpus being heterogeneous and noisy, the Open IE process led to a number of incorrect triples in the KG despite our best eforts to eliminate the noise at each step. Here, we perform a critical analysis and look deeper into the quality of the triples in the first version of the KG. For this, we sample few of the incorrectly extracted triples, to understand the nature of mistakes committed by the automated KG generation process. Table 2 presents some triples in the KG and the corresponding text snippets in the input data from which they were extracted.
In T1, even though the triple appears to be syntactically correct, the actual entity corresponds to the entire phrase The Third of May 1808 in Madrid which is an artwork, and thus the correct triples should relate this artwork to the corresponding artist Francisco de Goya, perhaps including the date 1814 as well. This example illustrates the dificulty of recognizing artwork titles, given that they usually contain other entities like Madrid (location). A similar mistake can be seen in T6. Here Appel was incorrectly recognized as a location instead of the surname of Karel Appel (person), and thus the triple represents the information to be an influence of an artist on a location, instead of between the artists.
Examples in T2 to T6 represent the triples and the supporting text snippets for the results of the query as depicted in Figure 4, which contains a mixture of factually correct, factually incorrect, and speculative facts. In T2, a relation was correctly extracted from the text, but the head entity was incorrectly recognized as ‘American’. This example speaks for the need for additional work on co-reference resolution, in order to properly follow the connections in the text. A more precise triple would have been ⟨Gorky, beInfluenceBy, Pablo Picasso ⟩.
T3 is an example in which the lack of context in the syntactic analysis of the sentence results in the assumption that the statement is true, although it is a suggestion by a specific person and therefore, not necessarily a true fact. A similar example is T4 in which the source text is Morris
Louis roman_art american
guevara beinfluenceby beinfluenceby Aubrey Beardsley b eifl n u e n c e b y y cb ne e u einfl b
Pablo Picasso b eifl n u e n c e b y y b e c n e u eifl n b philpot beinfluenceby beinfluenceby
Appel femme_ couchee
City Artist NORP None ⟨American, beInfluenceBy, Pablo Picasso⟩ The more one examines Gorky’s early works, the more they appear like Gorkys rather than like Picassos. Moreover, his unabashed borrowings can be seen as forward-looking: for an American to be influenced by Picasso in the heyday of American Scene painting was, art historian Meyer Schapiro points out, “an act of originality.” ⟨Pablo Picasso, beInfluenceBy, Morris Louis⟩ . . . to Andrew Hudson, art critic of The Washington Post, for suggesting that Pablo Picasso has been influenced by Morris Louis and Kenneth Noland, two leaders of the “post-painterly” Washington, D.C. ⟨Guevara, beInfluenceBy, Pablo Picasso⟩ It is probable that Guevara was influenced by Picasso to experiment with the encaustic technique, which had been practised in antiquity. Hot wax was used as a medium for mixing floral and vegetable dyes. ⟨Pablo Picasso, beInfluenceBy, Aubrey Beardsley⟩ Picasso was influenced doubtless by Aubrey Beardsley, who had died in 1899 at the age of twenty-six, but then what an excellent influence it proved to be for this portrait ! ⟨Appel, beInfluenceBy, Pablo Picasso⟩ In artistic respect, one could also see, that Karel Appel was strongly influenced in this period, by Picasso and Miro. explaining a potential influence relation between the artists, but it cannot be directly assumed that it is a fact. These two examples illustrate that the context of the actual text might get lost during the extraction process, which may lead to erroneous facts being represented in the KG. Thus, it is important to take into account the provenance information that can help the user understand the full context for obtaining the correct information.
A diferent scenario is depicted in T5, in which the text clearly confirms the validity of the fact. One interesting observation is regarding the syntactic structure of the relation phrase - the word ‘doubtless’ acts as an adverb emphasizing the validity of the fact, and although it divides the relation phrase ‘was influenced by’, the syntactic analyzer and the canonicalization step were able to normalize the relation to a canonical form. This is also evident in the diversity of relation phrases in this sample of texts. They are expressed in diferent tenses, with auxiliary verbs, and sometimes spread within a more complex sentence, as seen in T5. Examples T3 to T6 illustrate the need for fact-checking in our KG. Particularly, the facts in the KG could be presented to domain experts who would be able to easily look at the information in a user-friendly manner and then proceed to investigate further to either corroborate or even contradict the triples in the automatically generated KG. We envision the easy access and scrutiny of the information stored in large text collections to be the primary use-case of this automatically generated art-historic KG.
This work presented a first attempt at constructing a domain-oriented knowledge graph for
the art domain in an automated fashion with Open IE techniques. Due to the noisy and
heterogeneous dataset that is typical of digitized art-historic collections, we encountered challenges
at various steps of the KG construction process. During the very first step, it was dificult to
correctly identify the mentions of artworks (i.e. titles of paintings) in the dataset due to the
noise and inherent ambiguities. This domain-specific issue needs further attention in order to
improve the quality as well as coverage of the resulting KG, as discussed in detail by previous
work [
While the Open IE approach allowed for the extraction of a wide variety of entities and
relations, this led to canonicalization becoming a complicated task. We observed that existing
techniques for canonicalization on generic datasets, such as CESI, do not show comparable
performance for domain-specific dataset. It would be interesting to investigate if large
pretrained language models such as FastText and BERT could compete with the relatively older KG
embeddings that were employed in CESI for obtaining better clusters. There are other recent
works on canonicalization [
Moreover, we have only considered English texts in this work so far, since the existing methods show their best performance with English texts. However, our art-historic collection is comprised of multiple languages and we would like to expand the pipeline to process multilingual texts. Taking into account the existing limitations of the methods with domain-specific corpora, this seems to be an arduous but interesting research challenge.
With regard to the implementation of the KG pipeline, while we have so far used of-the-shelf
tools and libraries like SpaCy, Stanford CoreNLP and CESI, we plan to further fine-tune them
to the task of domain-specific KG construction. It will also be worthwhile to explore and
evaluate the performance with other available tools such as Flair [
The evaluation of the art-historic KG is also a crucial task worth discussing. While we have performed a semi-automated evaluation for the first version of our KG, a more rigorous and thorough evaluation of the correctness of the facts is certainly imperative before this KG can be useful to a non-expert user (as discussed in Section 5). One way to ensure this would be to maintain the provenance and of the facts in the KG, in terms of their source document as well as their confidence measure. This could also facilitate a fair and complementary manual evaluation in terms of precision and recall which could provide further insights. For this, we plan to closely collaborate with domain experts and enlist their help in the near future.
In this work, we have presented our approach to construct an art-historic KG from digitized texts in an automated manner. We have leveraged existing Open IE tools for various stages of the KG construction process and discussed the limitations and challenges while adapting these generic tools for domain-specific datasets. We have presented these insights with the hope of encouraging interesting dialogue and further progress along these lines. While our limited initial analysis and evaluation has shown encouraging results, it has also shown clear indications towards the points of improvement for creating a more refined and comprehensive version of an art-historic KG which could be used for downstream tasks such as search and querying.