<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A knowledge representation framework for managing Leonardo Da Vinci's Mona Lisa: case study of the hidden painting</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Alessia Amelio</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Gian Piero Zarri</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>InGeo, University “G. d'Annunzio” Chieti-Pescara</institution>
          ,
          <addr-line>Pescara 65127</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>STIH Laboratory, Sorbonne University</institution>
          ,
          <addr-line>Paris 75005</addr-line>
          ,
          <country country="FR">France</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>This paper explores the use of Artificial Intelligence/Knowledge Representation methods for digitally modeling the cultural heritage items. It fully complies with the concept of “Cultural Heritage Digital Twin”, which is characterized by a “physical” component of the cultural entity, concerning style, dimension, name of the artist, execution time, etc., and by an “immaterial” component representing, among other things, the emotional and intangible messages transmitted by the entity. The “Narrative Knowledge Representation Language” (NKRL) is then been adopted for digitally representing the two components of the twin and its immaterial component in particular, due to its ability to represent in a simple but rigorous and efficient way complex situations and events, behaviors, attitudes, etc. An experiment concerning the “hidden painting” that lies beneath the Mona Lisa (“La Gioconda”) image on the same poplar panel has been then realized, showing that NKRL is able, in fact, to successfully provide a suitable representation of at least some of the intangible elements of the “visual narrative” represented by this still largely undeciphered portrait.</p>
      </abstract>
      <kwd-group>
        <kwd>1 Cultural Heritage Digital Twin</kwd>
        <kwd>Knowledge Representation</kwd>
        <kwd>NKRL</kwd>
        <kwd>Mona Lisa</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        In recent years, knowledge representation and querying/inferencing proved to be important topics to be
dealt with in the cultural heritage domain, which raised attention by the European Commission. More
specifically, considering the recent Horizon 2020 Programme, and with respect in particular to the
specific 2020 challenges related to the topic “DT-TRANSFORMATIONS-12-2018-2020: Curation of
digital assets and advanced digitization”, the Commission expressed the need to create a comprehensive
representation of the studied assets, which includes not only the visual and structural information, but
also experiences and narratives stored in language data, together with their cultural and socio-historical
context, as well as their evolution over time [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
      </p>
      <p>
        This implies that more advanced methodologies should be introduced to describe and represent not
only the “physical” aspects of the cultural heritage items, including all those features usually employed
for characterizing the items, i.e., support, style, dimensions, name of the artist, execution technique,
information about the owner, location and collections the item is or was in the past, etc., but also the
“message” and the emotional states that these items express according to their cultural, historic and
social background. Recently, the term “Cultural Heritage Digital Twin” has been introduced, which
merges the concept of Digital Twin well-known in the context of Artificial Intelligence [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] and that of
Cultural Heritage, in order to provide a complete digital modeling and characterization of every Cultural
Heritage item. The twin should be characterized by the presence of two components, the first one
regarding the aforementioned physical description of the item, and the second one to be used for
representing the intangible (immaterial) and emotional messages transmitted by the item.
      </p>
      <p>
        In this paper, we make use of this new concept by focusing, in particular, on the improvement of the
digital modeling techniques to be used for those “iconographic entities” (paintings, drawings, frescoes,
mosaics, sculptures, murals etc.) that represent a particularly important component – both numerically
and culturally – of the whole Cultural Heritage domain. In this framework, an experiment has been
conducted in the context of a project concerning an advanced digital representation of the Mona Lisa
(“La Gioconda”) painting by Leonardo Da Vinci, taking into account in particular the “hidden painting”
that lies beneath Mona Lisa’s portrait on the same poplar panel. This work conforms to the
aforementioned concept of Cultural Heritage Digital Twin by emphasizing, in particular, the
importance of the “immaterial” and “emotional” components of this concept; in spite, in fact, of the
current literature where various works have been proposed for pattern extraction and exploration from
a “visual computing” perspective [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], this research direction is still little explored. Accordingly, it seems
there are no really suitable computer-usable descriptions of these emotional elements that can be
considered as sufficiently exhaustive (and interoperable) for the scope. To overcome this limitation, the
“Narrative Knowledge Representation Language”, NKRL [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] has been selected for digitally modeling
the whole Mona Lisa’s Cultural Heritage Digital Twin, due to the NKRL’s ability to represent in a
simple but rigorous and efficient way complex situations and events, behaviors, attitudes, etc., see also
our previous work in this context described in [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
      </p>
      <p>The paper is organized as follows. Section 1 provides the general background and motivation of this
work. Section 2 presents in some details the experiment about the digital modeling of the hidden
painting’s “narrative”. Eventually, Section 3 draws conclusions about the described experiment.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Experiment</title>
      <p>
        A problem that has troubled Mona Lisa’ specialists for a long-time concerns the identification of the
woman represented in the “hidden painting”, i.e., the portrait, visible to us only in x-rays, indubitably
painted by Leonardo and that lies beneath Mona Lisa on the same poplar panel see, e.g., [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. According,
e.g., to Lillian Feldmann Schwartz [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ], this woman has nothing to do with Mona Lisa and represents
probably Isabella d’Aragona, wife of the Duke of Milan Gian Galeazzo Sforza – end of 15th century,
Leonardo had worked for the Sforza family. Feldmann Schwartz’s hypothesis is based, essentially,
on the lack of correspondence between the facial characteristics of the hidden painting woman and
Mona Lisa. More exactly, eyes, mouth, nose tips, hairlines and chins do not match between Mona Lisa and
the unknown woman. A correspondence exists, on the contrary, between this last woman and the woman
represented on a Leonardo’s cartoon, a (very retouched) preparatory study for a portrait of Isabella
d’Aragona. Specifically, the eyes of the hidden painting woman were, in L.F. Schwartz’s words, “lined up
with the eyes on the cartoon, the hairlines fell into place and overlapped exactly”. Table 1 below reproduces
the NKRL narrative, gio3, see [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ], which formalizes (at least partly) the hidden painting issue.
gio3.c6: PRODUCE:
      </p>
      <p>SUBJ: LEONARDO_DA_VINCI
OBJ: PAINTING_2
TOPIC: (SPECIF portrait_ UNKNOWN_WOMAN_1)
Produce:Entity (6.2)
{ obs }
date-1: today_
date-2:
Own:CompoundProperty (5.42)
A painting (conventionally: PAINTING_2) concerning the portrait of a woman (conventionally: UNKNOWN_WOMAN_1) has been
produced by Leonardo da Vinci within the temporal interval 1497-1503.
gio3.c7: OWN:</p>
      <p>SUBJ: PAINTING_2
OBJ: property_
TOPIC: (COORD1 (SPECIF painted_on POPLAR_PLANK_1 (SPECIF under_ PAINTING_1)) (SPECIF labelled_as</p>
      <p>HIDDEN_PAINTING) (SPECIF visible_ x_ray_analysis))
We can remark today (modulator obs(erve)) that i) PAINTING_2 has been painted on the same poplar plank of PAINTING_1
(conventionally, Mona Lisa’s portrait), ii) it is located under PAINTING_1, iii) it is known as the HIDDEN_PAINTING and, iv) it is
visible only under x-ray analysis.
gio3.c3: MOVE:</p>
      <p>SUBJ: LILLIAN_FELDMANN_SCHWARTZ
OBJ: #gio3.c4
MODAL: (SPECIF SCIENTIFIC_PAPER_2 (SPECIF published_on THE_VISUAL_COMPUTER_JOURNAL))
date-1: 1/1/1988, 31/1/1988
date-2:
Move:GenericInformation (4.41)
In January 1988, Lillian Feldmann Schwartz has circulated the information described in gio3.c4 by means of a scientific paper
published in 1988 on “The Visual Computer” Journal.
gio3.c4: (COORD gio3.c8 gio3.c9)
gio3.c8: (CAUSE gio3.c10 gio3.c11)
The information spread by Lillian Feldmann Schwartz via her paper is formed of two parts.</p>
      <p>The first part of the disseminated information consists in stating that what is described in gio3.c10 is originated by the events
collected in gio3.c11.
gio3.c10: BEHAVE:</p>
      <p>SUBJ: (SPECIF UNKNOWN_WOMAN_1 (SPECIF identified_with MONA_LISA))
{ obs, negv }
date-1: 1/1/1988, 31/1/1988
date-2:
Behave:HumanProperty (1.1)
The elementary event represented by occurrence gio3.c10 is a “negated event” (modulator negv), i.e., UNKNOWN_WOMAN_1 is
not MONA_LISA.
gio3.c11: (COORD gio3.c12 gio3.c13)
The reasons for failing to identify UNKNOWN_WOMAN_1 with MONA_LISA are (partially, see below) collected in gio3.c11.
gio3.c12: OWN:</p>
      <p>SUBJ: (SPECIF eye_ (SPECIF cardinality_ 2) UNKNOWN_WOMAN_1)
OBJ: property_
TOPIC: (SPECIF different_from (SPECIF eye_ (SPECIF cardinality_ 2) MONA_LISA))
MODAL: x_ray_analysis
{ obs }
date-1: today_
date-2:
Own:CompoundProperty (5.42)
A dissimilarity exists between the eyes of UNKNOWN_WOMAN_1 and those of MONA_LISA.
gio3.c13: OWN:</p>
      <p>SUBJ: (SPECIF mouth_ UNKNOWN_WOMAN_1)
OBJ: property_
TOPIC: (SPECIF different_from (SPECIF mouth_ MONA_LISA))
MODAL: x_ray_analysis
{ obs }
date-1: today_
date-2:
Own:CompoundProperty (5.42)
A dissimilarity exists also between the mouths of UNKNOWN_WOMAN_1 and MONA_LISA. In reality, the COORD list (binding
occurrence) gio3.c11 should include three additional predicative occurrences, practically identical to gio3.c12 and gio3.c13, but
including nose_, chin_ and hairline_ instead of eye_ and mouth_. To the extent that the insertion of these three additional
occurrences would not introduce any substantial improvement with respect to the understanding of the logic of the NKRL
coding, they have been suppressed for the sake of clarity.
gio3.c9:
The second component of L.F. Schwartz’s message is represented by a REFER(ence) – weak causality – binding occurrence.
In a REFER(ence) type of relationship, the second argument is needed but not sufficient to explain the first.
gio3.c14: BEHAVE:</p>
      <p>SUBJ: (SPECIF UNKNOWN_WOMAN_1 (SPECIF identified_with ISABELLA_D_ARAGONA))
{ obs, poss }
date-1: 1/1/1988, 31/1/1988
date-2:
Behave:HumanProperty (1.1)
It is possible (modal modulator poss(ible)) that the “unknown woman” corresponds to Isabella d’Aragona.
gio3.c15: (COORD gio3.c16 gio3.c17 gio3.c18)
The reasons for this (possible) identification are given in the three predicative occurrences listed in gio3.c16.
gio3.c16: PRODUCE:</p>
      <p>SUBJ: LEONARDO_DA_VINCI
OBJ: CARTOON_DRAWING_1
TOPIC: (SPECIF portrait_ ISABELLA_D_ARAGONA)
date-1: 1/1/1490, 31/12/1495
date-2:
Produce:Entity (6.2)
In the period 1490-1495, Leonardo da Vinci realized, in the form of a cartoon drawing, the portrait of Isabella d’Aragona.
gio3.c17: OWN:</p>
      <p>SUBJ: (SPECIF eye_ (SPECIF cardinality_ 2) ISABELLA_D_ARAGONA): (CARTOON_DRAWING_1)
OBJ: property_
TOPIC: (SPECIF coincident_with (SPECIF eye_ (SPECIF cardinality_ 2) UNKNOWN_WOMAN_1))
MODAL: x_ray_analysis
{ obs }
date-1: today_
date-2:
Own:CompoundProperty (5.42)
There is a correspondence between the eyes of Isabella d’Aragona and those of the unknown woman.
gio3.c18: OWN:</p>
      <p>SUBJ: (SPECIF hairline_ ISABELLA_D_ARAGONA): (CARTOON_DRAWING_1)
OBJ: property_
TOPIC: (SPECIF coincident_with (SPECIF hairline_ UNKNOWN_WOMAN_1))
MODAL: x_ray_analysis
{ obs }
date-1: today_
date-2:
Own:CompoundProperty (5.42)
There is a correspondence between the hairline of UNKNOWN_WOMAN_1 and the hairline of WOMAN_54.
gio3.c5: BEHAVE:</p>
      <p>SUBJ: (SPECIF individual_person art_historian (SPECIF cardinality_ several_))
OBJ: LILLIAN_FELDMANN_SCHWARTZ
MODAL: endorsement_
TOPIC: (SPECIF ISABELLA_D_ARAGONA_HYPOTHESIS LILLIAN_FELDMANN_SCHWARTZ)
{ for }
date-1: 1/1/1988, 31/1/1988
date-2: today_
Behave:FavourableAttitude (1.311)
Several art historians agree with Lillian Feldmann Schwartz about her Isabella d’Aragona hypothesis.</p>
      <p>It can be interesting to remark how the NKRL representation of a full (visual or textual) narrative can
always be represented under tree-shaped format – as a knowledge graph, then, to make use of a now very
popular terminology. This possibility represents, among other things, a simple way to immediately check
the logical coherence of a generic digitalized narrative. Figure 1 shows the representation under tree format
of the narrative of Table 1.</p>
      <p>
        We will now comment and explain the above encoding trying, at the same time, to highlight the interest
of the NKRL’s knowledge representation tools for symbolizing/managing complex “narrative” contents
independently from the type of their support. That is, independently from the fact that these contents are
disclosed to the public through visual-oriented techniques (i.e., making use of paintings, drawings, frescoes,
mosaics, sculptures, murals etc.) as in the case examined in the present paper – see also [
        <xref ref-type="bibr" rid="ref5 ref7">5, 7</xref>
        ] – or through
textual-oriented techniques (i.e., via books, papers, articles, news, all sort of written documents, etc.).
2.1
      </p>
    </sec>
    <sec id="sec-3">
      <title>Basic Remarks about the NKRL’s Knowledge Representation Structures</title>
      <p>From an ontological point of view, the most interesting characteristic of NKRL concerns the addition
of an ontology of elementary events to the usual ontology of concepts. This last is called HClass
(hierarchy of classes) in an NKRL context and allows us to represent the strictly lexical aspects of the
specific information to be dealt with. HClass presents some interesting, original aspects from an
ontological point of view with respect, e.g., to the representation of difficult notions like substances and
colors [4: 132-137]. However, HClass’ architecture is relatively traditional, and its concepts are
represented, to a large extent, according to the usual Semantic Web “binary” model.</p>
      <p>
        A pure binary-based approach faces, however, major difficulties when the entities to be represented
are not simple notions/concepts that can be defined a priori and inserted then in a graph-shaped static
ontology, but denote instead dynamic situations. These last are characterized, in fact, by the presence
of complex spatio-temporal information and of mutual relationships among their constituent elements
(including, e.g., intentions and behaviors). In specifying then the ontology of elementary events of
NKRL, an augmented n-ary approach has been chosen. In this approach, n-ary means to make use of
a formal representation where a given predicate can be associated with multiple arguments –
representing the n-ary purchase relation implies, e.g., associating with a purchase-like predicate several
arguments as a seller, a buyer, a good, a price, a timestamp, etc. Augmented means that – see in this
context the seminal “What’s in a Link” paper by William Woods [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] – within n-ary representations, the
logico-semantics links between the predicate and its arguments are explicitly represented making use
of the notion of functional role [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. The NKRL representation of a simple narrative like “Peter gives a
book to Mary” will include, then, the indication that Peter plays the role of subject/agent of the action
of giving, book is the object of this action and Mary the beneficiary.
      </p>
      <p>In the NKRL ontology of elementary events, the nodes are then represented by augmented n-ary
knowledge patterns called templates – this ontology is known then as HTemp, the hierarchy of
templates. Templates denote formally general classes of elementary states/situations/actions/episodes
etc. (designated collectively, for simplicity, as elementary events). Examples can be “be present in a
place”, “experience a given situation”, “have a specific attitude”, “send/receive messages”, etc.
Templates’ instances – called predicative occurrences – describe formally the meaning of specific
elementary events pertaining to one of these classes. The general representation of a template is given
by Eq. 1.</p>
      <p>(Li (Pj (R1, a1) (R2, a2) … (Rn, an))).
(1)</p>
      <p>In Eq. 1, Li is the symbolic label identifying a given template. Pj is a conceptual predicate. Rk is a
generic functional role, used to specify the logico-semantic function carried out by its filler ak, a
standard predicate argument, with respect to predicate Pj. We can note that, to avoid the ambiguities of
natural language and any possible combinatorial explosion problem, see Zarri [4: 56-61], both the
conceptual predicate of Eq. 1 and the associated functional roles are represented as primitives.
Predicates Pj belong then to the closed set {BEHAVE, EXIST, EXPERIENCE, MOVE, OWN, PRODUCE,
RECEIVE}, and the roles Rk to the set {SUBJ(ect), OBJ(ect), SOURCE, BEN(e)F(iciary), MODAL(ity),
TOPIC, CONTEXT}. As a consequence, the HTemp hierarchy is structured into seven branches, where
each of them includes only the templates created – see Eq. 1 – around one of the seven predicates Pj.
HTemp includes presently (September 2022) more than 150 templates, very easy to specialize and
customize.</p>
      <p>A simple example extracted from Table 1 above – a predicative occurrence identified by the
symbolic label (Li) gio3.c16 – will allow us to clarify what the use of Eq. 1 above implies from a concrete
point of view.</p>
      <p>When the NKRL template Produce:Entity corresponding to the general syntax of Eq. 1, see Table 2,
is instantiated to provide the representation of the elementary event, “In the period 1490-1495, Leonardo
da Vinci realized, in the form of a cartoon drawing, the portrait of Isabella d’Aragona”, we can see from
gio3.c16 that the predicate Pj of Eq. 1 (PRODUCE in this case) introduces three arguments ak, namely
LEONARDO_DA_VINCI, CARTOON_DRAWING_1 and (SPECIF portrait_ ISABELLA_D_ARAGONA)
via, respectively, the functional roles (Rk) SUBJ(ect), OBJ(ect) and TOPIC. As it appears clearly from
Table 2, in the actual representation of all the NKRL’s templates, the ak terms of Eq. 1 are implemented
under the form of variables vari associated with constraints expressed by HClass terms or combinations
of these terms; these constraints must be satisfied when the occurrences are created.
The Produce:Entity template
name: Produce:Entity
father: Produce:
position: 6.2
NL description: Creation of Generic Entities, Elements, etc.</p>
      <p>PRODUCE:
SUBJ: var1: [(var2)]
OBJ: var3
[SOURCE: var4: [(var5)]]
[BENF: var6: [(var7)]]
[MODAL: var8]
[TOPIC: var9]
[CONTEXT: var10]
{ [ modulators ], ≠abs }
var1: human_being_or_social_body
var3: artefact_, information_content, economic/financial_entity
var4: human_being_or_social_body
var6: human_being_or_social_body
var8: artefact_, activity_, process_, temporal_development
var9: information_content, physical_appearance, situation_
var10: situation_, symbolic_label
var2, var5, var7: location_</p>
      <p>
        In our example, the individual LEONARDO_DA_VINCI is an instance (through intermediate HClass
concepts like individual_person) of the high-level concept human_being_or_social_body. In NKRL,
individuals are denoted using upper-case characters, and concepts are in lower-case. It satisfies then the
constraint imposed on var1 of the Produce:Entity template – and, as a consequence, on all the possible
fillers of the SUBJ(ect) roles in all the predicative occurrences derived from this template. Similarly,
CARTOON_DRAWING_1 is an instance of cartoon_drawing, a specific HClass term included in the
branch of the HClass ontology having at the top the (high-level) concept artefact_ and integrating
intermediate items like drawing_ and artistic_artefact; it satisfies then the constraint on var3, the
OBJ(ect) filler of the template. With respect now to the TOPIC filler of gio3.c16, (SPECIF portrait_
ISABELLA_D_ARAGONA), we can note first that main element of this “structured argument” or
“expansion”, i.e., portrait_, is a HClass concept instead of an individual as in the previous two cases.
This is in conformity with an important NKRL principle that requires to limit as much as possible any
unnecessary proliferation of individuals in the context of concrete NKRL applications because of the
difficulties of creating and managing in a coherent way large amounts of this sort of entities, see [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] in
this context. Anyway, portrait_ is a specific term of image_, a concept pertaining to the
information_content subtree of HClass, and the constraint on var9 of Produce:Entity is then satisfied.
      </p>
      <p>Returning now to the structured arguments/expansions and their syntax, we can note that this
particular kinds of ak arguments take the form of recursive lists introduced by the four AECS operators,
the alternative operator ALTERN(ative) = A, the distributive operator ENUM(eration) = E, the collective
operator COORD(ination) = C and the attributive operator SPECIF(ication) = S. This last operator is
widely used in the predicative occurrences of Table 1; in the example of gio3.c16, it is used to exactly
specify that the portray we are speaking of is the Isabella d’Aragona’s portrait. An example of the use
of the operator COORD(ination) is given in the gio3.c7 predicative occurrence: in this case, the TOPIC’s
filler is represented by three SPECIF lists where the first tell us that PAINTING_2 (the hidden painting)
is painted on the same poplar plank utilized for the Mona Lisa’s portrait and, more precisely, that is
located under this portrait, the second that this specific painting is named “the hidden painting”, and the
third that PAINTING_2 is visible only under x-ray. This particular expansion allows us to introduce also
the so-called priority rule, see [4: 68-70], which supervises the interweaving of the AECS operators
within a structured argument by forbidding, e.g., to use a list of the COORD type within the scope of a
list SPECIF. The inverse is obviously admitted, as the structured argument of gio3.c7 demonstrates.</p>
      <p>Once a given template has been instantiated in order to obtain a valid n-ary predicative occurrence,
this last is reified by means of a symbolic name like gio3.c16, corresponding then to Li of Eq. 1 and
allowing the inclusion of this occurrence within wider conceptual structures, see next subsection. With
respect to templates and their instantiation procedures, we can also add that, in Table 2, the elements in
square brackets are facultative: this means that, for example, in the predicative occurrences derived
from the Produce:Entity template, only the SUBJ and OBJ roles and their associated elements are
obligatorily present.</p>
      <p>When needed, determiners (attributes) can be added to templates and predicative occurrences to
introduce further details about the basic core, “symbolic label – conceptual predicate – functional roles
– arguments of the predicate”, see Eq. 1, of their formal representation. In the template of Table 2, e.g.,
the variables var2, var5 and var7 denote, e.g., determiners/attributes of the location type represented, in
the corresponding predicative occurrences, by specific terms of the HClass concept location_ or by
individuals derived from these terms. Modulators represent an important category of
determiners/attributes that – unlike the location determiners that can be associated only to the fillers of
the SUBJ, OBJ, SOURCE and BENF functional roles – apply to a full, well-formed template or
predicative occurrence to particularize its meaning. They are classed into three categories, temporal
(begin, end, obs(erve)), deontic (oblig(ation), fac(ulty), interd(iction), perm(ission)) and modal
(abs(olute), against, for, main, ment(al), poss(ible), wish, etc.), see [4: 71-75]. Several examples of use
of the different categories of modulators appear in Table 1. In particular, the temporal modulator
obs(erve) is frequently used to denote that, at the specific date associated with the attribute date-1 (see
below), the elementary event corresponding to the predicative occurrence we are dealing with (e.g.,
gio3.c7) is in progress, without making any assumptions about the existence of this event before and
after the given date. The modal negv (negated event) modulator used in gio3.c10 is particularly
important, given that it is used to point out that the elementary event corresponding to the particular
predicative occurrence where negv can be found did not take place. In the gio3.c7 example,
UNKNOWN_WOMAN_1 has not been recognized as Mona Lisa.</p>
      <p>
        A last category of attribute/determiners concerns the two operators date-1, date-2. They are
necessarily associated with any well-formed NKRL predicative occurrence, see Table 1, and are used
in general to materialize the temporal interval normally associated with the elementary event
corresponding to a particular occurrence. Linking a specific date (full date, uncertain data, reconstructed
date etc.) associated with date-1 with one of the three temporal modulators introduced above allows us
to denote the beginning of a specific event (modulator begin), the termination of an event (end) or, as
already stated, the fact that a specific event is in progress at the date-1 date. A detailed description of
the formal system used in NKRL for the representation and management of temporal information can
be found, e.g., in [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ].
2.2
      </p>
    </sec>
    <sec id="sec-4">
      <title>Second order Conceptual Structures</title>
      <p>In the context of accurate, complete and digitally exploitable representations of any sort (pictural,
textual etc.) of complex and structured narrative events, it is evident the need for efficient tools able to
collect and join together, within a unified and coherent framework, all their possible basic, formalized
fragments that could be equated, in some way, to NKRL’s predicative occurrences. NKRL is endowed
with several tools capable to satisfy this need; this represents a further proof of the advantage of
NKRL’s approach with respect to other proposals committed to the formalization of narrative-like
information.</p>
      <p>A first, elementary way to associate together predicative occurrences in an NKRL context concerns
the possibility of making use of a sort of co-reference mechanism allowing us to logically associate two
or more predicative occurrences where the same individual(s) appear(s). In Table 1 for example, we
have utilized the individual UNKNOWN_WOMAN_1 to denote the unknown personage whose portrait
lies beneath Mona Lisa’s portrait. The consistent utilization of this artifice allows us, then, to associate
together seven predicative occurrences of Table 1 – gio3.c6, gio3.c10, gio3.c12 etc. – and to create then,
at lower costs, a (possibly interesting) thematic cluster. With respect to the above remark about the
practical dangers associated with an unjustified proliferation of individuals, we can note that the
creation of the individual UNKNOWN_WOMAN_1 is here absolutely necessary and then totally
justified. The introduction of a generic unknown_woman concept instead of the individual within the
seven predicative occurrences would have made, in fact, the management problems of the HClass terms
even worse by forcing the developers to create useless and expansive inference procedures to verify,
when finding unknown_woman in one of the seven occurrences, that this woman corresponds really to
those mentioned in the other ones.</p>
      <p>The most general and interesting way of logically associating single predicative occurrences is,
however, to make use of second order structures created through the reification of the single
occurrences. These structures reflect, at the digital formal level, surface linguistic connectivity
phenomena like causality, goal or indirect speech. “Reification” is intended here, as usual in a
Computer Science context, as the possibility of creating new objects out of already existing entities and
to say something about them without making reference to the original entities. In NKRL, reification is
implemented using the symbolic labels (the Li terms of Eq. 1) of the predicative occurrences according
to two different conceptual mechanisms.</p>
      <p>A first solution concerns the possibility of referring to an elementary or complex event as an
argument of another (elementary) event – a complex event corresponds to a coherent set of elementary
events. The (surface) connectivity phenomenon involved here is the indirect speech. An informal
example can be that of an elementary event X describing someone who speaks about Y, where Y is itself
an elementary/complex event. In NKRL, this mechanism is called completive construction, see [4:
8791]. It is illustrated, e.g., by the association of the occurrences gio3.c3/gio3.c4 in Table 1, where gio3.c4
– a binding occurrence (see below) – is introduced by gio3.c3 as filler of its OBJ(ect) role; the # prefix
in gio3.c3 indicates that its associated term is not a HClass item but an occurrence label. Some formal
restrictions must be respected. For example, only the OBJ, MODAL, TOPIC and CONTEXT functional
roles of a predicative occurrence pci can accept as filler the symbolic label Lj of a (generic) cj occurrence:
this last can then be, as in the case of the gio3.c3/gio3.c4 association in Table 1, one of those binding
occurrences we will introduce below.</p>
      <p>The second mechanism allows us to associate together, through several types of connectivity
operators, elementary/complex events that, unlike the previous case, can still be regarded as fully
independent entities. This mechanism – binding occurrences, see [4: 91-98] – is realized under the form
of lists formed of a binding operator Bni and its Li arguments, see Eq. 2:
(Lbk (Bni L1 L2 … Ln)) .
(2)</p>
      <p>Lbk is the symbolic label identifying the global binding structure: unlike templates and predicative
occurrences, binding occurrences are characterized by the absence of any predicate and functional
role. The Bnj operators are: ALTERN(ative), COORD(ination), ENUM(eration), CAUSE,
REFER(ence), the weak causality operator, GOAL, MOTIV(ation), the weak intentionality operator,
COND(ition); their precise logico-semantic definitions can be found in [4: 92]. The binding
occurrences bci must necessarily conform to a set of mandatory restrictions, like the following:
● Each term (argument) Lj that, in a binding list, is associated with one of the above operators denotes
exactly a single predicative or binding occurrence cj that is defined externally to the list. Therefore,
the arguments Lj are always single terms and cannot consist of lists of symbolic labels.
● Within binding occurrence of the ALTERN, COORD and ENUM type – see in the previous
subsection the corresponding AECS operators introduced, with the same basic meaning, in a structured
arguments/expansions context – no restrictions are imposed on the cardinality of the list, i.e., on the
possible number of arguments Lj.
● In the binding occurrences labeled with CAUSE, REFER, GOAL, MOTIV and COND only two
arguments Lm and Ln are admitted. The occurrences labeled with these five operators are then simply
of the form, (Lbk (Bni Lm Ln)). In these lists, the arguments Lm and Ln can denote, in general, either a
predicative or a binding occurrence: an exception are the COND(ition) binding occurrences, where
the first argument, Lm, must necessarily correspond to a predicative occurrence, see [4: 93-95].</p>
      <p>Several binding occurrences can be found in Table 1, most of the COORD type, see gio3.c1, gio3.c2,
gio3.c4, gio3.c11 etc. gio3.c5, i.e., (CAUSE gio3.c10 gio3.c11), is a binding occurrence of the “causal”
type. As explained above, it necessarily includes only two arguments, the general meaning being that
the event corresponding to its first argument gio3.c10, recognizing that UNKNOWN_WOMAN_1 is not
Mona Lisa, is caused by all the incoherencies described by the occurrences included in another binding
occurrence gio3.c11. In a context of formal representation of causal situations, it may be worth
emphasizing the difference between the conceptual meaning associated with the two binding operators
CAUSE and REFER(ence) – the syntax of the corresponding binding occurrences being identical, i.e.,
(Lbk (Bni Lm Ln)), with Bni corresponding to CAUSE or REFER. In the CAUSE case, Ln is both necessary
and sufficient to explain Lm while, in the REFER case, Ln is necessary but not sufficient to explain Lm.
This means that, in the interpretation of the Mona Lisa’s expert who has inspired the Table 1 coding,
and making reference to the REFER occurrence of Table 1, i.e., gio3.c9: (REFER gio3.c14 gio3.c15),
all the explications put forward by Lillian Feldmann Schwartz to support her Isabella d’Aragona
hypothesis are necessary but not sufficient to definitely validate this hypothesis.</p>
      <p>
        To conclude, we can remark that these binding structures are very important in an NKRL context
given that, in particular, the top-level formal structure introducing the NKRL representation of any sort
of complex narrative has obligatory the form of a binding occurrence, see gio3.c1 in Table 1. In this
COORD binding occurrence, the three main components of the whole representation are specified: they
are introduced, respectively, by gio3.c2, gio3.c3 and gio3.c5. Note that, even if several NKRL
representations of full narratives are introduced, as in Table1, by a COORD binding occurrence, any
other sort of binding occurrence can be used to this aim, e.g., a CAUSE binding occurrence, should the
narrative be reduced to the description of a particular event and of its causes. More in general, we can
also remark that the (although simplified) description of the second order features of NKRL presented
in this subsection should make particularly evident the advantages of using NKRL to formally describe
complex narrative information with respect to possible alternative solutions, for example those of
Semantic Web origin. These features compare favorably, e.g., with the RDF-star initiative [
        <xref ref-type="bibr" rid="ref11 ref12">11, 12</xref>
        ], the
last avatar of the continuous (and largely unsuccessful see, e.g., [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]) efforts made by the SW
community to get rid of all the problems generated by a too strict adherence to the “binary RDF” dogma.
RDF-star is an extension of the RDF conceptual data model enabling the creation of triples that
reference other triples as “subject” and “object” (in the RDF/SW meaning) resources; triples that
include a triple as subject or object are denoted as RDF-star triples. Apart from the possible logical
incoherencies of this approach derived from the syntactic/semantic limitations associated with its
strictly binary framework, and the lack of any real originality giving that these association techniques
are in reality very old ones dating back to at least the sixties (see, e.g., [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ] in this context), we can note
that the resulting RDF-star nested structures are still simply “triples”. In the NKRL case, on the contrary,
the entities related through the second-order structures are formalized complex events of any possible
degree of sophistication.
      </p>
    </sec>
    <sec id="sec-5">
      <title>3. Conclusion</title>
      <p>
        In this paper, we have proposed the use of advanced digital techniques to consent a generalized access
to all the important features, both physical and symbolic/conceptual, of any kind of Cultural Heritage
(CH) entities, and to make possible their global fruition. According then to the so-called Digital
Cultural Heritage Twin approach [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ], we suggest to conceive the digitalized image of any CH entity
as the association of two terms, the first describing the physical (in the largest leaning of this word)
properties of this entity, and the second supplying the formal description of all the immaterial/symbolic
aspects of the same entity. NKRL, the Narrative Knowledge Representation Language, has been chosen
for the implementation, in particular, of the immaterial component of this approach, because of its
wellknown capacity for dealing with important immaterial/symbolic aspects of many different realities.
      </p>
      <p>
        To demonstrate the viability of this methodology, we have illustrated in some depth the use of NKRL
to represent in digital form complex expressive cultural heritage entities, with particular reference to
those “iconographic narratives” entities corresponding to stories told in visual form that are conveyed
by works of art like paintings, drawings, frescoes, mosaics, sculptures, murals etc. and that represent a
particularly important component – both numerically and culturally – of the whole Cultural Heritage
domain. More specifically, an experiment has been conducted in the context of a project that concerns
the digital modeling of the Mona Lisa (“La Gioconda”) painting by Leonardo Da Vinci, see also [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] in
this context, according to the Digital Cultural Heritage Twin approach: this experiment was devoted to
represent in NKRL format the main characteristics of the “hidden painting” that lies beneath Mona Lisa
on the same poplar panel. As a result of this study, the well-known generality and efficiency properties
of NKRL with respect to the representation and management of any possible form of “narrative” (both
in written and pictorial form) have been confirmed, both from a specific knowledge representation and
an inferential point of view – with respect to this last point, not dealt with explicitly in this paper for
space limitations see, e.g., [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. The potential of NKRL overcomes, in fact, the limitations of simple
description under metadata form or knowledge representation tools unsuitable for representing
complex situations/events, allowing us to explore the “immaterial” components (including its emotional
factors) associated to any possible Cultural Heritage entities in a more expressive, complete and
meaningful form. We can also note that the above NKRL’s features do not concern merely specific
characteristics of the Iconographic Cultural Heritage domain, but represent in general distinguishing
qualities of Human Sciences in their totality. NKRL could be borne in mind, then, as an attractive
possibility for implementing really accurate digitization activities, and interesting applications of the
corresponding results, for the whole Human Sciences field.
      </p>
    </sec>
    <sec id="sec-6">
      <title>4. References</title>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>European</given-names>
            <surname>Commission</surname>
          </string-name>
          ,
          <year>Horizon 2020</year>
          ,
          <string-name>
            <given-names>Work</given-names>
            <surname>Programme</surname>
          </string-name>
          2018-2020 - 13.
          <article-title>Europe in a changing world-Inclusive, innovative</article-title>
          and reflective societies,
          <year>2019</year>
          . Available online: https://ec.europa.eu/research/participants/data/ref/h2020/wp/2018-2020/main/h2020-wp1820
          <source>societies_en.pdf (accessed on 3 September</source>
          <year>2022</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <surname>Grieves</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <article-title>Origins of the Digital Twin Concept</article-title>
          - Working Paper; Florida Institute of Technology: Melbourne,
          <string-name>
            <surname>FL</surname>
          </string-name>
          , USA,
          <year>2016</year>
          . Available online: https://www.researchgate.net/publication/307509727
          <source>(accessed on 3 September</source>
          <year>2022</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>Amelio</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <article-title>Exploring Leonardo Da Vinci's Mona Lisa by Visual Computing: a Review</article-title>
          ,
          <source>in: Proc. of the 1st Int. Workshop on Visual Pattern Extraction</source>
          and
          <article-title>Recognition for Cultural Heritage Understanding (VIPERC 2019), CEUR-WS</article-title>
          .org Vol-
          <volume>2320</volume>
          , Aachen,
          <year>2019</year>
          , pp.
          <fpage>74</fpage>
          -
          <lpage>85</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Zarri</surname>
            ,
            <given-names>G.P.</given-names>
          </string-name>
          ,
          <source>Representation and Management of Narrative Information - Theoretical Principles and Implementation</source>
          , Springer-Verlag, London,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <surname>Amelio</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Zarri</surname>
            ,
            <given-names>G.P.</given-names>
          </string-name>
          , Conceptual Encoding and
          <article-title>Advanced Management of Leonardo da Vinci's Mona Lisa: Preliminary Results</article-title>
          ,
          <source>Information MDPI</source>
          <volume>10</volume>
          (
          <issue>10</issue>
          ),
          <volume>321</volume>
          ,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <surname>Schwartz</surname>
            ,
            <given-names>L.F.F.</given-names>
          </string-name>
          ,
          <article-title>The Mona Lisa identification: Evidence from a computer analysis</article-title>
          ,
          <source>The Visual Computer</source>
          <volume>4</volume>
          (
          <issue>1</issue>
          ),
          <fpage>40</fpage>
          -
          <lpage>48</lpage>
          ,
          <year>1988</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <surname>Zarri</surname>
            ,
            <given-names>G.P.</given-names>
          </string-name>
          ,
          <article-title>Use of a Knowledge Patterns-Based Tool for Dealing With the “Narrative Meaning” of Complex Iconographic Cultural Heritage Items</article-title>
          ,
          <source>in: Proc. of the 1st Int. Workshop on Visual Pattern Extraction</source>
          and
          <article-title>Recognition for Cultural Heritage Understanding (VIPERC 2019), CEURWS</article-title>
          .org Vol-
          <volume>2320</volume>
          , Aachen,
          <year>2019</year>
          , pp.
          <fpage>25</fpage>
          -
          <lpage>38</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Woods</surname>
            ,
            <given-names>W.A.</given-names>
          </string-name>
          ,
          <article-title>What's in a Link: Foundations for Semantic Networks, in: Representation and Understanding: Studies in Cognitive Science</article-title>
          , Academic Press, New York,
          <year>1975</year>
          , pp.
          <fpage>35</fpage>
          -
          <lpage>82</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <surname>Zarri</surname>
            ,
            <given-names>G.P.</given-names>
          </string-name>
          ,
          <article-title>Functional and Semantic Roles in a High-Level Knowledge Representation Language</article-title>
          ,
          <source>Artificial Intelligence Review</source>
          <volume>51</volume>
          (
          <issue>4</issue>
          ),
          <fpage>537</fpage>
          -
          <lpage>575</lpage>
          ,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <surname>Zarri</surname>
            ,
            <given-names>G.P.</given-names>
          </string-name>
          ,
          <article-title>Modelling and Exploiting the Temporal Information Associated with Complex 'Narrative' Documents</article-title>
          ,
          <source>International Journal of Knowledge Engineering and Data Mining</source>
          <volume>6</volume>
          (
          <issue>2</issue>
          ),
          <fpage>135</fpage>
          -
          <lpage>167</lpage>
          ,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <surname>Hartig</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <article-title>Foundations of RDF⋆ and SPARQL⋆ (An Alternative Approach to Statement-Level Metadata in RDF)</article-title>
          .
          <source>In: Proc. of the 11th Alberto Mendelzon Int. Workshop on Foundations of Data Management and the Web, CEUR-WS.org</source>
          Vol-
          <volume>1912</volume>
          , Aachen,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>Hartig</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Champin</surname>
          </string-name>
          , P.-A.,
          <string-name>
            <surname>Kellog</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Seaborne</surname>
            ,
            <given-names>A</given-names>
          </string-name>
          . (eds.)
          <string-name>
            <surname>Arndt</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Broekstra</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>DuCharme</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Patel-Schneider</surname>
            ,
            <given-names>P.F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Prud</surname>
          </string-name>
          'hommeaux, E.,
          <string-name>
            <surname>Thibodeau</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Thompson</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          (
          <article-title>Authors) and additional contributors, RDF-star and SPARQL-star</article-title>
          ,
          <source>Draft Community Group Report 13 April</source>
          <year>2021</year>
          . Available online: https://w3c.github.io/rdf-star/cg-spec
          <source>/2021-04-13.html (accessed on 3 September</source>
          <year>2022</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Trame</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kessler</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kuhn</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          ,
          <article-title>Linked Data and Time - Modeling Researcher Life Lines by Events</article-title>
          .
          <source>In: Proc. of COSIT</source>
          <year>2013</year>
          , LNCS 8116, Springer, Berlin,
          <year>2013</year>
          , pp.
          <fpage>205</fpage>
          -
          <lpage>223</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>Ceccato</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          (
          <year>1964</year>
          ).
          <source>Automatic Translation of Languages. Information Storage and Retrieval</source>
          ,
          <volume>2</volume>
          (
          <issue>3</issue>
          ),
          <fpage>105</fpage>
          -
          <lpage>158</lpage>
          ,
          <year>1964</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>Zarri</surname>
            ,
            <given-names>G.P.</given-names>
          </string-name>
          , Digital Cultural Heritage Twins:
          <article-title>New Tools for a Complete Fruition of the Cultural Heritage Entities</article-title>
          .
          <source>In: Intelligent Decision Technologies: Proceedings of the 14th KES-IDT 2022 Conference, Smart Innovation, Systems and Technologies</source>
          , vol
          <volume>309</volume>
          , Springer, Singapore,
          <year>2022</year>
          , pp.
          <fpage>37</fpage>
          -
          <lpage>48</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>