<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Annotation of Metadata for Dramatic Texts: the POP-ODE Initiative</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Vincenzo Lombardo</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Rossana Damiano</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Antonio Pizzo</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Carmi Terzulli</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Eleonora Ceccaldi</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Giacomo Albert</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Davide Pulizzotto</string-name>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>CIRMA and Dipartimento di Informatica, Universita di Torino</institution>
          ,
          <addr-line>Torino</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>CIRMA and Dipartimento di Studi Umanistici, Universita di Torino</institution>
          ,
          <addr-line>Torino</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>Departement de Philosophie, Universite du Quebec a Montreal</institution>
        </aff>
      </contrib-group>
      <fpage>30</fpage>
      <lpage>42</lpage>
      <abstract>
        <p>This paper addresses the problem of the metadata annotation for the dramatic texts. Metadata for drama describe the dramatic qualities of a text, connecting them with the linguistic expressions. Relying on an ontological representation of the dramatic qualities, the paper presents an annotation environment for the creation of a corpus of annotated texts.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
    </sec>
    <sec id="sec-2">
      <title>Drama and annotation</title>
      <p>
        A drama is a story conveyed through characters who perform live actions: for
example, theatrical plays (Shakespeare's Hamlet ), TV series (HBO's Sopranos4),
but even reality shows (CBS's Survivor 5), and games (Ubisoft's Assassin's Creed
6). Drama has been pivotal for storytelling across all cultures and ages [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ],
growing through di erent media and being most pervasive, from theater and cinema
to TV and videogames [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] . Along these media, a single drama can assume several
forms, ful lling a number of its core conditions. For example, the abstraction of
the oral tale Cinderella has, e.g., Perrault's [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ] and Disney's [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] versions.
Metadata annotation for dramatic texts must encode the major concepts and relations
of the drama domain, which have been shared by a majority of scholars in the
drama literature. Here, we refer to the so{called dramatic qualities, that is those
elements that are necessary for the existence of a drama, which can be found
in several drama analyses, e.g. [
        <xref ref-type="bibr" rid="ref22 ref23 ref7 ref9">9, 22, 7, 23</xref>
        ]. All the initiatives on this topic have
shared similar sets of elements, namely story units, characters or agents, actions,
intentions or plans, goals, con icts, values at stake, emotions. These elements
are annotated in connection with media chunks (e.g., text paragraphs), often
with the goal of constructing corpora of annotated narratives and the study of
the relationships between the linguistic expression of the story in the narrative
and its content.
      </p>
      <p>In order to clarify the dramatic elements, we anticipate an informal
annotation of a scene taken from Shakespeare's Hamlet : the so called \nunnery" scene.
In this scene, situated in the Third Act, Ophelia is sent to Hamlet by Polonius
(her father) and Claudius (Hamlet's uncle, the king) to con rm the
assumption that Hamlet's madness is caused by his rejected love. According to the two
conspirators, Ophelia should induce him to talk about his inner feelings. At the
same time, Hamlet tries to convince Ophelia that the court is corrupted and that
she should go to a nunnery. In the middle of the scene, Hamlet puts Ophelia to
a test to prove her honesty: guessing (correctly) that the two conspirators are
hidden behind the curtain, he asks the girl to reveal where her father Polonius
is. She decides to lie, by replying that he is at home. Hamlet realizes from the
answer that also Ophelia is corrupted and consequently becomes very angry,
realizing that there is no hope to redeem the court. The climax incident in the
scene consists of a question-answer pair:
{ Hamlet: \Where is your father?"
{ Ophelia: \At home, my Lord!"</p>
      <p>This is a (very relevant) story unit: boundaries are decided through the
detection of a speci c goal pursuit, distinct from the goals pursued in the previous
unit. Here Hamlet, one of the two characters in the unit, is pursuing the goal of
proving Ophelia's honesty. Honesty is a value for Hamlet, and Ophelia's
behavior is putting at stake such a value. So, he decides to pursue the goal of proving
Ophelia honesty through a plan in which he asks a question he knows the answer
of, i.e. the current location of her father Polonius (Hamlet is sure that Polonius is
in the same room, behind a curtain). and Ophelia lies, by answering with a false
5 http://www.cbs.com/shows/survivor/, visited on 21 July 2017
6 https://www.ubisoft.com/en-US/game/assassins-creed/, visited on 21 July 2017
location, i.e. Polonius' home. So, we can annotate this unit with the following
metadata:
- Unit: Hamlet tests Ophelia for honesty
- Characters: Hamlet, Ophelia
- Hamlet values at stake: Honesty
- Hamlet goal: Prove Ophelia honesty
- Hamlet plan: Asking Ophelia a rhetorical question
- Hamlet plan accomplishment: FALSE
- Ophelia values at stake: father's authority, honesty
- Ophelia goal: Respect father's authority
- Ophelia plan: Lying about presence of Polonius in the room
- Ophelia plan accomplishment: TRUE
- Conflict: Hamlet plan VS. Ophelia plan
- Hamlet emotions: Distress, Reproach, Anger
- Ophelia emotions: Disappointment, Joy, Shame</p>
      <p>
        In particular, notice that emotions arise from the accomplishment of the
characters with respect to their values, an adaptation of OCC emotional theory
[
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]. So, Hamlet feels Distress because his plan fails, feels Reproach because
Ophelia is putting at stake honesty (an important value for him), and Anger
as a consequence of Distress and Reproach. Ophelia, though feels Joy because
she achieved her goal to respect father's authority (the value with maximum
priority here), she is disappointed because her hope to convince Hamlet to talk
about his feelings failed and is ashamed because she put at stake another value
of hers, namely honesty (with a lower priority here, but always present). There
also are long spanning values at stake, goals, and plans of the characters as well
as con icts.
      </p>
      <p>
        These elements have been taken into account by several annotation projects,
with several non-empty intersections. Project DramaBank, which has proposed
a template based language for describing the intentional content of textual
narratives, is a standalone downloadable application relying on an internal,
nonstandardized representation format [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. A media-independent model of story is
provided by the OntoMedia ontology [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ], exploited across di erent projects
(such as the Contextus Project7) to annotate the narrative units of di erent
media objects, ranging from written literature to comics and TV ction. In the
eld of cultural heritage dissemination, the StorySpace ontology supports
museum curators in linking the content of artworks through stories [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ], with the
ultimate goal of enabling the generation of user tailored content retrieval. Some
initiatives also rely on automatic annotation approaches, which can overcome the
di culties of recruiting annotators, especially when minimal schemata targeted
at grasping the regularities of written and oral narratives at the discourse level
can be worked out [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. Finally, Drammar8 is an ontology of drama, speci cally
conceived to annotate dramatic media [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], that makes the knowledge about
7 http://www.contextus.net, visited on 21 July 2017
8 https://www.di.unito.it/wikidrammar, visited on 21 July 2017
LDaryaemratic
Timeline
Layer
Motivational
Layer
      </p>
      <p>Scene
Timeline1</p>
      <p>Unit
DrammarScene</p>
      <p>spans</p>
      <p>Timeline2
Timeline123
hasPreconditions</p>
      <p>Unit
hasMember
Action Action</p>
      <p>Unit
hasMember</p>
      <p>Action Action
Consistent</p>
      <p>StateSet
StateStateState contains contains contains contains</p>
      <p>Directly Directly Directly Directly
hasPreconditions ExePculatnable ExePculatnable ExePculatnable ExePculatnable
list of plans list of plans
tree of scenes
hasEffects
Consistent</p>
      <p>StateSet
StateStateState
hasEffects
knows
Belief
Value hasValue</p>
      <p>Agent1intends achiAevbPessltarnact isMotivatedBy AbPsltaranacchtieves intenAdgsent2</p>
      <p>ConflictSet
hasValue Value
knows</p>
      <p>Belief
hasGoal</p>
      <p>Goal inConflictWith Goal</p>
      <p>
        hasGoal
drama available as a vocabulary for the linked interchange of annotations and
readily usable by automatic reasoners, such as, e.g., the calculation of
characters' emotions [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]. The usage of Drammar introduces two advantages: rst, it
is complete with respect to dramatic elements and non-dependent with respect
to di erent media, considered as a mapping object from the metadata
representation; second, as we have seen above, being a formal ontological model we can
exploit the automatic reasoning capabilities for the discovery of new metadata.
      </p>
      <p>
        However, the use of ontology editors and reasoning tools is challenging for
drama experts [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ]. For the accomplishment of the annotation task, it is crucial
to provide a friendly environment with metaphors and interfaces that directly
descend from the drama scholarship, which abstracts the annotator from the
details of the ontology representation. Here we describe a pipeline and system
for the metadata annotation of dramatic texts.
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>The Drammar ontology</title>
      <p>
        In order to build a formal encoding of the dramatic elements, Drammar (see [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]
and [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] for thorough descriptions) resorts to a set of theories and models that are
well established in Arti cial Intelligence and Computer Science. The ratio of this
design strategy is twofold: on the one side, it relies on widespread, sound models,
with formal properties that have been investigated in depth; on the other side, it
augments the interoperability of the representation with other encodings, which
can be contributed by several disciplines, such as, e.g., interactive storytelling
and procedural animation.
      </p>
      <p>
        The design of Drammar ontology relies on three representation layers (see
Figure 1 for a synoptic overview). The rst, the closest to the drama
document to be annotated, is the observable timeline (middle of Figure 1), appraised
through a literary text or an audiovisual medium, a succession of the incidents
(or actions) that happen in the drama. Incidents are assembled into discrete
structures, called units. Each succession of incidents forms a sub-timeline of the
whole timeline of the drama. This level is formalized through the Situation
Calculus paradigm ([
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]): with sub-timelines that function as operators advancing
the story world from one state to another (states aggregated in consistent state
sets, ellipses in the gure), that work as preconditions and e ects of some
subtimeline of incidents. The actions result from the deliberation process of the
characters, named agents here.
      </p>
      <p>
        The deliberation process is represented by the motivational layer (bottom of
Figure 1), which centers upon the notion of the character's intention in achieving
(or trying to achieve) a goal. The intention, or the commitment of the character,
is represented by a plan, which consists of the actions that are to be carried out
in order to achieve some goal; plans are organized hierarchically, with high-level
behaviors (abstract plans) formulated as lists of lower-level plans, or subplans,
until the directly executable plans, which directly contain actions. Goals
originate from the values of the characters that are put at stake and need to be
restored, given the beliefs (i.e. the knowledge) of the agents. This level is
formalized through the rational agent paradigm, or BDI (Belief, Desire, Intention)
paradigm ([
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]) (which is also applied in the computational storytelling
community ([
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]; [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]). This is why characters are encoded as agents in Drammar
(bottom of Figure 1). The agent is characterized by goals, beliefs, values engaged,
and plans; values can be at stake or in balance; plans can be in con ict with other
plans, possibly of other agents; a con ict set aggregates all the plans, agents and
goals that determine a dramatic scene (DrammarScene), through the game of
alternate accomplishments. The plan is the major structure of the Motivational
Layer, where all the other entities participate ([
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]); plan hierarchies are trees
of plans, with abstract plans that recursively dominate children subplans, until
directly executable plans with actions that are actually performed by the agents
in the drama; each plan hierarchy pertains to a single agent; several hierarchies
(pertaining to several agents) project onto the same portion of the timeline,
often with goals in con ict (actually, con icts motivate a dramatic scene). The
success/failure in achieving goals as well as in supporting own values is
responsible for agents' appraisal of the drama incidents. Plans have preconditions and
e ects, which are consistent sets of states (where consistent means that there no
two states in contradiction within the set); when some plan motivates a
timeline, its preconditions and e ects (the consistent state sets mentioned above) are
included in the preconditions and e ect of a timeline.
      </p>
      <p>The dramatic layer (top of Figure 1), which is directly inspired by the
literature on drama theory, accounts for the hierachical structure of the scenes:
scenes are recursively composed of daughter scenes. Scenes span timelines, that
is sequences of units. Some scenes are called DrammarScenes, meaning that they
are motivated by some con ict over the characters' intentions, which is the
characterization of scenes according to the Drammar ontology.</p>
      <p>
        The abstract ontology, expressed as a set of logical speci cations of classes
and properties, is expressed through a formal language to become a digital,
textual artifact that can be fed to a software program (for manipulation, querying,
comparison, etc.). In particular, Drammar is expressed through the ontology
language, which has been designed as part of the Semantic Web project and allows
conceptual models to be described in an unambiguous way, open to
understanding and manipulation by both human users and software programs. The concepts
and relations introduced above are encoded in the ontology Drammar, written
in the Semantic Web language known as OWL (Ontology Web Language). In
particular, Drammar is written in a speci c sub-language, OWL2 RL (Rule
Language), a syntactic and semantic restriction of OWL 2 ([
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]), which provides the
adequate tradeo between expressivity and complexity with respect to the
requirements of the drama domain (see ([
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]) for an introduction to computational
ontologies). Also, Drammar includes classes that are intended as an interface
between the drama domain concepts and the linguistic and common sense types
of knowledge that express the content of the drama when instantiated in media,
according to the paradigm of linked data ([
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]).
4
      </p>
    </sec>
    <sec id="sec-4">
      <title>The POP-ODE initiative</title>
      <p>The POP-ODE initiative consists of several aspects: the pipeline, the web
interface, the visualization tool, the corpus.
4.1</p>
      <p>
        The POP-ODE pipeline
POP-ODE consists of a pipeline and a number of tools for the accomplishment
of the annotation task of metadata for dramatic texts (see Figure 2). The
Figure 2 shows the pipeline vertically, on the left, from top to bottom. A drama
encoding annotator (at the top, left) works through a web-based interface to ll
the tables of a data base built according to the tenets of ontology Drammar,
which encodes the elements mentioned above, namely story units, characters,
actions, intentions or plans, goals, con icts, values at stake (emotions are
calculated automatically from these data). At the same time, the annotator can
select the text chunks that correspond to some annotation (from the .txt le).
The ontology axioms have already been encoded by the drama scholar (possibly
supported by the ontology engineers), through the well-known Protege editor9.
Exploiting these axioms (contained in a conceptual model, OWL le), the
mapper module DB2OWL converts the data base tables into an OWL le, actually
a Drammar Instantiated Ontology le (OWL DIO le). A further software
module, OWL2CHART, extracts the individuals and properties in a XML Drammar
Chart le, which is then visualized by the interactive chart module, the user
can interact with [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. The interactive chart module, developed for visualization
purposes and as a teaching device, here supports the validation of the produced
9 http://protege.stanford.edu, visited on 21 July 2017.
ontology, allowing for a fast checking of the encoded axioms. On right column,
the gure shows examples from each step. The top of the column shows a
thumbnail of the web interface (see detail below). As example for the data base, there
are two example tables, Agent and Value, connected through the Agent identi er
(Hamlet has honesty as a value). The assertion example from the DIO OWL le
concerns the agent Marcellus who intends the plan of reaching the guard post.
The example from the XML chart representation shows the attributes of the
plan above, which determine its visualization color and shape in the interactive
chart.
4.2
      </p>
      <p>The POP-ODE web interface
Figure 3 shows the web interface for the annotation. The top of the gure shows
the text selector: on the left, the Hamlet text from an authoritative source
(Shakespeare's navigators), on the right, the text chunk that pertains to the
unit selected below. The middle of the gure shows the unit annotation, that is
the actions that have been identi ed by the annotator in the selected segment
of the text, recognized as a bounded unit. On the left and the right of the unit
annotation are the previous and the following unit in the story timeline, with the
values that are at stake or at balance before and after the current unit. So, in this
example, the unit concerns Polonius that asks Ophelia about her feelings; it
occurs after Polonius blesses Laertes on his departure and before Ophelia promises
to avoid Hamlet. The bottom of the gure concerns the plans that motivate such
a unit. In particular, going from left to right, we see that, Ophelia (the agent
or character shown at the left), who has the goal of meeting Hamlet, has the
plan of convincing her father Polonius that Hamlet is reliable, and this plan is
in con ict with Polonius' plan who wants to convince Ophelia that she is too
candid for Hamlet. As we know from the following unit, Polonius will succeed in
convincing Ophelia, and actually Ophelia's plan fail (see \accomplished? NO"
at the far right).
4.3</p>
      <p>The POP-ODE visualization tool
Figure 4 shows a detailed visualization of the \nunnery" scene. The timeline
of the story units (middle of gure) is the pivotal element onto which the
upper and lower part of the interface hinge. The header on the left contains the
description of each row (from top to bottom: play, acts, scenes, timeline and
characters, represented by their initials). The upper part of the interface
contains the incident structure, organized as a recursive hierarchy of acts, scenes
and units that acknowledges both the tradition of theatrical writing and the
most recent theories of scriptwriting. At each level of hierarchy, an arc marks
the presence of some segment of text: the grey box situated on each arc can
be clicked to display information about the segment (the use of boxes to signal
text content is consistently repeated through all the components). Each arc is
marked by a number of segments of di erent colors, which are intended as visual
cues of the participation of the characters to the segment. The lower part of</p>
      <sec id="sec-4-1">
        <title>Text</title>
        <p>.txt
annotator</p>
      </sec>
      <sec id="sec-4-2">
        <title>Conceptual</title>
      </sec>
      <sec id="sec-4-3">
        <title>Model .owl</title>
        <p>Protégé</p>
      </sec>
      <sec id="sec-4-4">
        <title>Web-based interface DB .sql</title>
        <p>DB2OWL
mapper
DIO
.owl</p>
      </sec>
      <sec id="sec-4-5">
        <title>OWL2CHART extractor iChart .xml</title>
      </sec>
      <sec id="sec-4-6">
        <title>Interactive chart</title>
        <p>VALUE
id
name
descr
idAge
nt
1
honesty
Hamlet
honesty
3</p>
        <p>AGENT
id
name
print
3
Hamlet
Hamlet,
prince
&lt;ObjectPropertyAssertion&gt;
&lt;ObjectProperty</p>
        <p>IRI="drammar.owl#isIntendedBy"/&gt;
&lt;NamedIndividual</p>
        <p>IRI="#reachGuard216Pl"/&gt;
&lt;NamedIndividual</p>
        <p>IRI="#Marcellus134Ag"/&gt;
&lt;/ObjectPropertyAssertion&gt;
&lt;plan
id=”reachGuard216Pl"
agent="Marcellus134Ag"
goal=”startGuard94Go"
accomplished=”T" … &gt;
&lt;/plan&gt;
user
the interface contains the characters' individual tracks. The alignment with the
storyline shows how the characters' intentions motivate each segment of the
storyline. Plans that are abandoned by characters, because something went wrong
during the execution, are represented by incomplete arcs marked by a crossed
box. For example, Ophelia abandons the intention to return the gifts to Hamlet
at some point. Also, notice the hierarchical representation of the intentions of
the characters, with more complex intentions encompassing shorter, simpler
intentions that only span one unit of the timeline. This visualization shows clearly
the con ict between Ophelia and Hamlet in this scene. Notice, for example, that
her intentions span longer subsequences of the timeline, and that their
beginning is always followed by the beginning of Hamlets' intentions, signaling the
fact that he is mostly reactive in this scene.</p>
        <p>The number of plan failures spanning the same segment, moreover, signal
the high level of con ict that characterizes this part of the play. The vertical
alignment of the characters' intentions distributed along the tracks with the story
incidents along the timeline make the audience perceive the logical sequencing
of actions, and represents the credibility of the story in terms of a consistent
list of incidents. Long{term intentions denote a structured deliberation phase:
in the visualization of the nunnery scene the intentions attributed to Ophelia are
of a higher level with respect to Hamlet's ones, thus showing that she holds the
overarching goal in the scene (although she fails ultimately). Opposite, Hamlet
display lower hierarchies, thus showing that he is mainly reactive to a situation
designed by others. The synchronous occurring of two characters' intentions
(such as the ones of Hamlet and Ophelia in the "nunnery" scene) reveals the
orchestration of con icts. In this case, two arcs of two di erent characters' tracks
happen to span the same scene or incident onto the timeline; and possibly have
opposite result on accomplished, with one of the two failing (barred rectangle and
interrupted arc). This means that there is a con ict between the characters and
our visualization provides a clear image of the orchestration of con icts and their
execution. For example, on the left, in the scene "returning gift", the con ict is
between the intention of Ophelia of returning Hamlet the gifts he gave her and
the intention of Hamlet of refusing the gifts, by saying that they were not his, and</p>
        <p>Hamlet is successful in his intention; on the right, in the scene "where question",
the intention of Hamlet of testing Ophelia's honesty fails, because Ophelia lies,
with the intention of respecting her father Polonius' authority. The succession
of intentions displayed by a character's track represents the character's changes
through planning and re{planning because of the con icts with other characters,
thus stress the emotional charge of the drama. This is particular evident in
the case of Ophelia (Figure 4): as we have seen, she has the highest level of
intentions in this scene, composed of two main intentions (bottom of the gure)
separated by a gap lled by one of Hamlet's intentions. This shows that Ophelia
has to execute some sort of re{planning, given the failure of the rst (bottom
left), so to regain the lead of the scene with another overarching plan (bottom
right). Moreover, all along the scene we see that there is a large number of failed
intentions (rectangles barred with a cross); hence, the visualization reveals the
inner nature of this scene with failed attempts on both sides to achieve their
goals: on the one hand, Ophelia wants to discover the motivation for Hamlet's
madness, on the other, Hamlet wants to send the "fair Ophelia" to the nunnery,
but discovers that she is not honest at all.
5</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>The corpus of annotated drama</title>
      <p>The pipeline above has been used to build a crowdsourced corpus of annotated
drama. Currently, there are a few ongoing projects in annotating drama from
classic repertoire, used in theatre, cinema and media programmes. Students,
about fty per year, receive a focussed short training in formal representation
and predicate logic; then, they are assigned a scene (actually, a unit) from a
classical drama to be accessed through the web interface. They ll the forms
concerning units (upper part, with previous and next unit in the timeline), on
one hand, and plans, goals, agents, and values (lower part, selected through
plans), on the other. They also annotate con icts over plans and values that are
put at stake or in balance by the incidents in the units.</p>
      <p>Inter-annotator agreement is managed by a supervisor, who is expert in
drama studies. The intervention of the supervisor is necessary to understand
whether some annotation is a paraphrase of another and whether the two
annotations can be reduced to one; in case this is not possible, the two versions
remain. A typical case that occur is the segmentation of a unit into incidents:
some students only nd a single incident within a unit (so, the unit is reduced to
one incident); other students encode several incidents within a unit, and usually
partial overlap boundaries of incidents. The policy of the supervisor has been
to identify the minimal units, and each segmentation proposal is expressed in
terms of the basic units built arti cially.</p>
      <p>Although the task looked very challenging, students with many kinds of
backgrounds (psychologies, media studies, philosophy, media studies) could perform
the task. The tool has proven to be e ective in inferring a number of classes and
relations of the ontology that are syntactically important for the coherence of the
representation but are cumbersome and error-prone for the task of a manual (or
semi-manual) annotator. For example, when an annotator states that some scene
is spanning from this to to that unit, the tool automatically creates a timeline.
We are going to make a vast and e ective test of the annotation tool over several
student classes, together with questionnaires and etnographic observations, to
evaluate the functioning of the tool and to create a vast corpus for studies in the
digital humanities.</p>
      <p>
        The current corpus of annotated drama documents consists of a small number
of video and textual drama documents, respectively (see table 1). Though we
have not carried a thorough evaluation of the annotation, we have employed the
annotated documents in two applicative tasks: the rst is the calculation of the
emotions felt by the characters through automatic reasoning, on the basis of the
events and the intentions manually annotated [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]; the second is the realization of
printed charts of the characters' intentions, aligned with the timeline of incidents
[
        <xref ref-type="bibr" rid="ref13">13</xref>
        ], currently employed in the didactics of drama writing at the University
of Torino. We are going to evaluate the appropriateness of Drammar on the
adequacy of description from the point of view of research on the humanities.
6
      </p>
    </sec>
    <sec id="sec-6">
      <title>Conclusion</title>
      <p>In this paper, we have described the POP-ODE initiative for the metadata
annotation of dramatic texts. We have described the annotation pipeline for drama
documents and a web-based annotation tool. The tool implements a visual
interface for the representation of the intentional motivations of the characters
(agents) to act within the drama. The tool has proven to be very e ective in
Medium Work Fragment
Text Hamlet (Shakespeare) whole text
Text Mother Courage (Brecht) whole text
Text L'Arialda (Testori's Italian neorealism) whole text
Movie Apocalypse now helicopter attack scene (ride of valkyries)
Movie Taxi driver \Are you talkin' me?" scene
Movie Matrix bullet time scene
Movie La Dolce Vita Trevi fountain scene
Movie The Clockwork Orange Flat Block Marina scene
Movie Blade Runner \I've seen things ..." scene
Movie The deer hunter Russian roulette scene
Movie The Godfather Sollozzo omicide scene
Movie The Snatch dog VS. rabbit scene
Movie Kill Bill - Vol. 2 \losing the other eye" scene
Musical video clip Taylor Swift's \You belong with me' 3-min video
Advertisement clip \Zippo" lighter commercial 30-sec video
Animation short Oktapodi 2:30-min video
inferring a number of classes and relations of the ontology that are
syntactically important for the coherence of the representation but are cumbersome and
error-prone for the task of a manual (or semi-manual) annotator. We are going
to make a vast and e ective test of the annotation tool over several student
classes, together with questionnaires and ethnographic observations, to evaluate
the functioning of the tool and to create a vast corpus for studies in the digital
humanities.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1. Cinderella.
          <article-title>animated movie produced by Walt Disney, distributed by RKO Radio Pictures Release (</article-title>
          <year>1950</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Bratman</surname>
            ,
            <given-names>M.E.</given-names>
          </string-name>
          : Intention, Plans, and
          <string-name>
            <given-names>Practical</given-names>
            <surname>Reason</surname>
          </string-name>
          . Harvard University Press, Cambridge (MA) (
          <year>1987</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Elson</surname>
            ,
            <given-names>D.K.</given-names>
          </string-name>
          :
          <article-title>Dramabank: Annotating agency in narrative discourse</article-title>
          .
          <source>In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC</source>
          <year>2012</year>
          ). Istanbul, Turkey (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Esslin</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>The Field of Drama</article-title>
          . Methuen, London (
          <year>1988</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Fikes</surname>
            ,
            <given-names>R.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nilsson</surname>
            ,
            <given-names>N.J.:</given-names>
          </string-name>
          <article-title>Strips: A new approach to the application of theorem proving to problem solving</article-title>
          .
          <source>Arti cial Intelligence</source>
          <volume>2</volume>
          (
          <issue>3</issue>
          {4),
          <volume>189</volume>
          {
          <fpage>208</fpage>
          (
          <year>1971</year>
          ), http: //www.sciencedirect.com/science/article/pii/0004370271900105
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Guarino</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oberle</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Staab</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>What is an ontology?</article-title>
          <source>In: Handbook on Ontologies. Springer, 2nd edition edn</source>
          . (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Hatcher</surname>
            ,
            <given-names>J.:</given-names>
          </string-name>
          <article-title>The Art and Craft of Playwriting</article-title>
          . Story Press, Cincinnati, Ohio (
          <year>1996</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Heath</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bizer</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Linked data: Evolving the web into a global data space</article-title>
          .
          <source>Synthesis Lectures on the Semantic Web: Theory and Technology</source>
          pp.
          <volume>1</volume>
          {
          <issue>136</issue>
          (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Lavandier</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>La dramaturgie</article-title>
          .
          <source>Le</source>
          clown et l'enfant,
          <string-name>
            <surname>Cergy</surname>
          </string-name>
          (
          <year>1994</year>
          ), http://www. clown-enfant.com/leclown/dramaturgie/
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Lombardo</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Battaglino</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pizzo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Damiano</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lieto</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Coupling conceptual modeling and rules for the annotation of dramatic media</article-title>
          .
          <source>Semantic Web Journal</source>
          <volume>6</volume>
          (
          <issue>5</issue>
          ),
          <volume>503</volume>
          {
          <fpage>534</fpage>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Lombardo</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pizzo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Multimedia tool suite for the visualization of drama heritage metadata</article-title>
          .
          <source>Multimedia Tools and Applications</source>
          <volume>75</volume>
          (
          <issue>7</issue>
          ),
          <volume>3901</volume>
          {
          <fpage>3932</fpage>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Lombardo</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pizzo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Damiano</surname>
          </string-name>
          , R.:
          <article-title>Safeguarding and accessing drama as intangible cultural heritage</article-title>
          .
          <source>ACM Journal on Computing and Cultural Heritage</source>
          <volume>9</volume>
          (
          <issue>1</issue>
          ),
          <volume>1</volume>
          {
          <fpage>26</fpage>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Lombardo</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pizzo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Damiano</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Terzulli</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Albert</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          :
          <article-title>Interactive chart of story characters' intentions</article-title>
          .
          <source>In: Interactive Storytelling, 9th International Conference on Interactive Digital Storytelling, ICIDS</source>
          <year>2016</year>
          , Los Angeles, CA, USA, November
          <volume>15</volume>
          {
          <fpage>18</fpage>
          ,
          <year>2016</year>
          , Proceedings. vol.
          <volume>10045</volume>
          , pp.
          <volume>415</volume>
          {
          <fpage>418</fpage>
          . Springer International Publishing,
          <source>Cham { CHE (November</source>
          <volume>15</volume>
          {
          <fpage>18</fpage>
          ,
          <year>2016</year>
          ), http://dx.medra.
          <source>org/ 10</source>
          .1007/978-3-
          <fpage>319</fpage>
          -48279-8_
          <fpage>39</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Mamet</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Three Uses of the Knife: On the Nature and Purpose of Drama</article-title>
          .
          <source>The Columbia Lectures on American Culture</source>
          , Columbia University Press (
          <year>1998</year>
          ), http: //books.google.it/books?id=YZRODCVYACsC
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>McCarthy</surname>
            ,
            <given-names>J.C.</given-names>
          </string-name>
          :
          <article-title>Mental situation calculus</article-title>
          .
          <source>In: Proceedings of the 1986 Conference on Theoretical Aspects of Reasoning About Knowledge</source>
          . pp.
          <volume>307</volume>
          {
          <fpage>307</fpage>
          . TARK '
          <volume>86</volume>
          , Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (
          <year>1986</year>
          ), http://dl. acm.org/citation.cfm?id=
          <volume>1029786</volume>
          .
          <fpage>1029815</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Norling</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sonenberg</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>Creating Interactive Characters with BDI Agents</article-title>
          .
          <source>In: Proceedings of the Australian Workshop on Interactive Entertainment IE2004</source>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Ortony</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Clore</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          , Collins,
          <string-name>
            <surname>A.</surname>
          </string-name>
          :
          <article-title>The Cognitive Structure of Emotions</article-title>
          . Cambrigde University Press (
          <year>1988</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Peinado</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cavazza</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pizzi</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Revisiting Character-based A ective Storytelling under a Narrative BDI Framework</article-title>
          .
          <source>In: Proc. of ICIDIS08</source>
          . Erfurt,
          <string-name>
            <surname>Germany</surname>
          </string-name>
          (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Perrault</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          : Contes, chap. Cendrillon, pp.
          <volume>274</volume>
          {
          <fpage>79</fpage>
          .
          <string-name>
            <surname>Flammarion</surname>
          </string-name>
          , Paris (
          <year>1989</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Rahimtoroghi</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Corcoran</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Swanson</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Walker</surname>
            ,
            <given-names>M.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sagae</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gordon</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Minimal narrative annotation schemes and their applications</article-title>
          .
          <source>In: AAAI Publications, Seventh Intelligent Narrative Technologies Workshop</source>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Rissen</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lawrence</surname>
            ,
            <given-names>K.F.</given-names>
          </string-name>
          :
          <article-title>Re-imagining the creative and cultural output of the bbc with the semantic web</article-title>
          .
          <source>In: Digital Humanities</source>
          <year>2010</year>
          (
          <year>2010</year>
          ), http://dh2010.cch.kcl.ac.uk/academic-programme/abstracts/papers/html/ ab-878.html$\#$
          <fpage>d760e1214</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Ryngaert</surname>
          </string-name>
          , J.:
          <article-title>Introduction a l'analyse du thea^tre</article-title>
          . Collection Cursus. Serie Litterature, Armand
          <string-name>
            <surname>Colin</surname>
          </string-name>
          (
          <year>2008</year>
          ), http://books.google.it/books?id= WldsIAAACAAJ
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Spencer</surname>
            ,
            <given-names>S.:</given-names>
          </string-name>
          <article-title>The Playwright's Guidebook: An Insightful Primer on the Art of Dramatic Writing</article-title>
          . Faber &amp;
          <string-name>
            <surname>Faber</surname>
          </string-name>
          (
          <year>2002</year>
          ), http://books.google.it/books?id= nDrHmckSqi4C
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Varela</surname>
            ,
            <given-names>M.E.:</given-names>
          </string-name>
          <article-title>Interoperable performance research promises and perils of the semantic web</article-title>
          .
          <source>The Drama Review</source>
          <volume>60</volume>
          (
          <issue>3</issue>
          ) (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Wol</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mulholland</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          , Collins,
          <string-name>
            <surname>T.</surname>
          </string-name>
          :
          <article-title>Storyspace: A story-driven approach for creating museum narratives</article-title>
          .
          <source>In: Proceedings of the 23rd ACM Conference on Hypertext and Social Media</source>
          . pp.
          <volume>89</volume>
          {
          <issue>98</issue>
          (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>