English. Verb argument positions can be described by the semantic types that characterise the words filling that position. We investigate a number of linguistic issues underlying the tagging of an Italian corpus with the semantic types provided by the T-PAS (Typed Predicate Argument Structure) resource. We report both quantitative data about the tagging and a qualitative analysis of cases of disagreement between two annotators.
Words that fill a certain verb argument position
are characterised for their semantic properties.
For instance, the fillers of the object position of
the verb “eat” are typically required to share the
fact that they are edible objects, like “meat” and
“bread”. There has been a vast literature in
lexical semantics addressing, under different
perspectives, this issue, including the notion of
selectional preferences
In this paper we assume a corpus-based
perspective, and we focus on manually tagging verb
argument positions in a corpus with their
corresponding semantic classes, selected from those
used in the T-PAS resource
A mid term goal of this work is the extension of the T-PAS resource with a corpus of annotated sentences aligned with the T-PASs of the verbs (see section 2). This would have a twofold impact: it would allow a corpus based linguistic investigation, and it would provide a unique dataset for training semantic parsers for Italian.
The paper is structured as follows. Section 2 introduces T-PAS and the ontology of semantic types used in the resource. Section 3 describes the annotation task and the guidelines for annotators. Section 4 presents the annotated corpus and the data of the inter-annotator agreement. Finally, Section 5 discusses the most interesting phenomena that emerged during the annotation exercise. 2
The T-PAS resource is an inventory of 4241
Typed Predicate Argument Structures (T-PASs)
for example [[Human]] partecipa a ‘takes part
in’ [[Event]] - for 1000 average polysemy
Italian verbs, acquired from the ItWaC corpus
The main goal of the annotation effort reported in this paper is to enrich the annotation already present in the examples associated with each TPAS. Specifically, given a T-PAS of a verb and an example from the corpus, we annotate the lexical items (in the example) generalised by the STs (in the T-PAS).
For instance, Example (1) shows the T-PAS#1 of the verb vendere (Eng. ‘to sell’), and a sentence associated to it. The task consists in annotating prodotti tipici (Eng. ‘traditional products’) as a lexical item for [[Inanimate]]-obj.
(1) [[Human j Business Enterprise]] vendere [[I.n.a. n..i m..a.t.e. j Animal]] “[..] il nome di un’associazione brasiliana che vendeva anche .p.r.o.d. o..tt.i.t.i.p.i.c.i” 1
We annotate the content word(s) that is the head-noun both in case of the noun-phrases (NP) (e.g. give a c..a.k.e.) and in case of prepositionalphrases (PP) (e.g. give a c.a..k.e. to his little s.o..n.). In the case the head-noun is a quantifier, the quantifier is not tagged but the quantified element is (e.g. to give a piece of c..a.k.e.).
Notice that more than one token can be annotated, e.g. in the case of multiword expressions such as p.r.o. d..o.t.ti. .t.ip..ic..i in Example (1), and more than one item can be tagged for the same argument position, e.g. in case of coordination, such in [..] che vendeva anche p..ro..d.o.t.t.i.t.ip..ic..i e c.a.r.t.o.l.i.n.e.” 2.
In the case an argument is not present in the sentence (for instance, when the subject of the verb is unexpressed), we do not signal this lack.
On the other hand, the annotation accounts for the following cases.
notated according to the T-PAS; however, the annotator can use a different ST, if she/he thinks the one specified in the T-PAS does not apply. For instance, Example (2) reports another instance of T-PAS#1 of vendere in which lavoro has been annotated as [[Activity]], a ST not selected by the T-PAS#1 of vendere in object position (see the TPAS in Example (1)).
(2) “il l.a.v. o..r.o.come qualsiasi altra cosa puo` essere acquistato e venduto.”3
Syntactic mismatches. We account for cases in which the syntactic role of the lexical items does not match with the one proposed in the T-PAS, e.g. in cases of passive forms of verbs, where the subject and prepositional phrase introduced by da correspond respectively to the object and the subject of the active construction. In Example (2), lavoro is the syntactic subject of the passive clause, and it is generalized by [[Activity]]) in the object position of the T-PAS. In such cases we annotate both the ST of the lexical item and its grammatical relation using the one in the T-PAS.
Pronouns. In case the argument of the verb is realised as a pronoun, we tag the pronoun without assigning a ST. The pronoun is then linked to the noun(s) it refers to, and this noun is actually 1Eng. ‘[..] the name of that Brazilian association that was selling t.ra..d.i.ti.o. n..a.l.p.r. o.d..u.c.t.s.’
2Eng. ‘[..] that was selling t.r.a.d.i.t.io..n.a.l. .p.r.o. d..u.c.t.s and p..o.s.t.c.a.r.d.s.’ 3Eng. ‘jobs can be sold and bought just like anything.’ tagged with the ST label. In case the pronoun is agglutinated to the verb (i.e. it is found in the same token of the verb, e.g. venderla, Eng. ‘to sell it’), the part of the token corresponding to the pronoun is specified and, as just specified, the noun is annotated with the ST.
sonal constructions with an indefinite pronoun, the pronoun is annotated and the ST it refers to is specified: e.g. In Germania [..] si vende a 10 euro al chilo 4, si is annotated with [[Human]].
We annotated the examples in T-PAS using CAT
(Content Annotation Tool)5, a general-purpose
text annotation tool
The pilot annotation consisted in a selection of 3554 sentences extracted from the current version of T-PAS6 associated to 25 Italian verbs, selected with different levels of polysemy (from a minimum of 2 to a maximum of 10 T-PASs), and argument structure. The average polysemy of the 25 verbs (i.e. number of senses divided by the number of verbs) is 4.08, and for each T-PAS (sense) we have an average of 34.84 annotated sentences.
The annotation was carried out by a master student in linguistics, who was trained on the T-PAS resource, but had no previous experience in annotation. The annotator was able to tag the 3554 sentences in one month.
Table 1 shows the main data of the pilot annotation. Overall, we annotated 5342 argument positions expressed in the 3554 sentences, with an average of 1.5 argument per sentence. Out of the 230 Semantic Types available in the T-PAS ontology, 99 have been selected during the annotation, which means that we used about 40% of the STs contained in the hierarchy.
Data # Verbs # T-PASs # Examples # Examples per T-PAS # Semantic Types used 4Eng. ‘In Germany, they sell it at 10 euro per kilo’. 5https://dh.fbk.eu/resources/ cat-content-annotation-tool 6http://tpas.fbk.eu
In order to assess the reliability of the annotated data, we run an Inter-Annotator Agreement (IAA) test.7 We asked a second annotator to annotate a sample of 11 T-PASs associated to 3 different verbs (i.e., pulire, vendere and sbottonare). These verbs were chosen because they correspond to about 10% of the annotated sentences. Moreover, we selected them because they present a low or middle degree of polysemy with respect of the group of 25 verbs initially annotated. The second annotator was provided with the task guidelines and a training session was done to solve potential uncertainties in annotation. The second annotator was trained on a selection of corpus instances derived from verb lemmas, which are not included in the evaluation we report here.
Table 2 shows the results of the IAA for each
T-PAS. We measured both the agreement on
argument annotation, calculated with the Dice’s
coefficient
T-PAS Pulire, T-PAS#1 Pulire, T-PAS#2 Sbottonare, T-PAS#1 Sbottonare, T-PAS#2 Sbottonare, T-PAS#3 Sbottonare, T-PAS#4 Vendere, T-PAS#1 Vendere, T-PAS#2 Vendere, T-PAS#3 Vendere, T-PAS#4 Vendere, T-PAS#5 Overall average
Argument ST Dice’s value Accuracy 0.83 0.74
1 1 0.94 0.89 0.95 0.98
1 1 0.88 0.90 0.87 0.81 0.33 0.5 0.8 1 1 1 1 1 0.87 0.83
A special case is vendere T-PAS#2, which shows the lowest score for both argument and STs annotation. The annotation task allowed annotators to discard sentences which according to their opinion did not fit the sense of the T-PAS taken into consideration. Vendere T-PAS#2 has only a few corpus instances, which were mostly discarded or 7Cinkova´ et al. (2012) held an IAA on patternidentification using the CPA procedure in 30 English verbs. tagged differently by the two annotators, causing low agreement in the results for this T-PAS. 5
This Section discusses the most interesting phenomena that emerged during the annotation exercise, particularly in light of the Inter-annotator Agreement. 5.1
In this paragraph, we focus on the disagreements we found in argument tagging. The annotation task was difficult because the annotators had to identify the semantic structure of the verbs, using syntactic criteria to distinguish whether a lexical element was an argument or not.
Annotating pronouns was also a very demanding process since it implies the identification of co-reference chains. Differences in argument annotation between the two annotators, that impact the arguments Dice score, lie mainly in the annotation of pronouns and in the identification of co-referents. One annotator usually tends to annotate all the pronouns contained in an utterance whereas the other tags only the pronoun which is an argument of the verb taken into consideration. In addition, one usually does not identify co-referents which are lexically realised at great distance of words from the tagged verb, whereas the other sometimes annotates co-referents even if the argument has already been identified. There are also differences concerning the extension of annotation e.g. one interpreted prodotti tipici as multiword expression and the other did not. Overall, we obtained good agreement results, although some disagreements still remain even if we tried to reduce potential differences in annotation treating as many cases as possible in the guidelines. 5.2
The main goal of this section is to analyse the results of IAA on ST selection. Annotators used approximately 40 STs even though their expected number (according to the T-PAS resource) was 11. Table 3 represents the ST usage in the IAA experiment for each T-PAS.
Annotators used approximately the expected number of semantic types with some T-PASs, while with others they used many more. To a higher number of STs employed corresponds a lower ST accuracy score (see Table 1), more Pulire, T-PAS#1 Pulire, T-PAS#2 Sbottonare, T-PAS#1 Sbottonare, T-PAS#2 Sbottonare, T-PAS#3 Sbottonare, T-PAS#4 Vendere, T-PAS#1 Vendere, T-PAS#2 Vendere, T-PAS#3 Vendere, T-PAS#4 Vendere, T-PAS#5 specifically this correlation is shown by pulire T-PAS#1, sbottonare T-PAS#1,#4, vendere TPAS#1. There are a number of reasons that justify this STs usage. In some cases one annotator tends to tag the entity denoted by single lexical items instead of the generalisations made by the TPASs. This causes a sentence specific annotation that employs STs that are end nodes in the hierarchy, which do not correspond to the ones in the reference T-PAS. As future work, we plan to develop a methodology to normalize the STs to the appropriate level of abstraction.
There are also linguistic reasons that intervene
in the assignment of different STs to the same
lexical element. Annotators captured repeatedly the
phenomenon known as inherent polysemy by
tagging the same lexical elements in two totally
different ways. An inherent polysemous noun
denotes, depending on the context, a single aspect
of an entity which is inherently complex, i.e. that
can be described simultaneously by more than
one ST (see
For instance in example (3) one annotator tagged lente as [[Artifact]] highlighting its nature of manufactured object, whereas the other has annotated the lexical item as [[Physical Object Part]] focusing on its nature of constituent element of a bigger object.
(3)
“Giles pulisce una l.e.n.t.e. dei suoi
occhiali.”8
Moreover, there are differences is ST assignment
caused by regular polysemy
We performed a pilot experiment to tag the arguments of verbs, as recorded in the T-PAS resource, with their associated semantic type. We obtained good result in the annotation. By analyzing the cases of inter annotator disagreement, we were able to identify phenomena which lie at the core of such disagreements, such as the presence of inherent polysemous words. Ongoing work includes spelling out the rules for polysemous words tagging more clearly in the guidelines. 8Eng.‘Giles cleans a lens of his glasses’