English. This paper describes the first resource annotated for multiword expressions (MWEs) in Italian. Two versions of this dataset have been prepared: the first with a fast markup list of out-of-context MWEs, and the second with an in-context annotation, where the MWEs are entered with their contexts. The paper also discusses annotation issues and reports the inter-annotator agreement for both types of annotations. Finally, the results of the first exploitation of the new resource, namely the automatic extraction of Italian MWEs, are presented.
Italiano. Questo contributo descrive
la prima risorsa italiana annotatata con
polirematiche. Sono state preparate due
versioni del dataset: la prima con una
lista di polirematiche senza contesto, e
la seconda con annotazione in contesto.
Il contributo discute le problematiche
emerse durante l’annotazione e riporta
il grado di accordo tra annotatori per
entrambi i tipi di annotazione. Infine
vengono presentati i risultati del primo
impiego della nuova risorsa, ovvero
l’estrazione automatica di polirematiche
per l’italiano.
ever, despite being desiderata for linguistic
analysis and language learning, as well as for
training and evaluation of NLP tasks such as term
extraction (and Machine Translation in multilingual
scenarios), resources annotated with MWEs are a
scarce commodity
This paper outlines the development of a new language resource for Italian, namely a corpus annotated with Italian MWEs of a particular class: verb-noun expressions such as fare riferimento, dare luogo and prendere atto. Such collocations are reported to be the most frequent class of MWEs and of high practical importance both for automatic translation and language learning. To the best of our knowledge, this is the first resource of this kind in Italian.
The development of this corpus is part of a
multilingual project addressing the challenge of
computational treatment of MWEs. It covers English,
Spanish, Italian and French and its goal is to
develop a knowledge-poor methodology for
automatically identifying MWEs and retrieving their
translations
Two versions of the above resource have been produced. The first version consists of lists of MWEs annotated out-of-context with a view to performing fast evaluation of the developed methodology (out-of-context mark-up). The second version consists of annotated MWEs along with their concordances (in-context annotation). The latter type of annotation is time-consuming, but provides the contexts for the MWEs annotated. 2
After more than two decades of computational
studies on MWEs, the lack of a proper gold
standard is still an issue. Lexical resources like
dictionaries have limited coverage of these
expressions
Most previous studies on the computational
treatment of MWEs have focused on extracting
types (rather than tokens)1 of MWEs from corpora
As for the automatic identification of the tokens of MWEs, Fazly et al. (2009) make use of both linguistic properties and the local context, in determining the class of an MWE token. They report an unsupervised approach to identifying idiomatic and literal usages of an expression in context. Their method is evaluated on a very small sample of expressions in a small portion of the British National Corpus (BNC), which were annotated by humans. Schneider et al. (2014a) developed a supervised model whose purpose is to identify MWEs in context. Their methodology results in a corpus of automatically annotated MWEs. It is not clear, however, if the methodology is able to tag one specific expression as an MWE in one context and non-MWE in another. The PARSEME shared task2 is also devoted to annotating verbal 1Type refers to the canonical form of an expression, while token refers to each instance (usage) of the expression in any morphological form in text.
2http://typo.uni-konstanz.de/parseme/index.php/2-general/ 142-parseme-shared-task-on-automatic-detection-of-verbal-mwes MWEs in several languages. The shared task, while having interesting discussions on the area, has embarked upon the labour-intensive annotation of verbal MWEs.
Since there is no list of verb-noun MWEs in
Italian, we first automatically compile a list of
such expressions, to be annotated by human
experts. This is based on previous attempts at
extracting a lexicon of MWEs (as in
However, we believe that idiomaticity is not a
binary property; rather it is known to fall on a
continuum from completely semantically transparent,
or literal, to entirely opaque, or idiomatic
Highly polysemous verbs, such as give and take
in English and fare and dare in Italian widely
participate in Verb+Noun MWEs, in which they
contribute a broad range of figurative meanings that
must be recognised
We design an annotation task, in which we provide a sample of all usages of any type of Verb+Noun expression to be annotated. For this purpose, we employ the SketchEngine to list all the concordances of each verb when it is followed by a noun. Concordances include the verb in focus with almost ten words before and ten words after that. The SketchEngine reports only 100; 000 concordances for each query. Among them, we filter out the concordances that include Verb+Noun expressions with frequencies lower than 50 and we randomly select 10% of the concordances for each verb. As a result, there are 30; 094 concordances to be annotated. The two annotators annotate all usages of Verb+Noun expressions in these concordances, considering the context that the expression occurred in, marking up MWEs with 1 and expressions which are not MWEs, with 0. Table 1 reports on the details of annotation tasks and Table 2 shows the agreement details for them. 2.4
As seen in Table 2, the inter-annotator agreement is significantly higher when annotating the expressions in context. One of the main causes of disagreements in out-of-context annotation is concerned with abstract nouns. The annotation of expressions composed of a verb followed by a noun with an abstract meaning is a more complicated process as the candidate expression may carry a figurative meaning. Each annotator uses their intuition to annotate them and it leads to random tags for these expression (e.g. fare notizia, dare identità, prendere possesso) when they are out-ofcontext. However, in the case of in-context annotation, concordances composed of abstract nouns have been annotated in the majority of cases with 1 by both annotators.
In-context annotation is also very helpful for annotating expressions with both idiomatic and literal meanings. An interesting observation, reported in Table 3, is related to the number of expressions that are detected with the two different usages of idiomatic and non-idiomatic, in context.
As can be seen in Table 3,3 among the 1; 649 types of expressions in concordances, 530 (32%) of them could be MWEs in some context and nonMWEs in others (context-depending), according to the first annotator. This annotator has annotated only 3% of the expressions with tag ‘2’ without context.
3Note that the numbers in Table 3 cannot be interpreted to validate agreement between annotators, i.e. no conclusion about agreement can be derived from 3.
In our multilingual project (see Section 1) we regard the automatic translation of MWEs as a twostage process. The first stage is the extraction of MWEs in each of the languages; the second stage is a matching procedure for the extracted MWEs in each language which proposes translation equivalents. In this study the extraction of MWEs is based on statistical association measures (AMs).
These measures have been proposed to
determine the degree of compositionality, and fixedness
of expressions. The more compositional or fixed
expressions are, the more likely it is that they are
MWEs
In the first experiment, the list of all extracted Verb + Noun combinations (as explained in Section 2.1) are ranked according to the above measures that are computed from itWaC as a reference corpus. To perform the evaluation against the list of annotated expressions, we process all 2,415 expressions for which the annotators agreed on tags 0 or 1. After ranking the expressions by the measures, we examine the retrieval performance of each measure by computing the 11-point Interpolated Average Precision (11-p IAP). This reflects the goodness of a measure in ranking the relevant items (here, MWEs) before the irrelevant ones. To this end, the interpolated precision at the 11 recall values of 0, 10%, ..., 100% is calculated. As detailed in Manning et al. (2008), the interpolated precision at a certain recall level, r, is defined as the highest precision found for any recall level r0 r. The average of these 11 points is reported as 11-p IAP in Table 4.
As can be seen in Table 4, the selected association measures generally perform with similar performance in ranking this type of MWEs, with M I3 performing slightly better than others. In the second experiment, we seek to establish the effect of these measures on identifying the usages of MWEs in our dataset of in-context annotations. We set a threshold for each score that we have computed for Verb+Noun expression types. By setting thresholds we compute the classification accuracy of the measures to identify MWEs among the usages of Verb+Noun expressions in a corpus. Specifically, each candidate of a Verb+Noun in the concordances is automatically tagged as an MWE if its lemmatised form has a score higher than the threshold, and as a nonMWE, otherwise. For each measure, we compute the arithmetic mean (average) of all the values of that measure for all expressions, and set the resulted average value as a threshold.
The accuracies of classifying the candidate Verb+Noun expressions are computed based on the human annotations of the concordances and are shown in Table 5. The classification accuracies of AMs are also very close to each other (see Table 5); however, this time Log-likelihood and F req fare slightly better than others in classifying tokens of Verb+Noun expressions. Our new resource of concordances contains useful linguistic information related to usages of expressions and as such important features can be extracted from the resource to help identifying MWEs. One of these features can be obtained from the statistics of different possible inflections of the verb component of an expression. Based on the premise of the fixedness of MWEs, we expect that the verb component of a verb-noun MWE occurs only in a limited number of inflections. We implement this feature by dividing the frequency of occurrences of each expression by the number of inflections that the verb component occurs in. Note that to count the number of different inflections of the verb component, we rely on the subcorpus of concordances that we gathered.
We evaluate this approach only on 1,077 expressions that occur in concordances. We rank the expressions according to this newly computed score and we call this score, which depends on the inflection varieties, INF-VAR. For all verbs, the INF-VAR performs comparably to Frequency in ranking MWEs higher than non-MWEs, but for the verb trovare, we obtain better 11-p IAP using this score than by using Frequency (see Table 6).
In this paper, we outline our work towards a goldstandard dataset which is tagged with Italian verbnoun MWEs along with their contexts. We show the reliability of this dataset by its considerable inter-annotator agreement compared to the moderate inter-annotator agreement on annotated verbnoun expressions presented without context. We also report the results of automatic extraction of MWEs using this dataset as a gold-standard. One of the advantages of this dataset is that it includes both 0-tagged and 1-tagged tokens of expressions and it can be used for classification and other statistical NLP approaches. In future work, we are interested in extracting context features from concordances in this resource to automatically recognise and classify the expressions that are MWEs in some contexts but not MWEs in others.
Ted Dunning. 1993. Accurate methods for the statistics of surprise and coincidence. COMPUTATIONAL LINGUISTICS, 19(1):61–74.