This paper represents the current state of the research project aimed at modifying the MIPVU protocol for metaphor annotation for usage on Czech-language texts. Three annotators were trained to use metaphor identification procedure MIPVU and annotated 2 short text excerpts of about 600 tokens length, then the reliability of annotation was measured using Fleiss' kappa. The resultant interannotator agreement of 0.70 was below kappa values reported by annotators of VU Amsterdam Metaphor Corpus (Steen et al., 2010) and very similar to the agreement that researchers (Badryzlova et al., 2013) got in their first reliability test with unmodified MIPVU procedure applied on Russian texts. Some modifications of the annotation procedure are proposed in order for it to be more suitable for Czech language. The modifications are based on the observations made by annotators in error analysis and by authors of similar projects aimed to transfer MIPVU procedure to Slavic/inflected languages. The functionality of the annotation procedure refinements now have to be tested in the second reliability test.
This paper represents the current state of the research project aimed at modifying the MIPVU protocol for metaphor annotation for usage on Czech-language texts. It is the initial stage of creation of Czech metaphor corpus which could be a very valuable resource for several fields of linguistic research (such as computational, cognitive and corpus linguistics).
This initial stage includes: 1) Modification of the MIPVU protocol for reliable linguistic metaphor identification in Czech 2) Introducing an alternative tag (located in parallel to original MIPVU tags) which, if needed, will allow us to filter out the highly conventionalized cases of metaphors.
The process of modifying the MIPVU procedure is described in the following parts of this work. The addition of the alternative tag for highly conventionalized metaphors is motivated by the desire to use the resulting corpus for training of systems for automatic identification of metaphor.
Lexicalized cases of metaphors can be successfully
interpreted using standard word sense disambiguation
techniques
Since early ninety-eighties, when conceptual metaphor
theory
Their method called MIP (Metaphor Identification
Procedure;
If basic meaning of a word is: a) more concrete; what it evokes is easier to imagine, see, hear, feel, smell and taste; b) related to bodily action; c) more precise (as opposed to vague); the word is marked as MRW.
The history of a lexical unit is usually not taken into account, which is one of the differences between MIP and MIPVU. 2.2
Yulia Badryzlova and her colleagues (2013) modified the MIPVU protocol for Russian-language texts and attempted to extend annotation to the level of conceptual mappings “deep annotation”.
They measured the inter-annotator agreement on texts
using original MIPVU and their modified version and
compared it with the results of the same tests made by
Steen and his colleagues (2010) in the process of
establishing MIPVU procedure. In the second test their
resulting inter-annotator agreement outperformed the
agreement reported for VUAMC. The project was then
discontinued, but recently
Justina
For the current stage of our project we are using a model
similar to work of
3.
We annotated two text excerpts each of about 600 tokens
length. First excerpt (598 tokens) belonged in the fiction
genre and was taken from short story “Zasraný vánoce”
by Michal Viewegh. The second one (611 tokens) was
taken from a transcription of proceedings of European
Parliament. These transcriptions are available from the
parallel corpus InterCorp
Dictionary of Standard Czech Language
Two of the 3 annotators were Ph.D. students and the remaining one was a Master's student, all of them in the field of linguistics and with prior experience in conceptual metaphor studies.
The reliability of the annotation was measured using
Fleiss' kappa, a statistical measure of inter-annotator
agreement which corrects for chance agreement between
analysts
In this first reliability test, the annotators were trained in MIPVU protocol and instructed to follow it. The annotation was performed in the manner similar to reliability tests in the process of making VUAMC, which means the annotators worked only with plain text and marked each lexical unit with either 1 (MRW) or 0 (nonMRW). The Fleiss' kappa calculation as well as determination of the cases of disagreement was carried out by a Python program designed specifically for this task.
The results can be seen in Tab. 1.
The minimum thresholds accepted for Fleiss' kappa are
commonly stated to be 0.67, 0.7 or 0.8
4.
4.1
The table 3 shows disagreement count for both annotated texts in total and in respect of different parts of speech. Part of speech which in both annotated excerpts manifested most of the disagreement were verbs, followed by prepositions in case of the fiction text by Michal Viewegh, and by nouns in the case of European Parliament proceedings.
Viewegh
Europarl POS Nouns Verbs Adjectives Adverbs Prepositions Conjunctions All POS It is noteworthy that while the annotated excerpt of European Parliament proceedings shows more disagreements in annotation it nevertheless shows higher inter-annotator agreement (as seen in Tab. 1). This is caused by the fact that more than twice as many metaphors are present in the text compared to the other excerpt. This corresponds with the findings of Steen and his colleagues (2010) that from the four registers, (academic, news, fiction, and conversation) only conversation had lower frequency of MRWs than fiction texts.
Part of the disagreement in verb annotation seems to be caused more by a bias of individual annotators than a systematic pattern in the annotation protocol. In case of the European Parliament proceedings one of the annotators did not marked several metaphorically used lexical units as MRWs. The reason was that in case of some verbs the annotator overlooked personifying connection between the verb and its subject if the latter was highly abstract (e.g. luck, possibility, right or freedom), the annotator have realized this omission immediately after the annotation course was finished. The approach we have chosen for dealing with disagreements in preposition annotation is showed in chapter 4.2. 4.2
In English and presumably in many languages,
prepositions are the most metaphor-rich part of speech as
they are reported to account for 38.5-46.9% of
metaphorrelated words in VUAMC
Just like Badryzlova and her colleagues (2013) did, we
made a list of major prepositions’ basic meanings. We
followed the Czech linguistic tradition where
prepositions’ meanings are distinguished by grammatical
case
Take for example these expressions containing preposition “za”. While it is clear that in sentences 3) and 4) “za” is a MRW, in the case of 1) and 2) both meanings are clearly distinct but equally concrete and bodily related. 1) Petr stojí za mnou; Petr stands behind me 2) Chytil jsem ho za nohu; I caught him by the leg 3) Za 2 roky to bude hotové; It will be done in 2 years 4) Vyměnil jsem kolo za auto; I traded the bike for the car If we distinguish between “za” in instrumental (expression 1)) and in accusative 2), we can have basic meaning for each one, moreover “accusative za” standing for basic meaning of this preposition in sentences 3) and 4) which both are MRWs. 4.3
Reflexive pronouns “se/si” are used either when the subject and object of the sentence are identical 5) or as an integral part of a reflexive verb whose lexical meanings they often determine. The presence of a reflexive pronoun “se/si” can result in a complete shift of meaning as illustrated in 6).
5) umyji se; I will wash myself 6) rozvést / rozvést se; to develop (an idea) / to divorce Expectably, the original MIPVU procedure does not account for this phenomenon. The table 4 shows its effect on an actual annotated sentence.
Annotated sentence Když se před třemi lety rozvedl [...] Original MIPVU 0 0 1 0 0 1 Modified MIPVU 0 0 1 0 0 0 The highlighted tokens, when treated as separate lexical units, will render the basic meaning of the word “rozvedl” to be “he developed/expanded (something)” and the contextual meaning which is “he got divorced” should therefore be a MRW. On the other hand, the expression “se” + “rozvedl”, when counted as one lexical unit which is distinct from “rozvedl”, has an equal basic meaning to the contextual one, so it is annotated as not-MRW, which matches better with the general sense of the sentence.1 Similarly, Czech auxiliary verbs such as “bych” are considered integral parts of the full verb’s conjugation forms.
Therefore for reflexive pronouns “se/si” and auxiliary verbs we applied the same policy as annotators of VUAMC used for phrasal verbs in English, which means that they count as one lexical unit altogether with the full verb.
On the other hand, meanings commonly expressed by phrasal verbs in English tend to be expressed by prefixes in Czech which are already parts of the word as seen in 7).
7) zesílit; turn up 4.4
Dealing with set expressions, we followed remarks on
MIPVU made recently by the main author of VUAMC
So far, we have applied MIPVU on Czech texts and tested
inter-annotator agreement. Direct transferability of the
MIPVU procedure to Czech language turned out to be
problematic, which we expected, as the same
complications were reported by researchers applying the
procedure on Russian
After the error analysis, we have proposed several minor modifications of the guidelines in order to make them more suitable for Czech and we plan to conduct second reliability test as soon as possible.
The next step after successfully transferring MIPVU to be used on Czech texts would be to annotate the data with an additional tag for highly lexicalized metaphors. It is meant to work not by asking whether the contextual meaning is different from basic one but rather whether there is a literal word in use which can express the given contextual meaning. If there is not, it is probably a highly conventionalized metaphor.
Nevertheless, there are several yet unanswered questions regarding this approach, the most important one being if annotators will agree sufficiently on those cases.
This work was funded by Ministry of Education, Youth and Sport of the Czech Republic as a part of the project “Počáteční fáze tvorby korpusu metafory v češtině” (Grant number IGA_FF_2018_026). 1 In the first course of annotation the interconnection of words is realized simply by giving the reflexive pronoun (or an auxiliary verb) always the same value of metaphoricity which is given to its corresponding verb. This naive method is justifiable because this stage of the project only serves to refine the annotation manual. It is not suitable for actual corpus generation as it would influence the metaphor usage statistics.