This paper presents a corpus annotated for the task of directspeech extraction in Croatian.The paper focuses on the annotation of the quotation, co-reference resolution, and sentiment annotation in SETimes news corpus in Croatian and on the analysis of its language-speci c differences compared to English. From this, a list of the phenomena that require special attention when performing these annotations is derived. The generated corpus with quotation features annotations can be used for multiple tasks in the eld of Natural Language Processing.
Quotes are an essential part of news articles and stories made by the media,
individuals, or other organisations. The reproduction of the spoken-text includes
public opinion which expresses personal and subjective information about events
of the world surrounding us. Political analysts and researchers have a major
interest in analysing quotations [
Although recent advances in Machine Learning and Natural Language Processing do provide ample support for extracting quotes and identifying the speakers in English texts, very little research has been done for other languages. Therefore, this paper proposes an original approach for the creation of a corpus of news texts with quotes annotations for the Croatian language, with a deep analysis of the corpus genesis process. This news corpus is tagged with quotations, verbcues, and speakers' identi cation. It also includes co-reference resolution in case ? Copyright © 2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). of pronouns involved in the quotations. Finally, the quotes were tagged in terms of sentiment (positive, negative, or neutral) of the spoken text at the sentence level.
This paper is organized into 6 sections. Section 2 summarizes the related work and the dataset alongside the annotation process methodology is described in Section 3. Section 4 presents several statistical information concerning the generated corpus, followed by Section 5, illustrating the various phenomena that were encountered. Finally, in Section 6, the main conclusions of this study are presented. 2
The following sentences are examples of the most common syntactic arrangements of direct quotations that can be found in news articles in English. 1. Michelle Mcgrath said, \We stand ready to support you in every way ". 2. \We stand ready to support you in every way " Blair said. 3. Tony Mcgrath visited Iraq... He said, \We stand ready to support you in every way ". 4. Tony Mcgrath visited Iraq... \We stand ready to support you in every way " the Prime Minister said. 5. \I'm really happy for Fabio" Materazzi told the Apcom news agency Friday. \I feel part of this distinction because I think that all the Azzurri helped a great champion like Cannavaro win an important prize ".
The main task of quote extraction is composed of the following sub-tasks [
Previous attempts of automatic extraction of quote-speaker pairs from news
sources have been well-researched for English [
The SETimes Croatian corpus was used as the basis for the annotation process.
SETimes is a parallel corpus (CC-BY-SA license) [
We merge all the sentences belonging to a single article into one document by concatenating with a space delimiter. From the whole recomposed corpus, a sub-corpus of 140 random documents was selected for the annotation task. 3.3
INCEpTION tool [
Anaphoric - The connection of a pronoun present either in the quoted text and/or as the speaker via a co-reference chain.
Uses-verb - Connects the speaker and verb-cue as a chain.
Verb-spoken-text - Connects the verb-cue to the spoken-text span as a chain.
The annotations of the Croatian corpus were performed by a Croatian native speaker. These annotations serve as a preliminary step to try and assess the possible problems that could arise during the annotation processes and what features of quoting in Croatian should be noted when building a tool for automatic processing of quotes. 4
This section describes the overall statistics of the created dataset. We have a total of 2497 annotations concerning speaker and quote identi cation and types relations. In total, 469 quotes were found in 140 di erent documents, around 3 quotes per article. In terms of sentiment tagging, we have annotated 875 individual sentences. 1 The corrected version of SETimes corpus where diacritics and encoding system have been corrected is available from nlp. zg.hr. This version is considered in this paper.
Annotation Class Speakers Spoken-text Verb-cue Anaphoric Relation Uses-verb Relation Verb-spoken-text Relation Total
No of Annotations
446
469
468
13
431
515
2497
Quotes can be categorized into di erent categories, depending on which feature
one would like to emphasize. In this work, the categorization stated in O'Keefe
et al. [
The work described in this paper is focused only on direct quotes. Quote and quotation are used interchangeably, and both can indicate the whole quote (speaker+verb-cue+spoken-text) or only the quote content (`spoken-text'), depending on the context. The ideal quotation would be the one where the speaker, verb-cue, and spoken-text would be consecutively positioned in a sentence and not broken apart in any way. Often this is not the case in the corpus (nor real life), as appositions, time and date details, and other sentence parts are positioned in between the speaker, verb-cue, and spoken-text. To try and grasp all potential issues for automatic detection, attribution, and extraction of quotes, everything that was not deemed as the ideal quotation was written down as a problem during the manual annotation, separately from the annotations described above. This led to many instances being marked as potential problems, but by identifying even the smallest deviations from the envisioned standard we can more easily group problems and try to adapt the automatic processing to most commonly found issues. Croatian is a South Slavic language and therefore has some distinct linguistic features when compared to English. Some problems encountered in English automatic quotation processing, such as pronoun disambiguation, do not concern Croatian (at least not to the same extent), and viceversa - instances which do not exist in English pose a problem in Croatian. In the following subsections, we will brie y describe all of the problems encountered during the annotation process, with a more detailed focus on language-speci c ones, except for one problem because it is strictly a problem with the INCEpTION tool.
Altogether, 16 problem clusters were recognized { 15 in the Quote Fine layer and one in the Quote Simple layer. All of the problems are listed in Table 3, along with the percentage of documents2 in which they were noticed. It is worth mentioning that this is merely an indicator of how many documents had a certain problem, and not how prevalent a problem might be inside a certain document (i.e. in one document the problem might arise 10 times, and only once in another document). Still, it was assessed that this could be useful as a rough estimate of problem distribution throughout the annotated documents.
All of these phenomena were named by the authors for the purpose of this work (except for cross-branching and passive) and are not o cial names for such occurrences. The tags in examples were generated by the ReLDI tagger3 for Croatian. 5.1
Indirectly correlated speaker and Indirectly correlated verb-cue were noted as two di erent problems as the former deals only with the speaker, and the latter only with verb-cue issues. However, they will be described together as they essentially represent the same issue, i.e. one of the other two constituents of a quote being separated from the spoken-text. This could create di culties with automatic extraction because the speaker and verb-cue can in some cases be far away from the spoken-text. This can happen with the insertion of time and date details, appositions, or other sentence parts between the speaker, verb-cue, and spokentext (the quoted content itself). While a human reader probably would not have any problems connecting the correct speaker or verb-cue to its spoken-text, the presumption was that this could pose a problem for the process of automatic extraction and disambiguation of quotes. While it would be possible to lter out the most common words and expressions separating the two, such as wellknown appositions or time/location expressions, some less frequent instances could create a problem. This is why it was decided to note every case where the 2 This percentage was calculated with dividing the number of documents containing a certain problem with the number of documents which had direct quotes (120 documents), since only those documents were actually annotated. 3 ReLDI tagger, available on http://www.clarin.si/services/web/query
Case % of documents Quote on multiple pages [tool problem] Indirectly correlated speaker 61.67 Indirectly correlated verb cue 12.50 Speaker or verb-cue in the middle of the quoted text 41.67 Quotes without an apparent speaker or verb-cue 11.67 Quotes marking something other than speech 33.33 Quotes from parts of documents or a collective 27.50 Seems to be quoted but no quotes 1.67 Speaker mentioned multiple times di erently 63.33 Apposition(s) alongside speaker 78.33 Mixed quotes 54.17 Non-person speaker 9.17 Anonymous speaker 15.83 Cross-branching 5 Passive 18.33
Sentence parts di erently annotated 32.50 speaker or verb-cue were separated from the spoken-text, with no regard to if there was only one word separating them or whole paragraphs. This can o er an insight into how this separation happens (in our dataset) and how it could be tackled.
Example 6. demonstrates an occurrence of Indirectly correlated speaker. `Sarah Lum' is the speaker whose words were quoted in the following sentence of the document, meaning that there are ve tokens separating the speaker from her quote and the verb-cue. In this case, there is an apposition with additional details about the institution and its location (`predstavnica Americkog ureda u Pristini', en. `representing the US O ce in Pristina') between the speaker and the quote. 6. Croatian:
Slicnu similar-Agpfsay \QUOTE", quote
izjavu statement-Ncfsa
dala je give-Vmp-sf be-Var3s
Sarah Lum, speaker predstavnica Americkog ureda u representative-Ncfsn American-Agpmsgy o ce-Ncmsg in-Sl
Pristini.
Pristina-Npfsl kazala je. say-Vmp-sf be-Var3s v e r b - c u e
English: Similar comments came from Sarah Lum, representing the US o ce in Pristina. \QUOTE," she said.
In Example 7., a case of Indirectly correlated verb-cue in a mixed quote can be seen. Verb-cue `izjavio je' (en. `has stated') is separated from the direct part of the mixed quote by the indirect part of the quote. Mixed quotes were not the focus of this work, but the direct part of a mixed quote was nevertheless annotated along with its speaker and verb-cue when judged possible or needed. It is interesting to note that there were no annotated documents which had Indirectly correlated verb-cue where there was not also an Indirectly correlated speaker 4, even though there were many documents with instances of Indirectly correlated speaker, but no Indirectly correlated verb-cue. Since these observations are made only on the document, and not on the quote-level, more detailed research would be needed to establish whether this is merely a coincidence or not.
7. Croatian: English: \QUOTE" are Ankara's Balkan diplomacy goals, according to Gurkan Zengin (...) 5.2
This problem cluster grouped together all cases where one quote's content was separated by other text in which both the speaker and/or the verb-cue occurred. This type of quoting could be seen in Example 5. in Section 2. This meant that a quote that would be understood by a human reader as one individual quote was separated into multiple spoken-texts. This could create problems with automatic extraction of quotes because it could happen that only a part of the quote's content would be extracted with the speaker and verb-cue, while the other part could be ignored, extracted without speaker/verb-cue, or with them wrongly attributed. Example 8. demonstrates a quote from one Chernomorets resident, whose statement was separated by a short text with time (`krajem rujna', en. `end of September') and source details (`za SETimes', en. `for SETimes'), but also mentions speaker and verb-cue. Tags `Quote1.1' and `Quote1.2' in the example 4 There was one exception, but the Indirectly correlated verb-cue occurred in a passive clause with no agent, so there was no speaker in the rst place. tag the rst and the second part of his quote, respectively. After the second part of the quote, the article in question continues with its topic with no further reference to the sherman or his words. Since there are no clear indications about speaker and verb-cue for the second part of the quote it could, in theory, easily be mistakenly extracted as a quote with no speaker/verb-cue (or simply ignored because of \lack" of these constituents) or some other speaker and verb-cue could falsely be attributed to it. This is why it was thought important to note these cases so some data about how to most successfully try and add separated quote contents to each other in later steps of our work was available.
8. Croatian: \QUOTE1.1", part 1/2 of the quote
rujna September-Ncmsg
80-godisnji 80-year- old-Agpmsny
rekao say-Vmp-sm v e r b - c u e
English: \QUOTE1.1," Angel Kishev, an 80-year-old sherman from the town of Chernomorets, told SETimes in late September. \QUOTE2.2" 5.3
Problem cluster Quotes without an apparent speaker or verb-cue deals with those quotes whose speaker and/or verb-cue would be apparent to a human reader, but not necessarily to a tool for automatic quotations extraction. This could lead to quotes which are part of this cluster to be ignored or wrongly extracted/attributed. This was already partially discussed in Subsection 5.2, where it was described how some parts of the separated quote could fall into this category. However, there are other cases in which this problem arises. Example 9. shows a sentence before the problematic quote. After the quote, the article continues with no additional information about its speaker or verb-cue. The sentence shown in the example contains a quote from, presumably, a di erent source - it seems to list the UN's endorsed standards. The real quote from Holkeri follows after that sentence. The system or the tool for automatic processing of quotes would rst have to recognize that the inserted quote is not Holkeri's quote, but rather a part of some document. The next step would be to attribute Holkeri's actual quote to him, even though it does not have a speci c verb-cue, is not in the same sentence as the speaker, and one has to infer that this indeed is Holkeri's quote.
za for-Sa ciljevi goal-Ncmpn
UN { UN-Npmpn
za for-Sa 9. Croatian:
Plan, plan-Ncmsn
cilj goal-Ncmsan
dodao add-Vmp-sm
ima have-Vmr3s standarda koje standard-Ncmpg which-Pi-mpa
English: The plan, Holkeri said, aims to ensure that the goal of the UNendorsed standards { \to make Kosovo a better place for everyone: safe, stable and prosperous" { will become a reality. \QUOTE" 5.4
As the name would suggest, this problem cluster dealt with quotation marks marking something other than direct speech. These could be quotes marking informal style, quotes around some named entity etc. Quotes marking something other than speech might \confuse" the system or the tool for automatic processing of quotes and it could extract those non-speech instances as speech. Additionally, in some cases it is very hard to judge whether the quotation marks were used as an indication of some kind of metaphor or exaggeration, or if they actually were some speaker's literal words. This dilemma would especially be important for those dealing with all types of quotes, including mixed quotes, but it is not that crucial for direct quotation processing. Example 10. shows quotation marks which are marking a workshop's name instead of a quote.
Example 11. represents a more ambiguous case. It is unclear whether the quotation marks around `pribliznu istinu' (en. `approximate truth') are someone's literal words (if they are, whose?), an informal re ection of the author's or the public's feelings towards this subject or something else. Furthermore, this is (...)
organizirali su radionicu organize-Vmp-pm be-Var3p workshop-Ncfsa \Arhitektura, tradicije, sjecanje". architecture-Ncfsn tradition-Ncfsg memory-Ncnsn quotes marking something other than speech
English: Students (...) organised a workshop \Architecture, Traditions, Memory". an opening statement of the text (perhaps it was the article's title), and it does not get mentioned again in the text's content, making the decision even harder. 10. Croatian:
Studenti student-Ncmpn 11. Croatian:
Novi new-Agpmsny
izravno directly-Rgp dokumentirajuci document-Rr
bi be-Vaa3s usredotocio focus-Vmp-sm
English: A new panel would focus squarely on the victims in documenting an \approximation of the truth". 5.5
Quotes from a collective are not considered as \real" quotes by the guidelines of
Agence France-Presse because their real source is unclear [
Seems to be quoted, but no quotes refers to sentences or paragraphs that seem like they could be someone's direct statement, but they are missing quotation marks. When the annotation process rst started, it seemed like many documents have such occurrences. Later on, it was realized those sentences were actually source information for the article, and it was simply the reporter's name and the (presumed) title, subtitle, or introductory sentence of the article. After ignoring those introductory details of articles, only two instances of this problem were found. One such instance can be seen in Example 12., where the quote came after another quote, speaker, and verb-cue, but the quotation marks on the left side of this quote were forgotten. This could lead to the quote being missed in the automatic extraction process.
12. Croatian:
Celnici moraju leader-Ncmpn must- Vmr3p
voditi, lead-Vmn
a and-Cc
ne not-Qz samo only-Rgp
pratiti svoje pristase".
follow-Vmn one's own-Px-mpa follower-Ncmpn
English: Leaders must lead, and not merely follow their followers". 5.7
In news articles, one individual speaker is often referred to in many ways after he was rst introduced to the text. In English, this might be done with the surname of the speaker, his title (such as \prime minister") or a pronoun, which then creates a problem because pronoun disambiguation is needed. In Croatian, it is not usual to use pronouns when referring to a person without using their name/title because the predicate form often indicates gender, so pronoun disambiguation was not something crucial to have in mind at this stage. In the rare article that this was the case, the pronoun was annotated as having an anaphoric relationship with the name. Surnames and titles are also often used to refer to the speaker. Creating some sort of connections between surnames/titles and the speaker's full name will be needed in the future to identify the speaker's full name for all his or her quotes found in the text. In general, di erent versions of someone's name were not annotated as anaphoric (so, surnames were not annotated as having an anaphoric relationship with the prename or full name). When the speaker of the quote was the speaker's title, the title and the name were tagged as having an anaphoric relationship. 5.8
Many times when a source is quoted, his or her title or some other description of who this person is (like the `80-year-old sherman' in Example 8.) are also mentioned. All of these occurrences were grouped together under the problem cluster Apposition(s) alongside speaker and marked when an apposition of any kind would appear next to the speaker. This problem cluster is not exactly a problem because appositions alongside speakers would not in any way intrude the process of automatic extraction/disambiguation of quotations. On the contrary, they were marked to gain a rough overview of what kinds of expressions could be used instead of the speaker's name when referring to him/her and how to extract and connect them to the speaker's name. The potential dataset one could gather from a collection of appositions could be used for resolving issues described in Subsection 5.7. 5.9
Mixed quotes are quotes that combine direct and indirect quoting styles. While this work is currently focused only on direct quotations, it was strived to annotate direct parts of mixed quotes whenever it was judged possible or necessary to do so, most often when the meaning or purpose of the direct part would be clear enough on its own. Example 13. demonstrates a mixed quote - parts of the speaker's statement were put in quotation marks, and other parts (the speaker expressing condolences) were indirectly quoted. The mixed quote in this example is even more problematic than others could be because the directly quoted statement is expressed in the 3rd person singular, and not in 1st person singular as one would expect of someone speaking for himself. Based on that, it can be concluded that even this direct part was changed and possibly ltered by the article's author.
13. Croatian:
Rekavsi say-Rr
English: Saying he \remembers and aware of the depth (...)," Peres extended condolences (...) Direct parts of mixed quotes sometimes do not have their own verb-cue because the verb-cue with which they were introduced is not a typical quoting verb. Example 14. portrays this nicely, as `izrazio zaljenje' (en. `voiced regret') is far from what one would consider a typical quoting verb. Often these annotated direct parts of mixed quotes were annotated only partially - without verb-cue or without the connections in the Quote Co-reference layer.
14. Croatian:
English: (...) Holkeri also voiced regret over the fact that \not every community" participated in the development of the plan.
These two problems were marked down separately, but both concern issues with speaker disambiguation and attribution so they will brie y be described together. Non-person speaker was used to mark those sources which were not persons, but rather collectives or documents. Because it was often found alongside Quotes from parts of documents or a collective, it was used as an addition to that problem cluster.
Anonymous speaker problem cluster marked all speakers who were not mentioned by their name, but rather with an apposition, title etc. Example 15. has a quote whose source is a collective - `dvoje celnika' (en. `two leaders') - and additionally, they are not named. Future work should decide on how to treat such \speakers" and their quotes, as already mentioned in Subsection 5.5. 15. Croatian:
English: \QUOTE" the two leaders said in a statement after the meeting.
This phenomenon is also referred to as \discontinuous constituents"5 and occurs
when some sentence parts split other sentence parts, such as predicates or noun
5 The term cross-branching is used in [
As evident in Example 16., the auxiliary part of the predicate is separated from the participle part by four other tokens (in the example, the notation P means predicate). This is a problem for verb cue classi ers because they would have to recognize such instances and lter out the full predicate from the sentence. However, this could be easily solved by automatically adding the auxiliary verb/copula when what seems like a lone participle is found. The gender of the auxiliary verb can be easily deduced from the morphological form of the participle, since parts of a predicate must agree in gender.
16. Croatian:
Vojnici soldier-Ncmpn
angaziranje engagement-Ncnsa
opisali describe-Vmp-pm participle part of the P
su be-Var3p aux. part of the P
svoje their-own-Px-nsa
English: The soldiers described the engagement in Iraq as \QUOTE"
Passive in Croatian usually has no agent, meaning that the quotes whose
verbcue is in passive would be without a speaker. The Croatian school grammar
states that \passive is used when the agent is unknown or one doesn't wish to
speci cally emphasize the agent" [
navodi se u priopcenju. state-Vmr3s self-Px{sa in- Sl statement-Ncnsl
p a s s i v e 5.13
Some of the annotated quotes presented more than one sentiment due to a change of tone or topic. Such sentences were segmented into parts which then received di erent sentiment annotations. These types of sentences can usually be easily spotted in Croatian because they use so-called \contrary conjunctions" which link constituent-sentences of contradictory sentiments. Contrary sentences are a type of complex sentences in which all of the clauses can be independent and convey contrary meanings. An example can be seen in Example 18., where `ali' (en. `but') is the contrary conjunction. The part before `ali' was annotated as positive and the rest of the sentence as negative. For easier understanding, the positive clause has been coloured with green, while the negative one was coloured with orange in the example given.
18. Croatian:
\Vjernik believer-Ncmsn
time, it-Pd-nsi
ne no-Qz
sam be-Var1s
i and-Cc p o s i t i v e
ponosim proud- Vmr1s English: \I am faithful and proud of it, & but killing innocent people can not have (...)" 6
In our paper, a detailed presentation of how a Croatian corpus of 140 documents annotated in terms of direct quotation features was created and what procedures were employed is o ered. Potential problems for future work in the Quote Fine and Quote Simple layer were recognized and described, along with their examples in Croatian and English. The work described in this paper provides merely a starting point in creating a system or a tool for automatic extraction and attribution of quotations in Croatian. In the future, this gold standard data will be used to automatically annotate the remaining un-tagged documents of Croatian SETimes collection and create a silver standard dataset for Croatian. Furthermore, the annotations will be projected to the SETimes English parallel corpus and Bulgarian, Bosnian, Greek, Macedonian, Romanian, Albanian, Serbian, and Turkish. 7
The work presented in this paper has received funding from the European Union's Horizon 2020 research and innovation program under the Marie SklodowskaCurie grant agreement no. 812997 and under the name CLEOPATRA (Crosslingual Event-centric Open Analytics Research Academy).