Subjectivity is a feature of language: when making an utterance, the speaker simultaneously expresses their position, attitude, and feelings towards the utterance, thus leaving their own mark [1]. Subjectivity Detection (SD) is the task of distinguishing objective content from subjective one. Previous SD approaches can be divided into syntactic and semantic [2]. The first category relies on keyword spotting [3,4] or lexicons [5,6,7] as standard practice. However, these solutions are known to be language-specific unless some intermediate lossy translation procedure is considered [8]. Likewise, lexicon-based approaches require an external knowledge base which limits their applicability. In contrast, semantic approaches tackle SD via statistical [9,10] or neural [11,12,13] methods for text representation by relying on labeled training corpora. This requirement is either addressed by considering domain-specific assumptions [9] In: R. Campos, A. Jorge, A. Jatowt, S. Bhatia, M Litvak (eds.): Proceedings of the Text2Story'23 Workshop, Dublin (Republic of Ireland), 2-April-2023 * Corresponding author. Envelope federico.ruggeri6@unibo.it (F. Ruggeri); francesco.antici@unibo.it (F. Antici); a.galassi@unibo.it (A. Galassi); aikaterini.korre2@unibo.it (K. Korre); arianna.muti2@unibo.it (A. Muti); a.barron@unibo.it (A. Barrón-Cedeño) Orcid 0000-0002-1697-8586 (F. Ruggeri); 0000-0002-1125-0588 (F. Antici); 0000-0001-9711-7042 (A. Galassi); 0000-0002-9349-9554 (K. Korre); 0000-0002-3387-6557 (A. Muti); 0000-0003-4719-3420 (A. Barrón-Cedeño) or designing annotation guidelines [11,14,15,16].

Despite their independence from linguistic tools and allowing cross-lingual applicability with minor efforts [16,17,18], semantic approaches face a crucial yet demanding issue: the perception of subjectivity is itself subjective [2] and, thus, it is affected by interpretation bias [19], annotation ambiguity, and edge cases. As a result, defining practical, non-language-specific, and largely applicable annotation guidelines is a well-known challenge [15].

In this work, we adopt a prescriptive approach [20] and frame SD for a specific task to downplay annotation ambiguity [21], describing a method for the development of task-oriented annotation guidelines based on three key aspects: schematic case-based guidelines, iterative refinement, and reliable annotation. We also consider a preliminary case study on fact-checking to empirically evaluate the proposed methodology and elaborate on the encountered open challenges.

We identify three key aspects for developing task-oriented SD annotation guidelines. We follow the prescriptive paradigm [20] to impose a specific and consistent conceptualization of subjectivity for annotation.

Given a task that partially relies on SD, it is necessary to define subjectivity according to the task's objectives. It is, therefore, necessary to define annotation guidelines that are schematic and based on specific real cases. This formulation is less sensitive to domain-or language-specific cues and eases the annotators' training process. Moreover, these properties could foster collecting large corpora for SD based on annotation guidelines rather than relying on domain-dependent assumptions [22].

Iterative refinement. Agreeing on a set of validated annotation guidelines is a collaborative refinement process. Such a process has the objective of discovering annotation edge cases, i.e. instances that are not covered by annotation guidelines resulting in high inter-annotator disagreement. Indeed, a preliminary version of annotation guidelines is unlikely to thoroughly cover all possible cases. For this reason, guideline refinement is an iterative process consisting of multiple annotation pilot studies since edge case discovery depends on the nature of sampled annotation data [23]. The pilot studies are designed to instruct annotators and reach a common set of validation annotation guidelines [24], and are iterated until a sufficient level of agreement is reached [25]. This formulation is in line with the prescriptive paradigm [20], where annotator disagreement is a call to action to refine annotation guidelines.

Reliable annotation. The last key aspect concerns the data annotation task. First, annotators are provided with refined annotation guidelines to instruct them. Second, text instances are assigned to multiple annotators to downplay the impact of noisy labels and annotators' bias [19]. This process allows for discriminating edge cases from instances with a unanimous or almost perfect agreement. Tracking of individual annotations per instance is considered a measure of quality assurance [20,26]. Eventually, labels can be aggregated via voting strategies for training machine learning models [27]. In case of disagreement, a discussion phase among annotators takes place to agree on a solution. An additional annotator to label these instances is considered if an agreement is not reached. To address the problem of noisy labels, it is possible to discard those assigned by annotators that strongly disagree with each other [28] and explicitly report for which instances the discussion phase did not solve ambiguities [29,30].

We elaborate on the presented methodology by discussing a case study on fact-checking. We consider a pipeline for fact-checking where SD is performed to discriminate between objective sentences that can be directly verified and subjective sentences that must be processed or rewritten to extract the objective claim or information. The detection and processing of subjective content have the final purpose of creating an objective narrative upon which factchecking relies [31]. We consider the task of labeling sentences in English and Italian news articles targeting ongoing controversial topics, such as political affairs, Covid-19, civil rights, and economics (see Appendix A).

We initially design a set of preliminary annotation criteria suitable for fact-checking purposes (see Appendix B). These guidelines are mainly derived from existing work on SD on related domains [11,32]. We recruit six human annotators with native or near-native knowledge of the English and Italian languages. After two annotation pilot studies, annotators agree on a common set of annotation criteria. We keep track of inter-annotator agreement (IAA) over pilot studies to validate their efficacy. In particular, the average Cohen's kappa over annotator pairs is 0.39 (fair agreement) and 0.53 (moderate agreement) for the first and second pilot studies, respectively. We consider both Italian and English annotations when computing the IAA and observe comparable results between languages. The observed 14% gain between the two studies denotes a significant improvement in the annotation criteria.

During the pilot studies, we discuss the importance of contextual information (Section 3.1) for annotation and address several edge cases (Section 3.2). These observations are consistent in both languages, proving the efficacy of our methodology regardless of the language.

The lack of context may lead to ambiguous annotation cases, depending on the chosen input granularity [33,34]. In our setting, we consider sentence-level granularity as common practice [31]. This choice represents a suitable testing ground for evaluating context importance given the limited scope of a sentence. For this purpose, in the second pilot study, we arrange annotators into two groups. Half of them label input sentences in order of appearance, while the remaining half labels sentences in random order, neglecting any contextual information as done in the first pilot study. We observe a 0.38 and 0.53 average Cohen's kappa over annotator pairs for the context and non-context groups, respectively.

Our findings contrast the results of Ljubešić et al. [35], suggesting that context may be useful only in certain tasks or specific scenarios. Moreover, we identify two additional reasons in favor of a non-contextual annotation formulation. First, the use of context leads to an increased

Example of edge cases encountered in our case study.

He looked like he was on the verge of crying.

(b) Quotes "Crosbie is an extremely violent man who has no place in society, and we welcome the jury's verdict today." (c) Intensifiers

Recognising that, last Friday the US announced a further $600m of military aid to Ukraine, including more Himars rockets that have so damaged Moscow's logistics and its ability to resist. (d) Speculations Putin will hope to sow uncertainty in the eyes of policymakers' meetings in New York.

annotator's workload. Consequently, it negatively affects the applicability of annotation guidelines to multiple scenarios. Second, contextual information may not be available in certain domains and settings, as in tweets [2]. These observations and the higher IAA suggest that a non-contextual annotation for SD is a preferred formulation.

During our pilot studies, we identify four edge cases, as reported in Table 1.

Emotions. Statements carrying emotions convey a subjective point of view [36,37] but they cannot be verified or confuted by a fact-checking system since they are based on the author's beliefs and sensations only. Since it is impossible to provide such information in a more objective form, we label these statements as objective.

Quotes. In news sources, authors frequently use quotes to support their thesis. Even if the quoted content may be subjective, the task concerns detecting subjectivity only for the article's author. For this reason, we label quoted content as objective.

Intensifiers. We identify intensifiers as indicators of subjectivity since their presence could be symptomatic of the author's personal point of view. For example, in Table 1 (c) it is difficult to state if the expression "so damaged" conveys the author's personal point of view or, rather, is descriptive and can be re-formulated as "that have in this way damaged".

Speculations. Annotators often struggle to judge implicit statements without leveraging their own interpretation bias [38]. We consider speculation as a subjectivity indicator, since authors make use of it to allude to their own interpretation of events and consequences. The expression "will hope to sow uncertainty" in Table 1 (d) is an example.

We have presented our ongoing work on developing annotation guidelines for task-oriented SD. In particular, we introduced a methodology based on the prescriptive paradigm [20] to provide a task-specific definition of subjectivity via schematic and language-independent annotation criteria. These criteria are developed to cover annotation edge cases and downplay annotators' interpretation biases. The application of our methodology to a preliminary case study on fact-checking in two different languages allowed us to reduce the ambiguity of the annotation by identifying edge cases and addressing them through the definition of specific guidelines. In future works, we will extend our approach to further languages.