<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>The Roots of Doubt. Fine-tuning a BERT Model to Explore a Stylistic Phenomenon</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>MargheritaParigini</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Mike Kestemont</string-name>
        </contrib>
      </contrib-group>
      <fpage>72</fpage>
      <lpage>91</lpage>
      <abstract>
        <p>The narrative work of well-known Italian author Italo Calvino (1923-1985) features a phenomenon that literary critics refer to as “dubitative text”: this stylistic device consciously hinders the narrative progression of a story, by questioning its own content. We report on an attempt to model the presence of dubitative text in Calvino's 昀椀ctional oeuvre and examine whether this model can also be used to retrieve dubitative instances in his essayistic oeuvre. We hypothesize that precisely the category of the dubitative text yields interesting points of intersection between both writing modes. We 昀椀ne-tuned a BERT model based on a manually annotated dataset and report inter-annotator scores. We situate our 昀椀ndings and model criticism in the current landscape of Calvino scholarship. While detecting dubitative text is challenging, our model provides fresh insights into the device's surface features.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Italian literature</kwd>
        <kwd>Italo Calvino</kwd>
        <kwd>BERT</kwd>
        <kwd>entity recognition</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        The project looks at the manifestation of the phenomenon in 昀椀ction only, without considering
his essays production. Nevertheless, it is important to underline that Calvino, responding to a
trend of the time, is leaning over the course of his career towards hybrid forms that mix the
techniques of both his 昀椀ctional and essayistic writing (e.g. [
        <xref ref-type="bibr" rid="ref13">38, 18, 24</xref>
        ]).4 A recent study devoted to
the author’s non昀椀ction production showed how he tends to express himself by adopting
di昀erent «forms of perplexity» [6, p. 10][own translation]: in the essays, «the reader is accompanied
by a subtle enunciative game relying on questions and stylistic devices of doubt and
questioning» [6, p. 11][own translation]. Our hypothesis is that these “forms of perplexity” could be
related to the dubitative text.
      </p>
      <p>The dubitative phenomenon belongs to the 昀椀ctional universe, yet it might have its roots
within the essays, or vice versa it may have been able to propagate into non昀椀ction writing:
however, it seems clear to us that a link exists. This research intends to explore the nature of
this link and aims to assess what stylistic and structural components the dubitative text might
have in common with the non昀椀ction genre. To investigate this hypothesis, we 昀椀ne-tuned a
BERT model to automatically detect the presence of dubitative text in a cross-genre setting.
A昀琀erwards we annotated a representative sample of texts in both genres for dubitativity (§2.2.)</p>
      <p>One of the de昀椀ning characteristics of the dubitative phenomenon is that a昀ects “the main
progression of thestory” (§2.1.). The genre of the essay possesses a mobile de昀椀nition in the
various critical traditions, which, however, has some constants: in particular, this form of discourse
«is based on the implicit prescription not to invent a 昀椀ctional world to convey re昀氀ections on
the world» 3[1, p. 152][own translation]. Consequently, trying to analyze the essays using the
category of the dubitative text might seem like a stretch. However, we created a validation set
of essays by collecting dubitative occurrences as if they were 昀椀ctional texts, with two speci昀椀c
purposes: (a) 昀椀eld-testing the eventual limits of the dubitative category; (b) having a reference
point for more thoughtful evaluation of the model results.</p>
      <p>Instead of focusing on model performance, our aim was to derive interpretative insights into
the dubitative text in both genres, in order to deepen our understanding as to which stylistic
or structural elements are associated with this phenomenon.</p>
    </sec>
    <sec id="sec-2">
      <title>2. An Ambiguous Category</title>
      <sec id="sec-2-1">
        <title>2.1. Dubitative Text</title>
        <p>The dubitative text is a category of analysis which draws inspiration on multiple linguistic
and narrative dimensions of a text. It is a challenging notion to operationalize in a formal
annotation framework4[5]. We set up an annotation process with the ultimate aim of training
a machine learning model that can automatically detect its presence in an unseen text. We focus
speci昀椀cally on a stylistic conceptualization of this label in the sense proposed by Herrmann who
argue that «style is a property of texts constituted by an ensemble of formal features which
can be observed quantitatively or qualitatively2»5[, p. 44].
4«Scholars such as Thomas Pavel have recognized in the mixture of narrative and non昀椀ction meditation a decisive
character of the modern and contemporary novel (think of Marcel Proust, Thomas Mann, Robert Musil, Jorge
Luis Borges, and, among more recent examples, Thomas Bernhard, Milan Kundera, or Enrique Vila-Mat1a9s),» [
p. VII][own translation]4[0].</p>
        <p>To consider a string of text as dubitative, we require that the progression of the story (or a
cohesive section of it) must be based on the deconstruction of its content. The dubitative text
is therefore characterized by the following linguistic features:
• expressions of epistemic modality4[2, pp. 54-61];
• discourse connectives related to the argumentative genr3e0[];
• punctuation marks (parentheses, dashes, question marks, ellipsis);</p>
        <p>These features serve as useful surface cues in a text to 昀氀ag any dubitative occurrences in
Calvino’s writings. However, these characteristics in isolation do not automatically entail the
presence of dubitativity – the 昀椀rst condition regarding the deconstruction of the narrative
progress must still be met. Let us take an example.</p>
        <p>UN-DUBITATIVE
«piccole increspature che si propagavano
sott’acqua: forse il battere di coda d’un pesce,
forse un bambino che riempiva un secchiello
su una riva»
[small ripples that propagated underwater:
perhaps the tapping of a fish's tail, perhaps a
child filling a bucket on a shore] [own
translation]
Com’era grande il mare, 1948</p>
        <p>DUBITATIVE
«Una bestia si mosse in fondo a un cespo
d’eriche: forse una lepre, forse una volpe,
forse un tedesco coricato tra gli arbusti che lo
prendeva di mira »
[An animal moved at the back of a bush of
heather; perhaps it was a hare, perhaps a fox,
perhaps a German lying in the thickets keeping
him covered]</p>
        <p>Paura sul sentiero, 1946</p>
        <p>The stylistic form adopted in the two passages is almost identical. A movement whose
origins are unknown is described («piccole increspature», «una bestia si mosse»), and three
hypotheses introduced by as many adverbs of doubt («forse») are presented. The di昀erence
between these two extremely similar texts lies in the way the respective plots are developed.
Com’era grande il mare is a short story about a couple ascending a headland on a scorching
summer day, who, blinded by the sun and mugginess, cannot always see clearl1y1,[p. 855]. Paura
sul sentiero narrates the nighttime crossing of a forest by a partisan dispatcher: darkness, sleep
and fear generate a deeply doubting attitude toward what surrounds the partisan, thus
making the progression of the narration coincide with a concatenation of uncertainti9e,sp[. 250].
While in the case of the 昀椀rst short story we are dealing with a circumscribed visual di昀케culty,
in the second the thematization of doubt is the pivot around which the narrative revolves.
Determining which text can be considered dubitative is an intersubjective, interpretative act that
needs a case-by-case inspection during a manual annotation process.</p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. The Inter-Annotator Agreement</title>
        <p>
          Two annotators have independently annotated a representative sample of text to be able to
assess the feasibility of this annotation task. Each was asked to analyze a di昀erent set of texts,
selecting the parts considered as dubitativ5e.Their work was subsequently compared to the
group of annotations linked to the corresponding texts in the dubitative text dataset derived
from the Atlante Calvino project (§3.1.). Annotator A was in charge of the two related novels
Il castello dei destini incrociati and La taverna dei destini incrociati. Annotator B analyzed the
collectionPalomar. We decided to apply Cohen’s Kappa 1[4] [
          <xref ref-type="bibr" rid="ref18">22</xref>
          ] to measure the IAA.
        </p>
        <p>Annotator A obtained a score of 0.54 and Annotator B of 0.50. The di昀케culty of the task is
illustrated by the weak agreement observe6dW.e were therefore aware from the outset that we
had to manage our expectations regarding the performance of the automated model, deciding
to «exploi[t] disagreement between crowd workers as a signal, rather than try to eliminate it»
[2, p. 23].7</p>
        <p>
          Dubitative text is an elusive category that presents classi昀椀cation problems. We tried to
understand what could be the causes of such a low score, analyzing the discrepancies: (a) within
an area considered unanimously dubitative, the boundaries of occurrences are not always clear
(cf. Fig. 2); (b) it seems to us that annotations diverge when confronted with sentences with
a strong semantic component of uncertainty but without clear stylistic markers (Fcifg.. 3).
Both of these problems later resurfaced in the results of the model, studying the false positives
and the false negatives (§6.).
5To further explore the units of analysis used in content analys2i7s,[pp. 97-110].
6As reference points for evaluating the scores obtaine3d2[
          <xref ref-type="bibr" rid="ref26">, 28</xref>
          ].
7We do not want to claim that the «exploratory tool» produced, «even if wrong», is «intrinsically valuable because
exploration is intrinsically valuable1»5,[p. 602] Rather, we want to go beyond the numbers, as daunting as they
may seem, to check their cognitive reliability. Actually, «Parsing f-scores, and precision and recall values enable
a certain degree of understanding of the overall performance of the classi昀椀er and the function of the work昀氀ow as
a whole, but this leaves aside many other post-classi昀椀cation metrics that can be directly applied to the underlying
data model. These data can tell us much about our input datasets and the criteria by which the classi昀椀er made its
classi昀椀cations» [
          <xref ref-type="bibr" rid="ref12">17</xref>
          ].
Example
1
        </p>
        <p>annotation number 1
start/end of annotation</p>
        <p>ANNOTATOR A</p>
        <p>ANNOTATION ATLANTE
A ben vedere, tanto per l’alchimista quanto per il cavaliere errante</p>
        <p>1 1</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Data</title>
      <sec id="sec-3-1">
        <title>3.1. Dataset of the Dubitative Text</title>
        <p>The annotated training dataset was obtained from the Atlante Calvino project. It is composed
of almost 5,000 occurrences of dubitative text, derived from the 昀椀ctional work published when
Calvino was alive (205 short stories and 10 novels). Occurrences vary in length, including
groups of words, sentences, or even whole paragrap8hs.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Corpus of Essays</title>
        <p>Calvino produced numerous essays and articles throughout his career, about 400 of which
constitute the two volumes edited by Mario Barengh1i2[]. Nevertheless, we limited our study
to the three volumes set up by the author himself as coherent ensembleUs:na pietra sopra.
Discorsi di letteratura e società, published in 1980 and containing 42 essays written between 1955
and 1978; Collezione di sabbia, published in 1984 and containing 23 essays written between 1980
and 1984; Lezioni americane. Sei proposte per il prossimo millennio, published posthumously in
1988, which collects 5 lectures that the author should have given at Harvard University in
1985-1986 as part of the Charles Eliot Norton Poetry Lectures. The volumes were purchased
in electronic format and automatically converted to plain text.</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. Validation Set</title>
        <p>We set up two validation sets for the model validation: one consisting of 昀椀ctional writings (and
thus closer to the training material) and one consisting of essays (and thus closer to the target
domain for our model) (§5.).</p>
        <p>Five di昀erent short stories were selected ensuring that they were distributed chronologically
across the entire time span of Calvino’s career and with a variable presence of the dubitative
phenomenon. Ten essays were selected following the same guidelines.
8The occurrences have been collected, using the guidelines mentioned above (§2.1.). For an accurate description
of the structure of the dataset cf. DOUBT – DATASET 2 inhttps://atlantecalvino.unige.ch/capta?lang=enThe
dataset will be available once the PhD of its author, Margherita Parigini, is completed. Instead, the corpus data
from which the IDs and collateral information were derived for this paper are accessibl4e9c]f.. [
title</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Fine-tuning BERT model</title>
      <sec id="sec-4-1">
        <title>4.1. Research background</title>
        <p>
          We 昀椀ne-tuned a Transformer BERT to detect the presence of the dubitative text adopting a
NER approach. BERT, which means Bidirectional Encoder Representations from Transformers
[
          <xref ref-type="bibr" rid="ref49">51</xref>
          ], is «designed to pretrain deep bidirectional representations from unlabeled text by jointly
conditioning on both le昀琀 and right context in all layers»16[]. One of the aspects that make
BERT such an e昀ective tool is that it can be easily 昀椀ne-tuned to accomplish a variety of speci昀椀c
tasks, «with only one additional output layer1»6[]. To conduct our research, we used BERT
as a sequence tagger. For the most part, we have reproduced1][.
        </p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Preprocessing Data</title>
        <p>
          During preprocessing, an important initial choice related to the relative proportion of
dubitative text presented to the model in comparison to the un-dubitative text; our corpus statistics
suggest a serious class imbalance – the proportion of dubitative text has an average occurrence
of only 20%. During training, we therefore arti昀椀cially varied the relative proportion of
dubitative text fed to the model as a hyperparameter46[
          <xref ref-type="bibr" rid="ref34">, 36</xref>
          ]. Regarding un-dubitative text, we
decided to provide the model with (a) the context of each occurrence, using the sentences
naturally coming before/a昀琀er the dubitative ones, and (b) examples of non-dubitative sentences
extracted from Calvino’s 昀椀ctional corpus.
        </p>
        <p>(a) To determine the size of the context, we calculated the average number of words in
relation to the total number of words in occurrences in the dataset, which correspond to
22;
(b) We randomly extracted sentences from the 昀椀ctional corpus, looking at the mean and
variance of the distribution of dubitative occurrences in the dataset, for a total of 5000
sentences. We also took into account the possible overlap with dubitative occurrences
and we made sure to avoid it;
UN-DUBITATIVE TEXT</p>
        <p>DUBITATIVE TEXT
Da Felice i sei occupavano il banco del
bar da una parte all’altra, con tutti quei
SENTENCE pantaloni bianchi e quei gomiti
appoggiati al marmo che sembrava
fossero in dodici.</p>
        <p>LEFT CONTEXT fsoirmseosusnea i nle pforen,dofoarseununcaesvpoolpde’e,rfiocrhsee: Tag IN: B-DOUBT
OCCURRENCE un tedesco coricato tra gli arbusti che lo I-DOUBT</p>
        <p>prendeva di mira. C’era un tedesco per
RIGHT CONTEXT ogni cespuglio, un
+
+</p>
        <p>Stavano appollaiati lassù da alcuni mesi,
SENTENCE confidando nel clima mite, in un
prossimo decreto d’amnistia di Carlos III
e nella provvidenza divina.</p>
        <p>Tag OUT: O
Tag OUT: O</p>
        <p>To structure the 昀椀les needed to 昀椀ne-tune the BERT model, we interspersed a non-dubitative
sentence before and a昀琀er each group of tokens formed by context and occurrence.</p>
        <p>Next, we were careful to stratify the data to form a training set and a test set, adopting the
proportions 80% and 20%, respectively, without shu昀툀ing the tokens.</p>
      </sec>
      <sec id="sec-4-3">
        <title>4.3. Training Steps</title>
        <p>
          Our model was developed through numerous iterations to determine the optimal
(hyper)parameters on the validation data, also in the light of the prior literature on the topic (e.g.
[
          <xref ref-type="bibr" rid="ref24 ref33 ref49">26, 35, 51</xref>
          ]). Here, we brie昀氀y report on the main insights derived from this process.
        </p>
        <p>
          The BERT Lang Street platform3[
          <xref ref-type="bibr" rid="ref2">7</xref>
          ], developed by a group of researchers within the Bocconi
University of Milan, allowed us to identify the most suitable model to perform our type of task.
A昀琀er an initial attempt with the multilingual version, we then opted for thitealian XXL cased
model. The model trained with the starting distribution of dubitative text obtained an F1 of
0.49. Because of the fairly low Kappa’s score and the scores of previous experiences performed
on similar tasks 4[6], our expectations linked to the model were modest from the beginning.
Next, we experimented with the proportion of the doubt/no doubt instances in the training
data.
        </p>
        <p>As soon as we manipulated the internal distribution of the data, moving away from the
values established at the outset, the model went into over昀椀tting.</p>
        <p>M</p>
        <p>We then took the trained model with the best score so far and we decided to change the
training score from the traditional F1 to precision score, so that the model selected during
昀椀netuning would be as reliable as possible. In fact, our goal at this stage of the research was to
obtain su昀케ciently accurate results to be able to interpret the model selection processes.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Results</title>
      <sec id="sec-5-1">
        <title>5.1. Model Criticism</title>
        <p>To analyze the results we decided to select the three models with the best scores in the
昀椀netuning phase, namely Model1 (M1), Model2 (M2) and Model3 (M3). We wanted to take an
experimental approach, going beyond the numerical results that would lead us to choose a
single model. We wanted to see if, by combining the di昀erent results, we could learn more
about the dubitative text and how it was decoded.</p>
        <p>
          At 昀椀rst, we noticed a correlation between the density rati9o[
          <xref ref-type="bibr" rid="ref48">50</xref>
          ] of the dubitative text and
the scores of the validation set: this aspect is probably related to the strong imbalance in the
9Given two data samples 1 and 2 from unknown distributions, the density ratio is equal t(o:1 + 2)/ 1, where
1 in this case are the dubitative occurrences and2 are the non-dubitative occurrences cfh. ttps://github.com/h
oxo-m/densratio_py.
density ratio
density ratio
source data. Both models generally perform better when the presence of the phenomenon is
frequent (e.g. S159), but their performance weakens in the most extreme phenomenon
distribution situations: when its concrete presence is minor (e.g. S007), the models tend to predict more
than necessary; in the same way, when confronted with a highly dubitative text, the models
tend to categorize as dubitative those parts of the text with marked stylistic features (especially
graphical marks and argumentative connectives), leaving out more ambiguous expressions of
doubt (e.g. S142).
        </p>
        <p>We also observed a general decrease of the scores once the models were applied to the
essays.10 Rather than being alarming, this 昀椀nding allowed us to better understand a fundamental
10This could be explained with the literary genre of the training data: the dubitative text is a phenomenon that
takes place in the narrative corpus; there is therefore a mismatch between the material used to train the models
aspect of the phenomenon which we have already had occasion to mention (§3.3.): the
models numerically con昀椀rmed the di昀케culty in transporting this category into a realm other than
椀昀ction. However, our goal was not to verify the existence of the dubitative text in essays, but
to explore the possible connections between the phenomenon and essayistic expression. An
initial numerical data con昀椀rms the high compatibility between these two stylistic modes: in
fact, we can easily notice an increase in the percentage of essay text recognized as dubitative
compared to 昀椀ctional text.</p>
        <p>low
&lt;10%
medium
&lt;19%
hight
20%&gt;
titles
E006
E011
E019
E038
E044
E046
E070
E072
E078
E082</p>
        <p>VALIDATION SET: ESSAY
Model 1 Model 2 Model 3
18% 26% 16%
20% 31% 21%
16% 23% 13%
30% 21% 13%
21% 25% 13%
18% 23% 6%
11% 10% 11%
21% 19% 17%
15% 36% 17%
16% 21% 14%</p>
        <p>All models react with more greediness once they are applied to essays, recognizing a strong
presence of the phenomenon. This 昀椀rst 昀椀nding brought us to guess the existence of a point
of contact between the category of the dubitative text and the essay genre. The essay is a text
with a reasoning framework that attempts to approximate reality, weighing certain aspects of
it [19, p. VII]. It is a «genre of inquiries and questions», «intimately doubtful2»1,[ p. 186,
197][own translation]. These characteristics are highly compatible with our de昀椀nition of the
dubitative phenomenon. Yet the essay, however hybrid and 昀氀uctuating in nature, belongs to the
椀昀eld of argumentation. Applying the model trained to recognize the dubitative text to essays
allowed us to verify the presence of certain argumentative techniques that 昀椀lter into narrative
production, redirecting the development of the sto1r1y.</p>
        <p>Dobson states that «determining the meaning of textual data requires both computational
knowledge that can determine signi昀椀cance within the model and domain-speci昀椀c contextual
knowledge that can be applied to the understanding of these features1»7[]. A昀琀er evaluating the
numerical performance of the models, we then made a second reading, studying the concrete
results. A careful analysis of the errors made, both on the narrative and essay validation sets,
allowed us:
• to verify the presence in the non昀椀ction genre of certain stylistic features related to the
phenomenon;
and the type of text on which the models are applied. This apparent discrepancy is part of the experience, as our
goal is precisely to use the models to understand which are stylistic features related to the argumentative genre
may be reused by the dubitative text.
11The mix between non昀椀ction and 昀椀ction, though not through the prism of the dubitative text, has been reported in
the 昀椀eld of Calvinian studies particularly with regardLtoa giornata di uno scrutatore (1963), «a kind of novel-essay,
suspended between testimony and re昀氀ection»[3, p. 56][own translation].
• to identify stylistic and argumentative predispositions that open up new research
directions.</p>
      </sec>
      <sec id="sec-5-2">
        <title>5.2. Model Inspection</title>
        <p>A manual inspection of the models’ output indicated that certain surface features provided
strong cues to the models to 昀氀ag text as dubitative (cf. Appendix): (A.1) the presence of
punctuation marks; (A.2) sentences with keywords related to modality. Additionally, a closer
scrutiny of the false positives allowed us to identify two additional dubitativity markers which
we named respectively cerebrality and 昀椀gurality:
• (A.3) Cerebrality: verbs related with a brain activity suchpaesnsare [to think] andsapere
[to know];
• (A.4) Figurality: verbsparere [to seem], guardare [to look, to watch],vedere [to see] and
adverbcome [like];</p>
        <p>The 昀椀rst aspect (A.3) suggests an intellectual distance between the narrated object and the
act of writing. From the essay, the dubitative text assimilates the reasoning attitude. The
essay is marked by a strong «experimental element», a type of «prose as a means of relating
to the world» 1[9, p. VII][own translation]. The dubitative text thus seems to insert a 昀椀rst
and foremost conceptual frontier into the narrative, anchoring the development of 昀椀ction to a
re昀氀exive logic. Narrative strategies are then hybridized to argumentative strategies based on
approximation to eventual truth.</p>
        <p>
          Regarding the second aspect (A.4), the role of the visual dimension has particular relevance
for Calvino 4[]. Thus, it is not surprising that the tendency to associate narrative objects based
on a 昀椀gurative logic would emerge in his work. However, it is interesting that this same logic
is associated with the dubitative phenomenon, moreover in an essayistic dimension: the
author uses comparison, metaphor, and analogy as conceptual bridg12esT. here is no unanimous
agreement on the role that rhetorical 昀椀gures should play in argumentation, because their
presence does not correspond to the need for clarity associated with the genre: metaphor, for
example, «is both true and false, guilty of ambiguity and categorical erro4r3»][o[wn
translation][
          <xref ref-type="bibr" rid="ref42">44</xref>
          ]. However, Fenoglio traveling «backwards along the chain of tradition in search of
the masters of essaisme modern», identi昀椀es a «metamorphic drive at the origins of the genre»
[21, p. 180][own translation]. The essay, we have said before, is «the quintessential questioning
genre» [24, p. 31][own translation], marked by «the uncertainty, the contradictory, the
swarming of opinions and points of view»2[4, p. 33][own translation]. It is a type of text designed
to willingly accommodate both visual and conceptual refraction. And the dubitative text has
absorbed this characteristic, readily rehousing it in the realm of 昀椀ction, which is particularly
predisposed to accommodate this kind of stylistic momentum5][.
12This aspect had been pointed out by Pier Vincenzo Mengaldo, speaking of the «nexus between analogy-based
椀昀gurality and precision» [33, pp. 253-55][own translation].
6. Conclusion
«The opacity and ine昀ability of the text and the ethical demand to attend to it remain central
to practices of literary interpretation today2»9,[ p. 371]. Love’s ”today” is now twelve years
old, yet the issue continues to polarize our attention. The act of interpretation means «always
wrestling with the text» 3[4, p. 33][own translation], a strenuous struggle aimed at taking it
apart in an attempt to better understand the internal mechanisms that govern it. Currently, it
seems that research e昀orts have shi昀琀ed, focusing not so much on discovering the mechanisms,
but rather on the principles that govern this struggle. Establishing incontrovertible and
transparent rules is the new criterion by which to attribute value to an experience, relegating the
act of interpretation to an entirely secondary level.
        </p>
        <p>In the current research landscape of the Digital Humanities the debate resonates around the
proper weight of the interpretive act of the researcher and, implicitly related, the role of1e3rror.
Da negatively judges the way that for CLS «misclassi昀椀cations become object of interest,
imprecisions become theory»1[5, p. 602]. But one would have to wonder how error is perceived
in the more traditional humanistic sphere. Literary criticism is not based on the revelation of
absolute evidence, but on the demonstration of a point of view: «the 昀椀eld of argumentation is
that of the plausible, the probable, insofar as the latter escapes the certainties of calculation»
[41, p. 3][own translation].</p>
        <p>The perspective in which this research 昀椀ts does just what Da argues should not be done: we
did not think of the «statistical tools» relying on their so-called «true functio1n5s,»p[p.
619620]. Rather, «from a humanist perspective, we might want to think of data models created
from textual sources as alternative representations of supplie1d7»].[</p>
        <p>The trained model BERT was a way of viewing a research problem of literary criticism in
an innovative way. In this sense we tried, following Dobson’s suggestion, a «making
computational work interpretable»17[]. Through this experience, we learned that the point of contact
between the dubitative mechanism, belonging to the universe of 昀椀ction, and the non昀椀ction
genre is the latter’s peculiarity of possessing the «lens of a double visio2n1», [p. 196][own
translation]1.4 This characteristic of Calvino’s non昀椀ction is translated into the two
characteristics: 昀椀gurality on the one hand and cerebrality on the other.</p>
        <p>«The assumption of the essay is that the reader is reading things that are true, perhaps
paradoxical and provocative, but all the more so as they should be taken literally and be understood
in a dimension of reality. The assumption of the novel, on the other hand, is that, however
veraciously taken from reality, the things being read are to be understood on a plane of 昀椀ction23»,[
pp. 20-21][own translation]. The relationship between 昀椀ction and reality is thus put under
tension by a technique such as the dubitative text, which applies certain essayistic features within
13We 昀椀nd the attention to this issue that has emerged in recent CoP linked to some prestigious publication venues
particularly representativeW:orking on and with Categories for Text Analysis: Challenges and Findings from
and for Digital Humanities Practicefsor DHQ;Reproducibility and Explainability in Digital Humanitfioers IJDH.
Even among the topics of interest in the call for CHR22, two inherent items appeared: “development of empirical
methods for humanities research” and “modeling bias, uncertainty, and con昀氀icting interpretation in the
humanities”.
14«The essayist in e昀ect contemplates the gaze of others, thus his is a refracted gaze [...] he does not bend to the
evidence of things, he questions it, he subjects it to veri昀椀cation, that is, he looks at it from a foreshortened, unusual
and oblique point of view»2[1, p. 196][own translation].
a 昀椀ctional universe: this phenomenon implicitly prompts us to consider what is said within
the 昀椀ction as a truth to be proved. The dubitative text deceives the reader that a 昀椀ctional truth
can be reached by approximation, by adding and subtracting pieces of the di昀erent versions of
a single story that multiplies before our eyes through doubt.</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>7. Future work</title>
      <p>Through analysis of the results, we were able to see the strengths and weaknesses of the various
models. By cross-referencing the three models with each other, it is possible to derive a new
one in turn that could be more accurate in identifying dubitative occurrences. Once created,
this synthesis will be applied to the totality of the essays to study where the two features of
椀昀gurality and cerebrality appear and how they behave. At the same time, an analysis will be
conducted on the totality of argumentative connectives in the essays, leaning on the work of
[20], to understand which discourse relations are transported to the 昀椀ctional domain by the
dubitative text.</p>
      <p>Therea昀琀er, it is our intention to conduct new training by creating subsets of the starting
dataset based on the structural types of the occurrences: in fact, one of the major di昀케culties
when 昀椀ne-tuning the models is related to the occurrences’ uneven length. Our hope is that by
re昀椀ning the type of analysis, the model can improve its performance. Furthermore, in
analyzing false negatives, we realized that some features seem particularly di昀케cult to identify: (B.1)
negations; (B.2) questions; (B.3) alternatives; (B.4) sentences that rephrase something stated
earlier if taken in context, but once isolated possess no clear dubitative markers (cf. Appendix).
We hope that the creation of training subsets in the dataset, based on the structural types of
occurrences, can help us resolve these impasses in categorization.</p>
    </sec>
    <sec id="sec-7">
      <title>Acknowledgments</title>
      <p>We 昀椀rst thank Paavo Van der Eecken (University of Antwerp) for following this research project
from the very beginning, sharing his views on various conceptual junctures, and suggesting
ingenious technical solutions. We are grateful to Valeria Cavalloro (Università per Stranieri di
Siena) and Virginia Giustetto (Université de Genève) for their contribution as annotators. We
lastly thank Emilien Schultz (Université de Paris) and Simon Gabay (Université de Genève) for
constructive exchanges.
[6] S. Bozzola and C. D. CaprioF.orme e 昀椀gure della saggistica di Calvino . Roma: Salerno
Editrice, 2021.</p>
    </sec>
    <sec id="sec-8">
      <title>A. False Positives</title>
      <sec id="sec-8-1">
        <title>A.1. Interpunction</title>
        <p>Selection of examples to illustrate di昀erent types of false positives:
S159: “(A pagarla anche a suono d’unità a peso d’oro […] inevitabi)l”e;.S90: “Che cosa avviene
[…] mondo?” (M2); S101: “- una faccia […] faccia -” (M2); S142: “cosa intendo?” (M1, M2),
“perché intanto […] la polpa-” (M1, M3, M2), “- impercettibilmente o smisuratamente diversa
-” (M1), “- nella scelta che Priscilla fa di me” (M2); E006: “Ma a cosa ci porta il nostro continuo
nervoso sfogliare giornali freschi d’inchios?t”ro(M1, M2, M3), “Quale situazione migliore per
avere un’idea complessiva del mond?o” (M1, M3); E011: “(questa suggestione […] in po)i”
(M1, M2, M3), “(almeno all’immaginazion)e” (M1, M2, M3), “(non rare quando una coscienza
empirica e spontanea matura prima di una coscienza ri昀氀essa e organizzat)a” (M2, M3); E038:
“Il poeta(«la man che ci movea»)” (M1), “E l’arte?” (M1, M2, M3); E046: “(Popper dovrebbe
esser contento)” (M1, M2), “(vera)” (M1, M2), “(A Venezia nel Seicento il Vestri disegna una carta
delle correnti che ora le prospezioni via satellite compiute per determinare l’inquinamento della
Laguna confermano punto per punt)o” (M1), “(E’ interessante […] senza precedent)i” (M1, M2,
M3), “(cioè da 1 a 86.400)” (M1, M2).</p>
      </sec>
      <sec id="sec-8-2">
        <title>A.2. Sentences with argumentative connectives</title>
        <p>S007: “o tornare con la risposta” (M1),q“uasi con uno sciacquio” (M1), “forse già al Carmo”
(M1, M2, M3), “ma più si scava […] sottile” (M2), “Ma era anche il suo compito […] per il
椀昀eno” (M3), “ eppure riconosceva il sentiero, le pietre, gli alberi, il muschio” (M3); S1o59a: “
metà” (M1), “o pensato” (M1), “o sentire” (M1), “oppure uno smorzato […] operazione” (M3),
“o ancora […] del buio” (M3); S190: o“ per meglio […] a lui” (M3),o“ autolesionista” (M3); S101:
“quasi femminei” (M1), “ma quando poi […] fortemente” (M2), o“ troppo in frettao troppo
piano, senza libertà di movimenti, Amilcare doveva seguire la correonrteisalirla a fatica” (M1,
M2, M3); S142: “cioè che di momento in momento io non sono più lo stesso io e Priscilla non
è più la stessa Priscilla” (M1, M2, M3),m“ a forse sollevarlo non […] complicate” (M1, M3),
“cioè per via di quel che si dice” (M1, M3); E011: c“ioè a ribadire le proprie catene” (M1, M3),
“quasi diremmo extrastorica, catastro昀椀ca” (M1, M2, M3), “ma anzi forse con la possibilità di
capovolgere il rapporto tra i due termini” (M1, M2, M3a)n,z“i fa aggravare 昀椀no all’esplosione
naturale” (M1, M2, M3); E019: “Ma questo non è che il primo gradino della grammatica e
della sintassi narrativa;” (M2, M3); E044:o“una mescolanza cangiante di rosso bianco nero
grigio che sull’etichetta porta un nome ancor più policromo” (M1, M2, Mf3o),rs“e in mezzo
al deserto” (M1, M2, M3), E“ppure, chi ha avuto la costanza di portare avanti per anni questa
raccolta sapeva quel che faceva, sapeva dove voleva arrivare” (M1, M2, M3).</p>
      </sec>
      <sec id="sec-8-3">
        <title>A.3. Cerebrality</title>
        <p>S007: “Ma Binda orapensava […] sotterranei” (M1, M2); S159: “non so” (M2), “opensato
come in un delirio” (M2); S190: “Allorpaensa: «Se io vedo e penso e nuoto […] raggi” (M1),
“A ben pensarci, una tale situazione non è nuova:” (M2, M3); S142: E019: “Vediamo di tentare
un ragionamento opposto a quello che ho svolto 昀椀nora: questo è sempre il sistema migliore
per non restar prigioniero nella spirale dei proppernisieri” (M1); E038: “Talvolta iopenso e
immagino che tra gli uomini esiste una sola arte e scienza, e che questa sia il disegnare” (M1,
M2, M3); E082: “Se pensiamo che questa perorazione per una vera fraternità universale è stata
scritta quasi centocinquant’anni prima della Rivoluzione francese, vediamo come la lentezza
della coscienza umana a uscire dal suo parochialism antropocentrico può essere annullata in
un istante dall’invenzione poetica” (M1, M2, M3), “al contrarpioenso che la razionalità più
profonda implicita in ogni operazione letteraria vada cercata nelle necessità antropologiche a
cui essa corrisponde.” (M1, M2, M3), “p(enso naturalmente agli a昀ascinanti studi di Francis
Yates sulla 昀椀loso昀椀a occulta del Rinascimento e sui suoi echi nella letteratura)” (M2, M3).</p>
      </sec>
      <sec id="sec-8-4">
        <title>A.4. Figurality</title>
        <p>S007: “A voltepareva a Binda…” (M1); “come una scimmia aggrappata al collo” (M2)co“me
enormi ragni sotterranei” (M1, M2); S159:c“ome in trance” (M2, M3) “come un’altra parte di
me stesso cui corrispondono altre funzioni” (M2), “o pensactoome in un delirio” (M2),c“ome
un’altra parte di me stesso cui corrispondono altre funzioni” (Mc2o),m“e un organo della
mia persona” (M2); S101: “adesso magari faceva istintivamente per guardarle, ma subito gli
pareva che scorressero viacome un vento, senza dargli nessuna sensazione, e allora abbassava
indi昀erente le palpebre” (M3), “adesso il poterle vedere […] non più soltanto pglaireva un
vederle ma già addirittura un possederle’” (M2c)o,m“e se fosse la faccia tipica d’una categoria
di persone a lui estranea” (M2),c“ome un rimorso” (M2), E006: c“ome un limbo innocente
e funereo” (M2), “come se fossero stati rosi all’interno dalle termiti, appena gli s’avvicina la
mano non ne resta che polvere” (M2), “ma ormai, anche su questo terrenpoa,re che non possa
più crescere erba fresca” (M2, M3), E038: “Anziché il mondcoome oggetto rappresentabile
dall’arte e l’artceome rappresentazione del mondo, ci si apre un nuovo orizzonte in cui il
mondo vissuto è vistocome opera d’arte e l’arte propriamente detctaome arte al secondo
grado” (M1, M2, M3), “La lineacome segno del movimento,come godimento del movimento,
come paradosso del movimento” (M1), E082: “Non mpiare una forzatura connettere questa
funzione sciamanica e stregonesca documentata dall’etnologia e dal folklore con l’immaginario
letterario;” (M1, M3), “anzi, la leggerezza pensosa può far apparire la frivolceozmzae pesante e
opaca.” (M2, M3), “una fuga di immagini, che ècome un campionario delle bellezze del mondo”
(M2).</p>
      </sec>
    </sec>
    <sec id="sec-9">
      <title>B. False Negatives</title>
      <p>Selection of examples to illustrate di昀erent types of false negatives:</p>
      <sec id="sec-9-1">
        <title>B.1. Negations</title>
        <p>S007: “non d’agrifoglio”,n“on con acqua e rane”, “E arrivandonon avrebbe…”; S159: “non è
perché mi sia rimasto da dirti qualcosa d’indispensabilen”,é“è la nostra intimità interrotta al
momento della partenza che sono impaziente di ristabilire”; S190: “Tutto questo avvinenoen
sul mare,non nel sole, – pensa il nuotatore Palomar”, “il snoolen tramonta, il marenon ha quel
colore, le forme sono quelle che la luce proietta nella retina”; S1N01o:n“(li metteva sempre
[…] lontano)”, “non c’era dubbio […] diverso”, “Manon per i cambiamenti”, “ma un margine
di dubbio chenon fosse colui che credeva restava sempre”, “ma se veniva in qua adessno,on
poteva esser lei che aveva fatto tutto il giro”; S142: “Andando avanti vedremo cnhoen c’è
niente di fatto apposta, che nessuno ha messo lì niente”, “quindnion si sa quanti io ci siano a
monte dell’io che credo di essere io, e quante Priscilla a monte della Priscilla verso la quale io
sto credendo di stare correndo”</p>
      </sec>
      <sec id="sec-9-2">
        <title>B.2. Questions</title>
        <p>S159: “Sarà stato già tradotto in comandi ai selettori […] centrali di tran?s”i;toS190: “«E
quello il solo dato non illusorio, comune a tutti, il?b»usiiodomanda il signor Palomar”, “dove
椀昀nirebbe la spada ?”, “Un principio unico e assoluto da cui prendono origine gli atti e le fo?r”;me
S101: “Possibile che non l’avesse riconosciu?to”, “Ma com’era possibile scambiare Isa Maria per
Gigina?”
S007: “o l’avrebbe sentito a un rotolar di pietre che si metteva al suo 昀椀anco, a camminare
assieme a lui in silenzio.”; S159:o“ppure rivelarsi inaspettatamente attiva senz’aver dato prima
alcun segno di vita”, o“ quale altra città sia la tua”; S190: “depressivooautolesionista”; S101: o“
sono tutto”, “o non possono essere più niente”, o“ li si segue giorno per giorno”o, p“pure non
si riesce più a entrarci”, “se era stato visto no”; S142: “alla specieo all’ambienteo a noi due”,
“o già vicini una volta per tutte”, “fusioneo[mescolanza]o scambio”, “l’addensarsi in sciami
di cellule-semoi il concentrato maturare di cellule-uova”,
“’impercettibilmeontsemisuratamente,”, “o è avvenuto o avverrà”</p>
      </sec>
      <sec id="sec-9-3">
        <title>B.4. Sentences without clear dubitative markers</title>
        <p>S007: “Tanti lumi diversi, potevano essere, in marcia per tutti i sentieri di Tumena bassa”, “era
il 昀椀schio convenuto dei tedeschi che serravano intorno a lui, ecco che un altro 昀椀schio gli
rispondeva, era circondato!”; S190: ”«È un omaggio speciale che il sole fa a me personalmente», è
tentato di pensare il signor Palomar”, “«Tutti quelli che hanno occhi vedono il ri昀氀esso che li segue;
l’illusione dei sensi e della mente ci tiene sempre tutti prigionieri»”, “«E quello il solo dato non
illusorio, comune a tutti, il buio?» si domanda il signor Palomar”, “Tutto questo avviene non
sul mare, non nel sole, – pensa il nuotatore Palomar”; S142: “il nostro patrimonio genetico, tra
virgolette”, “E dicendo forma intendo tanto quella che si vede quanto quella che non si vede”,
“il rapporto tra i soli elementi di昀erenziali, perché quelli comuni si possono trascurare da una
parte e dall’altra”, “e allora bisogna vedere se si tratta di quelli comuni”, “la somma dei
caratteri dominanti del passato, il risultato d’una serie d’operazioni che davano sempre un numero
maggiore di zero”, “Tutto quel che possiamo dire è che in certi punti e momenti quell’intervallo
di vuoto che è la nostra presenza individuale viene s昀椀orata dall’onda che continua a rinnovare
le combinazioni di molecole”</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>W.</given-names>
            <surname>Amamou</surname>
          </string-name>
          .
          <article-title>How to Fine-Tune BERT Transformer with spaCy 3. A step-by-step guide on how to 昀椀ne-tune BERT for NER</article-title>
          .
          <year>2021</year>
          . url: https://towardsdatascience.com/how-to
          <article-title>-finetune-bert-transformer-with-spacy-3-6a90bfe5764.</article-title>
          <volume>7</volume>
          [2]
          <string-name>
            <given-names>L.</given-names>
            <surname>Aroyo</surname>
          </string-name>
          and
          <string-name>
            <given-names>C.</given-names>
            <surname>Welty</surname>
          </string-name>
          . “
          <article-title>Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation”</article-title>
          .
          <source>In: AI Magazine 36.1</source>
          (
          <issue>2015</issue>
          ), pp.
          <fpage>15</fpage>
          -
          <lpage>24</lpage>
          . doi:
          <volume>10</volume>
          .1609/aimag.v36i1.2564. url: https://ojs.aaai.org/index.php/aimagazine/article/view/256.4 [5] [20] [23] [24]
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>I.</given-names>
            <surname>Calvino</surname>
          </string-name>
          .Adam, One A昀琀ernoon . London: Vintage books,
          <year>2000</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>I.</given-names>
            <surname>Calvino</surname>
          </string-name>
          .Mr. Palomar. London: Vintage books,
          <year>1999</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>I.</given-names>
            <surname>Calvino</surname>
          </string-name>
          .
          <article-title>Romanzi e racconti</article-title>
          .
          <source>Vol. I. Milano: Mondadori</source>
          ,
          <year>1991</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>I.</given-names>
            <surname>Calvino</surname>
          </string-name>
          .
          <article-title>Romanzi e racconti</article-title>
          . Vol. Ii. Milano: Mondadori,
          <year>1992</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>I.</given-names>
            <surname>Calvino</surname>
          </string-name>
          .
          <article-title>Romanzi e racconti</article-title>
          . Vol. Iii. Milano: Mondadori,
          <year>1994</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>I. Calvino.Saggi. Vol.</given-names>
            <surname>Ii</surname>
          </string-name>
          . Milano: Mondadori,
          <year>1995</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>I.</given-names>
            <surname>Calvino</surname>
          </string-name>
          .
          <source>The Castle of Crossed Destinies. London: Vintage books</source>
          ,
          <year>1998</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>J.</given-names>
            <surname>Cohen</surname>
          </string-name>
          .
          <article-title>“A coe昀케cient of agreement for nominal scales”</article-title>
          .
          <source>In:Educational and Psychological Measurement 20.1</source>
          (
          <issue>1960</issue>
          ), pp.
          <fpage>37</fpage>
          -
          <lpage>46</lpage>
          . doi:
          <volume>10</volume>
          .1177/001316446002000104.
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>N. Z.</given-names>
            <surname>Da</surname>
          </string-name>
          . “
          <article-title>The Computational Case against Computational Literary Studies”</article-title>
          .
          <source>ICnr:itical Inquiry 45.3</source>
          (
          <issue>2019</issue>
          ), pp.
          <fpage>601</fpage>
          -
          <lpage>639</lpage>
          . doi:
          <volume>10</volume>
          .1086/702594.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>J.</given-names>
            <surname>Devlin</surname>
          </string-name>
          , M.-
          <string-name>
            <given-names>W.</given-names>
            <surname>Chang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Lee</surname>
          </string-name>
          , and
          <string-name>
            <surname>K.</surname>
          </string-name>
          <article-title>ToutanovaB</article-title>
          .ERT:
          <article-title>Pre-training of Deep Bidirectional Transformers for Language Understanding</article-title>
          .
          <year>2019</year>
          . doi:
          <volume>10</volume>
          .48550/arxiv.
          <year>1810</year>
          .
          <volume>04805</volume>
          . url: ht tps://arxiv.org/abs/
          <year>1810</year>
          .0480 5.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>J.</given-names>
            <surname>Dobson</surname>
          </string-name>
          . “Interpretable Outputs:
          <article-title>Criteria for Machine Learning in the Humanities”</article-title>
          .
          <source>In: Dhq 15.2</source>
          (
          <year>2021</year>
          ). url: http://www.digitalhumanities.org/dhq/vol/15/2/000555/000555.ht ml.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>S.</given-names>
            <surname>Ercolino.Il</surname>
          </string-name>
          romanzo-saggio.
          <source>Milano: Bompiani</source>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>S. C. Federico</given-names>
            <surname>Bertoni</surname>
          </string-name>
          and
          <string-name>
            <given-names>N.</given-names>
            <surname>Rubbi</surname>
          </string-name>
          .
          <article-title>“I con昀椀ni del saggio</article-title>
          .
          <article-title>Per un bilancio sui destini della forma saggistica”</article-title>
          .
          <source>In:Ticontre. Teoria Testo Traduzione</source>
          <volume>9</volume>
          (
          <year>2018</year>
          ), pp.
          <source>Vii-xii.</source>
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          <source>In: Proceedings of the Third Italian Conference on Computational Linguistics</source>
          (CLiC-it
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          2016. url: http://ceur-ws.
          <source>org/</source>
          Vol-
          <volume>1749</volume>
          /paper24.pd.f
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>C.</given-names>
            <surname>Fenoglio</surname>
          </string-name>
          . “
          <article-title>Euforia di un genere: il saggismo novecentesco”</article-title>
          .
          <source>IMn:ethodologica 19.2</source>
          (
          <year>2021</year>
          ). doi:
          <volume>10</volume>
          .6092/issn.1721-
          <issue>4777</issue>
          /10715.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [22]
          <string-name>
            <surname>G. Gagliardi.</surname>
          </string-name>
          “
          <article-title>Inter-Annotator Agreement in linguistica: una rassegna critica”</article-title>
          .PIrno:
          <article-title>- ceedings of the Fi昀琀h Italian Conference on Computational Linguistics (CLiC-it</article-title>
          <year>2018</year>
          ) .
          <year>2018</year>
          , pp.
          <fpage>206</fpage>
          -
          <lpage>212</lpage>
          . url: http://ceur-ws.
          <source>org/</source>
          Vol-
          <volume>2253</volume>
          /paper28.pd.f
          <string-name>
            <given-names>M. D.</given-names>
            <surname>Gesù</surname>
          </string-name>
          . “
          <article-title>Conversazione con Alfondo Berardinelli”</article-title>
          .
          <source>Ilns:aggio critico. Spunti</source>
          , proposte, riletture.
          <source>Palermo: Duepunti</source>
          ,
          <year>2007</year>
          , pp.
          <fpage>20</fpage>
          -
          <lpage>21</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <string-name>
            <given-names>M.</given-names>
            <surname>Graziano</surname>
          </string-name>
          .
          <article-title>Oltre il romanzo</article-title>
          . Racconto e pensiero in Musil e Svevo. Roma: Carocci,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          <string-name>
            <given-names>M.</given-names>
            <surname>Barenghi</surname>
          </string-name>
          .Calvino. Bologna: il Mulino,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          <string-name>
            <surname>M. BelpolitiL</surname>
          </string-name>
          .'occhio di Calvino.
          <source>Torino: Einaudi</source>
          ,
          <year>1997</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <source>In: Cahiers de Narratologie</source>
          <volume>35</volume>
          (
          <year>2019</year>
          ).
          <source>doi: 10.4000/narratologie.928.6</source>
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>B. J.</given-names>
            <surname>Herrmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Dalen-Oskam</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Schöch</surname>
          </string-name>
          . “
          <article-title>Revisiting Style, a Key Concept in Literary Studies”</article-title>
          .
          <source>In:Journal of Literary Theory</source>
          <volume>9</volume>
          (
          <year>2015</year>
          ). doi:
          <volume>10</volume>
          .1515/jlt-2015-0003.
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>A.</given-names>
            <surname>Kamsetty</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Fricke</surname>
          </string-name>
          , and
          <string-name>
            <given-names>R.</given-names>
            <surname>Liaw</surname>
          </string-name>
          .
          <article-title>Hyperparameter Optimization for Transformers: A guide</article-title>
          .
          <year>2020</year>
          . url: https://medium.com
          <article-title>/distributed-computing-with-ray/hyperparamete r-optimization-for-transformers-a-guide-c4e32c6c98</article-title>
          .
          <fpage>9b</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>K.</given-names>
            <surname>Krippendor昀. Content Analysis</surname>
          </string-name>
          :
          <article-title>An Introduction to Its Methodology</article-title>
          . 2nd. California: Sage Publication,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>K.</given-names>
            <surname>Krippendor昀</surname>
          </string-name>
          . “
          <article-title>Reliability in Content Analysis: Some Common Misconceptions and Recommendations”</article-title>
          .
          <source>In: Human Communication Research 30.3</source>
          (
          <issue>2004</issue>
          ), pp.
          <fpage>411</fpage>
          -
          <lpage>433</lpage>
          . doi:
          <volume>10</volume>
          .1111/j.1468-
          <fpage>2958</fpage>
          .
          <year>2004</year>
          .tb00738.x.
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>H.</given-names>
            <surname>Love</surname>
          </string-name>
          . “
          <article-title>Close but Not Deep: Literary Ethics and the Descriptive Turn”</article-title>
          .
          <source>INne:w Literary History 41.2</source>
          (
          <issue>2010</issue>
          ), pp.
          <fpage>371</fpage>
          -
          <lpage>391</lpage>
          . url: https://www.jstor.org/stable/4098382.7
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [30]
          <string-name>
            <given-names>T. S. e. A. M. Manfred</given-names>
            <surname>Stede</surname>
          </string-name>
          . “
          <string-name>
            <surname>Connective-Lex</surname>
          </string-name>
          :
          <article-title>A Web-Based Multilingual Lexical Resource for Connectives”</article-title>
          .
          <source>InD:iscours 46</source>
          (
          <year>2019</year>
          ). doi:
          <volume>10</volume>
          .4000/discours.10098.
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>L.</given-names>
            <surname>Marchese</surname>
          </string-name>
          . “
          <article-title>È ancora possibile il romanzo-saggio?” ITni:contre</article-title>
          .
          <source>Teoria Testo Traduzione</source>
          <volume>9</volume>
          (
          <year>2018</year>
          ), pp.
          <fpage>151</fpage>
          -
          <lpage>170</lpage>
          . doi:
          <volume>10</volume>
          .1007/s10503-020-09523-1.
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          [32]
          <string-name>
            <surname>M. L. McHugh.</surname>
          </string-name>
          “
          <article-title>Interrater reliability: the kappa statistic”</article-title>
          .
          <source>IBni:ochem Med 22.3</source>
          (
          <issue>2012</issue>
          ), pp.
          <fpage>276</fpage>
          -
          <lpage>282</lpage>
          . url: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900052. /
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [33]
          <string-name>
            <given-names>P. V.</given-names>
            <surname>Mengaldo</surname>
          </string-name>
          . “
          <article-title>Aspetti della lingua di Calvino”</article-title>
          .
          <source>ILna: tradizione del Novecento</source>
          .
          <source>Terza Serie. Torino: Einaudi</source>
          ,
          <year>1991</year>
          , pp.
          <fpage>227</fpage>
          -
          <lpage>291</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          [34]
          <string-name>
            <given-names>F.</given-names>
            <surname>Moretti</surname>
          </string-name>
          .
          <article-title>Falso movimento. La svolta quantitativa nello studio della letteratura</article-title>
          .
          <source>Milano: Nottetempo</source>
          ,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          [35]
          <string-name>
            <given-names>J.</given-names>
            <surname>Morris</surname>
          </string-name>
          .
          <article-title>Does Model Size Matter? A Comparaison of BERT and DistilBERT</article-title>
          .
          <year>2020</year>
          . url: https://wandb.ai
          <article-title>/jack-morris/david-vs-goliath/reports/Does-Model-Size-Matter-A-Co mparison-of-BERT-and-</article-title>
          <string-name>
            <surname>DistilBERT--VmlldzoxMDUxNz</surname>
          </string-name>
          .U
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          [36]
          <string-name>
            <given-names>T.</given-names>
            <surname>Nguyen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Nguyen</surname>
          </string-name>
          , and
          <string-name>
            <given-names>P.</given-names>
            <surname>Rao</surname>
          </string-name>
          . “
          <article-title>Adaptive Name Entity Recognition under Highly Unbalanced Data”</article-title>
          . In:CoRR abs/
          <year>2003</year>
          .10296 (
          <year>2020</year>
          ). arXiv:
          <year>2003</year>
          .10296. url: https://arxiv .org/abs/
          <year>2003</year>
          .10296.
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          <string-name>
            <given-names>D.</given-names>
            <surname>Nozza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Bianchi</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.</given-names>
            <surname>Hovy</surname>
          </string-name>
          . “
          <article-title>What the [MASK]? Making Sense of LanguageSpeci昀椀c BERT Models”</article-title>
          . In: CoRR abs/
          <year>2003</year>
          .02912 (
          <year>2020</year>
          ). arXiv:
          <year>2003</year>
          . 02912. url: https : //arxiv.org/abs/
          <year>2003</year>
          .0291 2.
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          [37] [39] [38]
          <string-name>
            <surname>C. de Obaldia</surname>
          </string-name>
          .
          <article-title>The Essaystic Spirit: literature, modern Criticism and the Essay</article-title>
          . Cotswolds: Clarendon Press,
          <year>1995</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          <string-name>
            <given-names>N.</given-names>
            <surname>Palmieri</surname>
          </string-name>
          . “
          <article-title>Il segno cancellato. Correzione e commento nella scrittura di Calvino”</article-title>
          . In: Studi sulla modernità (
          <year>1993</year>
          ), pp.
          <fpage>289</fpage>
          -
          <lpage>314</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref38">
        <mixed-citation>
          [40]
          <string-name>
            <given-names>T.</given-names>
            <surname>Pavel</surname>
          </string-name>
          .
          <source>The Lives of the Novel: A History</source>
          . Princeton: Princeton University Press,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref39">
        <mixed-citation>
          [41]
          <string-name>
            <given-names>C.</given-names>
            <surname>Perelman</surname>
          </string-name>
          and
          <string-name>
            <surname>L.</surname>
          </string-name>
          <article-title>Olbrechts-TytecaT.rattato dell'argomentazione. La nuova retorica</article-title>
          .
          <source>Torino: Einaudi</source>
          ,
          <year>1966</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref40">
        <mixed-citation>
          [42]
          <string-name>
            <given-names>P.</given-names>
            <surname>Pietrandrea</surname>
          </string-name>
          .
          <article-title>Epistemic modality. Functional properties and the italian system</article-title>
          . Amsterdam - Philadelphia: John Benjamins Publishing Company,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref41">
        <mixed-citation>
          [43]
          <string-name>
            <given-names>C.</given-names>
            <surname>Plantin</surname>
          </string-name>
          . “
          <article-title>Un lieu pour les 昀椀gures dans la théorie de l'argumentation”</article-title>
          .
          <source>AInrg:umentation et Analyse du Discours</source>
          (
          <year>2009</year>
          ). doi:
          <volume>10</volume>
          .4000/aad.215.
        </mixed-citation>
      </ref>
      <ref id="ref42">
        <mixed-citation>
          [44]
          <string-name>
            <given-names>L. V.</given-names>
            <surname>Poppel</surname>
          </string-name>
          . “
          <article-title>The Study of Metaphor in Argumentation Theory”</article-title>
          .
          <source>IAn:rgumentation 35</source>
          (
          <year>2021</year>
          ), pp.
          <fpage>177</fpage>
          -
          <lpage>208</lpage>
          . doi:
          <volume>10</volume>
          .1007/s10503-020-09523-1.
        </mixed-citation>
      </ref>
      <ref id="ref43">
        <mixed-citation>
          [45]
          <string-name>
            <given-names>A.</given-names>
            <surname>Roque</surname>
          </string-name>
          . “
          <article-title>Towards a computational approach to literary text analysis”</article-title>
          .
          <source>PIrno:ceedings of the NAACL-HLT 2012 Workshop on Computational Linguistics for Literature. Montréal</source>
          , Canada: Association for Computational Linguistics,
          <year>2012</year>
          , pp.
          <fpage>97</fpage>
          -
          <lpage>104</lpage>
          . urhl:ttps://aclan thology.
          <source>org/W12-251</source>
          .
          <fpage>4</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref44">
        <mixed-citation>
          [46]
          <string-name>
            <given-names>R.</given-names>
            <surname>Ruiz-Dolz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Alemany</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. M. B.</given-names>
            <surname>Heras</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Gará</surname>
          </string-name>
          c-ıFornes. “
          <article-title>Transformer-Based Models for Automatic Identi昀椀cation of Argument Relations: A Cross-Domain Evaluation”</article-title>
          .
          <source>In: IEEE Intelligent Systems 36.6</source>
          (
          <issue>2021</issue>
          ), pp.
          <fpage>62</fpage>
          -
          <lpage>70</lpage>
          . doi:
          <volume>10</volume>
          .1109/mis.
          <year>2021</year>
          .
          <volume>3073993</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref45">
        <mixed-citation>
          [47]
          <string-name>
            <given-names>F.</given-names>
            <surname>Serra</surname>
          </string-name>
          .Calvino. Roma: Salerno Editrice,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref46">
        <mixed-citation>
          [48]
          <string-name>
            <given-names>F.</given-names>
            <surname>Serra</surname>
          </string-name>
          . “
          <article-title>La notte del morto nel paese nemico”</article-title>
          .
          <source>IPna:ragone 61</source>
          .
          <fpage>90</fpage>
          -91-
          <lpage>92</lpage>
          (
          <year>2010</year>
          ), pp.
          <fpage>125</fpage>
          -
          <lpage>134</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref47">
        <mixed-citation>
          [49]
          <string-name>
            <given-names>F.</given-names>
            <surname>Serra</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Cavalloro</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Giustetto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Parigini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Mauri</surname>
          </string-name>
          , T. Elli, Á. Briones, and
          <string-name>
            <given-names>B.</given-names>
            <surname>Gobbo</surname>
          </string-name>
          .
          <source>Corpus Atlante Calvino. Yareta</source>
          ,
          <year>2022</year>
          . doi:
          <volume>10</volume>
          .26037/yareta:4w2fzwds2jfj5atzzlhjcv4ntm.
        </mixed-citation>
      </ref>
      <ref id="ref48">
        <mixed-citation>
          [50]
          <string-name>
            <given-names>M.</given-names>
            <surname>Sugiyama</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Suzuki</surname>
          </string-name>
          , and T.
          <source>KanamoriD.ensity Ratio Estimation in Machine Learning</source>
          . Cambridge University Press,
          <year>2012</year>
          . doi1:
          <fpage>0</fpage>
          .1017/cbo9781139035613.
        </mixed-citation>
      </ref>
      <ref id="ref49">
        <mixed-citation>
          [51]
          <string-name>
            <given-names>S.</given-names>
            <surname>Vaswani</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Shazeer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Parmar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Uszkoreit</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Jones</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. N.</given-names>
            <surname>Gomez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Kaiser</surname>
          </string-name>
          ,
          <string-name>
            <surname>and I. Polosukhin.</surname>
          </string-name>
          “
          <article-title>Attention Is All You Need”</article-title>
          .
          <source>InC:oRR abs/1706</source>
          .03762 (
          <year>2017</year>
          ). arXiv:
          <volume>1706</volume>
          .
          <fpage>03</fpage>
          <lpage>762</lpage>
          . url: http://arxiv.org/abs/1706.03762.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>