<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>Corresponding author.
reiter-haas@tugraz.at (M. Reiter-Haas); beate.kloesch@uni-graz.at (B. Klösch); markus.hadler@uni-graz.at
(M. Hadler); elisabeth.lex@tugraz.at (E. Lex)
https://iseratho.github.io/ (M. Reiter-Haas); https://elisabethlex.info/ (E. Lex)</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Computational Narrative Framing: Towards Identifying Frames through Contrasting the Evolution of Narrations</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Markus Reiter-Haas</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Beate Klösch</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Markus Hadler</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Elisabeth Lex</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Graz University of Technology, Institute of Interactive Systems and Data Science</institution>
          ,
          <addr-line>8010 Graz, Sandgasse 36/III</addr-line>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>University of Graz, Department of Sociology</institution>
          ,
          <addr-line>8010 Graz, Universitätsstraße 15/G4</addr-line>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2024</year>
      </pub-date>
      <volume>000</volume>
      <fpage>0</fpage>
      <lpage>0001</lpage>
      <abstract>
        <p>Our understanding of the world is fundamentally shaped by language, with narrations being a central point, and in uenced by its framing. Recent advancements in language models gave rise to computational methods for both narrative understanding and framing analysis. Although given their overlap, these two strands are mostly researched independently. In this position paper, we argue for their consolidation in the form of narrative framing, i.e., the framing process driven by narrations. Herein, we outline similarities between both based on semantic elements. Besides, we discuss how di erent narratives might compete with each other, as well as evolve over time. Thereby, narratives inevitably change the framing, exemplarily depicted on the issue of climate change. We believe that the analysis of narrative frames will lead to a broader understanding of textual corpora as a whole rather than individual pieces of text.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Framing Theory</kwd>
        <kwd>Narrative Frames</kwd>
        <kwd>Competing Narrations</kwd>
        <kwd>Climate Change Framing</kwd>
        <kwd>Semantic Graphs</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Experiences in the real-world and narrative perception are inextricably linked in humans,
even on a neurological level [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. In a similar vein, the framing of narratives can act as a
device to blend ction and reality [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], consequently suggesting certain solutions to speci c
problems [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] and a ect the people’s choices [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. Unlike other types of frames, the pool of
options concerning narratives for framing is essentially endless. Although some works on
computationally extracting narrative framing have already been conducted [e.g., 5, 6, 7], the still
sparse body of research tends to favor one strand of research, i.e., either narrative understanding
or framing analysis, over the other.
      </p>
      <p>
        In this position paper, we present a basic theoretical framework for computational narrative
framing analysis, e ectively combining computational narrative understanding and
computational framing analysis research. We identify the commonalities between the two strands to
form an elementary understanding of the necessities for emerging approaches in this direction.
Moreover, we explore how such a framework enables contrasting the evolution of di erent
lines of narrative frames across important issues. Herein, we exemplarily discuss the narrative
change regarding climate change, i.e., the evolution from global warming to the more urgent
naming of climate catastrophe and similar [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
      </p>
      <p>As our main contribution, we want to provide an impulse towards further exploration of how
narrations are being used to frame long-term discourses. We hope that our work bridges the
gap between two similar but still distinct communities.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Background</title>
      <p>The present work comprises two strands of computational research, based on narrations and
frames, respectively. Speci cally, we focus on the parts where computational narrative
understanding and computational framing analysis mostly overlap.</p>
      <p>
        Computational Narrative Understanding (CNU) Narrations, being de ned by their
content and structure, are used to study many topics, with the policy process in the narrative policy
framework being a well-known example [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. Herein, the elements of narrativity have rst been
fully formalized by Piper et al. [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], with the minimal de nition being structured as"Someone
tells someone somewhere that || someone did something(s) [to someone] somewhere at some time
for some reason". Here, the left part (before the ||) is the perspective of narrating the story, while
the right part concerns the story itself (i.e., diegesis). In a similar vein, some strides have already
been made towards analyzing narrative frames [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. Overall, we observe that actors and events
are central components of narrations, which provides overlap with some computational framing
analysis approaches.
      </p>
      <p>
        Computational Framing Analysis (CFA) Framing deals with salience in communication [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]
and is concerned “how” a text is presented rather than “what” is apparent [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. The analysis of
framing can be seen as a task of natural language understanding (e.g., similar to tasks in the
GLUE benchmark [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ]). The notion of framing is very distinctively conceptualized in
computational literature, comprising supervised and unsupervised, as well as mixed-method based
approaches [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. As supervised approaches depend on corpora and codebooks, unsupervised
approaches are more in line with narrative understanding. For instance, DiMaggio et al. [13]
use topic modeling for framing analysis and equate certain topics with frames. Besides, they
de ne frames as comprising narratives among other cues, and also nd narratives as being
part of a particular topic. Other works consider semantic information, such as semantic role
labels [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] and semantic graphs [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] to analyze narratives directly.
      </p>
      <p>
        In the remainder of the paper, we use the theory presented by Piper et al. [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] for CNU and
the survey by Ali and Hassan [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] for CFA as cornerstones in their respective areas. Also,
when using the term narrative framing, we refer to the framing using narratives as device, thus
compounding both CNU and CFA. Herein, we focus on framing through semantic structure
(i.e., following Fillmore and Baker [14]) rather than other forms of framing.
      </p>
    </sec>
    <sec id="sec-3">
      <title>3. Computational Narrative Framing</title>
      <p>
        As a starting point for better understanding narrative framing, we analyze how their research
directions are entangled. Both, Piper et al. [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] and Ali and Hassan [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], identify a set of future
research endeavors by stating core challenges and open questions, respectively. We provide
an overview of these future directions in Table 1. Comparing them, we observe remarkable
overlap between the two strands that we summarize as key requirements.
      </p>
      <p>First (R1), there is the improvement of methods by considering ne-grained nuanced features,
e.g., latent features (CNU) and semantic relations (CFA). Herein, CNU focuses on understanding
deep stories via narrative structuring of higher-order organizing principles, while CFA focuses
on semantic relations going beyond words with the aim to better explore frames. Here, we
identify narrative structure as a key direction for future research.</p>
      <p>Second (R2), the relation between multiple documents (potentially even for distinct types)
for a broader understanding are established. CNU aims to understand narrative discourse by
studying the interaction of narrative features, even between di erent narrative products (e.g.,
movies vs. books). CFA questions how di erent documents can be connected or inform each
other. We reason that the understanding of narratives must go beyond individual narratives
and shift towards a focus on competing narratives.</p>
      <p>Third (R3), both emphasize the incorporation of more nuanced knowledge sources, e.g., past
events like wars (CNU), culture, and omission (CFA). CNU argues for more robust classi cation
of narrative types via interdisciplinary large-scale registers. CFA calls for a computational
model to construct frames via salience through various framing devices. We see the modeling
of the temporal evolution as a good starting point to capture more nuances.</p>
      <p>Based on these suggestions, we reason that computational narrative framing approaches
must go beyond simple feature analysis (e.g., on the word-level) of individual documents,
but rather analyze the corpus as a whole considering the nuances within. Speci cally, we
argue that narrative frames emerge from the temporal evolution of collections of documents
comprising structural elements. In the following, we aim to synthesize these requirements from
the bottom-up.</p>
      <p>Synthesized requirements</p>
      <p>Narrative Structure
Competing Narratives</p>
      <p>Temporal Evolution</p>
      <sec id="sec-3-1">
        <title>3.1. Narrative Structure</title>
        <p>
          To start, we establish narrative frames that go beyond word frequency, with structure being a
focal point. We base the analysis on our prior work [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ] using semantic graphs based on abstract
meaning representations [15].
        </p>
        <p>In Figure 1a, we depict an example that shows how complex such representations can be,
even for short sentences. Speci cally, we used a sentence from a recent news article on the
COP281:</p>
        <p>U.N. Secretary-General Antonio Guterres said on Monday (December 11th, 2023)
one key to success of the COP28 climate summit (in Dubai) was for nations to reach
agreement on the need to "phase out" fossil fuels.</p>
        <p>We transformed the text to a graph using [16]2 and present its linearized form for brevity. While
a detailed explanation is beyond the scope of this paper3, the key elements that such model
extracts are semantic frames (comprising verbs and senses) [14], concepts (nouns), contextual
information (time and location), as well as named entities. We want to highlight that the
model implicitly performs both simpli cations (e.g., singularization of nations to nation) and
generalizations (e.g., wiki cation of U.N. to United Nations), potentially in unison (e.g., stemming
and verbi cation of agreement to agree-01), to improve the resulting representations.</p>
        <p>Therefore, this or similar representations are necessary to ful ll the rst requirement for
computational narrative framing (R1). Note that, we used a straightforward parser here for
demonstration, but more recent language models, e.g., BART [17], might be better suited for
the task at hand.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Competing Narratives</title>
        <p>After having extracted the narratives of individual documents, we might compare them. In
most scenarios, narratives will cluster together and compete with each other, with narratives
1Taken from: https://www.reuters.com/business/environment/phasing-out-fossil-fuels-is-key-cop28-success%
2Dsays-uns-guterres-2023-12-11/ where we enhanced the text with meta-data from the article, i.e., time and
location, which we put in parentheses.
2Available as open tool at: https://bollin.inf.ed.ac.uk/amreager.html
3The guidelines are available at: https://github.com/amrisi/amr-guidelines/blob/master/amr.md
of conspiracy theories being an obvious instance. Especially regarding the topic of climate
change, conspiracy thinking seems larger than anticipated [18]. Even for COP28, conspiracy
narratives are spreading, such as relating to the fear of keeping the population captive4. Besides
considering conspiracies, many intra- and inter-corpus dependencies should also be considered,
with polarization [19] being another noteworthy example.</p>
        <p>To identify such competing narratives, we can rely on established methods for corpus analysis
(e.g., [20]). However, beyond applying them on lexical features (e.g., words), considering the
semantic level as established in 3.1 is important for the second requirement (R2).</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. Temporal Evolution</title>
        <p>While the third requirement (R3) contains many distinct points, we focus on the temporal
aspects that we see as the most common factor. Hence, the present should depend on the past,
while also account for irregularities like notable omissions of speci c narratives. Furthermore,
the evolution will depend on the competing narratives established in 3.2. For example, the
overall narrative framing might show a similar trend but at a di erent pace depending on the
cultural context, which we visually illustrate using an arti cial example in Figure1b. Notably,
certain events could lead to sudden shifts in trajectories that need to be accounted for.</p>
        <p>While methods like time-series analyses seems sound at rst glance, we believe that due to
discreteness of narrative frames, sequential modeling approaches [21] are a better t. In such
models, side-information such as relevant events could be utilized as well.</p>
      </sec>
      <sec id="sec-3-4">
        <title>3.4. Challenges in Narrative Framing Analysis</title>
        <p>Foremost, we acknowledge that the main challenges identi ed still remain unsolved. Beyond
that, detecting narrative frames is even more di cult to achieve than both CNU and CFA
individually. While data is sparse in both domains, there is a complete lack of ground truth data
to train algorithms for predicting the narrative framing. Moreover, classical machine learning
setups like classi cation would not work at all, as there is no complete set of narrative frames
due to their emergent properties. Finally, the validation, especially quantitatively, is unsolved
as the evolving nature of narrative frames hinders most (static) measures.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Learning from Evolving Narratives: The Case of Global</title>
    </sec>
    <sec id="sec-5">
      <title>Warming to Climate Catastrophe</title>
      <p>
        Following up on the topic of the example provided in Figure 1, we now brie y discuss how
considering computational narrative framing would support understanding the discourse on
climate change. The framing of climate change has gradually shifted from global warming
to climate change, and more recently towards climate crisis or even climate catastrophe [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
While anecdotally obvious, such patterns are notoriously challenging to detect computationally
when they are not known in advance. Climate change, in particular, is a long-term issue where
changes are noticeable even for laymen. Besides the reframing of the scienti c consensus
4https://phys.org/news/2023-11-climate-conspiracy-theories- ourish-cop28.html
towards increasing urgency, even the framing of climate change denial shifted their narrations
from outright denying climate change to denying human-made climate change. Supporting
such discourse analysis with computational methods would be very bene cial for identifying
narrative patterns for preemptive counteraction, as well as future predictions.
      </p>
    </sec>
    <sec id="sec-6">
      <title>5. Conclusion</title>
      <p>In this paper, we introduced computational narrative framing that combines the research of
computational narrative understanding with computational framing analysis. Herein, we
identied that both of their pressing future research directions overlap, which coincidentally situate
the main requirements for the task at hand. Speci cally, (i) narrative structure, (ii) competing
narratives, and the (iii) temporal evolution are fundamental for a thorough understanding.
We exemplarily support our reasoning concerning the evolution of competing narrations in
climate change discourse. Our hope is that this paper serves as a starting point for mutual
bene t between two distinct research strands that enables a broader understanding of important
societal topics.
and analysis platform for natural language understanding, arXiv preprint arXiv:1804.07461
(2018).
[13] P. DiMaggio, M. Nag, D. Blei, Exploiting a nities between topic modeling and the
sociological perspective on culture: Application to newspaper coverage of us government
arts funding, Poetics 41 (2013) 570–606.
[14] C. J. Fillmore, C. F. Baker, Frame semantics for text understanding, in: Proceedings of</p>
      <p>WordNet and Other Lexical Resources Workshop, NAACL, volume 6, 2001.
[15] L. Banarescu, C. Bonial, S. Cai, M. Georgescu, K. Gri tt, U. Hermjakob, K. Knight, P. Koehn,
M. Palmer, N. Schneider, Abstract meaning representation for sembanking, in: Proceedings
of the 7th linguistic annotation workshop and interoperability with discourse, 2013, pp.
178–186.
[16] M. Damonte, S. B. Cohen, G. Satta, An incremental parser for abstract meaning
representation, in: 15th EACL 2017 Software Demonstrations, Association for Computational
Linguistics (ACL), 2017, pp. 536–546.
[17] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L.
Zettlemoyer, Bart: Denoising sequence-to-sequence pre-training for natural language generation,
translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019).
[18] J. E. Uscinski, S. Olivella, The conditional e ect of conspiracy thinking on attitudes toward
climate change, Research &amp; Politics 4 (2017) 2053168017743105.
[19] P. DiMaggio, J. Evans, B. Bryson, Have american’s social attitudes become more polarized?,</p>
      <p>American journal of Sociology 102 (1996) 690–755.
[20] B. L. Monroe, M. P. Colaresi, K. M. Quinn, Fightin’words: Lexical feature selection and
evaluation for identifying the content of political con ict, Political Analysis 16 (2008)
372–403.
[21] I. Sutskever, O. Vinyals, Q. V. Le, Sequence to sequence learning with neural networks,
Advances in neural information processing systems 27 (2014).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>C.</given-names>
            <surname>Baldassano</surname>
          </string-name>
          ,
          <string-name>
            <given-names>U.</given-names>
            <surname>Hasson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K. A.</given-names>
            <surname>Norman</surname>
          </string-name>
          ,
          <article-title>Representation of real-world event schemas during narrative perception</article-title>
          ,
          <source>Journal of Neuroscience</source>
          <volume>38</volume>
          (
          <year>2018</year>
          )
          <fpage>9689</fpage>
          -
          <lpage>9699</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>N.</given-names>
            <surname>Igl</surname>
          </string-name>
          , Framing the narrative (
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>R. M.</given-names>
            <surname>Entman</surname>
          </string-name>
          , Framing:
          <article-title>Toward clari cation of a fractured paradigm</article-title>
          ,
          <source>Journal of communication 43</source>
          (
          <year>1993</year>
          )
          <fpage>51</fpage>
          -
          <lpage>58</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>A.</given-names>
            <surname>Tversky</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kahneman</surname>
          </string-name>
          ,
          <article-title>The framing of decisions and the psychology of choice</article-title>
          , science
          <volume>211</volume>
          (
          <year>1981</year>
          )
          <fpage>453</fpage>
          -
          <lpage>458</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>E.</given-names>
            <surname>Portelance</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Piper</surname>
          </string-name>
          ,
          <article-title>Understanding narrative: Computational approaches to detecting narrative frames</article-title>
          .,
          <source>in: DH</source>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>E.</given-names>
            <surname>Jing</surname>
          </string-name>
          , Y.-Y. Ahn,
          <article-title>Characterizing partisan political narrative frameworks about covid-19 on twitter</article-title>
          ,
          <source>EPJ data science 10</source>
          (
          <year>2021</year>
          )
          <fpage>53</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>M.</given-names>
            <surname>Reiter-Haas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Klösch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Hadler</surname>
          </string-name>
          , E. Lex,
          <article-title>Framing analysis of health-related narratives: Conspiracy versus mainstream media</article-title>
          ,
          <source>arXiv preprint arXiv:2401.10030</source>
          (
          <year>2024</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>M.</given-names>
            <surname>Schäfer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Hase</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Mahl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Krayss</surname>
          </string-name>
          , From “climate change” to “climate crisis”?,
          <string-name>
            <given-names>Bergen</given-names>
            <surname>Language</surname>
          </string-name>
          and
          <string-name>
            <surname>Linguistics Studies</surname>
          </string-name>
          (
          <year>2023</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>M. D.</given-names>
            <surname>Jones</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Smith-Walter</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. K.</given-names>
            <surname>McBeth</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. A.</given-names>
            <surname>Shanahan</surname>
          </string-name>
          ,
          <article-title>The narrative policy framework, in: Theories of the policy process</article-title>
          ,
          <source>Routledge</source>
          ,
          <year>2023</year>
          , pp.
          <fpage>161</fpage>
          -
          <lpage>195</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>A.</given-names>
            <surname>Piper</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. J.</given-names>
            <surname>So</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Bamman</surname>
          </string-name>
          ,
          <article-title>Narrative theory for computational narrative understanding</article-title>
          ,
          <source>in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</source>
          ,
          <year>2021</year>
          , pp.
          <fpage>298</fpage>
          -
          <lpage>311</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>M.</given-names>
            <surname>Ali</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Hassan</surname>
          </string-name>
          ,
          <article-title>A survey of computational framing analysis approaches</article-title>
          ,
          <source>in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>9335</fpage>
          -
          <lpage>9348</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>A.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Singh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Michael</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Hill</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Levy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. R.</given-names>
            <surname>Bowman</surname>
          </string-name>
          ,
          <article-title>Glue: A multi-task benchmark</article-title>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>