=Paper=
{{Paper
|id=Vol-3117/paper12
|storemode=property
|title=Text2Icons: linking icons to narrative participants (position paper)
|pdfUrl=https://ceur-ws.org/Vol-3117/paper12.pdf
|volume=Vol-3117
|authors=Joana Valente,Alípio Jorge,Sérgio Nunes
|dblpUrl=https://dblp.org/rec/conf/ecir/ValenteJN22
}}
==Text2Icons: linking icons to narrative participants (position paper)==
https://ceur-ws.org/Vol-3117/paper12.pdf
Text2Icons: linking icons to narrative participants
(position paper)
Joana Valente1,3 , Alípio Jorge1,3 and Sérgio Nunes2,3
1
FCUP - Faculty of Sciences of the University of Porto, Porto, Portugal
2
FEUP - Faculty of Engineering of the University of Porto, Porto, Portugal
3
INESC TEC, Porto, Portugal
Abstract
Narratives are used to convey information and are an important way of understanding the world through
information sharing. With the increasing development in Natural Language Processing and Artificial
Intelligence, it becomes relevant to explore new techniques to extract, process, and visualize narratives.
Narrative visualization tools enable a news story reader to have a different perspective from the traditional
format, allowing it to be presented in a schematic way, using representative symbols to summarize
it. We propose a new narrative visualization approach using icons to represent important narrative
elements. The proposed visualization is integrated in Brat2Viz, a narrative annotation visualization
tool that implements a pipeline that transforms text annotations into formal representations leading
to narrative visualizations. To build the icon visualization, we present a narrative element extraction
process that uses automatic sentence extraction, automatic translation methods, and an algorithm that
determines the actors’ most adequate descriptions. Then, we introduce a method to create an icon
dictionary, with the ability to automatically search for icons. Furthermore, we present a critical analysis
and user-based evaluation of the results resorting to the responses collected in two separate surveys.
Keywords
Narrative Visualization, Narrative Extraction, Icons, Natural Language Processing, Embeddings
1. Introduction
Narratives represent life events, convey information, and cultural values. Being an important
way to perceive the world through information sharing. Narrative extraction techniques are
being developed to better understand the story behind texts, for example from news articles [1]
and social media [2]. Narrative representation facilitates tasks such as reading a news article [3],
and detecting a patient’s diagnosis [4]. It is important to present narratives in new formats to
reach wider audiences, employing more appealing and expressive means of communication.
In this paper we introduce an automatic process to extract, process and present a story in a
graphical form using nearly universally understandable icons. The process starts with a text from
which the relevant narrative elements are identified and extracted. Next, an internal abstract
representation is generated and finally icons are linked to the story. The intermediate abstract
In: R. Campos, A. Jorge, A. Jatowt, S. Bhatia, M. Litvak (eds.): Proceedings of the Text2Story’22 Workshop, Stavanger
(Norway), 10-April-2022
Envelope-Open up201405224@edu.fc.up.pt (J. Valente); amjorge@fc.up.pt (A. Jorge); ssn@fe.up.pt (S. Nunes)
Orcid 0000-0002-2601-6487 (J. Valente); 0000-0002-5475-1382 (A. Jorge); 0000-0002-2693-988X (S. Nunes)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
111
representation we use to represent the extracted narrative elements is Discourse Representation
Structure (DRS), a formal linguistics framework for exploring meaning [5]. To generate the
final icon-based visualization we automatically link the narrative elements of news stories to
representative icons. For that we use an icon dictionary which is also automatically built and
easily expanded.
With this work, we produced a method to automatically transform a story into a graphical
representation using icons with little to none human intervention. Our contributions include: a
validated algorithm to resolve the most specific description from a set of descriptions of the same
actor/participant; a methodology to build dictionaries, not only for icons, but also for images; a
method to automatically obtain icons; a new icon-based visualization method integrated in the
pipeline, which can be seen in a vertical strip or in a slideshow.
2. Related Work
Knowledge and information visualizations are widely used in areas like education, in order to
help process, access, and handle complex knowledge and vast amounts of information [6]. The
importance and potential of storytelling has been explored for information visualization as an
efficient method of data representation, as shown by Figueiras [7] that addresses the benefits
of incorporating narrative elements in visualizations. Recent research includes: Graphical
Storytelling project1 with the generation of news comics from journalistic text; Campos et
al. [8] Time-Matters (system that scores the relevance of temporal expressions in a text); Pasquali
et al. [9] Conta-me Histórias (tool that automatically generates a temporal summarization of
news collections); and Ramesh et al. [10] text-to-image generation (DALL·E).
3. Narrative Extraction Pipeline
Our pipeline aims to transform text into an icon based representation. Our contribution focuses
on the visualization step. Brat2Viz is a narrative annotation visualization tool [11] proposed to
support the debugging of narrative annotation done with BRAT. It implements a pipeline from
annotated text to visualization, by transforming the annotation into a formal representation,
using DRS, and then, to visual representations. Brat2Viz2 has two modules: Brat2DRS creates a
DRS representation for each news story by parsing a BRAT annotated news text; and DRS2Viz
parses the DRS representation and deploys a web application with the generated visualization.
The main aim of the icon visualization is to obtain a graphic representation for each sentence
in the news story. Thus, automatic sentence extraction from the input text was added to
Brat2DRS. Given that existing icon databases are mostly in English, it was convenient to employ
automatic translation to be able to work with non English texts. For that, two automatic
translation tools were included in DRS2Viz: Hugging Face Transformers3 and Googletrans4 .
1
https://bbcnewslabs.co.uk/projects/graphical-storytelling/
2
https://github.com/LIAAD/brat2viz
3
https://github.com/huggingface/transformers
4
https://pypi.org/project/googletrans/
112
The narrative elements to be represented are currently the actors mentioned in the text. Since
the same actor can have different descriptions throughout the narrative (robber, assailant, man,
suspect), we devised an algorithm to resolve the most specific description of an actor. From a
set of actor descriptions, our method finds the most specific using WordNet5 similarity between
words and the super-subordinate relation (hyperonymy and hyponymy). To evaluate both
the effectiveness of the algorithm and which translation to use, 109 sets of actor descriptions
were analyzed. The resulting data comes from a manual analysis of the results obtained from
processing the actors’ descriptions using the algorithm and comparing them with the expected
descriptions. With 67.89% of well-defined actors, Googletrans was the chosen translation.
4. Linking Narrative Elements to Icons
Now we present the icon dictionary that allows the actors’ connection
to the icons, describe the steps and implementation6 taken to reach
the icon visualization, and present the evaluation of the obtained
results.
4.1. Icon Dictionary
The Icon Dictionary allows the search for icons to use in the visualiza-
tion. The sources integrated are emojidex7 , IconFinder8 , Icons89 ,
and Icons-5010 . The quality and availability of the icons in the dic-
tionary depends on its sources, being possible to add new ones. Icons
are searched by terms, adding a new icon when that term does not
yet have an icon that represents it in the dictionary. When adding (a) Vertical strip.
a new icon, a search is made on the available sources, with one of
the icons being saved. The search (implemented in DRS2Viz) can be
performed: semi-automatically (the user chooses the icon from the
list of results presented by the icon sources) or automatically (using
fastText embeddings to calculate the cosine similarity between the
searched term and each element of the lists returned from the icon (b) Slideshow.
sources, choosing the most similar icon to the searched one).
Figure 1: Visualization
options.
4.2. Visualization Deployment
Two types of icon visualization were created: vertical strip (presents the full news story with
the actors represented with icons), shown in Figure 1a, and slideshow (allows user interaction
by moving to the next and previous sentences) in Figure 1b, where the news text presented
5
https://wordnet.princeton.edu
6
Available at: https://github.com/LIAAD/Text2Icons
7
https://developer.emojidex.com/#api
8
https://developer.iconfinder.com/reference/overview-1
9
https://developers.icons8.com/docs/getting-started
10
https://www.kaggle.com/danhendrycks/icons50
113
can be translated as: “Thieves stole 500 cows from a New Zealand farm without the owner
noticing anything for weeks. The last time he counted the herd, in early July, the man had 1300
heads, but now just over 800 remain. Police are investigating the robbery but have few leads.
“Probably, they weren’t all taken at once” admit the agents, who don’t realize how the farmer
didn’t notice the robbery earlier.”.
4.3. Analysis and Evaluation
To assess if the generated visualizations represent the stories efficiently from the perspective of
a potential user, two surveys were launched: one for the ability of a set of icons to represent a
story scene; and another for the quality of the term-icon connection. We present users’ views
on the use of icons as a complement to news stories. The questions were designed to minimize
the possibility of biasing the answers towards favoring our hypotheses. The answers obtained
from the surveys were collected by a network of direct and indirect personal contacts, and by
the academic community of the University of Porto.
Visualization of News Stories To evaluate the quality of the results, participants rated from
1 to 5 how well a set of icons represented a sentence, with 149 responses collected. The survey
had 10 questions, each with a sentence, and 3 images of a generated set of icons. The average of
the participants scores was calculated, where the image with the highest score, i.e., the most
scored on average by the participants, corresponds to the one generated by the visualization
in 80% of the cases. This indicates that possible news readers or users of the tool identify the
majority of the icon visualizations generated as the most suitable, concluding that the story
representation as a whole is mostly in accordance with what users expect to see.
Term-Icon Connection To assess the quality of the term-icon connection, 291 participants
choose the best suited icon for a term in 20 questions each with: 1 term and 6 icons (one is
the most similar to the term, and the others are plausible representations). Fleiss’ kappa [12]
was calculated to understand if the answers were chosen randomly. 𝜅 = 0.3785551 is classified
as Fair agreement, according to Landis et al. [13], making the data suitable. The hit rate of
each question was calculated, where the average hit rate for each question is 53%, with 85% of
the cases falling into the first and second most voted option. Concluding that the results are
favorable, since the visualization shows icons that users identify as the most suitable.
Participants opinion The opinion of possible news readers and visualization users is crucial
to understand the scope of this work. 291 participants rated two questions from 1 to 5. One
about the usefulness in having news text accompanied by illustrative icons for themselves, and
the other, regarding another (children or people with reading difficulties). 58.4% answered with
4 and 5 (highest level of agreement) in favor of the usefulness for themselves. And with 89.7%
in options 4 and 5, we conclude that the participants’ standpoint on the usefulness for others
reflects the premise here addressed: the importance of narrative visualization.
114
5. Conclusion
In this paper, we described a new narrative visualization approach, integrated in Brat2Viz, by
representing key concepts of a narrative with icons. The process to extract narrative elements is
presented: sentence extraction; automatic translation methods integration; and an actor linking
algorithm to find the most specific description of the news actors. The icon dictionary designed
as a database supports four sources, and two ways to add new icons: semi-automatically or
automatically. These steps combined result in two visualizations: vertical strip and slideshow.
Finally, the results were evaluated from potential users with positive and promising results. It
might be interesting to expand the visualization to automatically generated icons. In addition
to representing the actors, it can be useful to represent the narrative events. The algorithm also
has room for improvement when it comes to being able to handle more complex actors.
Acknowledgments
This work was carried out as part of the project Text2Story, financed by the ERDF - European
Regional Development Fund through the North Portugal Regional Operational Programme
(NORTE 2020), under the PORTUGAL 2020 and by National Funds through the Portuguese
funding agency, FCT - Fundação para a Ciência e a Tecnologia within project PTDC/CCI-
COM/31857/2017 (NORTE-01-0145-FEDER-031857).
References
[1] J. Piskorski, V. Zavarella, M. Atkinson, M. Verile, Timelines: Entity-centric event extraction
from online news, in: R. Campos, A. M. Jorge, A. Jatowt, S. Bhatia (Eds.), Proceedings of
Text2Story - Third Workshop on Narrative Extraction From Texts co-located with 42nd
European Conference on Information Retrieval, Text2Story@ECIR 2020, Lisbon, Portugal,
April 14th, 2020 [online only], volume 2593 of CEUR Workshop Proceedings, CEUR-WS.org,
2020, pp. 105–114. URL: http://ceur-ws.org/Vol-2593/paper13.pdf.
[2] D. Semedo, J. Magalhães, Dynamic-keyword extraction from social media, in: L. Azzopardi,
B. Stein, N. Fuhr, P. Mayr, C. Hauff, D. Hiemstra (Eds.), Advances in Information Retrieval
- 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14-18,
2019, Proceedings, Part I, volume 11437 of Lecture Notes in Computer Science, Springer,
2019, pp. 852–860. URL: https://doi.org/10.1007/978-3-030-15712-8_62.
[3] D. Caswell, K. Dörr, Automated journalism 2.0: Event-driven narratives, Journalism
Practice 12 (2017) 477–496. URL: https://doi.org/10.1080/17512786.2017.1320773.
[4] A. Dudchenko, M. Ganzinger, G. Kopanitsa, Diagnoses detection in short snippets of
narrative medical texts, Procedia Computer Science 156 (2019) 150–157. URL: https:
//doi.org/10.1016/j.procs.2019.08.190.
[5] H. Kamp, U. Reyle, Introduction to Model Theoretic Semantics of Natural Language, Formal
Logic and Discourse Representation Theory, volume 42, Springer Netherlands, 1993.
[6] T. Keller, S. Tergan, Visualizing knowledge and information: An introduction, in: S. Tergan,
T. Keller (Eds.), Knowledge and Information Visualization, Searching for Synergies [out-
115
come of a workshop held in Tübingen, Germany, May 2004], volume 3426 of Lecture Notes
in Computer Science, Springer, 2005, pp. 1–23. URL: https://doi.org/10.1007/11510154_1.
[7] A. Figueiras, Narrative visualization: A case study of how to incorporate narrative elements
in existing visualizations, in: E. Banissi, M. W. M. Bannatyne, F. T. Marchese, M. Sarfraz,
A. Ursyn, G. Venturini, T. G. Wyeld, U. Cvek, M. Trutschl, G. G. Grinstein, V. Geroimenko,
S. Kenderdine, F. Bouali (Eds.), 18th International Conference on Information Visualisation,
IV 2014, Paris, France, July 16-18, 2014, IEEE Computer Society, 2014, pp. 46–52. URL:
https://doi.org/10.1109/IV.2014.79.
[8] R. Campos, J. Duque, T. Cândido, J. Mendes, G. Dias, A. Jorge, C. Nunes, Time-matters:
Temporal unfolding of texts, in: D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast,
F. Sebastiani (Eds.), Advances in Information Retrieval - 43rd European Conference on
IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II,
volume 12657 of Lecture Notes in Computer Science, Springer, 2021, pp. 492–497. URL:
https://doi.org/10.1007/978-3-030-72240-1_53.
[9] A. Pasquali, V. Mangaravite, R. Campos, A. M. Jorge, A. Jatowt, Interactive system for
automatically generating temporal narratives, in: L. Azzopardi, B. Stein, N. Fuhr, P. Mayr,
C. Hauff, D. Hiemstra (Eds.), Advances in Information Retrieval - 41st European Conference
on IR Research, ECIR 2019, Cologne, Germany, April 14-18, 2019, Proceedings, Part II,
volume 11438 of Lecture Notes in Computer Science, Springer, 2019, pp. 251–255. URL:
https://doi.org/10.1007/978-3-030-15719-7_34.
[10] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, I. Sutskever, Zero-
shot text-to-image generation, in: M. Meila, T. Zhang (Eds.), Proceedings of the 38th
International Conference on Machine Learning, volume 139 of Proceedings of Machine
Learning Research, PMLR, 2021, pp. 8821–8831. URL: https://proceedings.mlr.press/v139/
ramesh21a.html.
[11] E. Amorim, A. Ribeiro, B. S. Santana, I. Cantante, A. Jorge, S. Nunes, P. Silvano, A. Leal,
R. Campos, Brat2viz: a tool and pipeline for visualizing narratives from annotated texts,
in: R. Campos, A. M. Jorge, A. Jatowt, S. Bhatia, M. A. Finlayson (Eds.), Proceedings of
Text2Story - Fourth Workshop on Narrative Extraction From Texts held in conjunction
with the 43rd European Conference on Information Retrieval (ECIR 2021), Lucca, Italy,
April 1, 2021 (online event due to Covid-19 outbreak), volume 2860 of CEUR Workshop
Proceedings, CEUR-WS.org, 2021, pp. 49–56. URL: http://ceur-ws.org/Vol-2860/paper6.pdf.
[12] J. L. Fleiss, Measuring nominal scale agreement among many raters, volume 76, Psycho-
logical Bulletin, 1971, p. 378–382. URL: https://doi.org/10.1037/h0031619.
[13] J. R. Landis, G. G. Koch, The measurement of observer agreement for categorical data,
volume 33, Biometrics, 1977, pp. 159–174. URL: https://doi.org/10.2307/2529310.
116