Narratives are used to convey information and are an important way of understanding the world through information sharing. With the increasing development in Natural Language Processing and Artificial Intelligence, it becomes relevant to explore new techniques to extract, process, and visualize narratives. Narrative visualization tools enable a news story reader to have a diferent perspective from the traditional format, allowing it to be presented in a schematic way, using representative symbols to summarize it. We propose a new narrative visualization approach using icons to represent important narrative elements. The proposed visualization is integrated in Brat2Viz, a narrative annotation visualization tool that implements a pipeline that transforms text annotations into formal representations leading to narrative visualizations. To build the icon visualization, we present a narrative element extraction process that uses automatic sentence extraction, automatic translation methods, and an algorithm that determines the actors' most adequate descriptions. Then, we introduce a method to create an icon dictionary, with the ability to automatically search for icons. Furthermore, we present a critical analysis and user-based evaluation of the results resorting to the responses collected in two separate surveys.
Narratives represent life events, convey information, and cultural values. Being an important
way to perceive the world through information sharing. Narrative extraction techniques are
being developed to better understand the story behind texts, for example from news articles [
In this paper we introduce an automatic process to extract, process and present a story in a
graphical form using nearly universally understandable icons. The process starts with a text from
which the relevant narrative elements are identified and extracted. Next, an internal abstract
representation is generated and finally icons are linked to the story. The intermediate abstract
representation we use to represent the extracted narrative elements is Discourse Representation
Structure (DRS), a formal linguistics framework for exploring meaning [
With this work, we produced a method to automatically transform a story into a graphical representation using icons with little to none human intervention. Our contributions include: a validated algorithm to resolve the most specific description from a set of descriptions of the same actor/participant; a methodology to build dictionaries, not only for icons, but also for images; a method to automatically obtain icons; a new icon-based visualization method integrated in the pipeline, which can be seen in a vertical strip or in a slideshow.
Knowledge and information visualizations are widely used in areas like education, in order to
help process, access, and handle complex knowledge and vast amounts of information [
Our pipeline aims to transform text into an icon based representation. Our contribution focuses on the visualization step. Brat2Viz is a narrative annotation visualization tool [11] proposed to support the debugging of narrative annotation done with BRAT. It implements a pipeline from annotated text to visualization, by transforming the annotation into a formal representation, using DRS, and then, to visual representations. Brat2Viz2 has two modules: Brat2DRS creates a DRS representation for each news story by parsing a BRAT annotated news text; and DRS2Viz parses the DRS representation and deploys a web application with the generated visualization.
The main aim of the icon visualization is to obtain a graphic representation for each sentence in the news story. Thus, automatic sentence extraction from the input text was added to Brat2DRS. Given that existing icon databases are mostly in English, it was convenient to employ automatic translation to be able to work with non English texts. For that, two automatic translation tools were included in DRS2Viz: Hugging Face Transformers3 and Googletrans4. 1https://bbcnewslabs.co.uk/projects/graphical-storytelling/ 2https://github.com/LIAAD/brat2viz 3https://github.com/huggingface/transformers 4https://pypi.org/project/googletrans/
The narrative elements to be represented are currently the actors mentioned in the text. Since the same actor can have diferent descriptions throughout the narrative (robber, assailant, man, suspect), we devised an algorithm to resolve the most specific description of an actor. From a set of actor descriptions, our method finds the most specific using WordNet 5 similarity between words and the super-subordinate relation (hyperonymy and hyponymy). To evaluate both the efectiveness of the algorithm and which translation to use, 109 sets of actor descriptions were analyzed. The resulting data comes from a manual analysis of the results obtained from processing the actors’ descriptions using the algorithm and comparing them with the expected descriptions. With 67.89% of well-defined actors, Googletrans was the chosen translation.
Now we present the icon dictionary that allows the actors’ connection to the icons, describe the steps and implementation6 taken to reach the icon visualization, and present the evaluation of the obtained results.
The Icon Dictionary allows the search for icons to use in the visualization. The sources integrated are emojidex7, IconFinder8, Icons89, and Icons-5010. The quality and availability of the icons in the dictionary depends on its sources, being possible to add new ones. Icons are searched by terms, adding a new icon when that term does not yet have an icon that represents it in the dictionary. When adding a new icon, a search is made on the available sources, with one of the icons being saved. The search (implemented in DRS2Viz) can be performed: semi-automatically (the user chooses the icon from the list of results presented by the icon sources) or automatically (using fastText embeddings to calculate the cosine similarity between the searched term and each element of the lists returned from the icon sources, choosing the most similar icon to the searched one).
(a) Vertical strip.
(b) Slideshow.
Two types of icon visualization were created: vertical strip (presents the full news story with the actors represented with icons), shown in Figure 1a, and slideshow (allows user interaction by moving to the next and previous sentences) in Figure 1b, where the news text presented 5https://wordnet.princeton.edu 6Available at: https://github.com/LIAAD/Text2Icons 7https://developer.emojidex.com/#api 8https://developer.iconfinder.com/reference/overview-1 9https://developers.icons8.com/docs/getting-started 10https://www.kaggle.com/danhendrycks/icons50 can be translated as: “Thieves stole 500 cows from a New Zealand farm without the owner noticing anything for weeks. The last time he counted the herd, in early July, the man had 1300 heads, but now just over 800 remain. Police are investigating the robbery but have few leads. “Probably, they weren’t all taken at once” admit the agents, who don’t realize how the farmer didn’t notice the robbery earlier.”.
To assess if the generated visualizations represent the stories eficiently from the perspective of a potential user, two surveys were launched: one for the ability of a set of icons to represent a story scene; and another for the quality of the term-icon connection. We present users’ views on the use of icons as a complement to news stories. The questions were designed to minimize the possibility of biasing the answers towards favoring our hypotheses. The answers obtained from the surveys were collected by a network of direct and indirect personal contacts, and by the academic community of the University of Porto.
Visualization of News Stories To evaluate the quality of the results, participants rated from 1 to 5 how well a set of icons represented a sentence, with 149 responses collected. The survey had 10 questions, each with a sentence, and 3 images of a generated set of icons. The average of the participants scores was calculated, where the image with the highest score, i.e., the most scored on average by the participants, corresponds to the one generated by the visualization in 80% of the cases. This indicates that possible news readers or users of the tool identify the majority of the icon visualizations generated as the most suitable, concluding that the story representation as a whole is mostly in accordance with what users expect to see. Term-Icon Connection To assess the quality of the term-icon connection, 291 participants choose the best suited icon for a term in 20 questions each with: 1 term and 6 icons (one is the most similar to the term, and the others are plausible representations). Fleiss’ kappa [12] was calculated to understand if the answers were chosen randomly. = 0.3785551 is classified as Fair agreement, according to Landis et al. [13], making the data suitable. The hit rate of each question was calculated, where the average hit rate for each question is 53%, with 85% of the cases falling into the first and second most voted option. Concluding that the results are favorable, since the visualization shows icons that users identify as the most suitable. Participants opinion The opinion of possible news readers and visualization users is crucial to understand the scope of this work. 291 participants rated two questions from 1 to 5. One about the usefulness in having news text accompanied by illustrative icons for themselves, and the other, regarding another (children or people with reading dificulties). 58.4% answered with 4 and 5 (highest level of agreement) in favor of the usefulness for themselves. And with 89.7% in options 4 and 5, we conclude that the participants’ standpoint on the usefulness for others reflects the premise here addressed: the importance of narrative visualization.
In this paper, we described a new narrative visualization approach, integrated in Brat2Viz, by representing key concepts of a narrative with icons. The process to extract narrative elements is presented: sentence extraction; automatic translation methods integration; and an actor linking algorithm to find the most specific description of the news actors. The icon dictionary designed as a database supports four sources, and two ways to add new icons: semi-automatically or automatically. These steps combined result in two visualizations: vertical strip and slideshow. Finally, the results were evaluated from potential users with positive and promising results. It might be interesting to expand the visualization to automatically generated icons. In addition to representing the actors, it can be useful to represent the narrative events. The algorithm also has room for improvement when it comes to being able to handle more complex actors.
This work was carried out as part of the project Text2Story, financed by the ERDF - European Regional Development Fund through the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 and by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia within project PTDC/CCICOM/31857/2017 (NORTE-01-0145-FEDER-031857). come of a workshop held in Tübingen, Germany, May 2004], volume 3426 of Lecture Notes in Computer Science, Springer, 2005, pp. 1–23. URL: https://doi.org/10.1007/11510154_1. [7] A. Figueiras, Narrative visualization: A case study of how to incorporate narrative elements in existing visualizations, in: E. Banissi, M. W. M. Bannatyne, F. T. Marchese, M. Sarfraz, A. Ursyn, G. Venturini, T. G. Wyeld, U. Cvek, M. Trutschl, G. G. Grinstein, V. Geroimenko, S. Kenderdine, F. Bouali (Eds.), 18th International Conference on Information Visualisation, IV 2014, Paris, France, July 16-18, 2014, IEEE Computer Society, 2014, pp. 46–52. URL: https://doi.org/10.1109/IV.2014.79. [8] R. Campos, J. Duque, T. Cândido, J. Mendes, G. Dias, A. Jorge, C. Nunes, Time-matters: Temporal unfolding of texts, in: D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast, F. Sebastiani (Eds.), Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II, volume 12657 of Lecture Notes in Computer Science, Springer, 2021, pp. 492–497. URL: https://doi.org/10.1007/978-3-030-72240-1_53. [9] A. Pasquali, V. Mangaravite, R. Campos, A. M. Jorge, A. Jatowt, Interactive system for automatically generating temporal narratives, in: L. Azzopardi, B. Stein, N. Fuhr, P. Mayr, C. Hauf, D. Hiemstra (Eds.), Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14-18, 2019, Proceedings, Part II, volume 11438 of Lecture Notes in Computer Science, Springer, 2019, pp. 251–255. URL: https://doi.org/10.1007/978-3-030-15719-7_34. [10] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, I. Sutskever, Zeroshot text-to-image generation, in: M. Meila, T. Zhang (Eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, PMLR, 2021, pp. 8821–8831. URL: https://proceedings.mlr.press/v139/ ramesh21a.html. [11] E. Amorim, A. Ribeiro, B. S. Santana, I. Cantante, A. Jorge, S. Nunes, P. Silvano, A. Leal, R. Campos, Brat2viz: a tool and pipeline for visualizing narratives from annotated texts, in: R. Campos, A. M. Jorge, A. Jatowt, S. Bhatia, M. A. Finlayson (Eds.), Proceedings of Text2Story - Fourth Workshop on Narrative Extraction From Texts held in conjunction with the 43rd European Conference on Information Retrieval (ECIR 2021), Lucca, Italy, April 1, 2021 (online event due to Covid-19 outbreak), volume 2860 of CEUR Workshop Proceedings, CEUR-WS.org, 2021, pp. 49–56. URL: http://ceur-ws.org/Vol-2860/paper6.pdf. [12] J. L. Fleiss, Measuring nominal scale agreement among many raters, volume 76, Psychological Bulletin, 1971, p. 378–382. URL: https://doi.org/10.1037/h0031619. [13] J. R. Landis, G. G. Koch, The measurement of observer agreement for categorical data, volume 33, Biometrics, 1977, pp. 159–174. URL: https://doi.org/10.2307/2529310.