Towards Generative Illustration of Text

Abstract

Generative illustration of text is a task that can be viewed as related to both information visualisation and computational design. In this paper, we present an exploratory study towards text illustration, describing our experiments to visually represent data about characters, objects and emotions. We present our conclusions in regards to representing data using attributes such as colour, shape and position. Over the years, illustration has been influenced by technological and artistic advances. Like this, generative illustration emerges by technology's influence. This can be defined as a process of illustration, whose possible visual compositions are created from algorithms or set by rules. Programming languages such as Processing1 have significantly contributed to the simplification of generative illustration processes, enabling the creation of several artefacts eficiently. Some research has been conducted in the past in regards to generative illustration. For example, representing the text in book covers - e.g. [1] or Data Book Covers2 - and in representation of songs and lyrics in video [2, 3]. The project described in this paper was developed in the context of a master's dissertation in Design and Multimedia [4] and addresses the potential of illustration using technological processes, namely, generative techniques. We explore several possibilities that allow for the visual representation of characteristics of diferent nature associated with the content of a text. We believe generative illustration can be seen as an alternative to the traditional techniques of illustration and be included as a component of a computational design system.

Keywords

• eol>Computational Design
• Information Visualisation
• Illustration
• Generative Design

1. Introduction

2. Approach

The examples given in the previous section rely mostly on structure or form (rather than content) to illustrate/visualise text. Despite not considering them as inefective approaches, for our project we wanted to go further and also consider aspects more related to content, such as character development. This idea draws inspiration from the works of the Swiss illustrator Warja Lavater3, where characters are represented as geometric shapes [ 6 ]. In the illustrations produced, the author explores relationships among characters and represents them using visual transformations, e.g. scale increase to represent the power of one character over the other, or inclusion to represent a character being eaten. For our project, we wanted to focus our analysis of the text in three aspects: characters, other objects that appear in the story and emotion. We take into consideration aspects deemed relevant when the visually representing concepts – e.g. the semiotics of colour and shape [ 7 ].

To produce a system, we made the decision of following a multilayered approach. Each layer would be produced by analysing diferent aspects of the text, as identified below: • 1st Layer – Proper Nouns (charaters) • 2nd Layer – Common Nouns (e.g. objects) • 3rd Layer – Sentiment

By following this strategy, we wanted to achieve an overall representation of the text, mapping text properties (syntax, semantics and also emotional content) to visual properties (e.g. colour, shape and size). For the analysis of emotion in the text we used two lexica: NRC Word-Emotion Association Lexicon and NRC Word- Color Association lexicon. These are part of a collection of lexica developed by the National Research Council of Canada [ 5 ]. In the following sections, we will describe our explorations, which had as final goal to implement a system for text generative illustration.

2.1. 1st Layer – Proper Nouns

One of the aspects that has more relevance in the story is its characters. To identify the characters we used the RiTa Library4 to perform part-of-speech tagging and retrieve the proper nouns. In general, this analysis allows us to visualise the “surface” of the story, identifying the characters names and respective relevance (i.e. extrapolated from the number of times that they are mentioned).

At a first stage, we did a preliminary exploration with size, position and colour in character representation. The conclusions from this first analysis were: (i) position is useful to visually 3http://www.maeght.com/news/oct09_lavater/english-index.html 4https://rednoise.org/rita/ represent the location in the story (e.g. mapping the story into a canvas by using the left top corner as beginning and bottom right as the end); (ii) size can be used to represent the relevance in the text but has some restrictions as it directly influences how the story is mapped to the canvas; and (iii) colour can be used to distinguish between diferent characters.

We identified three aspects that needed to be addressed when representing characters (using proper nouns): identification, emotion and positioning.

The first approach to character identification was done by using shapes, in which the number of vertices was being mapped to the number of verbs – we used a minimum of 3 and a maximum of 15 vertices and then assigned the diferent shapes to the characters in order of verb count (the character with the highest count would get a shape with more vertices).

In addition, we started using the adjectives by mapping them to emotions and using colours associated with these emotions retrieved from [ 5 ] (see Fig. 1). We used these colours in combination with the shapes associated with the characters and we also changed their size depending on the number of occurrences. However, this strategy did not prove itself very useful, as the characters representation would change depending on the verbs and in small sizes it was not possible to distinguish between characters. 2.1.1. Second Approach Given that we had already concluded that colour is appropriate for character distinction, we decided to change the strategy. The new approach for character representation was based on two assumptions: (i) colour was to be used to distinguish between characters and (ii) a change in saturation was to be used to reflect the change in emotion. This latter aspect would enable us to see the evolution of the character. The first colour study can be seen in Fig. 2. In this figure our eforts to produce a colour palette can be observed – we tried to define a set of colours and respective change in saturation. However, the results obtained showed that the colours were too similar and would eventually lead to perception issues. For this reason, we decided to reduce the set, assuming that less important characters would not have great evolution. In Fig. 3 it is possible to see the final palette, in which five colours can be used with diferent saturation, while the others are static.

We decided to start by assigning emotion to shape. Initially, we started by using the emotions defined by Plutchik [ 8 ]: Anger, Fear, Sadness, Joy, Trust, Anticipation, Disgust and Surprise. However, some emotions could be interpreted as being dependent on a third party (e.g. trust in what?), so we decided to exclude “trust”, “anticipation”, “disgust” and “surprise”. With the remaining emotions, we conducted a perception study based on distortions applied to a circular shape and their perceived emotion (see Fig. 4). With this study, we tried to define how a shape should be distorted to represent each emotion.

The emotions were retrieved using the verbs and adjectives associated with the character. These were mapped to a shape. However, upon observing the shapes we came to the conclusion that emotion is not totally perceivable from shapes alone. For this reason, we decided to make use of the saturation palette previously described. The saturation value was established based on emotions associated with the character: positive ones would lead to more saturation and negative to less.

An example of a result obtained for proper nouns can be observed in Fig. 6.

2.2. 2nd Layer – Common Nouns

For the common Nouns layer we wanted to follow a diferent approach. By looking at the results obtained with the proper nouns, we concluded that the geometric shapes were a style we wanted to assign to the first layer. For this reason, we decided to explore an approach similar to collage of images. To do this, we implemented a method to retrieve images for the common nouns using the platform Pixabay5. By querying a word, we would receive images that represented it in some way. Initially, we started to explore the application of filters to these images in association with emotions, as observed in Fig. 5. However, we were unsatisfied with the results that we were obtaining and decided to leave this topic for future work.

2.3. 3rd Layer – Sentiment

For the 3rd layer we wanted to convey a more general sense of the sentiment of the text. As such, we analyse the words, retrieve their sentiment. To achieve a sense of overall sentiment, we used the background colour and assigned a warm colour to a positive sentiment and a cold colour to a negative one.

3. Closing Remarks

In this paper, we described a project that had as main goal the implementation of a system for text illustration. It is important to mention that this project results from an exploratory study in the context of a master thesis conducted by the first author [ 4 ]. Our system generates illustrations for texts and the results obtained show how diferent types of text lead to diferent illustrations, even being based on the same story. Figure 7 illustrates this by comparing the “Cinderella” form the Gutenberg Project6 with the “Cinderella arrives by car” from the Fairy Tales for the Disillusioned [ 9 ]. This is particularly interesting when one considers the existence of parodic retellings of classic tales [ 10 ], which can be used to generate illustrations and assess the impact of diferent text of similar stories on the final output of the system. Our end goal is to use our system as a component of a bigger framework for computational design of books. In our opinion, generative techniques have the potential to bring new possibilities to the field of graphic design [ 11 ].

Despite having reached some important conclusions, there is a considerable amount of work to be done. Some future developments include addressing relations between characters and a thorough study on perception of the generated illustrations.

Acknowledgments

This work is partially funded by the Foundation for Science and Technology, I.P., within the scope of the project CISUC - UID/CEC/00326/2020 and by European Social Fund, through the Regional Operational Program Centro 2020, and under the grant SFRH/BD/120905/2016.

References

[1] L. Duro , P. Machado , A. Rebelo , Graphic narratives: Generative book covers , in: ACM SIGGRAPH 2012 Posters, SIGGRAPH '12 , Association for Computing Machinery, New York, NY, USA, 2012 . doi: 10 .1145/2342896.2342925.

[2] M. Cruz , P. Hardman , J. M. Cunha , Computationally generating images for music albums , in: Proceedings of the Ninth International Conference on Computational Creativity , Salamanca, Spain, June 25-29, 2018 ., 2018 , p. 309 .

[3] M. Cruz , Os olhos também ouvem: Sistema de geração de imagens de acordo com texto e som para álbuns de música, Master's thesis , University of Coimbra, 2019 .

[4] C. G. Lopes , Ilustração Generativa com Base em Texto, Master's thesis , Universidade de Coimbra, 2019 .

[5] S. Mohammad , Colourful language: Measuring word-colour associations , in: Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics , 2011 , pp. 97 - 106 .

[6] W. Lavater , Le Petit Chaperon Rouge, A. Maeght , 1965 .

[7] J. M. Cunha , P. Martins , A. Cardoso , P. Machado , Generation of concept-representative symbols , in: Workshop Proceedings of the 23rd International Conference on Case-Based Reasoning (ICCBR-WS 2015 ), CEUR, 2015 .

[8] R. Plutchik , The emotions, University Press of America, 1991 .

[9] G. Schultz , L. Seifert, Fairy Tales for the Disillusioned: Enchanted Stories from the French Decadent Tradition , Princeton University Press, 2019 .

[10] C. Malarte-Feldman , Folk materials, re-visions, and narrative images: The intertextual games they play , Children's Literature Association Quarterly 28 ( 2003 ) 210 - 218 .

[11] A. V. Schimpf , Substantial Changes in Graphic Design with the Emergence of Generative Design Processes, Master's thesis , Universidad Internacional de La Rioja, 2019 .