Over the years, illustration has been influenced by technological and artistic advances. Like this, generative illustration emerges by technology's influence. This can be defined as a process of illustration, whose possible visual compositions are created from algorithms or set by rules. Programming languages such as Processing 1 have significantly contributed to the simplification of generative illustration processes, enabling the creation of several artefacts efficiently. Some research has been conducted in the past in regards to generative illustration. For example, representing the text in book covers -e.g. [1] or Data Book Covers 2 -and in representation of songs and lyrics in video [2,3].
The project described in this paper was developed in the context of a master's dissertation in Design and Multimedia [4] and addresses the potential of illustration using technological processes, namely, generative techniques. We explore several possibilities that allow for the visual representation of characteristics of different nature associated with the content of a text. We believe generative illustration can be seen as an alternative to the traditional techniques of illustration and be included as a component of a computational design system.
The examples given in the previous section rely mostly on structure or form (rather than content) to illustrate/visualise text. Despite not considering them as ineffective approaches, for our project we wanted to go further and also consider aspects more related to content, such as character development. This idea draws inspiration from the works of the Swiss illustrator Warja Lavater 3 , where characters are represented as geometric shapes [6]. In the illustrations produced, the author explores relationships among characters and represents them using visual transformations, e.g. scale increase to represent the power of one character over the other, or inclusion to represent a character being eaten. For our project, we wanted to focus our analysis of the text in three aspects: characters, other objects that appear in the story and emotion. We take into consideration aspects deemed relevant when the visually representing concepts -e.g. the semiotics of colour and shape [7].
To produce a system, we made the decision of following a multilayered approach. Each layer would be produced by analysing different aspects of the text, as identified below:
• 1st Layer -Proper Nouns (charaters) • 2nd Layer -Common Nouns (e.g. objects) • 3rd Layer -Sentiment By following this strategy, we wanted to achieve an overall representation of the text, mapping text properties (syntax, semantics and also emotional content) to visual properties (e.g. colour, shape and size). For the analysis of emotion in the text we used two lexica: NRC Word-Emotion Association Lexicon and NRC Word-Color Association lexicon. These are part of a collection of lexica developed by the National Research Council of Canada [5]. In the following sections, we will describe our explorations, which had as final goal to implement a system for text generative illustration.
One of the aspects that has more relevance in the story is its characters. To identify the characters we used the RiTa Library4 to perform part-of-speech tagging and retrieve the proper nouns. In general, this analysis allows us to visualise the "surface" of the story, identifying the characters names and respective relevance (i.e. extrapolated from the number of times that they are mentioned).
At a first stage, we did a preliminary exploration with size, position and colour in character representation. The conclusions from this first analysis were: (i) position is useful to visually represent the location in the story (e.g. mapping the story into a canvas by using the left top corner as beginning and bottom right as the end); (ii) size can be used to represent the relevance in the text but has some restrictions as it directly influences how the story is mapped to the canvas; and (iii) colour can be used to distinguish between different characters.
We identified three aspects that needed to be addressed when representing characters (using proper nouns): identification, emotion and positioning.
The first approach to character identification was done by using shapes, in which the number of vertices was being mapped to the number of verbs -we used a minimum of 3 and a maximum of 15 vertices and then assigned the different shapes to the characters in order of verb count (the character with the highest count would get a shape with more vertices).
In addition, we started using the adjectives by mapping them to emotions and using colours associated with these emotions retrieved from [5] (see Fig. 1). We used these colours in combination with the shapes associated with the characters and we also changed their size depending on the number of occurrences. However, this strategy did not prove itself very useful, as the characters representation would change depending on the verbs and in small sizes it was not possible to distinguish between characters.
Given that we had already concluded that colour is appropriate for character distinction, we decided to change the strategy. The new approach for character representation was based on two assumptions: (i) colour was to be used to distinguish between characters and (ii) a change in saturation was to be used to reflect the change in emotion. This latter aspect would enable us to see the evolution of the character. The first colour study can be seen in Fig. 2. In this figure our efforts to produce a colour palette can be observed -we tried to define a set of colours and respective change in saturation. However, the results obtained showed that the colours were too similar and would eventually lead to perception issues. For this reason, we decided to reduce the set, assuming that less important characters would not have great evolution. In Fig. 3 it is possible to see the final palette, in which five colours can be used with different saturation, while the others are static.
We decided to start by assigning emotion to shape. Initially, we started by using the emotions defined by Plutchik [8]: Anger, Fear, Sadness, Joy, Trust, Anticipation, Disgust and Surprise. However, some emotions could be interpreted as being dependent on a third party (e.g. trust in what?), so we decided to exclude "trust", "anticipation", "disgust" and "surprise". With the remaining emotions, we conducted a perception study based on distortions applied to a circular shape and their perceived emotion (see Fig. 4). With this study, we tried to define how a shape should be distorted to represent each emotion.
The emotions were retrieved using the verbs and adjectives associated with the character. These were mapped to a shape. However, upon observing the shapes we came to the conclusion that emotion is not totally perceivable from shapes alone. For this reason, we decided to make use of the saturation palette previously described. The saturation value was established based on emotions associated with the character: positive ones would lead to more saturation and negative to less.
An example of a result obtained for proper nouns can be observed in Fig. 6.
For the common Nouns layer we wanted to follow a different approach. By looking at the results obtained with the proper nouns, we concluded that the geometric shapes were a style we wanted to assign to the first layer. For this reason, we decided to explore an approach similar to collage of images. To do this, we implemented a method to retrieve images for the common nouns using the platform Pixabay 5 . By querying a word, we would receive images that represented it in some way. Initially, we started to explore the application of filters to these images in association with emotions, as observed in Fig. 5. However, we were unsatisfied with the results that we were obtaining and decided to leave this topic for future work.
For the 3rd layer we wanted to convey a more general sense of the sentiment of the text. As such, we analyse the words, retrieve their sentiment. To achieve a sense of overall sentiment, we used the background colour and assigned a warm colour to a positive sentiment and a cold colour to a negative one.
In this paper, we described a project that had as main goal the implementation of a system for text illustration. It is important to mention that this project results from an exploratory study in the context of a master thesis conducted by the first author [4]. Our system generates illustrations for texts and the results obtained show how different types of text lead to different illustrations, even being based on the same story. Figure 7 illustrates this by comparing the "Cinderella" form the Gutenberg Project 6 with the "Cinderella arrives by car" from the Fairy Tales for the Disillusioned [9]. This is particularly interesting when one considers the existence of parodic retellings of classic tales [10], which can be used to generate illustrations and assess the impact of different text of similar stories on the final output of the system. Our end goal is to use our system as a component of a bigger framework for computational design of books. In our opinion, generative techniques have the potential to bring new possibilities to the field of graphic design [11]. Despite having reached some important conclusions, there is a considerable amount of work to be done. Some future developments include addressing relations between characters and a thorough study on perception of the generated illustrations.
This work is partially funded by the Foundation for Science and Technology, I.P., within the scope of the project CISUC -UID/CEC/00326/2020 and by European Social Fund, through the Regional Operational Program Centro 2020, and under the grant SFRH/BD/120905/2016.