=Paper=
{{Paper
|id=Vol-2827/KBS-Paper_4
|storemode=property
|title=Towards Generative Illustration of Text
|pdfUrl=https://ceur-ws.org/Vol-2827/KBS-Paper_4.pdf
|volume=Vol-2827
|authors=Carolina Gonçalves Lopes,João M. Cunha,Pedro Martins
}}
==Towards Generative Illustration of Text==
https://ceur-ws.org/Vol-2827/KBS-Paper_4.pdf
Towards Generative Illustration of Text
Carolina Gonçalves Lopesa , João M. Cunhaa and Pedro Martinsa
a
CISUC, Department of Informatics Engineering
University of Coimbra, Portugal
Abstract
Generative illustration of text is a task that can be viewed as related to both information visualisation
and computational design. In this paper, we present an exploratory study towards text illustration, de-
scribing our experiments to visually represent data about characters, objects and emotions. We present
our conclusions in regards to representing data using attributes such as colour, shape and position.
Keywords
Computational Design, Information Visualisation, Illustration, Generative Design
1. Introduction
Over the years, illustration has been influenced by technological and artistic advances. Like
this, generative illustration emerges by technology’s influence. This can be defined as a process
of illustration, whose possible visual compositions are created from algorithms or set by rules.
Programming languages such as Processing1 have significantly contributed to the simplification
of generative illustration processes, enabling the creation of several artefacts efficiently. Some
research has been conducted in the past in regards to generative illustration. For example,
representing the text in book covers – e.g. [1] or Data Book Covers2 – and in representation of
songs and lyrics in video [2, 3].
The project described in this paper was developed in the context of a master’s dissertation
in Design and Multimedia [4] and addresses the potential of illustration using technological
processes, namely, generative techniques. We explore several possibilities that allow for the
visual representation of characteristics of different nature associated with the content of a text.
We believe generative illustration can be seen as an alternative to the traditional techniques of
illustration and be included as a component of a computational design system.
2. Approach
The examples given in the previous section rely mostly on structure or form (rather than
content) to illustrate/visualise text. Despite not considering them as ineffective approaches, for
Joint Proceedings of the ICCC 2020 Workshops (ICCC-WS 2020), September 7-11 2020, Coimbra (PT) / Online
" uc2013150510@student.uc.pt (C. G. Lopes); jmacunha@dei.uc.pt (J. M. Cunha); pjmm@dei.uc.pt (P. Martins)
~ https://cdv.dei.uc.pt/ (J. M. Cunha)
0000-0001-6502-3500 (J. M. Cunha); 0000-0002-3630-7034 (P. Martins)
© 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings CEUR Workshop Proceedings (CEUR-WS.org)
http://ceur-ws.org
ISSN 1613-0073
1
https://processing.org
2
http://pmcruz.com/work/book-covers
Figure 1: Color-emotion association based on [5]
our project we wanted to go further and also consider aspects more related to content, such
as character development. This idea draws inspiration from the works of the Swiss illustrator
Warja Lavater3 , where characters are represented as geometric shapes [6]. In the illustrations
produced, the author explores relationships among characters and represents them using visual
transformations, e.g. scale increase to represent the power of one character over the other, or
inclusion to represent a character being eaten. For our project, we wanted to focus our analysis
of the text in three aspects: characters, other objects that appear in the story and emotion. We
take into consideration aspects deemed relevant when the visually representing concepts – e.g.
the semiotics of colour and shape [7].
To produce a system, we made the decision of following a multilayered approach. Each layer
would be produced by analysing different aspects of the text, as identified below:
• 1st Layer – Proper Nouns (charaters)
• 2nd Layer – Common Nouns (e.g. objects)
• 3rd Layer – Sentiment
By following this strategy, we wanted to achieve an overall representation of the text, mapping
text properties (syntax, semantics and also emotional content) to visual properties (e.g. colour,
shape and size). For the analysis of emotion in the text we used two lexica: NRC Word-Emotion
Association Lexicon and NRC Word- Color Association lexicon. These are part of a collection of
lexica developed by the National Research Council of Canada [5]. In the following sections, we
will describe our explorations, which had as final goal to implement a system for text generative
illustration.
2.1. 1st Layer – Proper Nouns
One of the aspects that has more relevance in the story is its characters. To identify the
characters we used the RiTa Library4 to perform part-of-speech tagging and retrieve the proper
nouns. In general, this analysis allows us to visualise the “surface” of the story, identifying the
characters names and respective relevance (i.e. extrapolated from the number of times that they
are mentioned).
At a first stage, we did a preliminary exploration with size, position and colour in character
representation. The conclusions from this first analysis were: (i) position is useful to visually
3
http://www.maeght.com/news/oct09_lavater/english-index.html
4
https://rednoise.org/rita/
Figure 2: First Approach to a set of colours to be assigned to characters
Figure 3: Second Approach to a set of colours to be assigned to characters
represent the location in the story (e.g. mapping the story into a canvas by using the left top
corner as beginning and bottom right as the end); (ii) size can be used to represent the relevance
in the text but has some restrictions as it directly influences how the story is mapped to the
canvas; and (iii) colour can be used to distinguish between different characters.
We identified three aspects that needed to be addressed when representing characters (using
proper nouns): identification, emotion and positioning.
The first approach to character identification was done by using shapes, in which the number
of vertices was being mapped to the number of verbs – we used a minimum of 3 and a maximum
of 15 vertices and then assigned the different shapes to the characters in order of verb count
(the character with the highest count would get a shape with more vertices).
In addition, we started using the adjectives by mapping them to emotions and using colours
associated with these emotions retrieved from [5] (see Fig. 1). We used these colours in combi-
nation with the shapes associated with the characters and we also changed their size depending
on the number of occurrences. However, this strategy did not prove itself very useful, as the
characters representation would change depending on the verbs and in small sizes it was not
possible to distinguish between characters.
2.1.1. Second Approach
Given that we had already concluded that colour is appropriate for character distinction, we
decided to change the strategy. The new approach for character representation was based on
two assumptions: (i) colour was to be used to distinguish between characters and (ii) a change
in saturation was to be used to reflect the change in emotion. This latter aspect would enable us
Figure 4: Shapes assigned to emotions
to see the evolution of the character. The first colour study can be seen in Fig. 2. In this figure
our efforts to produce a colour palette can be observed – we tried to define a set of colours
and respective change in saturation. However, the results obtained showed that the colours
were too similar and would eventually lead to perception issues. For this reason, we decided to
reduce the set, assuming that less important characters would not have great evolution. In Fig. 3
it is possible to see the final palette, in which five colours can be used with different saturation,
while the others are static.
We decided to start by assigning emotion to shape. Initially, we started by using the emotions
defined by Plutchik [8]: Anger, Fear, Sadness, Joy, Trust, Anticipation, Disgust and Surprise.
However, some emotions could be interpreted as being dependent on a third party (e.g. trust
in what?), so we decided to exclude “trust”, “anticipation”, “disgust” and “surprise”. With the
remaining emotions, we conducted a perception study based on distortions applied to a circular
shape and their perceived emotion (see Fig. 4). With this study, we tried to define how a shape
should be distorted to represent each emotion.
The emotions were retrieved using the verbs and adjectives associated with the character.
These were mapped to a shape. However, upon observing the shapes we came to the conclusion
that emotion is not totally perceivable from shapes alone. For this reason, we decided to make
use of the saturation palette previously described. The saturation value was established based
on emotions associated with the character: positive ones would lead to more saturation and
negative to less.
An example of a result obtained for proper nouns can be observed in Fig. 6.
2.2. 2nd Layer – Common Nouns
For the common Nouns layer we wanted to follow a different approach. By looking at the
results obtained with the proper nouns, we concluded that the geometric shapes were a style
we wanted to assign to the first layer. For this reason, we decided to explore an approach
similar to collage of images. To do this, we implemented a method to retrieve images for the
Figure 5: Exploration with filters applied to common nouns
Figure 6: Screenshot of the representation of proper nouns
common nouns using the platform Pixabay5 . By querying a word, we would receive images
that represented it in some way. Initially, we started to explore the application of filters to these
images in association with emotions, as observed in Fig. 5. However, we were unsatisfied with
the results that we were obtaining and decided to leave this topic for future work.
2.3. 3rd Layer – Sentiment
For the 3rd layer we wanted to convey a more general sense of the sentiment of the text. As
such, we analyse the words, retrieve their sentiment. To achieve a sense of overall sentiment,
we used the background colour and assigned a warm colour to a positive sentiment and a cold
colour to a negative one.
5
https://pixabay.com/
Figure 7: Generation based on “Cinderella” (left) and on “Cinderella arrives by car” (right)
3. Closing Remarks
In this paper, we described a project that had as main goal the implementation of a system
for text illustration. It is important to mention that this project results from an exploratory
study in the context of a master thesis conducted by the first author [4]. Our system generates
illustrations for texts and the results obtained show how different types of text lead to different
illustrations, even being based on the same story. Figure 7 illustrates this by comparing the
“Cinderella” form the Gutenberg Project6 with the “Cinderella arrives by car” from the Fairy
6
https://www.gutenberg.org
Tales for the Disillusioned [9]. This is particularly interesting when one considers the existence
of parodic retellings of classic tales [10], which can be used to generate illustrations and assess
the impact of different text of similar stories on the final output of the system. Our end goal is
to use our system as a component of a bigger framework for computational design of books. In
our opinion, generative techniques have the potential to bring new possibilities to the field of
graphic design [11].
Despite having reached some important conclusions, there is a considerable amount of work
to be done. Some future developments include addressing relations between characters and a
thorough study on perception of the generated illustrations.
Acknowledgments
This work is partially funded by the Foundation for Science and Technology, I.P., within the
scope of the project CISUC - UID/CEC/00326/2020 and by European Social Fund, through the
Regional Operational Program Centro 2020, and under the grant SFRH/BD/120905/2016.
References
[1] L. Duro, P. Machado, A. Rebelo, Graphic narratives: Generative book covers, in: ACM
SIGGRAPH 2012 Posters, SIGGRAPH ’12, Association for Computing Machinery, New
York, NY, USA, 2012. doi:10.1145/2342896.2342925.
[2] M. Cruz, P. Hardman, J. M. Cunha, Computationally generating images for music al-
bums, in: Proceedings of the Ninth International Conference on Computational Creativity,
Salamanca, Spain, June 25-29, 2018., 2018, p. 309.
[3] M. Cruz, Os olhos também ouvem: Sistema de geração de imagens de acordo com texto e
som para álbuns de música, Master’s thesis, University of Coimbra, 2019.
[4] C. G. Lopes, Ilustração Generativa com Base em Texto, Master’s thesis, Universidade de
Coimbra, 2019.
[5] S. Mohammad, Colourful language: Measuring word-colour associations, in: Proceedings
of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, 2011, pp.
97–106.
[6] W. Lavater, Le Petit Chaperon Rouge, A. Maeght, 1965.
[7] J. M. Cunha, P. Martins, A. Cardoso, P. Machado, Generation of concept-representative
symbols, in: Workshop Proceedings of the 23rd International Conference on Case-Based
Reasoning (ICCBR-WS 2015), CEUR, 2015.
[8] R. Plutchik, The emotions, University Press of America, 1991.
[9] G. Schultz, L. Seifert, Fairy Tales for the Disillusioned: Enchanted Stories from the French
Decadent Tradition, Princeton University Press, 2019.
[10] C. Malarte-Feldman, Folk materials, re-visions, and narrative images: The intertextual
games they play, Children’s Literature Association Quarterly 28 (2003) 210–218.
[11] A. V. Schimpf, Substantial Changes in Graphic Design with the Emergence of Generative
Design Processes, Master’s thesis, Universidad Internacional de La Rioja, 2019.