In games, unlike in traditional linear storytelling media such as novels or lms, narrative events unfold in response to the player input. Therefore, the music composer in an interactive environment needs to create music that is dynamic and non-repetitive. We investigate how to express emotions and moods in music and how to apply this research to improve player experience in games. The main novelty in our approach, compared to most algorithmically generated music research, is our focus on a ective and cognitive modelling, coupled with real-time adjustment of the music. This focus on the emotional meaning that procedurally generated music should express has also been identi ed by Collins as one of the lacking features that prevent procedurally generated music to be more widely used in the game industry [1].
We aim to ll what we believe is the gap that's holding generative
procedural music generation back: emotion expression. A number of works have been
published in the area of a ect, semiotics and mood-tagging ([
The nal objective is to create a system where, by using a emotional model of the player, we would be able to identify the player's emotional state and be able to reinforce or manipulate it through the use of mood-expressive music to improve user experience. What this could achieve is the creation of better immersive experiences (reinforcement of current emotional state and play-style) and the development of tools/models to help designers create experiences where the players are put in a speci c emotional state.
A number of challenges arise from this objective, such as how to validate our mood expression model, improve our generator to be able to include harmony and melody, make the generated music more interesting and creating a cognitive model of the player, just to name a few. First we are going to validate the theory we used until now and better tune it to increase the mood recognition rate, then we will harmony generation in our generator. We are thinking about training a Markov chain model by using a database of chord successions which could consider, apart from the current chord, one or two previous ones.
Our current generator doesn't consider harmony, so we can't express these; still we will soon integrate harmony and melody generation in our framework, hopefully opening up even more possibilities for our research. Once we have improved our music generation to ideally being able to express easily identi able moods and creating interesting music, we will start working on the cognitive model of the player to nd ways to extrapolate his/her emotional state. This will be integrated into an a ective loop where music generation is used as part of an approach to player-adaptive games through content generation. We would rstly focus on one speci c game genre and, time permitting, expand our model to include more genres, making it more general. We would also like to continue our work on narrative cues expression through music, even if this direction of the research is not the main focus of the project. 2 2.1
Procedural generation of content is a booming eld that has seen a tremendous
growth lately. Applications can be: creating simple sound e ects, game levels,
entire game worlds, and more. While a good number of games use some sort of
procedural music structure, there are di erent approaches (or degrees), Wooller
et al. distinguish two of them: transformational algorithms and generative
algorithms[
Generative algorithms instead create the musical structure themselves, which leads to a higher degree of di culty in keeping the music consistent with the game events, and generally require more computing power as the musical materials have to be created in real-time. An example of this approach can be found in Spore: the music written by Brian Eno was created with Pure Data in the form of many small samples that created the soundtrack in real time. 2.2
Emotions have been extensively researched within psychology, although their
nature (and what constitutes the basic set of emotions) is still widely debated.
Lazarus argues that \emotion is often associated and considered reciprocally
in uential with mood, temperament, personality, disposition, and motivation"
[
A ect is generally considered to be the experience of feeling or emotion.
It is largely believed that a ect is post-cognitive; emotion arises only after an
amount of cognitive processing has been accomplished. With this assumption,
every a ective reaction (e.g., pleasure, displeasure, liking, disliking) results from
\a prior cognitive process that makes a variety of content discriminations and
identi es features, examines them to nd value, and weighs them according to
their contributions" [
Mood is an a ective state. However, while an emotion generally has a speci c
object of focus, moods tends to be more unfocused and di used [
In this paper, we focus on moods instead of emotions, for we expect that in games { where the player listens to the background music for a longer time, that a particular emotion is induced by the mood { and moods are more likely to be remembered by the players after their game-play. In addition, they are easier for game designers to integrate, since they represent longer-duration sentiments suitable for segments of game play. 2.3
The set of adjectives that describe music mood and emotional response is
immense and there is no accepted standard vocabulary. For example, in the work of
[
[
A ect in music can in this way be divided into the four clusters based on
the dimensions of valence and arousal: Anxious/Frantic (Low Valence, High
Arousal), Depression (Low Valence, Low Arousal), Contentment (High Valence,
Low Arousal) and Exuberance (High Valence, High Arousal). These four clusters
have the advantage of being explicit and discriminable; also they are the basic
music-induced moods as described in [
We have created a rst prototype of a Moody Music Generator [
In response we designed a new open-ended study that, rather than directly attempting to validate that our two control parameters represent arousal and valence, crowd-sourced labels characterising di erent parts of this two-dimensional control space. Our aim was to characterise perception of the generators expressive space, without constraining listeners responses to labels speci cally aimed at validating the original arousal/valence motivation. Subjects were asked to listen to clips of generated music over the Internet, and to describe the moods with free-text labels. We nd that the arousal parameter does roughly map to perceived arousal, but that the nominal valence parameter has strong interaction with the arousal parameter, and produces di erent e ects in di erent parts of the control space. We believe that this characterisation methodology is general and could be used to map the expressive range of other parameterisable generators. This study has returned some positive results, yet suggests that we need to re ne our mood expression model, especially on expressing the valence axis.
Currently we have implemented a new AI system for music generation, which creates more interesting and musically complex music that might in uence our current a ective state expression theory. The objectives of our generator are: (i) to express di erent a ective states using a variety of AI techniques; (ii) to generate such music in real-time and (iii) to react in real-time to external stimuli. The architecture is comprised of three main components: the composition generator, the real-time a ective music composer and an archive of compositions. A novel feature of our approach is the separation of composition and a ective interpretation: the system creates abstractions of music pieces (what we call compositions) and interprets these compositions in real-time to achieve the desired a ective expression while also introducing stochastic variations. In terms of composition generation, we present a novel combination of Evolutionary Computation techniques to evolve melodies: the Feasible/Infeasible two population method (FI-2POP [23]) and Multi Objective Optimization [24]. 4
We are currently conducting an evaluation study on the music generation technique developed for our new generator. Soon we'll also study the a ect expression capabilities of our generator, as our theory will probably need to be adjusted to the higher complexity of the music produced.
Soon after we plan to decide a game to integrate with the generator, and start building an a ective model of the player based on that game. The a ect model should be able to tell us the emotional state of the player from in-game metrics and also give us information on how we can in uence the player's state through a ective expressive music. 5
This study will be (and already is) a contribution to the eld of music generation: while the eld is very active in the generation of music to emulate a speci c style, the creation of accompaniments to a melody, there is very little research that investigates the expression (and manipulation) of a ective states through procedurally generated music. We will also contribute to the eld of procedural content generation, in fact our research is strongly connected with the concept expressed by Yannakakis and Togelius of Experience-Driven Procedural Content Generation (EDPC) [25], where the content itself is tailored to the player in an attempt to create highly personalized content to improve player experience. We will create the rst player-adaptive a ective game music generator. We believe that this innovative research and our unique approach to it might also be interesting for other elds, such as musicology, human-computer interaction and computer science in general. 22. Scirea, M., Nelson, M.J., Togelius, J.: Moody music generator: Characterising control parameters using crowdsourcing. In: Evolutionary and Biologically Inspired Music, Sound, Art and Design. Springer (2015) 200{211 23. Kimbrough, S.O., Koehler, G.J., Lu, M., Wood, D.H.: On a feasible{infeasible two-population ( -2pop) genetic algorithm for constrained optimization: Distance tracing and no free lunch. European Journal of Operational Research 190(2) (2008) 310{327 24. Deb, K.: Multi-objective optimization using evolutionary algorithms. Volume 16.
John Wiley & Sons (2001) 25. Yannakakis, G.N., Togelius, J.: Experience-driven procedural content generation.
IEEE Transactions on A ective Computing 2(3) (2011) 147{161