<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>HarmonyMapper: Generating Emotionally Diverse Chord Progressions for Games</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Sara Cardinale</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Oliver Withington</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Queen Mary University of London</institution>
          ,
          <addr-line>London</addr-line>
          ,
          <country country="UK">UK</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>In this paper, we introduce our system HarmonyMapper, a system for generating diverse musical sequences. By combining the MAP Elites algorithm with insights from music theory, specifically Neo-Riemannian Theory, HarmonyMapper is able to generate sets of chord sequences with a high-level diversity in terms of the emotions they are expected to elicit. Our intention is that this system will form the basis for a mixed-initiative musical composition tool focused on generating diverse music for digital games. While HarmonyMapper is limited in the musical complexity it can produce, our pilot experiments showed that it is capable of generating large numbers of varied sequences with no input data and only limited set up and computational resources required.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Procedural Music</kwd>
        <kwd>Neo-Riemannian Theory</kwd>
        <kwd>MAP-Elites</kwd>
        <kwd>Computational Creativity</kwd>
        <kwd>Quality-Diversity Algorithms</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>AIIDE Workshop on Experimental Artificial Intelligence in Games,
October 08, 2023, University of Utah, Utah, USA
* Corresponding author.
† These authors contributed equally.
$ s.cardinale@qmul.ac.uk (S. Cardinale);
o.withington@qmul.ac.uk (O. Withington)
 saracardinalemusic.com/ (S. Cardinale); owithington.co.uk
(O. Withington)
0000-0002-seven00seven-5193 (O. Withington)</p>
      <p>© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)</p>
    </sec>
    <sec id="sec-2">
      <title>2. Related Work</title>
      <p>and compound NROS employed by composers and the
specific events or emotions they aimed to portray [ 12]. By
2.1. Music Generation Systems identifying the NROs and CNROs used in a progression
we can to an extent infer the likely emotional response
The generation of chord progressions has been investi- of a potential listener.
gated for the automatic generation of musical harmony NRT has been used as the basis of generative music
to accompany various forms of media. Bernardes et. al. for games before in the work of Cardinale and Colton
[6] implemented the D’accord harmony generation sys- [13]. They introduced procedural NRT and suggested
tem which worked over a perceptually motivated tonal that it could be used in conjunction with Lehman’s
obserinterval space. Monteith et al. [7] generated music to in- vations to write event-based music for media, including
duce targeted emotions, using statistical techniques such games. This led to the development of their GENRT
as HMMs, and applied this in [8] to produce afective system which used NRT to procedurally generate
musimusic to accompany the audio of fairy tales being read. cal sequences which matched with a given specification</p>
      <p>Numerous generative systems have been used to cre- such as matching a specified tempo, or by incorporating
ate soundtracks for videogames. For instance, the game changes in the music’s mood using Lehman’s descriptors
GhostWriter (1998) [9] uses a generative approach to at specific time stamps. We expand on these ideas by
create the music. The system implements a rule-based generating sets of diverse sequences rather than
individalgorithm that maps the in-game tension to the music. ual sequences, though the current version of our system
To create a chord progression, the system implements produces simpler music than GENRT which is capable of
Schoenberg’s Theory of Harmony [10]. While the com- generating sequences with a melody, harmony, bassline,
positional aspect is conducted by a generative system, and percussion rather than just a chord sequence.
the detection of in-game tension is not an automated
process: a director identifies and maps the level of suspense
in the music to the in-game events. 2.3. MAP-Elites and Music Generation</p>
      <p>More recently, Lopez Duarte [11] introduced an
algorithmic music generator that makes use of a second
or third-order Markov model to manage instrumental
range and quantity, RNN for melody and bass, and an
Augmented Transition Network (ATN) to provide a tonal
grid. Duarte states that a system which combines
generative and algorithmic (rule-based systems, deterministic
approaches) could provide extensive musical variety in
the structure and musicality while supporting the style
and ofering continuity [11].</p>
      <p>The algorithm at the centre of this work’s music
generation system is the ‘Multi-dimensional Archive of
Phenotypic Elites’ algorithm, more commonly referred to
as MAP-Elites. MAP-Elites is the most prominent
algorithm within a family of relatively modern algorithms
referred to as ‘Quality-Diversity’ (QD) algorithms. These
algorithms work similarly to traditional genetic search
methods, which aim to find high-performing solutions to
problems by combining and mutating the best solutions
found so far, in a way heavily influenced by real-world
evolution. Where QD algorithms distinguish themselves
2.2. Neo-Riemannian Theory is by generating populations of solutions rather than fit
individuals, and by aiming to maximise the phenotypic
Neo-Riemannian Theory (NRT) is a branch of transforma- diversity of this population.
tional theory that provides analytical methods to study MAP-Elites achieve quality and diversity by
discretischromatic chord progressions that deviate from tradi- ing the search space for solutions into an evenly spaced
tional tonality. Conventional music theory, which relies grid, with each cell in the grid defined by a range of
calon keys and modes to analyze harmonic developments, culable phenotypic traits (Phenotype/phenotypic in this
often struggles to efectively analyze these types of pro- context meaning characteristics of the actual generated
gressions. NRT introduces Neo-Riemannian Operators artefact, in our case a musical sequence. This is opposed
(NROs) to examine chord changes. These operators de- to genotype/genotypic which refers to the encoded form
scribe the transitions between chords in a sequence, al- of the artefact). Classically this grid is defined by two
lowing for the analysis of chord progressions that go dimensions, though there are implementations that work
beyond traditional tonal frameworks. These NROs can in higher dimensions [14]. Each cell in the grid is only
be applied individually to a chord to produce a new chord, allowed to contain a fixed number of solutions, and the
or several can be applied at once to produce a Compound membership of the cell is determined by a fitness
funcNRO (CNRO) which is more distant from the initial chord. tion. If a new solution is generated and its phenotypic</p>
      <p>NRT does not just allow for the description of non- traits mean it should be added to a cell that is already
traditional progressions; it also allows them to be anal- at capacity, then the fitness of all solutions is calculated
ysed. Through the analysis of diverse film scores using and the least fit solution is removed. Each loop of the
NRT, Lehman observed a link between the chosen NROs algorithm previously generated solutions are selected
from the archive, new solutions are generated from them
and then they are added to the archive. By repeating this
process until a stopping condition is met such as a certain
number of loops, the intention is to arrive at a final grid
populated with a set of solutions that are both high
performing and diverse in terms of designer-selected traits.</p>
      <p>MAP Elites is a relatively new algorithm, having been
introduced by Mouret and Clune in 2015 [15]. However
it and its variants have already seen significant success
in domains as varied as generating robot designs [16],
generating digital artworks [17], controlling swarms of
micro-robots [18], and many others. It has also seen
significant popularity in the generation of content for
digital games, especially game levels [19, 20, 21].</p>
      <p>A domain that has seen little use of QD algorithms
is the domain of music generation (that the authors are
aware of). However, they have seen success in similarly
creative tasks such as digital drawing. McCormack et
al.[22] employ quality-diversity search techniques to
investigate a creative generative system based on an
agentbased line drawing model. The application of
qualitydiversity search allowed for the discovery of multiple
high aesthetic value phenotypes within the system.</p>
      <p>Berker and Colton have recently argued that QD
algorithms for music generation could be a good fit for the
ifeld [ 23]. We agree that it is a potentially good fit for
multiple reasons. There is a substantial amount of music
theory that can be leveraged to produce interesting
phenotypic metrics with which to evaluate generated music.</p>
      <p>Music can also be encoded genotypically in a form that
makes it easy to manipulate using conventional genetic
operators, especially when using NRT as we do in this
paper.
chord in the form of an array of midi note numbers. A
midi note number is a widely used encoding form within
music in which each note is assigned a number from 0 to
127, with middle C found at number 60. For example, a
C Major chord in the 4th octave composed of the notes
C4, E4, and G4 can be encoded as [60, 64, 67]. Each NRO
in traditional NRT can be described as a set of additions
and subtractions to the notes of a trichord in midi note
number form (see Table 1). This makes applying an NRO
to a trichord a mathematically trivial operation. However,
after applying an NRO to the midi note number, its form
may have to be reordered to place the trichord in its
root position. This is also complicated by the fact that
each NRO letter operates on note numbers diferently
depending on whether it is applied to a Major or a Minor
triad.</p>
      <sec id="sec-2-1">
        <title>3.2. Phenotypic Metrics and Fitness</title>
      </sec>
      <sec id="sec-2-2">
        <title>Function</title>
        <p>3. Methodology As discussed in Section 2.3, a central decision when
implementing MAP-Elites is on the choice of phenotypic
3.1. Musical Encoding metrics to use to determine where generated artifacts are
placed within the grid archive. One of the contributions
To generate our musical sequence we need a genotypic of this work is the introduction of metrics that can be
form for them which can be easily manipulated by ge- calculated for a musical sequence. These metrics are
innetic operators, while also being convertible into midi formed by musical theory and are intended to capture
audio with limited efort and compute. This is where NRT specific characteristics of musical pieces which are
meanworks well. NRT describes a series of Neo-Riemannian ingful to listeners, and therefore characteristics that a
Operators (NROs), conventionally labeled with a letter, game designer or composer might like to have diversity
which can operate on a trichord to produce a new tri- in to fit diverse game settings.
chord. These NROs can be combined together and applied
in sequence to a starting chord to produce a Compound 3.2.1. Fitness Function: Key Prediction
NRO (CNRO). The musical encoding we use in this
version of our system is to store tracks as an ordered array
of CNROs along with a starting chord. From this we can
generate the series of chords produced by applying these
CNROs in order, forming a musical sequence that can be
converted to midi at any point.</p>
        <p>To apply these CNROs to a chord, we need to store the</p>
        <sec id="sec-2-2-1">
          <title>To quantify the fitness of the generated chord sequences</title>
          <p>we experimented with several alternatives before
arriving at the option that gave the most appealing results:
the strength of fit between a chord sequence and its
predicted key. To analyse the predicted key of a piece we use
the Krumhansl-Schmuckler Key-Finding Algorithm [24].</p>
          <p>By comparing the distributions of pitches in a musical
Algorithm 1 Illustration of the Process for Generating a List of Chords from a Start Chord and a List of Compound
NRO Objects
Require: _,  _
ℎ_ = []
ℎ_ ← ℎ_ + _ℎ
for CNRO IN CNRO_List do
for NRO IN CNRO do
_ℎ ← ℎ(ℎ)
_  ←  (_ℎ)
if _  then</p>
          <p>_ℎ ←   ( , _ℎ)
else</p>
          <p>_ℎ ←   ( , _ℎ)
end if
 _ℎ ←    ( _ℎ)
ℎ_ ← ℎ_ +  _ℎ
end for
end for
return ℎ_
sequence with those of the fixed key profiles which they as you increase the size of the CNROs used, the resulting
developed, one can predict the key of any piece of music. chord tends to be further away from the original key. In</p>
          <p>
            In this work, we use Music21 [25], a popular mu- a series of experiments carried out by Krumhansl [26], it
sicology Python Library, and their implementation of was shown that tones less closely related to the tonality
Krumhansl-Schmuckler’s Algorithm. This not only gives are less stable than tones closely related to the tonality.
us a possible key of a piece, which is used later for pre- As tonality is inferred through harmonic context and
dicting the mode of our generated music, but it also gives the interplay of tension and release among surrounding
us the correlation coeficient of the confidence level of chords in relation to a specific tone or chord base [
            <xref ref-type="bibr" rid="ref5">27</xref>
            ],
the algorithm. The higher the coeficient, the better the the absence of a tonal centre can be said to create a sense
distribution of pitches matches the predicted key. This of uncertainty.
is important to know, as it is more than possible for a We operationalise this metric by summing all of the
piece to fit best with a certain key, but to still only weakly NROs in all CNROs that define a track, and then dividing
match its distribution. It is this correlation coeficient this number by the number of CNROs, giving us the mean
that we use as our fitness function, with the goal of push- number of NROs per chord transition. We also argue that
ing the system to produce music that is strongly within this is a natural characteristic to want to find diversity in
one key and therefore that is likely to sound more cohe- when generating video game music, as varying levels of
sive to a listener. However, this does significantly limit tension during the course of play is common to so many
the variety of music that will be generated, something genres of game.
we discuss further in Section 7 on our future plans with
this system. 4.0.2. Predicted Mode
          </p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>4. Implementation</title>
      <p>4.0.1. Average CNRO Shift</p>
      <sec id="sec-3-1">
        <title>The first of the two metrics that we use is what we term</title>
        <p>Average NRT Shift. This metric is based on the
observations of musicologist Frank Lehman [12] that there
is a relationship between how many NROs are used
between two chords in a sequence and how unsettling a
piece is. This could be attributed to the fact that when
using only one NRO, the resulting chord is more likely
to be in the same key or a closely-related key, whereas</p>
      </sec>
      <sec id="sec-3-2">
        <title>The second metric we use is the predicted mode of a piece.</title>
        <p>
          As discussed in Section 1, music theory describes seven
musical modes. These are highly useful for our purposes
as prior research has found that they can be ordered
from brightest-sounding, Lydian, to darkest-sounding,
Locrian, as shown in Figure 1 [
          <xref ref-type="bibr" rid="ref5">27</xref>
          ].
        </p>
        <p>
          There are several steps involved in HarmonyMapper’s
process for predicting the mode of a generated piece
of music. First, we take the predicted key of the piece
using the same Krumhansl-Schmuckler’s Algorithm [24]
which we use in calculating fitness. For this key, we
then generate the scale of that key for each of the seven
modes, giving us seven scales, each containing 7 notes, to
able to pick a mode and NRO shift that most represent the
in-game situation or the situation they want the player
to perceive. For example, in the song At Doom’s Gate
[
          <xref ref-type="bibr" rid="ref7">29</xref>
          ] from the game Doom [
          <xref ref-type="bibr" rid="ref8">30</xref>
          ] the main guitar rif is in
Locrian, giving a tense, dark mood to the level, whereas
the soundtrack frequently switches to brighter moods
for more triumphant moments.
        </p>
        <sec id="sec-3-2-1">
          <title>4.1. Genetic Operators</title>
        </sec>
      </sec>
      <sec id="sec-3-3">
        <title>In this system, we use two genetic operators to produce</title>
        <p>child solutions from pairs of parent tracks, namely a
mutation operator and a crossover operator both designed
to work on a sequence of CNROs. This parent pair are
selected at random from the MAP-Elites archive archive
population as in conventional MAP-Elites [15].</p>
        <p>The mutation operator works by replacing a CNRO in
a sequence with a new generated CNRO. This CNRO is
generated by first stochastically selecting a CNRO length
with a range between 1 and the max length for a CNRO
which is specified for the MAP-Elites run as a whole. We
then generate a sequence of NROs by selecting NROs
at random until we have enough to produce a CNRO of
the specified length. This new CNRO then replaces the
selected CNRO in the original sequence. This process has
a chance of being applied to every CNRO in a sequence,
so a mutation rate of X% means that every CNRO in a
selected sequence has an X% chance of being replaced
with a newly generated one.</p>
        <p>The crossover operator used is a one-point crossover.</p>
        <p>
          First, we generate a random crossover point for a selected
pair of parents which can be anywhere in the range 1
to the fixed track length minus 1. Two child CNRO
sequences are then generated by taking the portion of
parent 1s CNRO sequence up until the crossover point and
concatenating it with the portion of parent 2s following
the crossover point, and vice versa for child 2.
check. We then loop through every chord in the piece of
music and count each chord whose notes are all contained
within each mode’s scale. At the end of this process, we
then select the mode for which the most chords fit within. 4.2. Duplicate Chord Fitness Adjustment
In the case that there the chords fit within two or more An issue with our method for generating CNROs
stochasscales equally, we then check the frequency rate of the tically is that there is a chance that when a series of
appearance of a character note of each scale within the NROs are applied to a chord it can produce a loop that
chords. This note can be considered to be the signature produces the same chord as the initial chord. This can
note of a scale, and the one that most represents that be especially problematic in conjunction with the NRT
scale (see Table 2) [
          <xref ref-type="bibr" rid="ref6">28</xref>
          ]. For instance, the character note shift metric. As discussed, the goal for this phenotypic
of the Lydian mode is the raised fourth degree. metric is as a heuristic for how discomforting the track
        </p>
        <p>We again argue that this mode prediction would be might be to a listener based on the magnitude of the
potentially very useful to a game designer or composer, shifts between sequential chords. Due to this looping
as similarly to NRO shift, being able to target a variety of issue, however, it is possible to have long NRO sequences
moods would be very useful for many games and game which would cause higher scores on the NRT shift metric,
genres. While some games may not change their intended while producing duplicate chords which do not produce
mood much at all during play, especially traditional ar- the desired discomfort.
cade titles, these are in the minority in contemporary
video gaming. The composer or game designer would be</p>
        <p>
          To address this issue we opted to check each newly
generated track for the existence of duplicate chords in
sequence. Where a duplicate chord is found we force the performance of the system when it comes to the number
iftness of the track to be 0. This means the track can still of elites stored, but also the impact on system usability.
be stored in the MAP Elites grid, but it will always be One can imagine circumstances in which having a large
discarded when a suficient number of tracks without range of similar compositions with similar traits would
duplicates are generated within the same cell location. be useful to present a user with a suficient choice, but
While this approach is commonly used to handle infea- also circumstances in which a choice of any kind could be
sible individuals in genetic search and it undoubtedly redundant. However, for these initial experiments, this
achieves the goal of avoiding them being present in the decision was made purely based on system performance.
ifnal generated track set, it has also long been argued For the dimensions of the map, the predicted mode
against as an approach due to the loss of genetic informa- dimension was split into 7 increments, one for each
postion involved in efectively discarding these individuals sible mode, whereas the NRT Shift dimension was split
which may have made for useful parents [
          <xref ref-type="bibr" rid="ref9">31</xref>
          ]. As a re- into 10 increments evenly divided between 1 at the
botsult, we consider it to be an non-ideal solution to the tom end of the theoretical range if every CNRO in a track
problem and something we will look to improve in future was of the minimum length and 5 at the top of the range.
iterations of this system.
        </p>
        <sec id="sec-3-3-1">
          <title>5.1. Performance Measures</title>
        </sec>
      </sec>
      <sec id="sec-3-4">
        <title>To evaluate the performance of a generative run we cal</title>
        <p>culate two features: Average Grid Fitness and Coverage.</p>
        <p>To calculate the Average Grid Fitness we first sum up
the fitness scores of the fittest tracks found in each grid
cell, with empty cells receiving a fitness of 0. This number
is then divided by the total number of cells. To calculate
the Coverage we count the number of cells which have
any solution at all, and divide this count by the total
number of cells. Coverage is intended to measure how
efectively we are exploring the overall space, whereas
Average Grid Fitness looks at performance more
generally and requires both exploration but also the finding
of high-performing tracks. Coverage is commonly
calculated in MAP Elites implementations including in the
paper which introduced the algorithm [15], and
Average Grid Fitness is similar to the Reliability score which
appears in that paper and others, except that we do not
have an alternative algorithm to compare to.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>5. Experimental Configuration</title>
      <p>To conduct our initial experiments to explore this system
and its capabilities we first needed to decide on how
the system would be parameterised. Where possible we
based these decisions on quantitative data from pilot
experiments to indicate parameterisations that appeared
to lead to the best system performance, but where this
was not possible a more arbitrary decision was made.
5.0.1. Musical Parameterisation</p>
      <sec id="sec-4-1">
        <title>In terms of the parameterisation of the music tracks them</title>
        <p>selves, each track starts with a tri-chord in the 4th octave.</p>
        <p>This tri-chord is randomly selected for each track when
generating the starting population, and then inherited
from one of the two parents for tracks generated during
a run. Each track is composed of 20 CNROs giving a total
of 21 chords including the initial chord. The maximum
CNRO length was capped at 5. Decisions on track and
max CNRO length were made arbitrarily, and it is here 5.1.1. Run Length
where we expect that designers and composers using fu- In terms of experiment length, we let the process run
ture iterations of this system would have the most control. for 5,000 loops. This cutof point was selected after
iniHowever, it is more than possible there is a performance tial experimentation suggested that we were unlikely
impact induced by these choices, something we aim to to see Average Grid Fitness gains after this point. Each
examine in future work. loop involved the selection and generation of two tracks
to support the use of the crossover operator. Therefore
5.0.2. MAP Elites and Genetic Operator 10,000 tracks were generated in total. To seed the
ini</p>
        <p>Parameters tial map a starting population of size 500 was generated
The most important performance-afecting parameters for each run. This starting population was generated
are the mutation and crossover rates, as well as the num- through purely stochastic generation and concatenation
ber of elites that are stored per cell in the MAP Elites grid. of CNROs, as well as the selection of a random 4th-octave
Initial experiments using a grid search suggested that a start chord for each starting track.
mutation rate of 0.2, a crossover rate of 0.5, and storing a
maximum of 3 elites per cell led to the best overall perfor- 5.1.2. Computational Resources Used
mance. We note that in future iterations of this system All experiments were run on a Dell laptop with an
i5that are designed to meet the needs of game designers 10310U CPU with 16.0 GB of RAM. With the above
conand composers, we may have to consider not just the
of chords with a high level of tonal consistency.</p>
        <p>
          The system has the advantage of being fairly
computationally light and it was able to generate and evaluate
over 10 thousand chord sequences on a modestly
powered laptop in only 92 minutes. However, the process
would be too slow to be used in a generative system
which required quick iteration and generation such as a
mixed-initiative generative system with a human in the
loop (see [
          <xref ref-type="bibr" rid="ref10">32</xref>
          ] for an overview of mixed-initiative content
Figure 2: Figure showing the performance of the generative generation systems). Additionally, though we were able
process in terms of average grid fitness and coverage as loop to achieve a robust average fitness during our
experimencount increases. Measurements recorded and visualised for tal runs, Figure 6 suggests that a longer run time would
every 100th loop be required to achieve the maximum possible fitness as
average fitness and coverage were still increasing at the
termination point.
        </p>
        <p>As is evident in Figure 3 there were combinations of
metric values which the system struggled to find tracks
for. At the extreme low and high ends of the possible
NRT Shift values the system visibly struggles to further
populate the MAP Elites grid, especially in the 4.6 to 5
range in which it found 0 tracks. While one would expect
that the outlying locations of the possibility space would
naturally be harder to reach, we expect that this could
also be the result of the high mutation rate used and the
nature of our mutation operator. With the mutation rate
used of 20% for each of a track’s 20 CNROs, and with
the 80% chance that a generated CNRO will have less
than 5 NROs there is significant pressure pulling newly
generated tracks back to the centre of the grid.</p>
        <p>By inspecting the rules of our system, we can see that
it cannot produce chord progressions that are fully in the
Locrian mode. This is because it is impossible to produce
Figure 3: Figure showing the final MAP Elites archive in the its signature half-diminished interval tonic using only
form of a heat map. The color of each cell is determined by conventional NRO shifts. The best tracks found in this
the fittest track found for the cell part of the archive are comprised of chord sequences
that do not fit comfortably into any key when analysed
with Krumhansl-Schmuckler’s Algorithm but happen to
ifguration on this hardware, a full generative run took 1 contain chords that conform best to the Locrian scale
hour and 32 minutes. for the best-fitting key. However, this does not mean</p>
        <p>
          The system including all of its code, data for the results, they are actually in the Locrian mode. This highlights
as well as the generated chord progressions from this pa- the need for a fitness cutof before any generated chord
per’s experiment in both textual and .midi form is avail- sequences are actually selected for use, as the lower the
able at github.com/KrellFace/harmony-mapper/. fitness of a track, it both conforms less to a key, but also
less likely to be in the correct mode. The inability of
6. Results and Discussion the system to produce sequences in the Locrian mode is
a limitation we aim to address on as discussed later in
Here we discuss the overall performance of the system Section 7. This limitation could be improved by using the
including its strengths and weaknesses of it which are extended version of NRT by Colton and Cardinale [
          <xref ref-type="bibr" rid="ref11">33</xref>
          ]
highlighted by the results of our initial experiments. which is able to produce multiple chord types, including
        </p>
        <p>In terms of systems performance, we are cautiously diminished.
optimistic. It is able to produce sets of tracks with signif- The music produced by the system consists of chords
icant amounts of diversity present in terms of our two played on a piano. Each chord has an equal duration
metrics. The relatively high final average fitness score of and the instrumentation does not change throughout the
0.786 indicates that the system is able to produce series chord progression. This is a simplified version of the
system as proof of concept. In Section 7 we discuss the
future addition of melody components and improving the phrases that often represent a concept or character in
complexity of the music generated by HarmonyMapper. soundtracks. Firstly we aim to explore the generation</p>
        <p>In terms of the metrics themselves, future listening of diverse harmonies that all include the same leitmotif,
studies are required to explore whether or not we are suc- mirroring how they are used in traditional composition
cessfully producing emotionally diverse chord sequences. to evoke diferent moods while still referencing the same
However, our initial impressions are that the NRT shift character or concept. More excitingly perhaps, we aim
metric did not perform as expected. We were aware that to investigate whether we can use HarmonyMapper to
there were limitations to it as a heuristic due to the way produce tracks that are blends of two leitmotifs by
develNROs operate. To take an example from one of the re- oping a heuristic for evaluating how much a composition
views of this paper, R and SSLL are equivalent in terms of confirms to either or both of two specified leitmotifs. This
the chord change they produce but they difer massively could allow us to generate musical sequences that both
in length. We were aware of this limitation but hoped evoke diferent moods but also evoke two contrasting
that larger shifts would still tend to indicate tracks with characters or concepts to a lesser or greater degree,
simichord transitions which would be perceived as more jar- lar to how composers write music to unite two or more
ring. To the authors’ ears, however, this was not always leitmotifs.
the case. As a result, improving on this distance metric
is going to be a focus of our future work.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>8. Conclusion</title>
    </sec>
    <sec id="sec-6">
      <title>7. Future Work</title>
      <sec id="sec-6-1">
        <title>The most important area in which to improve and expand</title>
        <p>HarmonyMapper is the musical complexity it is able to
generate. Specifically, we aim to extend the system with
the generation of melodies, bass lines and percussion to
pair with the diverse harmonies that are currently
generated. One of the potential end goals for this system is for
non-musicians to be able to use HarmonyMapper
musical sequences in their games without the need for any
alteration. In terms of how these additions are generated,
we could take inspiration from Cardinale and Colton’s
GENRT and generate them stochastically based on a set
of specifications [ 13]. Naturally, these musical additions
are also another facet of the music in which we can try
and promote interesting diversity. For instance, we plan
to produce deliberate diversity during the melody
generation process, by using similar heuristics to the ones
introduced in this paper to make the melody more or less
voice-led.</p>
        <p>
          We also plan to explore using diferent evaluation
metrics for our system. One substantial change we plan
to make is to use a diferent distance metric, such as
Lerdhal and Krumhansl’s model for tonal tension [
          <xref ref-type="bibr" rid="ref12">34</xref>
          ].
Lerdahl suggests that a multidimensional representation
of a pitch space allows for a more nuanced understanding
of tonal relationships and can capture the hierarchical
organization of tonal music [
          <xref ref-type="bibr" rid="ref13">35</xref>
          ]. This could allow us to
understand more in-depth the link between NRT chord
progressions, modes, and moods. This would potentially
also allow us to avoid the use of our duplicate chord
iftness adjustment and the subsequent loss of genetic
information.
        </p>
        <p>Another potential extension to HarmonyMapper
which we are keen to explore is the introduction of the
concept of leitmotifs. Leitmotifs are repeated musical</p>
      </sec>
      <sec id="sec-6-2">
        <title>In this paper, we introduced HarmonyMapper, a novel</title>
        <p>system for generating diverse chord sequences using
MAP Elites, and have argued that a Quality-Diversity
approach for music generation with music theory-inspired
metrics is a valuable basis for game music generation
systems. While the current output is limited to simple
chord progressions, their intentional diversity in terms of
the mood they convey to a listener makes them valuable
as a musical foundation for generating more complex
compositions. We look forward to expanding on this
approach to develop a controllable system for generating
diverse and appropriate music for games.</p>
      </sec>
    </sec>
    <sec id="sec-7">
      <title>Acknowledgements</title>
      <sec id="sec-7-1">
        <title>We thank the reviewers for providing insightful feedback.</title>
        <p>This work has been funded by UKRI and EPSRC as part
of the “UKRI Centre for Doctoral Training in Artificial
Intelligence and Music”, under grant EP/S022694/1, as well
as the EPSRC Centre for Doctoral Training in Intelligent
Games &amp; Games Intelligence (IGGI) [EP/S022325/1].
[5] D. Lewin, A formal theory of generalized tonal cial Intelligence in Music, Sound, Art and Design,
functions, Journal of Music Theory 26 (1982) 23– Springer International Publishing, Cham, 2022, pp.
60. 369–384.
[6] G. Bernardes, D. Cocharro, C. Guedes, M. Davies, [18] L. Cazenille, N. Bredeche, N. Aubert-Kato,
ExHarmony generation driven by a perceptually mo- ploring Self-Assembling Behaviors in a Swarm of
tivated tonal interval space, Computers in Enter- Bio-micro-robots using Surrogate-Assisted
MAPtainment 14(2) (2016). Elites, in: 2019 IEEE Symposium Series on
[7] K. Monteith, T. Martinez, D. Ventura, Automatic Computational Intelligence (SSCI), IEEE,
Xiageneration of music for inducing emotive response, men, China, 2019, pp. 238–246. URL: https:
in: Proceedings of the International Conference on //ieeexplore.ieee.org/document/9003047/. doi:10.</p>
        <p>Computational Creativity, 2010. 1109/SSCI44817.2019.9003047.
[8] K. Monteith, V. Francisco, T. Martinez, P. Gervás, [19] A. Khalifa, S. Lee, A. Nealen, J. Togelius, Talakat:
D. Ventura, Automatic generation of emotionally- bullet hell generation through constrained
maptargeted soundtracks, in: Proceedings of the Inter- elites, in: Proceedings of the Genetic and
Evolutionnational Conference on Computational Creativity, ary Computation Conference, ACM, Kyoto Japan,
2011. 2018, pp. 1047–1054. URL: https://dl.acm.org/doi/
[9] J. Robertson, A. de Quincey, T. Stapleford, G. Wig- 10.1145/3205455.3205470. doi:10.1145/3205455.
gins, Real-time music generation for a virtual 3205470.
environment, in: PROCEEDINGS OF ECAI-98 [20] A. Khalifa, M. C. Green, G. Barros, J. Togelius,
InWORKSHOP ON AI/ALIFE AND ENTERTAIN- tentional Computational Level Design, in:
ProMENT, 1998. ceedings of the Genetic and Evolutionary
Com[10] S. A., Theory of Harmony, University of California putation Conference, GECCO ’19, Association for</p>
        <p>Press, 1983. Computing Machinery, New York, NY, USA, 2019,
[11] L. Duarte, A. E., Algorithmic interactive music pp. 796–803. URL: https://doi.org/10.1145/3321707.
generation in videogames: A modular design for 3321849. doi:10.1145/3321707.3321849,
eventadaptive automatic music scoring, SoundEfects - place: Prague, Czech Republic.</p>
        <p>An Interdisciplinary Journal of Sound and Sound [21] M. C. Fontaine, R. Liu, A. Khalifa, J. Modi, J.
ToExperience 9 (2020) 38–59. doi:10.7146/se.v9i1. gelius, A. K. Hoover, S. Nikolaidis, Illuminating
118245. Mario Scenes in the Latent Space of a Generative
[12] F. Lehman, Film music and Neo-Riemannian The- Adversarial Network, in: 35th AAAI Conference
ory, Oxford Handbook (2014). on Artificial Intelligence, AAAI 2021, 35th AAAI
[13] S. Cardinale, S. Colton, Neo-riemannian theory Conference on Artificial Intelligence, AAAI 2021,
for generative film and videogame music, in: Pro- Association for the Advancement of Artificial
Intelceedings of the Int. Conference on Computational ligence, 2021, pp. 5922–5930.</p>
        <p>Creativity, 2022. [22] J. McCormack, C. C. Gambardella, S. J. Krol,
Cre[14] V. Vassiliades, K. Chatzilygeroudis, J.-B. Mouret, ative discovery using qd search, arXiv preprint
Using Centroidal Voronoi Tessellations to Scale Up arXiv:2305.04462 (2023).
the Multi-dimensional Archive of Phenotypic Elites [23] B. Banar, S. Colton, A Quality-Diversity-Based
Algorithm (2016). URL: https://arxiv.org/abs/1610. Evaluation Straetgy for Symbolic Music
Genera05729. doi:10.48550/ARXIV.1610.05729, pub- tion, 2022.</p>
        <p>lisher: arXiv Version Number: 2. [24] C. L. Krumhansl, M. A. Schmuckler, Key-finding in
[15] J.-B. Mouret, J. Clune, Illuminating search spaces music: An algorithm based on pattern matching to
by mapping elites, arXiv:1504.04909 [cs, q-bio] tonal hierarchies, in: 19th Annual Meeting of the
(2015). URL: http://arxiv.org/abs/1504.04909, arXiv: Society of Mathematical Psychology, Cambridge,
1504.04909. MA, 1986.
[16] J. Nordmoen, F. Veenstra, K. O. Ellefsen, [25] M. S. Cuthbert, C. Ariza, Music21: A toolkit for
K. Glette, MAP-Elites Enables Powerful computer-aided musicology and symbolic music
Stepping Stones and Diversity for Modular data., in: J. S. Downie, R. C. Veltkamp (Eds.), ISMIR,
Robotics, Frontiers in Robotics and AI 8 International Society for Music Information
Re(2021) 639173. URL: https://www.frontiersin. trieval, 2010, pp. 637–642. URL: http://dblp.uni-trier.
org/articles/10.3389/frobt.2021.639173/full. de/db/conf/ismir/ismir2010.html#CuthbertA10.
doi:10.3389/frobt.2021.639173. [26] C. L. Krumhansl, The psychological
rep[17] J. McCormack, C. Cruz Gambardella, Quality- resentation of musical pitch in a tonal
condiversity for aesthetic evolution, in: T. Martins, text, Cognitive Psychology 11 (1979) 346–
N. Rodríguez-Fernández, S. M. Rebelo (Eds.), Artifi- 374. URL: https://www.sciencedirect.com/science/</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>LookingGlass</given-names>
            <surname>Technologies</surname>
          </string-name>
          , System Shock,
          <year>1994</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>Looking</given-names>
            <surname>Glass</surname>
          </string-name>
          <string-name>
            <surname>Studios</surname>
          </string-name>
          ,
          <source>Thief: The Dark Project</source>
          ,
          <year>1998</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>Techland</surname>
          </string-name>
          ,
          <source>Dying Light 2</source>
          ,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>C.</given-names>
            <surname>Plut</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Pasquier</surname>
          </string-name>
          ,
          <article-title>Generative music in video games: State of the art, challenges, and prospects</article-title>
          ,
          <source>Entertainment Computing</source>
          <volume>33</volume>
          (
          <year>2020</year>
          )
          <article-title>100337</article-title>
          . URL: https://linkinghub. elsevier.com/retrieve/pii/S1875952119300795. doi:
          <volume>10</volume>
          .1016/j.entcom.
          <year>2019</year>
          .
          <volume>100337</volume>
          . article/pii/0010028579900161. doi:https://doi. org/10.1016/
          <fpage>0010</fpage>
          -
          <lpage>0285</lpage>
          (
          <issue>79</issue>
          )
          <fpage>90016</fpage>
          -
          <lpage>1</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>V.</given-names>
            <surname>Persichetti</surname>
          </string-name>
          ,
          <article-title>Twentieth century harmony: creative aspects and practice</article-title>
          , W. W. Norton &amp; Company,
          <year>1978</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>R.</given-names>
            <surname>Rawlins</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. E.</given-names>
            <surname>Bahha</surname>
          </string-name>
          ,
          <article-title>Jazzology: the encyclopedia of jazz theory for all musicians</article-title>
          ,
          <source>Hal Leonard Corporation</source>
          ,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>B.</given-names>
            <surname>Prince</surname>
          </string-name>
          , At doom's gate,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [30]
          <string-name>
            <surname>id</surname>
            <given-names>Software</given-names>
          </string-name>
          , Doom,
          <year>1993</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>Z.</given-names>
            <surname>Michalewicz</surname>
          </string-name>
          ,
          <article-title>Do not kill unfeasible individuals</article-title>
          ,
          <source>Proceedings of the Fourth Intelligent Information Systems Workshop (IIS'95)</source>
          (
          <year>1995</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [32]
          <string-name>
            <given-names>G.</given-names>
            <surname>Lai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F. F.</given-names>
            <surname>Leymarie</surname>
          </string-name>
          , W. Latham,
          <article-title>On mixedinitiative content creation for video games</article-title>
          ,
          <source>IEEE Transactions on Games</source>
          <volume>14</volume>
          (
          <year>2022</year>
          )
          <fpage>543</fpage>
          -
          <lpage>557</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [33]
          <string-name>
            <given-names>S.</given-names>
            <surname>Colton</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Cardinale</surname>
          </string-name>
          ,
          <article-title>Extending generative neoriemannian theory for event-based soundtrack production</article-title>
          , in: International Conference on Computational Intelligence in Music, Sound,
          <source>Art and Design (Part of EvoStar)</source>
          , Springer,
          <year>2023</year>
          , pp.
          <fpage>67</fpage>
          -
          <lpage>83</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [34]
          <string-name>
            <given-names>F.</given-names>
            <surname>Lerdahl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. L.</given-names>
            <surname>Krumhansl</surname>
          </string-name>
          , Modeling Tonal Tension,
          <source>Music Perception</source>
          <volume>24</volume>
          (
          <year>2007</year>
          )
          <fpage>329</fpage>
          -
          <lpage>366</lpage>
          . doi:
          <volume>10</volume>
          . 1525/mp.
          <year>2007</year>
          .
          <volume>24</volume>
          .4.329.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [35]
          <string-name>
            <given-names>F.</given-names>
            <surname>Lerdahl</surname>
          </string-name>
          , Tonal pitch space,
          <source>Music Perception: An Interdisciplinary Journal</source>
          <volume>5</volume>
          (
          <year>1988</year>
          )
          <fpage>315</fpage>
          -
          <lpage>349</lpage>
          . URL: http://www.jstor.org/stable/40285402.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>