=Paper=
{{Paper
|id=Vol-2068/milc8
|storemode=property
|title=Lumanote: A Real-Time Interactive Music Composition Assistant
|pdfUrl=https://ceur-ws.org/Vol-2068/milc8.pdf
|volume=Vol-2068
|authors=James Granger,Mateo Aviles,Joshua Kirby,Austin Griffin,Johnny Yoon,Raniero Lara-Garduno,Tracy Hammond
|dblpUrl=https://dblp.org/rec/conf/iui/GrangerAKGYLH18
}}
==Lumanote: A Real-Time Interactive Music Composition Assistant==
https://ceur-ws.org/Vol-2068/milc8.pdf
Lumanote: A Real-Time Interactive Music Composition
Assistant
James Granger, Mateo Aviles, Joshua Kirby, Austin Griffin, Johnny Yoon, Raniero Lara-Garduno,
Tracy Hammond
Sketch Recognition Lab
Texas A&M University
ragtimekeys@gmail.com, mateoavls12@hotmail.com, kirbyja721@gmail.com,
austinwgriffin@gmail.com, ydy728@gmail.com, raniero@tamu.edu, hammond@tamu.edu
ABSTRACT
Musical composition typically requires several years of
experience and knowledge in music theory that includes
but is not limited to chord progression, melody note theory,
and an understanding of whole-step/half-step passing tones
among others. Due to the complexity of musical composition,
even experienced musicians may find the writing process
challenging and time consuming. Songwriters may find the
need to employ trained pianists to aid in the theory behind
musical composition. We present Lumanote, an interactive
real-time music composition tool that aids songwriters in
their writing endeavors by presenting real-time suggestions on
appropriate melody notes and chord progression. Lumanote’s
color-coded design provides chords appropriate to chosen
scales, and melody notes appropriate to given chords in
real-time as musicians play chords and melody notes. We Figure 1. A composer playing a chord and one melody note in Lumanote.
performed a qualitative study to gauge the effectiveness of
this tool, including both experienced musicians and complete
beginners to assess the intuitiveness of the system at various
skill levels. Qualitative results helped in identifying potential on the development of automated or semi-automated music
issues on differing interpretations of color-coded suggestions, composition, taking with it some of the agency and in turn
and yielded insights on interface improvements and polish. some of the identity of music creation. Songwriters often hire
the assistance of professional pianists to aid in composing
ACM Classification Keywords a song, providing insights on chord progression and melody
H.5.5 Applied Computing: Sound and music computing; K.3.1 theory to harmonize with the given chords. Since musical
Computers and Education: Computer Uses in Education; composition is inherently a human endeavor, we believe
H.5.2 Information Interfaces and Presentation: User Interfaces significant care must be placed to develop software that helps
composers create melodies they can still consider completely
Author Keywords their own.
Music Composition, Creativity Tools, Music Education
RELATED WORK
INTRODUCTION Tools for Musical Analysis
Computer-aided musical composition remains a promising Demand for digital tools that analyze music has risen in recent
technology for the creation of the newest generation of years. Bello et al. details a system that extracts rhythmic
music. Many current technologies, however, tend to focus and harmonic information without attempting to transcribe the
entirety of a song [2]. Abdallah et al. introduces a model for
extracting structural components such as a verse, the chorus,
and the introduction of a piece of music for the purposes
of cataloging, music summarizing, and song identification
[1]. More recent developments have focused on automating
harmonic analysis using theories of tonal harmony as its
©2018. Copyright for the individual papers remains with the authors. backbone, such as the work of De Haas et al. [5]. While
Copying permitted for private and academic purposes. these systems are successful in applying signal processing
MILC ’18, March 11, 2018, Tokyo, Japan
Figure 2. Lumanote’s main interface, displaying melody note suggestions given a chord input using the "Expert" color palette.
algorithms and musical theory to identify song attributes, they children, and found that students were successful in developing
do not take an active approach in helping composers create music of sound form and structure [11]. Hickey employed
new music. Leveraging the technology behind understanding the researcher-designed "Music Mania" digital composition
the tonal and harmonic composition of a song to assist in tool and found that unobtrusive recording sessions for children
musical composition would greatly enhance the utility of this resulted in more successful inspiration for students over more
previous work. closely guided tasks [7]. Burnard explores the impact of
digital composition tools and online teaching courses on the
Tools for Musical Composition existing pedagogy of instrument and music theory teachings
Several technologies have been developed to help musical [4]. Burnard emphasizes the promise of the viability of
composition through procedural generation, with increasingly utilizing digital composition software for teaching purposes,
more complex methods of input and parameters that affect encouraging both teachers and pupils to remain engaged and
the resulting output for more musically interesting material. communicate both of their needs as research in this field
Diaz-Jerez leveraged his previous technology of FractMus into continues. These tools offer a glimpse on the potential that
a project called Melomics in order to assist in the procedural interactive digital composition tools might have on the learning
creation of music based on biologically inspired musical process. A digital composition tool that offers teaching
evolution [6]. While this produces interesting musical material, opportunities on the basics of authoring music could prove
it takes considerable creative input away from the composer. invaluable.
Hookpad is a composition tool that utilizes music theory
concepts to help composers construct melodies and chord DESIGN
progressions that are known to work well together in specific In this research we present Lumanote, an interactive online
keys 1 . The software, however, relies on a more thorough tool that integrates MIDI input to suggest chords and melodic
understanding of musical composition theory with an interface notes depending on the scale and type of user input. This
that is not particularly intuitive for beginners. Biles presented project was conceived to fill the need to enhance the
the GenJam project, a real-time interactive improvisation tool music composition process for beginner and intermediate
for Jazz music [3]. Its intent is to serve as a companion for composers. The software is intended for users to create
existing Jazz musicians, requiring considerable expertise in music with appropriate chord progressions and melodic
the genre. It is also restricted to the Jazz genre only. Klemenc choices within only minutes of starting to use the software.
has represented music with color for educational purposes Careful consideration of the existing digital composition tools
but does not use it for musical composition [10]. Musical and technologies has yielded the following set of design
recommendation systems include work from Huang et al.[8], requirements:
Simon et al. [12], and Kitahara et al. [9], but none are
designed for real-time suggestions to support composition • Create a practical piece of composition software that can
and improvisation. fully function with minimal cost, installation time and
effort
Teaching Systems
Computerized interactive teaching tools have promisingly • Allow users to interface with the software with or
emerged as low-cost, convenient solutions for learning about without a musical instrument with equivalent levels of
music theory and instrument playing. Nilsson et al. has productivity
explored the integration of professional computer musical
• Intuitively present appropriate chords and accompany-
sequencing programs with the creative composition process of
ing melodic notes to a user who may not have had previous
1 Hookpad: https://www.hooktheory.com/hookpad/new music composition experience
• Guide users toward the creation of a melody without taking Note Highlighting
control from the composer or sense of authorship Users can active Lumanote’s suggestion system in one of two
fashions: play a base chord, or a melody tone. Any keys
System Setup pressed by the user will be denoted by a column of color
In order to minimize software setup requirements, Lumanote that will extend vertically, clearly indicating to users which
is hosted on the website www.lumanote.net for access with keys they are currently playing. If a chord is played that is
any computer that has an active internet connection. It is one of the chords of the chosen musical key, any melody key
recommended that users access the website through the that the user then presses will highlight all adjacent notes
Google Chrome browser. The system’s first screen directs the that are appropriate for that chord and base melody note.
users to an optional video tutorial on how the use the software Similarly, if a melody note is played first, Lumanote highlights
in four minutes. The application then scans the USB input all root chord notes for that note’s accompanying chords.
devices for MIDI-type devices. Users who wish to instead This allows users to create music in both possible forms of
use the software with a computer keyboard or mouse will composition: if the user has a melody as inspiration, they can
be presented with instructions on which keyboard keys are quickly receive suggestions on appropriate chord progression,
mapped with which notes. Afterward, the user will see the and can similarly receive melody note suggestions if their
main interface screen. inspiration is a chord progression. This ensures that Lumanote
accommodates for both forms of the creative composition
process.
Interface Design In musical composition there exist varying degrees of what is
Notes are represented through the standard keyboard layout an "appropriate" note. While some notes are considered "safe"
due to the instrument’s naturally intuitive note representation (one of the chord notes), other notes like whole-step, half-
of one key per note, as well as the ease of mapping a USB step passing tones, and 7th/9th degree notes are used to strain
keyboard’s inputs with that of a digital on-screen keyboard. the melody before its resolution. Lumanote’s note and chord
The top bar controls the mode, musical key, a toggle for suggestion color codes the highlighted notes to indicate which
microphone input for singing, the synthesizer’s sound, and suggested notes are "safer" than others, so that composers
the color palette. The bottom left buttons are six chords that can adeptly alternate between safe notes, passing tones, and
correspond to the selected key. The bottom right buttons strained notes as they compose their music.
include a function to shift the "split point" (the note that The specific color scheme that differentiates between the safety
separates the chord notes with the melodic notes), a "panic" level of suggested notes depends on the "Theme" chosen. The
key that resets the interface who have become confused at their "beginner" theme highlights all in-scale notes the same color,
setting changes or encountered any interface bugs, a function whereas "Intermediate" and "Advanced" modes integrate more
that highlights all the notes of the selected musical key, and colors of the rainbow so that the least "safe" notes are closest
zoom-in and zoom-out buttons to alter how many keys are to red. "Rainforest" highlights the notes in a green and blue
shown on the screen at once. The system is also able to re- gradient, "Ocean" highlights notes in various shades of blue,
center the digital keyboard’s view in case a note that is out of and the "Sunset" palette integrates tones of red, yellow and
the screen’s view is played on the keyboard. purple. Differing color schemes are for the preference of the
composer only and do not affect the suggested notes or their
level of safety.
Musical Composition Process
The musical composition process involves the user exploring
suggested notes, which in turn suggests appropriate chords
and notes that accompany the pressed note. This allows
composers to evolve their composition and progress their
composition from one "step" to the next. Users may begin
the composition with a chord and a single melody note, and
follow the suggested highlighted notes until the user desires to
change the chord. Pressing the single melody tone at that point
will then suggest new chord root notes. The user can then pick
one of the root notes, play the chord appropriate to that root
note, and the new chord combined with the existing melody
note will suggest new melody notes that the composer can
then use. Lumanote works best when the composer utilizes
the suggested chords to select melody notes, and then in turn
Figure 3. All six color Themes depicting the same chord and melody
use those melody notes to select appropriate chords to advance
note input, and the same Lumanote suggestions. The color schemes are the progression of the composition.
as follows from top to bottom: Beginner, Intermediate, Advanced, Rain The musical composition process can be completed with a
Forest, Ocean, Sunset. musical MIDI controller such as a digital keyboard, but can
also be used with a computer typing keyboard. If using
a computer keyboard, users can press the numerical keys FUTURE WORK AND CONCLUSION
from 1 to 6 to have Lumanote automatically play every Lumanote’s powerful and flexible melody and chord
note in the corresponding chord. Users may also interact suggestion system presents significant potential in leveraging
with the software with a mouse by selecting melody notes digital technology to aid composers in creating music quickly
for chord suggestions, hovering over and clicking on listed and efficiently. Careful design decisions were made to place
chords on the bottom left to identify the appropriate chords, all composition decisions on the side of the user, to ensure the
and hovering over the "Keys" button on the lower right to author preserves their identity and can call the composition
highlight all the keys appropriate to any given scale. A singer their own. Future iterations of this system will include a
songwriter can also interface with the system by enabling the menu to select different styles of music, and UI elements
"Sing" option, which takes a singed note via microphone as for beginners to help them understand what makes a "good"
a melodic input and Lumanote will suggest accompanying melody, so that they know how often to pick safe vs. unsafe
chords. The numerous ways to interface with Lumanote have notes.
been developed with the intention to provide as flexible a piece
of software as possible. REFERENCES
1. Samer Abdallah, Katy Noland, Mark Sandler, Michael A.
Casey, and Christophe Rhodes. 2005. Theory and
SYSTEM EVALUATION
Evaluation of a Bayesian Music Structure Extractor. In
Composition Activity International Conference on Music Information Retrieval,
A system evaluation comprised of a user study of 20 Josh Reiss and Geraint Wiggins (Eds.). London, 420–425.
participants. Of these, 12 were non-pianist musicians, 6 were 2. Juan Pablo Bello and Jeremy Pickens. 2005. A Robust
pianist musicians, and 2 were non-musicians. They wwere Mid-Level Representation for Harmonic Content in
first asked to create a "simple melody" without the use of Music Signals.. In ISMIR, Vol. 5. 304–311.
Lumanote. We did not provide specifics on melody type, 3. John A Biles. 2002. Genjam: Evolutionary computation
key, or composition. Non-musicians had the most difficulty gets a gig. In Proceedings of the 2002 Conference for
with creating a melody without using Lumanote, generally Information Technology Curriculum, Rochester, New
creating melodies that had no structure and followed no key. York, Society for Information Technology Education.
4. Pamela Burnard. 2007. Reframing creativity and
More experienced musicians required some time to remember
technology: Promoting pedagogic change in music
specific chord progressions.
education. Journal of Music, Technology & Education 1,
After each participant completed what they considered a
1 (2007), 37–55.
simple melody, they were asked to repeat the process using
5. W Bas De Haas, José Pedro Magalhães, Frans Wiering,
Lumanote. They were given the choice to start with a new
and Remco C Veltkamp. 2013. Automatic functional
melody or refine their existing work. Nearly all participants
harmonic analysis. Computer Music Journal 37, 4 (2013),
opted to start a new composition. They were given no
37–53.
instructions on how to use the system in order for us to 6. Gustavo Diaz-Jerez. 2011. Composing with Melomics:
gauge its level of intuitiveness. Researchers answered the Delving into the computational world for musical
participants’ questions when asked. All participants, including inspiration. Leonardo Music Journal 21 (2011), 13–14.
experienced users, showed considerable improvement in the 7. Maud Hickey. 1997. The computer as a tool in creative
quality and musicality of their composition. music making. Research Studies in Music Education 8, 1
(1997), 56–70.
Suggestions and Changes 8. Cheng-Zhi Anna Huang, David Duvenaud, and
A four-minute instructional video was created as a need Krzysztof Z Gajos. 2016. Chordripple: Recommending
to address the most common questions from participants, chords to help novice composers go beyond the ordinary.
including questions about color scheme, explanation of the In Proceedings of the 21st International Conference on
Split Point line, questions about genre, and choosing notes. We Intelligent User Interfaces. ACM, 241–250.
initially had one single color palette, but soon observed most 9. Tetsuro Kitahara, Satoru Fukayama, Shigeki Sagayama,
users had different interpretations of colors for suggestions. Haruhiro Katayose, and Noriko Nagata. 2011. An
This prompted us to develop various color palettes to suit more interactive music composition system based on
individual’s personal interpretations. autonomous maintenance of musical consistency. In Proc.
Several UI additions were made for various use cases. Key Sound and Music Computing.
widths can now be changed to accommodate various computer 10. Bojan Klemenc, Peter Ciuha, and Franc Solina. 2011.
screen sizes and resolutions, the Split Point line location can Educational possibilities of the project Colour
be now changed at the request of more advanced participants, visualization of music. Organizacija 44, 3 (2011), 67–75.
and hovering over each chord to display the notes was added 11. Bo Nilsson and Göran Folkestad. 2005. Children’s
at the request of the intermediate users who otherwise needed practice of computer-based composition. Music
reminders on which chords to play. Finally, we added non- Education Research 7, 1 (2005), 21–37.
MIDI input devices with keyboard, mouse, and note singing 12. Ian Simon, Dan Morris, and Sumit Basu. 2008. MySong:
since participants frequently reported not having a MIDI automatic accompaniment generation for vocal melodies.
keyboard at their disposal if they were to use this system In Proceedings of the SIGCHI Conference on Human
at home. Factors in Computing Systems. ACM, 725–734.