=Paper=
{{Paper
|id=Vol-3657/paper10
|storemode=property
|title=Teach Me How to Improvise: Co-designing an Augmented Piano Training System for
Improvisation
|pdfUrl=https://ceur-ws.org/Vol-3657/paper10.pdf
|volume=Vol-3657
|authors=Jordan Aiko Deja,Sandi Štor,Ilonka Pucihar,Klen Čopič Pucihar,Matjaž Kljun
|dblpUrl=https://dblp.org/rec/conf/hci-si/DejaSPPK23
}}
==Teach Me How to Improvise: Co-designing an Augmented Piano Training System for
Improvisation==
https://ceur-ws.org/Vol-3657/paper10.pdf
Teach Me How to ImproVISe: Co-Designing an
Augmented Piano Training System for Improvisation
Jordan Aiko Deja1,2,∗ , Sandi Štor3 , Ilonka Pucihar4,5 , Klen Čopič Pucihar1,6,7 and
Matjaž Kljun1,7
1
University of Primorska, Faculty of Mathematics, Natural Sciences and Information Technologies, Koper, Slovenia
2
De La Salle University, Manila, Philippines
3
JazzObala, Portorož, Slovenia
4
Vrhnika Music School, Vrhnika, Slovenia
5
University of Ljubljana, Academy of Music, Ljubljana, Slovenia
6
Faculty of Information Studies, Novo Mesto, Slovenia
7
Stellenbosch University, Department of Information Science, Stellenbosch, South Africa
Abstract
Improvisation is a vital but often neglected aspect of traditional piano teaching. Challenges such as
difficulty in assessment and subjectivity have hindered its effective instruction. Technological approaches,
including augmentation, aim to enhance piano instruction, but the specific application of digital augmen-
tation for piano improvisation is under-explored. This paper outlines a co-design process developing
an Augmented Reality (AR) Piano Improvisation Training System, ImproVISe, involving improvisation
teachers. The prototype, featuring basic improvisation concepts, was created and refined through expert
interaction. Their insights guided the identification of objectives, tools, interaction metaphors, and
software features. The findings offer design guidelines and recommendations to address challenges in
assessing piano improvisation in a learning context.
Keywords
Augmented Reality, Piano, Improvisation, Training System, Music Learning
1. Introduction and Background
Improvisation is an important music skill [1] yet tends to be overlooked within conventional
piano instruction methods [2]. The challenges of assessing and judging improvisation skills,
along with subjective factors, have made it difficult to actually teach musical improvisation
especially for novices. For seasoned piano players, demonstrating improvisational skills signifies
a broad musical vocabulary. Piano teaching experts argue that improvisation supports creativity
of piano learners and performers at all levels. Teaching improvisation to novices and expe-
rienced performers enhances rhythmic accuracy, note-reading, concentration, self-reflection,
imagination, and bolsters confidence [3].
HCI SI 2023: Human-Computer Interaction Slovenia 2023, January 26, 2024, Maribor, Slovenia
Envelope-Open jordan.deja@famnit.upr.si (J. A. Deja); klen.copic@famnit.upr.si (K. Čopič Pucihar); matjaz.kljun@famnit.upr.si
(M. Kljun)
Orcid 0000-0001-9341-6088 (J. A. Deja); 0000-0002-7784-1356 (K. Čopič Pucihar); 0000-0002-6988-3046 (M. Kljun)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Figure 1: Features of ImproVISe: A - Augmented piano space with projector, B - Guided Press, C -
Rolling Improv, D - OnWait Improv Rolling, E - Expert Press visualisations.
It is known that using digital augmentation (e.g. augmented avatars, rolling visualisations) is
effective in teaching musical concepts such as hand-finger-arm synchronisation [4], increasing
motivation [5] and even supporting proper sight-reading [6]. However, to the best of our
knowledge, using augmentation to encourage users to improvise on the piano (as well as
other instruments) remains unexplored [7]. To effectively evaluate this potential, it is crucial to
understand how to properly-design AR training systems for teaching this skill. Considering there
is limited research alone on piano augmentations on improvisation, we posit that involving piano
teachers in a co-design process can help in the development of a training system specifically-
crafted to support improvisation teaching. We borrow from principles and techniques learned
in prior works on co-designing musical instruments and interfaces as seen in [8, 9, 10].
In summary, this paper presents the following contributions: a) narratives from our co-
design process of an AR training system for piano improvisation and its features, which
we refer to as ImproVISe, and b) guidelines and recommendations to address challenges in
assessing piano improvisation in the context of learning.
2. Co-Desigining ImproVISe
In our co-design, we followed a standard iterative procedure, having distinct phases: a) con-
ceptualising the design, b) translating the design into a high-fidelity prototype, c) engaging
with the prototype, and d) refining the prototype based on acquired insights. The first author
spearheaded this process with the mentorship of the fourth and fifth authors. The second and
third authors are the (impro) experts in the domain of jazz/pop/classical improvisation and
teaching improvisation. They have actively participated in all phases of the development of
the prototype, offering valuable feedback, and contributing their expertise to the process of the
designing the training system.
For the initial version of ImproVISe, we designed and implemented an interactive space
inspired by prior works [5, 11]. We connected an overhead projector, a modified Clavinova
with raised keys and a laptop computer in an interactive space (see Figure 1 A). We built a
Figure 2: Some impressions during the co-design sessions: interacting with the prototype, planning the
lessons, giving feedback, identifying lessons and features.
setup where animated visualisations are projected on the surface, and light certain keys on the
Clavinova. These visualisations are programmed in a Unity application specifically designed
for this setup.
The different visualisations projected and animations represent the different lessons and
metaphors in piano improvisation. Together with the impro experts, we considered basic
concepts such as harmony (specifically ii-V-i and ii-V-i-VI progressions), chord tone soloing
(also known as licks), and approaches (e.g., half-step and scale-above). These visualisations
were implemented in various lessons (which we refer to as modes) in the system namely Guided
Press , Rolling Improv , OnWait Roll and Express Press mode.
The Guided Press mode (Figure 1B) displays harmonic progressions and chord tones as
highlighted keys, allowing users to choose which keys to press [12]. In yellow, are the keys in
the harmonic progression while in pink are the “musically-correct” keys that may be pressed
for improvisation. In the Rolling Improv mode (Figure 1C), harmonic progressions appear
as falling piano keys in yellow (similar to [11, 5]), then revealing chord tones (in pink, like
in Guided Press mode) when the right time is triggered. OnWait Rolling mode (Figure 1D)
mirrors Rolling Improv but without the element of timing, allowing the learner to focus on
concept absorption without time pressure but with priming [13]. Finally, Expert Press mode
(Figure 1E) assumes user familiarity with chords, lighting up chord tones (in pink) when a
harmonic chord is recognised. Approaches (e.g. half-step or scale-above) are implemented as
additional guidance across all modes that the user can toggle on or off anytime and appear in a
darker shade of purple.
After implementing the first initial version, we then invited the second and third authors to
an interactive session (another set of phases c) and d)) with ImproVISe (as seen in Figure 2).
The co-design sessions consisted of a mix of different steps such as 1) using the prototype, 2)
interview on best practices and steps, 3) lesson building and 4) mapping of metaphors. Feedback
and insights were collected using multiple modalities (e.g. video, audio, notes, post-it’s). After
every interactive session, we went again through the phases c) and d) resulting to a total of at
least four co-designing sessions.
Table 1
Overview of Learnings and Features from Co-Design Sessions
Objectives Tools Feature Lesson
Learning modes Show different modes,
Lesson 01: Swing
and extensions play modes in swing
Repeat motifs, sequence the motifs,
Understand motifs Lesson 02: Motifs
learn how to form new motifs in Dorian scale.
Be familiar with
Practice with an audio accompaniement,
different rhythmic Lesson 03: Rhythmic patterns
repeat and invent motifs, questions and answers
patterns
Apply and learn a chosen chord progression, Lesson 04: Relationship between
Learn phrases
learn chord tones, repeat phrases over the chords. the melody and harmony
Repeat questions and repeat answers,
Learn basic Lesson 05: Composition
ask question and give your own answer,
composition techniques (Sequence, Q&A, Variation)
apply modes and be familiar with the vocabulary
Apply rhythmic patterns. Lesson 06: Improvise
Apply different styles
use tools and all above lessons (Compose in the moment)
3. Insights Learned and Discussion
The impro experts noted that important musical elements such as timing and patterns were
not very obvious in the initial version of ImproVISe. While different viz modes are adequate
representations of the lessons that they are teaching (e.g. highlights for soloing, piano roll
for timing and priming), being able to teach improvisation requires that the proper context is
provided to the learner. The experts noted that being confident to improvise is akin to how a
non-native speaker becomes confident when speaking a new language. From the repetitive
engage-refine sessions with our impro experts, newer lessons and approaches emerged. These
were then mapped into tools that can be converted into lessons which in turn can be implemented
in the next iteration of the prototype.
From these sessions, ImproVISe had to be re-organised into a different structure involving
not just lessons but also objectives, tools and new content to generate visualisations with. This
in turn will give the learner enough context and practice on how to use them. Table 1 lists
the mapping of these lessons, tools and metaphors. Experts noted that the goal of the system
should support rather than replace traditional teaching methods. In addition, understanding
the context behind each concept should still align with the prescribed theory and principle of
jazz improvisation (e.g. having metronome [14], having musical sheets [15] etc).
The iterative co-design sessions also resulted in specific set of practices that were composed
and recorded into MIDI files by one of the experts. Every time a new set of files are composed
and recorded, they are incorporated into the ImproVISe.
4. Conclusion and Future Work
In this work, we presented the initial features of ImproVISe, an AR training system aimed
towards encouraging piano learners to improvise. We also narrate the learnings from our
co-design sessions which informs the features of the next generation of improvisation training
systems. We intend to complete the prototype with the other features identified and perform a
longitudinal study to determine whether controlled usage of the prototype can support piano
improvisation learners.
Acknowledgments
We thank Matija Ratković for playing the role of a piano user in the videos and photos used in
this paper. This research was funded by the Slovenian Research Agency, grant number P1-0383,
P5-0433, IO-0035, J5-50155 and J7-50096. This work has also been supported by the research
program CogniCom (0013103) at the University of Primorska.
References
[1] J. A. Deja, Encouraging improvisation in piano learning using adaptive visualisations and
spatiotemporal models, in: Adjunct Publication of the 23rd International Conference on
Mobile Human-Computer Interaction, ACM Digital Library, Toulousse, France, 2021, pp.
1–4. doi:https://doi.org/10.1145/3447527.3474865 .
[2] J. A. Deja, S. Mayer, K. Č. Pucihar, M. Kljun, A survey of augmented piano prototypes: Has
augmentation improved learning experiences?, Proceedings of the ACM Human Computer
Interaction Volume ISS (2022) 1–28. doi:10.1145/3567719 .
[3] Y. E. Chyu, Teaching improvisation to piano students of elementary to intermediate levels,
Ph.D. thesis, Ohio State University, 2004.
[4] I. Barakonyi, D. Schmalstieg, Augmented reality agents in the development pipeline of
computer entertainment, in: F. Kishino, Y. Kitamura, H. Kato, N. Nagata (Eds.), Inter-
national Conference on Entertainment Computing, Springer Berlin Heidelberg, Berlin,
Heidelberg, 2005, pp. 345–356. doi:10.1007/11558651_34 .
[5] K. Rogers, A. Röhlig, M. Weing, J. Gugenheimer, B. Könings, M. Klepsch, F. Schaub,
E. Rukzio, T. Seufert, M. Weber, P.i.a.n.o.: Faster piano learning with interactive projection,
in: Proceedings of the Ninth ACM International Conference on Interactive Tabletops and
Surfaces, ITS ’14, Association for Computing Machinery, New York, NY, USA, 2014, p.
149–158. URL: https://doi.org/10.1145/2669485.2669514. doi:10.1145/2669485.2669514 .
[6] P.-Y. Chiang, C.-H. Sun, Oncall piano sensei: Portable ar piano training system, in:
Proceedings of the 3rd ACM Symposium on Spatial User Interaction, SUI ’15, Association
for Computing Machinery, New York, NY, USA, 2015, p. 134. URL: https://doi.org/10.1145/
2788940.2794353. doi:10.1145/2788940.2794353 .
[7] J. A. Deja, Piano learning and improvisation through adaptive visualisation and digital
augmentation, in: Companion Proceedings of the 2022 Conference on Interactive Surfaces
and Spaces, 2022, pp. 41–45.
[8] L. Turchet, M. Barthet, Co-design of musical haptic wearables for electronic music
performer’s communication, IEEE Transactions on Human-Machine Systems 49 (2018)
183–193.
[9] K. G. Chan, J. A. Deja, J. P. Tobias, A. V. Gonzales, M. A. Dancel, Applying user-centered
techniques in the design of a usable mobile musical composition tool, in: Proceedings
of the 5th International ACM In-Cooperation HCI and UX Conference, CHIuXiD’19,
Association for Computing Machinery, New York, NY, USA, 2019, p. 152–159. URL: https:
//doi.org/10.1145/3328243.3328263. doi:10.1145/3328243.3328263 .
[10] P. R. Cook, Remutualizing the musical instrument: Co-design of synthesis algorithms and
controllers, Journal of New Music Research 33 (2004) 315–320.
[11] M. Weing, A. Röhlig, K. Rogers, J. Gugenheimer, F. Schaub, B. Könings, E. Rukzio, M. Weber,
P.i.a.n.o.: Enhancing instrument learning via interactive projected augmentation, in:
Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct
Publication, UbiComp ’13 Adjunct, Association for Computing Machinery, New York, NY,
USA, 2013, p. 75–78. URL: https://doi.org/10.1145/2494091.2494113. doi:10.1145/2494091.
2494113 .
[12] S. Das, S. Glickman, F. Y. Hsiao, B. Lee, Music everywhere–augmented reality piano
improvisation learning system, in: Proceedings of the International Conference on New
Interfaces for Musical Expression, NIME ’17, PubPub, Cambridge, MA, USA, 2017, pp.
511–512.
[13] G. Moro, A. P. McPherson, Performer experience on a continuous keyboard instrument,
Computer Music Journal 44 (2020) 69–91.
[14] P. Dahlstedt, Mapping strategies and sound engine design for an augmented hybrid piano,
in: Proceedings of the International Conference on New Interfaces for Musical Expression,
NIME 2015, The School of Music and the Center for Computation and Technology (CCT),
Louisiana State University, Baton Rouge, Louisiana, USA, 2015, p. 271–276.
[15] F. E. Sandnes, E. Eika, Enhanced learning of jazz chords with a projector based piano
keyboard augmentation, in: International Conference on Innovative Technologies and
Learning, Springer, Cham, 2019, pp. 194–203.