Improvisation is a vital but often neglected aspect of traditional piano teaching. Challenges such as dificulty in assessment and subjectivity have hindered its efective instruction. Technological approaches, including augmentation, aim to enhance piano instruction, but the specific application of digital augmentation for piano improvisation is under-explored. This paper outlines a co-design process developing an Augmented Reality (AR) Piano Improvisation Training System, ImproVISe, involving improvisation teachers. The prototype, featuring basic improvisation concepts, was created and refined through expert interaction. Their insights guided the identification of objectives, tools, interaction metaphors, and software features. The findings ofer design guidelines and recommendations to address challenges in assessing piano improvisation in a learning context.
CEUR ceur-ws.org
Improvisation is an important music skill [
It is known that using digital augmentation (e.g. augmented avatars, rolling visualisations) is
efective in teaching musical concepts such as hand-finger-arm synchronisation [
In summary, this paper presents the following contributions: a) narratives from our codesign process of an AR training system for piano improvisation and its features, which we refer to as ImproVISe, and b) guidelines and recommendations to address challenges in assessing piano improvisation in the context of learning.
In our co-design, we followed a standard iterative procedure, having distinct phases: a) conceptualising the design, b) translating the design into a high-fidelity prototype, c) engaging with the prototype, and d) refining the prototype based on acquired insights. The first author spearheaded this process with the mentorship of the fourth and fith authors. The second and third authors are the (impro) experts in the domain of jazz/pop/classical improvisation and teaching improvisation. They have actively participated in all phases of the development of the prototype, ofering valuable feedback, and contributing their expertise to the process of the designing the training system.
For the initial version of ImproVISe, we designed and implemented an interactive space
inspired by prior works [
The diferent visualisations projected and animations represent the diferent lessons and metaphors in piano improvisation. Together with the impro experts, we considered basic concepts such as harmony (specifically ii-V-i and ii-V-i-VI progressions), chord tone soloing (also known as licks), and approaches (e.g., half-step and scale-above). These visualisations were implemented in various lessons (which we refer to as modes) in the system namely Guided Press, Rolling Improv, OnWait Roll and Express Press mode.
The Guided Press mode (Figure 1B) displays harmonic progressions and chord tones as
highlighted keys, allowing users to choose which keys to press [12]. In yellow, are the keys in
the harmonic progression while in pink are the “musically-correct” keys that may be pressed
for improvisation. In the Rolling Improv mode (Figure 1C), harmonic progressions appear
as falling piano keys in yellow (similar to [
After implementing the first initial version, we then invited the second and third authors to an interactive session (another set of phases c) and d)) with ImproVISe (as seen in Figure 2). The co-design sessions consisted of a mix of diferent steps such as 1) using the prototype, 2) interview on best practices and steps, 3) lesson building and 4) mapping of metaphors. Feedback and insights were collected using multiple modalities (e.g. video, audio, notes, post-it’s). After every interactive session, we went again through the phases c) and d) resulting to a total of at least four co-designing sessions. Apply and learn a chosen chord progression, Lesson 04: Relationship between learn chord tones, repeat phrases over the chords. the melody and harmony aaRspekppleqyautmeqsotudioeesnstiaaonnndds gbainevdefarymeopiuleiraaortwawnnitshawnteshrwse,evro,cabulary (LSeesqsounen0c5e:, CQo&mAp,oVsaitriioantion) Apply rhythmic patterns. Lesson 06: Improvise use tools and all above lessons (Compose in the moment)
The impro experts noted that important musical elements such as timing and patterns were not very obvious in the initial version of ImproVISe. While diferent viz modes are adequate representations of the lessons that they are teaching (e.g. highlights for soloing, piano roll for timing and priming), being able to teach improvisation requires that the proper context is provided to the learner. The experts noted that being confident to improvise is akin to how a non-native speaker becomes confident when speaking a new language. From the repetitive engage-refine sessions with our impro experts, newer lessons and approaches emerged. These were then mapped into tools that can be converted into lessons which in turn can be implemented in the next iteration of the prototype.
From these sessions, ImproVISe had to be re-organised into a diferent structure involving not just lessons but also objectives, tools and new content to generate visualisations with. This in turn will give the learner enough context and practice on how to use them. Table 1 lists the mapping of these lessons, tools and metaphors. Experts noted that the goal of the system should support rather than replace traditional teaching methods. In addition, understanding the context behind each concept should still align with the prescribed theory and principle of jazz improvisation (e.g. having metronome [14], having musical sheets [15] etc).
The iterative co-design sessions also resulted in specific set of practices that were composed and recorded into MIDI files by one of the experts. Every time a new set of files are composed and recorded, they are incorporated into the ImproVISe.
In this work, we presented the initial features of ImproVISe, an AR training system aimed towards encouraging piano learners to improvise. We also narrate the learnings from our co-design sessions which informs the features of the next generation of improvisation training systems. We intend to complete the prototype with the other features identified and perform a longitudinal study to determine whether controlled usage of the prototype can support piano improvisation learners.
We thank Matija Ratković for playing the role of a piano user in the videos and photos used in this paper. This research was funded by the Slovenian Research Agency, grant number P1-0383, P5-0433, IO-0035, J5-50155 and J7-50096. This work has also been supported by the research program CogniCom (0013103) at the University of Primorska. Association for Computing Machinery, New York, NY, USA, 2019, p. 152–159. URL: https: //doi.org/10.1145/3328243.3328263. doi:10.1145/3328243.3328263. [10] P. R. Cook, Remutualizing the musical instrument: Co-design of synthesis algorithms and controllers, Journal of New Music Research 33 (2004) 315–320. [11] M. Weing, A. Röhlig, K. Rogers, J. Gugenheimer, F. Schaub, B. Könings, E. Rukzio, M. Weber, P.i.a.n.o.: Enhancing instrument learning via interactive projected augmentation, in: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, UbiComp ’13 Adjunct, Association for Computing Machinery, New York, NY, USA, 2013, p. 75–78. URL: https://doi.org/10.1145/2494091.2494113. doi:10.1145/2494091. 2494113. [12] S. Das, S. Glickman, F. Y. Hsiao, B. Lee, Music everywhere–augmented reality piano improvisation learning system, in: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME ’17, PubPub, Cambridge, MA, USA, 2017, pp. 511–512. [13] G. Moro, A. P. McPherson, Performer experience on a continuous keyboard instrument,
Computer Music Journal 44 (2020) 69–91. [14] P. Dahlstedt, Mapping strategies and sound engine design for an augmented hybrid piano, in: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2015, The School of Music and the Center for Computation and Technology (CCT), Louisiana State University, Baton Rouge, Louisiana, USA, 2015, p. 271–276. [15] F. E. Sandnes, E. Eika, Enhanced learning of jazz chords with a projector based piano keyboard augmentation, in: International Conference on Innovative Technologies and Learning, Springer, Cham, 2019, pp. 194–203.