Teach Me How to ImproVISe: Co-Designing an Augmented Piano Training System for Improvisation Jordan Aiko Deja1,2,∗ , Sandi Štor3 , Ilonka Pucihar4,5 , Klen Čopič Pucihar1,6,7 and Matjaž Kljun1,7 1 University of Primorska, Faculty of Mathematics, Natural Sciences and Information Technologies, Koper, Slovenia 2 De La Salle University, Manila, Philippines 3 JazzObala, Portorož, Slovenia 4 Vrhnika Music School, Vrhnika, Slovenia 5 University of Ljubljana, Academy of Music, Ljubljana, Slovenia 6 Faculty of Information Studies, Novo Mesto, Slovenia 7 Stellenbosch University, Department of Information Science, Stellenbosch, South Africa Abstract Improvisation is a vital but often neglected aspect of traditional piano teaching. Challenges such as difficulty in assessment and subjectivity have hindered its effective instruction. Technological approaches, including augmentation, aim to enhance piano instruction, but the specific application of digital augmen- tation for piano improvisation is under-explored. This paper outlines a co-design process developing an Augmented Reality (AR) Piano Improvisation Training System, ImproVISe, involving improvisation teachers. The prototype, featuring basic improvisation concepts, was created and refined through expert interaction. Their insights guided the identification of objectives, tools, interaction metaphors, and software features. The findings offer design guidelines and recommendations to address challenges in assessing piano improvisation in a learning context. Keywords Augmented Reality, Piano, Improvisation, Training System, Music Learning 1. Introduction and Background Improvisation is an important music skill [1] yet tends to be overlooked within conventional piano instruction methods [2]. The challenges of assessing and judging improvisation skills, along with subjective factors, have made it difficult to actually teach musical improvisation especially for novices. For seasoned piano players, demonstrating improvisational skills signifies a broad musical vocabulary. Piano teaching experts argue that improvisation supports creativity of piano learners and performers at all levels. Teaching improvisation to novices and expe- rienced performers enhances rhythmic accuracy, note-reading, concentration, self-reflection, imagination, and bolsters confidence [3]. HCI SI 2023: Human-Computer Interaction Slovenia 2023, January 26, 2024, Maribor, Slovenia Envelope-Open jordan.deja@famnit.upr.si (J. A. Deja); klen.copic@famnit.upr.si (K. Čopič Pucihar); matjaz.kljun@famnit.upr.si (M. Kljun) Orcid 0000-0001-9341-6088 (J. A. Deja); 0000-0002-7784-1356 (K. Čopič Pucihar); 0000-0002-6988-3046 (M. Kljun) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Figure 1: Features of ImproVISe: A - Augmented piano space with projector, B - Guided Press, C - Rolling Improv, D - OnWait Improv Rolling, E - Expert Press visualisations. It is known that using digital augmentation (e.g. augmented avatars, rolling visualisations) is effective in teaching musical concepts such as hand-finger-arm synchronisation [4], increasing motivation [5] and even supporting proper sight-reading [6]. However, to the best of our knowledge, using augmentation to encourage users to improvise on the piano (as well as other instruments) remains unexplored [7]. To effectively evaluate this potential, it is crucial to understand how to properly-design AR training systems for teaching this skill. Considering there is limited research alone on piano augmentations on improvisation, we posit that involving piano teachers in a co-design process can help in the development of a training system specifically- crafted to support improvisation teaching. We borrow from principles and techniques learned in prior works on co-designing musical instruments and interfaces as seen in [8, 9, 10]. In summary, this paper presents the following contributions: a) narratives from our co- design process of an AR training system for piano improvisation and its features, which we refer to as ImproVISe, and b) guidelines and recommendations to address challenges in assessing piano improvisation in the context of learning. 2. Co-Desigining ImproVISe In our co-design, we followed a standard iterative procedure, having distinct phases: a) con- ceptualising the design, b) translating the design into a high-fidelity prototype, c) engaging with the prototype, and d) refining the prototype based on acquired insights. The first author spearheaded this process with the mentorship of the fourth and fifth authors. The second and third authors are the (impro) experts in the domain of jazz/pop/classical improvisation and teaching improvisation. They have actively participated in all phases of the development of the prototype, offering valuable feedback, and contributing their expertise to the process of the designing the training system. For the initial version of ImproVISe, we designed and implemented an interactive space inspired by prior works [5, 11]. We connected an overhead projector, a modified Clavinova with raised keys and a laptop computer in an interactive space (see Figure 1 A). We built a Figure 2: Some impressions during the co-design sessions: interacting with the prototype, planning the lessons, giving feedback, identifying lessons and features. setup where animated visualisations are projected on the surface, and light certain keys on the Clavinova. These visualisations are programmed in a Unity application specifically designed for this setup. The different visualisations projected and animations represent the different lessons and metaphors in piano improvisation. Together with the impro experts, we considered basic concepts such as harmony (specifically ii-V-i and ii-V-i-VI progressions), chord tone soloing (also known as licks), and approaches (e.g., half-step and scale-above). These visualisations were implemented in various lessons (which we refer to as modes) in the system namely Guided Press , Rolling Improv , OnWait Roll and Express Press mode. The Guided Press mode (Figure 1B) displays harmonic progressions and chord tones as highlighted keys, allowing users to choose which keys to press [12]. In yellow, are the keys in the harmonic progression while in pink are the “musically-correct” keys that may be pressed for improvisation. In the Rolling Improv mode (Figure 1C), harmonic progressions appear as falling piano keys in yellow (similar to [11, 5]), then revealing chord tones (in pink, like in Guided Press mode) when the right time is triggered. OnWait Rolling mode (Figure 1D) mirrors Rolling Improv but without the element of timing, allowing the learner to focus on concept absorption without time pressure but with priming [13]. Finally, Expert Press mode (Figure 1E) assumes user familiarity with chords, lighting up chord tones (in pink) when a harmonic chord is recognised. Approaches (e.g. half-step or scale-above) are implemented as additional guidance across all modes that the user can toggle on or off anytime and appear in a darker shade of purple. After implementing the first initial version, we then invited the second and third authors to an interactive session (another set of phases c) and d)) with ImproVISe (as seen in Figure 2). The co-design sessions consisted of a mix of different steps such as 1) using the prototype, 2) interview on best practices and steps, 3) lesson building and 4) mapping of metaphors. Feedback and insights were collected using multiple modalities (e.g. video, audio, notes, post-it’s). After every interactive session, we went again through the phases c) and d) resulting to a total of at least four co-designing sessions. Table 1 Overview of Learnings and Features from Co-Design Sessions Objectives Tools Feature Lesson Learning modes Show different modes, Lesson 01: Swing and extensions play modes in swing Repeat motifs, sequence the motifs, Understand motifs Lesson 02: Motifs learn how to form new motifs in Dorian scale. Be familiar with Practice with an audio accompaniement, different rhythmic Lesson 03: Rhythmic patterns repeat and invent motifs, questions and answers patterns Apply and learn a chosen chord progression, Lesson 04: Relationship between Learn phrases learn chord tones, repeat phrases over the chords. the melody and harmony Repeat questions and repeat answers, Learn basic Lesson 05: Composition ask question and give your own answer, composition techniques (Sequence, Q&A, Variation) apply modes and be familiar with the vocabulary Apply rhythmic patterns. Lesson 06: Improvise Apply different styles use tools and all above lessons (Compose in the moment) 3. Insights Learned and Discussion The impro experts noted that important musical elements such as timing and patterns were not very obvious in the initial version of ImproVISe. While different viz modes are adequate representations of the lessons that they are teaching (e.g. highlights for soloing, piano roll for timing and priming), being able to teach improvisation requires that the proper context is provided to the learner. The experts noted that being confident to improvise is akin to how a non-native speaker becomes confident when speaking a new language. From the repetitive engage-refine sessions with our impro experts, newer lessons and approaches emerged. These were then mapped into tools that can be converted into lessons which in turn can be implemented in the next iteration of the prototype. From these sessions, ImproVISe had to be re-organised into a different structure involving not just lessons but also objectives, tools and new content to generate visualisations with. This in turn will give the learner enough context and practice on how to use them. Table 1 lists the mapping of these lessons, tools and metaphors. Experts noted that the goal of the system should support rather than replace traditional teaching methods. In addition, understanding the context behind each concept should still align with the prescribed theory and principle of jazz improvisation (e.g. having metronome [14], having musical sheets [15] etc). The iterative co-design sessions also resulted in specific set of practices that were composed and recorded into MIDI files by one of the experts. Every time a new set of files are composed and recorded, they are incorporated into the ImproVISe. 4. Conclusion and Future Work In this work, we presented the initial features of ImproVISe, an AR training system aimed towards encouraging piano learners to improvise. We also narrate the learnings from our co-design sessions which informs the features of the next generation of improvisation training systems. We intend to complete the prototype with the other features identified and perform a longitudinal study to determine whether controlled usage of the prototype can support piano improvisation learners. Acknowledgments We thank Matija Ratković for playing the role of a piano user in the videos and photos used in this paper. This research was funded by the Slovenian Research Agency, grant number P1-0383, P5-0433, IO-0035, J5-50155 and J7-50096. This work has also been supported by the research program CogniCom (0013103) at the University of Primorska. References [1] J. A. Deja, Encouraging improvisation in piano learning using adaptive visualisations and spatiotemporal models, in: Adjunct Publication of the 23rd International Conference on Mobile Human-Computer Interaction, ACM Digital Library, Toulousse, France, 2021, pp. 1–4. doi:https://doi.org/10.1145/3447527.3474865 . [2] J. A. Deja, S. Mayer, K. Č. Pucihar, M. Kljun, A survey of augmented piano prototypes: Has augmentation improved learning experiences?, Proceedings of the ACM Human Computer Interaction Volume ISS (2022) 1–28. doi:10.1145/3567719 . [3] Y. E. Chyu, Teaching improvisation to piano students of elementary to intermediate levels, Ph.D. thesis, Ohio State University, 2004. [4] I. Barakonyi, D. Schmalstieg, Augmented reality agents in the development pipeline of computer entertainment, in: F. Kishino, Y. Kitamura, H. Kato, N. Nagata (Eds.), Inter- national Conference on Entertainment Computing, Springer Berlin Heidelberg, Berlin, Heidelberg, 2005, pp. 345–356. doi:10.1007/11558651_34 . [5] K. Rogers, A. Röhlig, M. Weing, J. Gugenheimer, B. Könings, M. Klepsch, F. Schaub, E. Rukzio, T. Seufert, M. Weber, P.i.a.n.o.: Faster piano learning with interactive projection, in: Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces, ITS ’14, Association for Computing Machinery, New York, NY, USA, 2014, p. 149–158. URL: https://doi.org/10.1145/2669485.2669514. doi:10.1145/2669485.2669514 . [6] P.-Y. Chiang, C.-H. Sun, Oncall piano sensei: Portable ar piano training system, in: Proceedings of the 3rd ACM Symposium on Spatial User Interaction, SUI ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 134. URL: https://doi.org/10.1145/ 2788940.2794353. doi:10.1145/2788940.2794353 . [7] J. A. Deja, Piano learning and improvisation through adaptive visualisation and digital augmentation, in: Companion Proceedings of the 2022 Conference on Interactive Surfaces and Spaces, 2022, pp. 41–45. [8] L. Turchet, M. Barthet, Co-design of musical haptic wearables for electronic music performer’s communication, IEEE Transactions on Human-Machine Systems 49 (2018) 183–193. [9] K. G. Chan, J. A. Deja, J. P. Tobias, A. V. Gonzales, M. A. Dancel, Applying user-centered techniques in the design of a usable mobile musical composition tool, in: Proceedings of the 5th International ACM In-Cooperation HCI and UX Conference, CHIuXiD’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 152–159. URL: https: //doi.org/10.1145/3328243.3328263. doi:10.1145/3328243.3328263 . [10] P. R. Cook, Remutualizing the musical instrument: Co-design of synthesis algorithms and controllers, Journal of New Music Research 33 (2004) 315–320. [11] M. Weing, A. Röhlig, K. Rogers, J. Gugenheimer, F. Schaub, B. Könings, E. Rukzio, M. Weber, P.i.a.n.o.: Enhancing instrument learning via interactive projected augmentation, in: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, UbiComp ’13 Adjunct, Association for Computing Machinery, New York, NY, USA, 2013, p. 75–78. URL: https://doi.org/10.1145/2494091.2494113. doi:10.1145/2494091. 2494113 . [12] S. Das, S. Glickman, F. Y. Hsiao, B. Lee, Music everywhere–augmented reality piano improvisation learning system, in: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME ’17, PubPub, Cambridge, MA, USA, 2017, pp. 511–512. [13] G. Moro, A. P. McPherson, Performer experience on a continuous keyboard instrument, Computer Music Journal 44 (2020) 69–91. [14] P. Dahlstedt, Mapping strategies and sound engine design for an augmented hybrid piano, in: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2015, The School of Music and the Center for Computation and Technology (CCT), Louisiana State University, Baton Rouge, Louisiana, USA, 2015, p. 271–276. [15] F. E. Sandnes, E. Eika, Enhanced learning of jazz chords with a projector based piano keyboard augmentation, in: International Conference on Innovative Technologies and Learning, Springer, Cham, 2019, pp. 194–203.