-This paper describes the design and the implementation approach of a piano training application. HoloKeys is an Augmented Reality tool which is capable to superimpose the keys to be played on a real piano. Musical pieces are loaded as MIDI files, interpreted and can be displayed in two different ways. This prototype provides many possibilities for extension which can make it a powerful teaching tool.
Augmented Reality (AR), described by Azuma as a
technology where the user sees ’the real world, with virtual objects
superimposed upon or composited with the real world’ [
We present HoloKeys, a prototypical implementation of an AR training tool for learning the piano. HoloKeys runs on an HMD which the user is wearing while sitting in front of a physical piano. The application indicates notes that are supposed to be played by displaying virtual keys superimposing the physical keyboard with two different approaches. Acquiring the musical data dynamically by loading and processing MIDI (Musical Instrument Digital Interface) files, the application is fully agnostic considering the musical pieces to be trained. To achieve the required precision for the augmentations on the piano, the application was implemented using fiducial marker tracking. Since this application is a prototype, an extensive collection of possible enhancements and prospects for the future is given.
The remainder of this paper is structured as follows: The next chapter provides an overview of the related work in music teaching applications. Chapter III will introduce the conceptual design of the application describing the architecture and the user interface. Implementation details are provided in Chapter IV. Finally conclusions are drawn and an outlook into the future work is given.
Music education has a long tradition in the field of AR.
In an early approach Cheng and Robinson provided a visual
sheet music overlay displayed planar in the visual field of the
user. The display of the augmentations is triggered when he
looks at the hands. The type of sheet is depending on which
hand he looks. The augmentation is not registered (meaning
it is not directly spatially interconnected) to a real object
opposed to the approach presented in this publication. An
HMD is used for display [
In order to avoid the use of fiducial markers on the piano
Huang et al. use their knowledge on the application domain
and track the keys of the piano for pose estimation with
the help of natural feature recognition [
Chow et al. focus on the educational level of AR piano
teaching showing that with the help of augmentations and
gamification components the motivation and interest in
learning the piano could be increased. They provided a system
illustrating the notes to be played by lines approaching the
keys. Their findings also indicate that notation literacy does not
increase using their system of illustration [
Opposed to this visualisation approach Torres-Fernandez et
al. introduce a virtual character which illustrates how well the
piano player has performed. To interpret the played music they
compare the input from a MIDI keyboard with an initially
loaded MIDI file [
In terms of visualisation Weing et al. demonstrate a system
in the area of Spatial Augmented Reality where they project
the keys to be pressed directly on the piano. Different modes
show for example the current and the next keys to be pressed.
If a wrong key is pressed it is highlighted in red to provide
feedback to the user [
Zhang et al. use a completely virtual keyboard and track the
hand of the user with fiducial markers and the finger positions
with a self-developed data glove. Their approach targets the
rehabilitation of the motor function of stroke survivors rather
than teaching the piano [
Compared to these existing and presented approaches our system is unique in terms of used display technology.
The application’s setup is illustrated in Fig. 1 and consists of the following two hardware components.
1) The Piano: The core component is a physical piano which is used for the actual playing. Underneath the piano keyboard which is usually made of 88 keys a fiducial marker is placed which is used by the application for tracking. The keys of a regular piano are standardized in size which makes the application fully independent considering the type of piano. In case a keyboard is used the key width can be adjusted.
the piano and wears an HMD on which the application runs. Through the HMD the user sees augmentations in the form of highlighted keys on top of the real keyboard. The HMD also handles tracking by recognizing the image marker with the help of computer vision algorithms. The HMD therefore keeps track of the player’s position and displays the augmentations accordingly. Additionally, the HMD is responsible for sound output of the music to be played. This gives the user an impression on how the piece is supposed to sound and makes it easier to play along with it.
In order to manage different settings and control the
playback, a simple user interface was implemented. The originally
two-dimensional UI is placed inside the 3D scene using
worldstabilized coordinates. Considering the usually static setup of
the application with the user sitting in front of the piano, the
world-stabilized menu is a reasonable approach [
1) The Main Menu: The initial scene of the application is the main menu. There the user can select the musical piece to play as well as the desired playback speed. By pressing the start button the application will switch to playback mode and begin visualizing and playing the musical piece.
2) Playback Mode: In playback mode the user sees the augmentations of the keys to be played superimposing the physical keyboard. Additionally a timeline shows the current playback position and gives the user the option to jump to different positions inside the piece. With the pause button the user is able to interrupt the playback or return to the main menu.
3) Calibration Mode: In calibration mode the application displays an augmentation of only one key, the middle C. The user can adjust the position of the marker until the virtual key perfectly fits the real one. This is useful to setup the optimal position of the marker on the piano. Additionally the user can also adjust the pitch of the virtual piano sound in calibration mode because this does not necessarily match with the real piano. Playback volume can be adjusted in the HMD.
Generally the HMD displays an augmentation of a bright green key to indicate that the actual key on that position has to be pressed. Two different approaches as seen in Fig. 2 were tested and both have their advantages and disadvantages concerning predictability and Field Of View (FOV) limitations.
1) The Instant Approach: The moment a key is supposed to be pressed it becomes highlighted. Once it is supposed to be released it switches back to normal. This way the user can more or less observe the playing of the piece in real-time, comparable to watch the fingers of an actual pianist. While this approach can be useful for advanced players, it is hardly possible to learn a new piece or even to play along with it, because the player has no way of predicting the next notes. Still, observing this looks great and could be used for showcase purposes (self-playing piano), as the limited FOV is also less of a problem there.
2) The Beatmania Approach: Note objects are created far in
the distance and from there start moving towards the particular
keys. As soon as the virtual object reaches the real key, the
note should be played. With this approach, which became
popular with the game ’Beatmania’ [
This chapter goes into detail regarding the concrete implementation of HoloKeys. It starts with a brief overview of used hardware and software tools followed by an in-depth description of the two main development tasks, visualization and MIDI processing.
The application was developed for tablet devices as well as the HoloLens. The tablet approach is mainly used for demonstration purposes, rather than actual training.
• HoloLens1
The HoloLens as a current AR HMD provides good sensory support as well as spatial audio and stereoscopic display capabilities. Its main disadvantage the limited FOV poses an issue to the applicability of this use case. 2) Software: To allow cross-platform and cross-device development the following set of tools and libraries was used. • Unity2
Unity is traditionally a game engine which has found
wide adoption in the whole domain of Mixed Reality
[
The Augmented Reality part of the project is based on Vuforia, an AR tracking library which perfectly integrates 1https://www.microsoft.com/en-us/hololens 2https://unity3d.com/ 3https://www.vuforia.com/ with Unity. Vuforia supports several different tracking methods ranging from recognizing plain images to complex objects. With a specific setup, Vuforia can also be used on the HoloLens. • C# Synth Project and MIDI Support4
The C# Synth Project is an open-source library which is used for processing MIDI data and synthesizing it to audio data. MIDI is an industry standard for interconnection between musical instruments and digital devices. Its file format represents musical information like notes values, volume and tempo. Although MIDI is a complex format, it is still the most popular and commonly used format to store musical data. For piano pieces the format is usually sufficient because only one channel is required to store a series of notes and tempo changes.
The application’s visuals consist of a Unity 3D scene which renders the virtual keys, combined with Vuforia’s tracking abilities to provide the information on where to render the keys.
1) Vuforia’s image target: For this application tracking via fiducial marker and image target was used. The image target in Unity is a planar object in 3D space which is associated with a set of 2D images. These images represent the markers that are placed somewhere in the real world. Once the camera recognizes a marker the application can trace back the position of the HMD and can therefore project all augmented objects accordingly.
2) Tracking setup: Marker images and other tracking settings can be configured in Vuforia’s web interface. This configuration with all related assets is then compiled into a Unity package that can be imported into Unity after that. In Unity two components of Vuforia, ARCamera and ImageTarget, are used. Subordinate objects of the ImageTarget become affected by the marker-related projection.
3) Generating the keyboard: In order to display the currently played keys, first an entire virtual keyboard is displayed half-transparently superimposing the real one. A script takes care of automatically generating all 88 key objects. One base key object is placed in the scene and aligned at around 90 degrees relative to the ImageTarget. This registration has to match with the real world relation between marker and piano keyboard. All other keys are then generated as duplicates of the base object with respective offset and color (black or white).
The two core components of the C# Synth Project library are the MidiSequencer which handles loading and processing MIDI data and the MidiStreamSynthesizer which handles the actual audio playback.
4https://csharpsynthproject.codeplex.com/ 1) Handling key actions: During playback the MidiSequencer fires two events that are relevant for this application: MidiNoteOn and MidiNoteOff. These two events are respectively fired when the playback of a note is triggered or terminated and therefore indicate exactly the time when a key is pressed and released. In the implementations of these two event handlers the MIDI code of the affected note is passed as a parameter. The only operation is to map this MIDI code to our according key object and set its material color to either green (in NoteOn) or the default color (in NoteOff).
sizer creates actual audio data based on the sequencer’s input. To make sure that this audio data is actually redirected to Unity’s audio source, the special method OnAudioFilterRead has to be implemented. This method supports direct writing into the audio buffer and therefore redirect the contents of the StreamSynthesizer to Unity’s audio source.
As a prototype the application serves well, but due to the limited FOV, which will most likely increase in the next years with the following generations of AR hardware, its real world usage could be doubted. Furthermore, an evaluation of the different augmentation methods would be useful. Especially when trying out a few more possible approaches, a user test could find out which of the methods are most likely to work in a real-world scenario. A more in-depth study of musical augmentation methods would also be useful for teaching other instruments or even in completely different areas of music.
A long-term vision could be the creation of a full-featured virtual piano teacher using AR. Especially early-stage piano learning contains many tasks that could be implemented with AR technologies like the one explained in this paper combined with gamification elements.
• Learning notes and the piano keyboard
Simple exercises or games to recognize the note names and match it with the proper keys could really increase the early-stage learning rate. For beginners the note names could be augmented on top of every key until they become familiar with it. • Learning easy to intermediate musical pieces Especially for smaller pieces the AR learning approach could surpass traditional learning by music sheets. Beginners who are not used to reading music yet, would still be able to learn pieces quickly on their own. Additionally a lot more useful information like fingering, expression and dynamics could be displayed during playback. • Technical exercises
The importance of regular technical exercises for piano students is huge but generally underestimated and disliked. With the introduction of AR and gamification, a whole lot of enjoyable and still pianistically valuable exercises could be realized. By adding some sort of level system, the student would be even more aware of his progress and more likely to remain motivated. • Dictionary of chords, scales etc.
A very useful utility not only for beginners but also for advanced pianists would be a piano dictionary. The player could look up all possible chords and scales and would be able to see them highlighted right on top of his keyboard. Especially for jazz piano where complex chords and scales are common, this technology would be of great service.
• Using music sheets as markers
The use of music sheets, perhaps in the form of a special music book, as fiducial markers could eliminate the need for additional markers placed on the piano. It could not only automatically detect the musical piece to be played but also indicate, when to turn the sheets or even highlight musical attributes on the sheets. • Checking the learning performance
Real-time feedback of the user’s playing could greatly
contribute to the learning experience. This could be
achieved on the one hand by using MIDI keyboards
to directly receive the MIDI input of pressed keys or
on the other hand by recording and deconstructing the
audio data. The first approach would be technologically
straight-forward but would limit the application to
electronic keyboard instruments while the second approach
would be more flexible but complicated to implement and
perhaps inaccurate [
The possibilities of the virtual piano teacher are enormous but all are based on the core concept of the technique explained in this paper. As soon as there are improvements in AR hardware, especially concerning FOV, virtual piano teachers can be implemented and actually start to become a helpful tool.