The rhythm or groove in music is an intuitive concept that is really hard to put into practice without a good level of exercise. It consists of the skills needed by a player for using a rhythm sequence that is repeated inside the song temporal flow, maintaining the coherence in the sound progression. All musicians must learn or develop these skills since it is required for mastering an instrument.
Usually, these skills are developed through self-study and occasional lessons with a music teacher. Therefore, in most of the time spent in practising the student is alone and this leads to situations where mistakes are not noticed by students.
The evolution of interaction modalities that are currently supported by the web opens this environment to more advanced applications for learning music theory. In this work, we introduce SnAIR Drum, a gestural interface combined with score visualisation for practising the drums. The application interface supports the user in learning how to follow the specified rhythm, how to read the score relying on the body coordination.
The idea of applying gestural interaction for recognising movements related to the percussive instruments has been widely investigated in the literature. The recognition technology has a huge impact on the effectiveness of these interfaces since the precision and the latency has a key role in their usability. Tindale et al. [7] discuss different tracking techniques and provide an evaluation of their pros and cons. Bott et al. [1] used a general purpose gesture input device such as the Nintendo WiiMote for controlling playing a drum game, studying the spatial positioning of the interactive elements for remotes. A similar attempt with a different gestural device (Microsoft Kinect) is discussed in [2]. The designed interface allows the collaboration of more than one musician.
Commercial applications for playing the drums through gestural interaction exist in the market. In particular, we can mention here AirBeats [3], a Leap Motion-based application that allows users to create rhythms with different styles: Hip-Hop, Dubstep, Trance etc. AirBeats allows using fingers and sticks for playing virtual pads with mid-air movements.
In this work we set a different objective for the interface: we do not focus on the expressiveness of the musician, but on the exploitation of the gestural input for teaching purposes, including score reading.
SnAIR Drum is a web application that on relies on gestural input for completing percussion exercises. It combines both a simulation of the spatial positioning of the different parts, together with the visualisation of the score and a support for reading it during the exercise.
The SnAIR drum interface is shown in Figure 2. In the left part, it visualises the different drum pieces, while the right part shows the score for the exercise. The user plays the virtual drum with his/her fingertips: the movements are shown in Figure 1. In order to support the coordination between the hand movements and the effects on the virtual pads, the interface shows a stylised representation of the user's hands (see Figure 2). The drum parts positioning tries to balance two different needs: the first is the correspondence between the positioning of a piece in a real drum set and in the virtual environment. The second is the gestural tracking device limitation, that provides information only on the hand movements. Therefore, we cannot rely on legs for controlling e.g., the kick and the hit-hat.
We positioned the drum pieces in three concentric circles (see Figure 2, left part): in the inner one, we positioned those controlled with feet and the snare, in the middle one the toms, while cymbals are in the outer circle. The interaction with the pieces is very simple: each one is contained in a box. When one of the fingertips touches one these boxes, the application emits the corresponding piece sound and changes the colour of the corresponding bounding box. The exercise support works as follows. The user loads through a menu the audio file containing the exercise musical base and the score of the drum part in the song. When the user pushes the play button (Figure 2 top part), the application plays the exercise base and the score representation highlights the current note that should be played by the user. According to the user's input, the tool shows a green feedback when if the user played the right piece at the right time, while it shows a red icon otherwise. In Figure 2 we show an example of error feedback.
The adherence of the user to the score is computed automatically by the application defining time buffers around the note duration. The scheme is shown in Figure 3: according to the note duration and to the song tempo, the application creates a buffer around the note duration, in order to be tolerant to small imperfections in the user's performance. If the user beats the correct piece in the green interval, the application considers the execution correct, and wrong otherwise. It is worth pointing out that there may be differences between the time when the user performs the movement and the instant sensed by the application due to input and elaboration latency, as we discuss in the following section.
A demo of the application is available at the following URL: https://youtu.be/P9Gm5CEAvI4.
The implementation consists of four main components, which control the main elements of the interface.
The first is the gestural input component, based on the Leap Motion SDK [5]. It supports the hand tracking, providing a hand-skeleton abstraction. The SDK input stream updates the position and orientation of the on-screen stylised representation of the user's hands.
The 3D elements in the user interface rely on Three.JS [6], a high-level Javascript library for managing WebGL [4] contents inside HTML pages. This component draws the hand representation, the cubes related to the drum kit parts, detecting the collision with the fingertips.
The music score and playing component controls the on screen score visualisation, the execution feedback, the base and the drum playing. The visualization exploits the Abcweb.js library [8]. It is able to read the XML definition of the score and to synchronise it with a musical base, moving a cursor on the different notes according to song tempo.
From the same score description, the application derives a queue of events corresponding to the correct execution of the song, which is the temporal distribution of the green areas in Figure 3. This event queue, together with the ones raised by the input device, is used for checking the user's performance.
In this paper, we introduced SnAIR Drum, a web-based application for supporting rhythm exercises through gestural interaction. It is built on top of standard web technologies and it is easy to distribute as teaching material, for complementing existing text, video and audio files.
The application is designed for supporting a proper music training, including means for enhancing the physical coordination and the user's ability to read and understand the score. In addition, it supports an automated performance checking that provides feedback to the learner.
In future work, we aim at refining the platform, adding the content authoring part. Complementing this part of the application would enable the creation of an exercise sharing platform for promoting the instrument learning. Finally, we are planning to evaluate the exercise platform with different user tests with both student and music teachers.