The availability of standard devices for gestural interaction supports the distribution on the web of interactive teaching material for learning music. In this paper, we focus on rhythm practice, introducing an exercise platform that exploits gestural interaction for tracking the user's movements and a web-based environment for loading exercises and checking his/her performance. We discuss the design and the technical implementation of the solution.
Copyright is held by the authors/owner(s). CHItaly ’17, September 18-20, 2017, Cagliari, Italy.
The rhythm or groove in music is an intuitive concept that is really hard to put into practice without a good level of exercise. It consists of the skills needed by a player for using a rhythm sequence that is repeated inside the song temporal flow, maintaining the coherence in the sound progression. All musicians must learn or develop these skills since it is required for mastering an instrument.
Usually, these skills are developed through self-study and occasional lessons with a music teacher. Therefore, in most of the time spent in practising the student is alone and this leads to situations where mistakes are not noticed by students.
The evolution of interaction modalities that are currently supported by the web opens this environment to more advanced applications for learning music theory. In this work, we introduce SnAIR Drum, a gestural interface combined with score visualisation for practising the drums. The application interface supports the user in learning how to follow the specified rhythm, how to read the score relying on the body coordination.
The idea of applying gestural interaction for recognising
movements related to the percussive instruments has been
widely investigated in the literature. The recognition
technology has a huge impact on the effectiveness of these
interfaces since the precision and the latency has a key role
in their usability. Tindale et al. [
Commercial applications for playing the drums through
gestural interaction exist in the market. In particular, we can
mention here AirBeats [
SnAIR Drum is a web application that on relies on gestural input for completing percussion exercises. It combines both a simulation of the spatial positioning of the different parts, together with the visualisation of the score and a support for reading it during the exercise.
The SnAIR drum interface is shown in Figure 2. In the left part, it visualises the different drum pieces, while the right part shows the score for the exercise. The user plays the virtual drum with his/her fingertips: the movements are shown in Figure 1. In order to support the coordination between the hand movements and the effects on the virtual pads, the interface shows a stylised representation of the user’s hands (see Figure 2). The drum parts positioning tries to balance two different needs: the first is the correspondence between the positioning of a piece in a real drum set and in the virtual environment. The second is the gestural tracking device limitation, that provides information only on the hand movements. Therefore, we cannot rely on legs for controlling e.g., the kick and the hit-hat.
We positioned the drum pieces in three concentric circles (see Figure 2, left part): in the inner one, we positioned those controlled with feet and the snare, in the middle one the toms, while cymbals are in the outer circle. The interaction with the pieces is very simple: each one is contained in a box. When one of the fingertips touches one these boxes, the application emits the corresponding piece sound and changes the colour of the corresponding bounding box.
The exercise support works as follows. The user loads through a menu the audio file containing the exercise musical base and the score of the drum part in the song. When the user pushes the play button (Figure 2 top part), the application plays the exercise base and the score representation highlights the current note that should be played by the user. According to the user’s input, the tool shows a green feedback when if the user played the right piece at the right time, while it shows a red icon otherwise. In Figure 2 we show an example of error feedback.
The adherence of the user to the score is computed automatically by the application defining time buffers around the note duration. The scheme is shown in Figure 3: according to the note duration and to the song tempo, the application creates a buffer around the note duration, in order to be tolerant to small imperfections in the user’s performance. If the user beats the correct piece in the green interval, the application considers the execution correct, and wrong otherwise. It is worth pointing out that there may be differences between the time when the user performs the movement and the instant sensed by the application due to input and elaboration latency, as we discuss in the following section. A demo of the application is available at the following URL: https://youtu.be/P9Gm5CEAvI4.
The implementation consists of four main components,
which control the main elements of the interface.
The first is the gestural input component, based on the
Leap Motion SDK [
The 3D elements in the user interface rely on Three.JS [
The music score and playing component controls the on
screen score visualisation, the execution feedback, the
base and the drum playing. The visualization exploits the
Abcweb.js library [
In this paper, we introduced SnAIR Drum, a web-based application for supporting rhythm exercises through gestural interaction. It is built on top of standard web technologies and it is easy to distribute as teaching material, for complementing existing text, video and audio files.
The application is designed for supporting a proper music training, including means for enhancing the physical coordination and the user’s ability to read and understand the score. In addition, it supports an automated performance checking that provides feedback to the learner.
In future work, we aim at refining the platform, adding the content authoring part. Complementing this part of the application would enable the creation of an exercise sharing platform for promoting the instrument learning. Finally, we are planning to evaluate the exercise platform with different user tests with both student and music teachers.