A Web-Based System For Suggesting New Practice Material To Music Learners Based On Chord Content Johan Pauwels Mark B. Sandler Centre for Digital Music Centre for Digital Music Queen Mary University of London Queen Mary University of London j.pauwels@qmul.ac.uk mark.sandler@qmul.ac.uk ABSTRACT educational context, music students are presented with new In this demo paper, a system that suggests new practice material in class and are then expected to rehearse on their material to music learners is presented. It is aimed at music own by repeating a small number of pieces. The reason for practitioners of any skill set, playing any instrument, as long this limited number of pieces is purely practical. It is hard to as they know how to play along with a chord sheet. Users find suitable material that is representative for a particular need to select a number of chords in a web app, and are technique or musical concept without resorting to compos- then presented with a list of music pieces containing those ing pieces especially for that purpose. Current music search chords. Each of those pieces can then be played back while interfaces are not very suitable for the process of finding its chord transcription is displayed in sync to the music. practice material. While it is easy to find specific songs using This enables a variety of practice scenarios, ranging from meta-data or audio fingerprinting, discovering new music following the chords in a piece to using the suggested music based on musical content is not directly supported. as a backing track to practice soloing over. We set out the However, repeating the same pieces over and over again various interface elements that make up this web application can make practising monotonous and disengaging. At the and the thoughts that went behind them. Furthermore, we same time, presenting the technique under study in more and touch upon the algorithms that are used in the app. Notably, varied contexts is likely to improve the effectiveness of the the automatic generation of chord transcriptions – such that rehearsal. For these reasons, we are investigating how music large amounts of music can be processed without human pieces can be recommended to learners based on their choice intervention – and the query resolution mechanism – finding of a set of chords as input. The main motivation behind the appropriate music based on the user input and transcription system we created is to answer the question: “What other quality – are discussed. music can I play with the chords I already know?” Chords are chosen for querying because they are musi- CCS CONCEPTS cally speaking on a high enough level of abstraction such • Applied computing → Sound and music computing; that multiple pieces can match a query and because they Education; • Human-centered computing → Web-based make multiple practice scenarios possible. Users can for in- interaction. stance play along with the chords to practice transitioning between them or improve their improvisation skills. We be- KEYWORDS lieve this to be a novel type of music recommendation, as music recommendation, music education, web application it is traditionally based on lower-level features or collabo- rative filtering [3]. Meanwhile, other applications of chord ACM Reference Format: recognition are song-centric, meaning that you first decide Johan Pauwels and Mark B. Sandler. 2019. A Web-Based System For Suggesting New Practice Material To Music Learners Based On which song you want to learn and then retrieve its chords, Chord Content. In Joint Proceedings of the ACM IUI 2019 Workshops, without chord-based recommendation [4]. Los Angeles, USA, March 20, 2019 , 4 pages. In this paper, we present a web-based system that suggest music to users based on their selection of chords. We start 1 INTRODUCTION by discussing the architecture of the application in section 2, Regular practice is an intrinsic part of music education and followed by a breakdown of the various screens and user absolutely necessary in order to improve. In a traditional interface elements in section 3. We then move on to the algorithms involved in the back-end of the app in section 4 and end with a conclusion and some points for future work IUI Workshops’19, March 20, 2019, Los Angeles, USA © 2019 Copyright for the individual papers by the papers’ authors. Copying in section 5. permitted for private and academic purposes. This volume is published and copyrighted by its editors. IUI Workshops’19, March 20, 2019, Los Angeles, USA Johan Pauwels and Mark B. Sandler 2 APPLICATION ARCHITECTURE The web app consists of a client-side front-end written in plain JavaScript as a single-page application, such that transi- tions between the three views that the app is comprised of are seamless. The server-side back-end consists of two APIs that are called in sequence. The first is our own query API that returns suggested music pieces based on the user input. This API returns an ordered list of identifiers and their chord transcriptions. These identifiers are then passed on to the API of a music content provider to retrieve the corresponding metadata, artwork and streaming audio. The front-end is built on the Bootstrap 3 framework1 with a custom developed theme coded as SASS2 rules. Our back- end is a light-weight API server built in Python using the Flask3 framework. It is little more than a single – albeit com- Figure 1: The query screen with the Cmajor – triads & tetrads plicated – call to a MongoDB4 database containing the chord preset activated. transcriptions, which then get returned as JSON5 . 3 USER INTERFACE ELEMENTS and to minimise the number of clicks when selecting com- mon combinations, we decided to offer presets that select all As mentioned earlier, the user interface consists of three chords in a given key. screens: a query screen where the user makes a selection of The available keys are formed by combining a major or chords, a list of results presenting music pieces that contain minor mode with all combinations of seven natural tonics (A, the requested chords and a music player showing the chords B, C, D, E, F, G) and three accidentals (♮, ♭, ♯). For each of these in sync with the music for every piece. The current version keys, the diatonic triads, diatonic tetrads or a combination of of the interface was developed based on the user feedback both can be selected. In total this gives a total of 2 × 7 × 3 × and user observations we got from an earlier prototype [7]. 3 = 126 presets. After activating a key preset, the resulting Each of the three screens will now be presented in further selection can be edited further. For instance, you can start detail. from the chords in a key and replace some that you don’t Query Screen want to practice with other ones not in that key. The usage of key presets is illustrated in Figure 1. As shown in Figure 1, users are presented with a grid of chord Selecting a key preset only results in a number of chords names on which they can click to add that chord to their being activated. This key information is not passed in the selection (and click again to remove it). The grid contains query and therefore won’t give different results compared to sixty chord names, organised as the twelve possible roots manually selecting the same chords, nor guarantee that the in the columns and five chord types in the rows. The chord returned music is in that key. Notably, any major key preset types are major, minor, dominant seventh, major seventh gives the same results as its relative minor key. and minor seventh in that order such that simpler triads Furthermore, for simplicity all chords are currently spelled are displayed above more complex triads. These particular with flats, both in the query screen and in the chord tran- five chord types were chosen because they came out as the scriptions that are returned as results. In the future, we plan most popular types in an analysis of popular music [2] and to make the spelling adapt automatically to the selected key together they cover almost the entire duration of that corpus. or to user preferences for sharps or flats when making a free Users can make a free selection of chord combinations. selection. Similarly, user preferences for chord type format- However, during our first trials of the interface, we noticed ting will be taken into account in the future, e.g. using △ that users mostly wanted to select chords that go well to- instead of maj7. gether. Depending on their level of musical skill, recalling good combinations could involve some effort. Therefore, Results List After submitting a chord selection to the system, users are 1 https://getbootstrap.com/ 2 https://sass-lang.com/ directed to the results screen shown in Figure 2. By clicking 3 http://flask.pocoo.org/ on individual results, users are taken to a player screen which 4 https://www.mongodb.com/ allows to play back the song and its chords. Additionally, 5 https://www.json.org/ more results can be loaded (not depicted in the figure) as Suggesting New Practice Material To Music Learners IUI Workshops’19, March 20, 2019, Los Angeles, USA Figure 2: The list of results for a particular chord query. Figure 3: The music player displaying the chords in sync with the music. long as there are more pieces corresponding to the query. Results are returned in batches of five by the query API, overview of the chord changes by indicating them as coloured but the front-end checks whether the corresponding music sections on the waveform. pieces are still available from the content provider API before Metadata and cover art of the piece is displayed under the displaying them, which can result in smaller, irregular batch music player followed by the set of chords appearing in the sizes. piece. The latter is shown because the chords of the piece do The reason for the unavailability of a piece through the not necessarily correspond exactly to the chords selected for provider API is typically that the music is temporarily or per- the query. The matching algorithm will try to find music that manently removed from the catalogue in the time between contains all of the requested chords, but failing that, might the indexation of the chord transcriptions and the user’s propose a partial match. Especially when a large number of query. Ideally, the indexation would keep track of changes chords is selected this might be the case. in the catalogue and remove pieces from the chord transcrip- Not shown in the image is that below the chords in the tion database when they disappear from the content provider, piece, the other results of the query are shown in a way sim- which would lead to a more consistent number of results ilar to the initial list of results. This can be used to navigate being returned. However, for the time being the indexation easily to other results of the query. Alternatively, you can go requires manual intervention and has only been performed back to the full list of results by clicking the text at the top once at the start of development. of the page. Finally, users can also return to the query screen in order to modify their chord selection. 4 UNDERLYING ALGORITHMS Two algorithms are working in tandem to suggest practice Music Player material to the users: an automatic chord estimator that The player screen, as depicted in Figure 3, displays the chord generates chord transcriptions for a large dataset beforehand, transcription as text labels in boxes that scroll by in time and a query matching mechanism that finds appropriate with the music. The current chord label is displayed bigger pieces based on the transcriptions. and has a progress indicator at the bottom of its box. This allows users to anticipate the timing of a chord change and Chord Estimation Algorithm makes it easier to play along. The chord estimation algorithm is used to create a large col- Below the chord boxes, a waveform is shown using the lection of chord transcriptions. Because a large collection of wavesurfer.js6 library. It gives a wider overview of the cur- transcriptions is needed to make all (or at least most) queries rent position with respect to the overall track duration and successful and creating this collection manually would be clicking on it allows you to move forwards and backwards too time consuming, we resorted to automatic chord esti- through the music. In the future, we’d like to give a global mation. Furthermore, since the transcriptions need to be time-aligned to the audio, machine-readable and available 6 https://wavesurfer-js.org/ for less popular music too, this makes it impossible to use IUI Workshops’19, March 20, 2019, Los Angeles, USA Johan Pauwels and Mark B. Sandler any of the existing commercial or crowd-sourced chord sheet for genre or difficulty (measured as the rate of harmonic repositories. change, for instance). However, the current state of the art for automatic chord estimation is far from perfect, so transcription errors can be 5 CONCLUSION AND FUTURE WORK expected. Therefore we use an algorithm [6] that outputs a In this paper, we demonstrated a web-based application that confidence measure in addition to the chord transcription. recommends new music to learners for practising their in- This confidence in the quality of the transcription will be struments with those pieces. Users need to make a selection taken into account when suggesting music pieces to users. of chords and the system then returns a list of music pieces The current version of the app is built by processing 100K that contain them. A variety of practice scenarios are en- tracks of the Jamendo7 catalogue, a music platform that cen- visioned, from beginners playing along with the chords to tralises Creative Commons music. They also provide an API8 more advanced learners playing arpeggios or soloing over to retrieve metadata, cover art and streaming audio from. the changes. Work is ongoing to scale up the size of the underlying dataset With this interface, we plan to perform an extensive user and to include other music catalogues into the application. study to evaluate the quality of the underlying algorithms, both the automatic chord estimation and the query resolu- Query Matching Algorithm tion, and the holistic system, including the user interface as The query matching algorithm is running in realtime every a whole. time a query is made by a user. It aims to find music pieces Further planned improvements include changing the for- that are the best match for the requested chords, but also bal- mat of the chord labels according to user preferences or ances this with the chord transcription quality as predicted depending on key, thereby adding support for transposing by the confidence measure in order to give the best possible instruments. Another potential topic for research is how to user experience. Details of the algorithm can be found in [5]. assist users in the free exploration of chord selections and The decision was made to only return pieces that contain no if providing visual hints for chord combinations that are other chords than the ones that present in the user selection. common in the data would be useful for that. The reasoning is that beginners might not know how to play In its current state, the system just recommends music to all possible chords yet and that returning pieces containing play along with, but it could be extended to a more complete chords they do not know how to play would lead to a bad music education app by letting it listen to the user’s per- user experience. An option for more advanced musicians to formance and return automatic scoring and feedback. The include one or more wildcard chords in the query is planned guitar-specific system presented in [1] may provide a way for a future version. towards this goal. A consequence of this decision is that the more chords REFERENCES are selected, the larger the subset of music pieces is that [1] Shunya Ariga, Masataka Goto, and Koji Yatani. 2017. Strummer: An the system can suggest. Therefore the query screen includes interactive guitar chord practice system. In Proceedings of the IEEE the instruction to select at least three chords. This is not a International Conference on Multimedia and Expo (ICME). technical requirement, but selecting a single chord would [2] John Ashley Burgoyne, Jonathan Wild, and Ichiro Fujinaga. 2011. An Ex- give the user music pieces consisting only of a single chord, pert Ground-Truth Set for Audio Chord Recognition and Music Analysis. not very suitable for practising, and therefore lead to an In Proceedings of the 12th ISMIR Conference. [3] Òscar Celma. 2010. Music Recommendation and Discovery: The Long Tail, unsatisfactory experience. Long Fail, and Long Play in the Digital Music Space. Springer-Verlag Preference is given to music pieces that contain as much of Berlin Heidelberg. the requested chords as possible. There might not be a music [4] W. Bas de Haas, José Pedro Magalhães, Dion ten Heggeler, Gijs piece available in the dataset with the exact chord combina- Bekenkamp, and Tijmen Ruizendaal. 2014. Chordify: Chord Transcrip- tion that’s requested, especially if the selected chords do not tion for the Masses. In Proceedings of the 13th ISMIR Conference, Late Breaking and Demo Session. go well together musically, but pieces with the largest subset [5] Johan Pauwels, György Fazekas, and Mark B. Sandler. 2018. Recom- of possible chords will be suggested in that case, all while mending songs to music learners based on chord content. In Proceedings considering the confidence to return high quality transcrip- of the 2018 Joint Workshop on Machine Learning for Music. tions. [6] Johan Pauwels, Ken O’Hanlon, György Fazekas, and Mark B. Sandler. We’d like to give users more control over the matching 2017. Confidence Measures and Their Applications in Music Labelling Systems Based on Hidden Markov Models. In Proceedings of the 18th algorithm in the future, by letting them specify preferences ISMIR Conference. 279–285. [7] Johan Pauwels, Anna Xambó, Gerard Roma, Mathieu Barthet, and György Fazekas. 2018. Exploring Real-time Visualisations to Support 7 https://www.jamendo.com/ Chord Learning with a Large Music Collection. In Proceedings of the 4th 8 https://developer.jamendo.com/v3.0/docs Web Audio Conference.