TCSL at the MediaEval 2014 C@merata Task Nikhil Kini Tata Consultancy Services Ltd. Innovation Labs, Thane nikhil.kini@tcs.com ABSTRACT separated by a comma. For example, "quarter note then half note We describe a system to address the MediaEval 2014 C@merata then quarter note in the tenor voice" is output as "(DUR, quarter task of natural language queries on classical music scores. Our note) (SEQ, then) (DUR, half note) (SEQ, then) (DUR, quarter system first tokenizes the question to tag the musically relevant note) in the (PRT, tenor voice)". Another example is "melodic features in the question using pattern matching. In this stage octave" becomes "(HRML, melodic) (INT, octave)". suitable word replacements are made in the question based on a list of synonyms. Using the tokenized sentence we infer the question type using a set of handwritten rules. We then search the input music score based on the question type to find the musical features requested. MIT's music21 library [2] is used for indexing, accessing and traversing the score. 1. INTRODUCTION To those interested in studying music scores, and especially in the field of Musicology, it is often necessary to search for, or refer to, particular sections of a music score that represent relevant musical features, just as a data scientist might look for interesting patterns in an abundance of data. Manually going through the score is prone to inefficiency, oversight, and requires the expert knowledge of actually understanding the score. This paper aims to develop specifications for tools that automate the task of search and retrieval of musical passages from a score with natural language queries. The problem may be defined as: given a Figure 1. System for natural language sheet music querying computer representation of music as a score in a particular format (in our case, musicXML), and given a short English noun phrase 2.2 Synonyms List referring to musical features in the score, search and list the A list of synonyms is referred to during tokenizing for substituting location of all occurrences of the said musical features in the words that refer to the same feature. This serves two purposes: 1) score. A complete description of the task can be found at [1]. to cover all manners of asking for the same feature and 2) standardizing the different ways of asking for the same thing so While a large body of work is available in other domains for that specifying the subsequent modules becomes simpler. The list Natural language understanding, as well as for searching through of synonyms can be updated as new ways of asking the same sheet music scores, we did not come across any work that feature are discovered when users actually query the system. combines these aspects. A natural language understanding systems survey may be found in [3], and work done in non-trivial 2.3 Inferring the question type search on sheet music scores can be found in [4], [5], [6], [7]. The tokenized output (with synonym list substitution) is the input to the module which infers the question type. A handcrafted set of 2. APPROACH rules was used to guess what type of question is asked based on Figure 1 presents the main modules of our system. Since we treat the constituent tokens (see section 2.4). Looking at all questions the problem as one of natural language understanding (of the available to us so far - task description, training set, test set - we question) and searching (through the musicXML), we define a set specify the following types of questions: simple note , note with of question classes based on the searchable musical features, and expression, interval (harmonic), interval (melodic), lyrics, extrema propose a specific search method for each type of question. The (highest or lowest note), time signature, key signature, cadence, main operations performed by our system are as follows: triads, texture, bar with dynamics, consecutive notes, combination of the above. 2.1 Identifying tokens in the question 2.4 Question rules In the tokenizing step, words representing musically important Based on the tokens present in the question phrase, we can write features are marked/tokenized. We use 3 or 4 letter markers for rules to guess the type of the question. For simple questions made the token class. After tokenization, the sentence will contain up of only one question this is straightforward. For the phrases tokens grouped with the value of the token, each token-value pair which contain a combination of elementary question types, some grouped by parentheses, and the token and the value will be parsing capability might be necessary. We will address this in future work. Copyright is held by the author/owner(s). MediaEval 2014 Workshop, October 16-17, 2014, Barcelona, Spain. 2.5 Search scope is a type with our system too, and the system performs decently on An important part to getting the right answer is limiting the search both these types. scope. For example, in the question "A sharp in the Treble clef", Although search was not implemented for harmonic_interval, we are not just looking for any A#, but particularly in the Treble cadence_spec, triad_spec and texture_spec, nearly all questions clef. Our tokens PRT and CLF can be used to scope the search. for these types were correctly classified by our system. No We look only in these parts during searching or we filter only answers were returned for these types of questions, which results those search results as answers which are within this search scope. in the zero scores seen in the table. Only 8 questions remained unclassified in the test data. 2.6 Searching for the answer The last step is searching the musicXML score for the identified 4. CONCLUSION token/token combination. This step is still a work in progress. We The system implemented based on the specifications in this paper make extensive use of music21 capabilities. performs decently on single musical feature retrieval. A study of the errors in this implementation might even be able to take the 2.7 Score index precision and recall for such simple types to 1, and this will be the This is a list of all the notes in the score stored with the following aim of the next cycle of development. associated information for each note: note name, note letter, accidental, pitch class, note octave, bar, offset, note length, part While our system performs well on the simple question phrases, number, part id and whether this is a rest or a note. (This the more complex question phrases still need work. As a question terminology is as defined in music21). grows more complicated to include multiple musical features, we will need to evolve a more complex parsing strategy to identify questions. It is possible that the specification of the system will 3. RESULTS AND DISCUSSION need to be revisited to take into account all the possibilities. Upon release of the results, we saw that the organizers had also used a scheme of classification for the questions. Reconciling the The scope of the system specification is limited mainly to what we organizers' and our question types, we saw that have observed in the task description and the training set, and as far as the test questions go, we had all possibilities covered. these are in no way exhaustive of the types of queries that can be asked. Table 1. Test set results Beat Measure # Question type P R P R 5. ACKNOWLEDGMENTS Many thanks to Dr. Sunil Kumar Kopparapu, my supervisor, for 1 simple_length 0.979 0.988 0.991 1 his help in shaping this paper. 2 simple_pitch 0.959 0.963 0.982 0.986 3 pitch_and_length 0.723 0.892 0.754 0.93 4 stave_spec 0.661 0.987 0.661 0.987 6. REFERENCES 5 melodic_interval 0.894 0.683 0.904 0.691 [1] R. Sutcliffe, T. Crawford, C. Fox, D. L. Root and E. Hovy. The C@merata Task at MediaEval 2014: Natural language 6 followed_by 0.733 0.688 0.842 0.789 queries on classical music scores. In MediaEval 2014 7 word_spec 0.261 1 0.261 1 Workshop, Barcelona, Spain, October 16-17 2014. 8 perf_spec 0.066 0.897 0.066 0.897 9 harmonic_interval 0 0 0 0 [2] Cuthbert, M. S., & Ariza, C. 2010. music21: A toolkit for 10 cadence_spec 0 0 0 0 computer-aided musicology and symbolic music data. In ISMIR 2010, Utrecht, Netherlands, August 9-13 2010 (637- 11 triad_spec 0 0 0 0 642). 12 texture_spec 0 0 0 0 13 all 0.633 0.821 0.652 0.845 [3] Allam, A. M. N., & Haggag, M. H. 2012. The Question Our classes corresponding to the organizers' "Question type" in Answering Systems: A Survey. In International Journal of Table 1 are: 1, 2, 3 - simple note, 4 - simple note with expression, Research and Reviews in Information Sciences 5 - simple note with a staff scope, 6 - simple note with a lyrics (IJRRIS), 2(3). scope, 7 - consecutive notes, 8 - interval (melodic), 9 - interval [4] Gabriel, J. 2013. Large data sets & recommender systems: A (harmonic), 10 - cadence, 11- triad, 12 - texture. feasible approach to learning music. In Proceedings of the Table 1 shows beat and measure precision recall scores for results Sound and Music Computing Conference 2013, SMC 2013, produced by our system for the test set. The strongest performance Stockholm, Sweden, p.701–706 . is seen in the 'simple notes' category (simple pitch, simple length, [5] Downie, J. S. 1999. Evaluating a simple approach to music pitch and length). This is no surprise as these question phrases are information retrieval: Conceiving melodic n-grams as text the easiest to handle. Perf_spec questions are simple note with (Doctoral dissertation, The University of Western Ontario). expression type questions (involving for example, mordant and [6] Viro, V. 2011. Peachnote: Music Score Search and Analysis trill). Word_spec is pitch/length occurring over a certain word in Platform. In ISMIR 2011, Miami, Florida (USA) October the lyrics. Although these were not handled by our 24–28, 2011 (359-362). implementation, some results were returned because the system fell back to the simple note type, which explains the non-zero [7] Ganseman, J., Scheunders, P., & D'haes, W. 2008. Using precision and recall. For e.g. “F trill” returns all F notes. XQuery on MusicXML Databases for Musicological Followed_by is equivalent to consecutive notes. Melodic_interval Analysis. In ISMIR 2008, Philadelphia, Pennsylvania (USA) September 14-18, 2008 433-438).