This paper summarises the C@merata task in which participants built systems to answer short natural language queries about classical music scores in MusicXML. The task thus combined natural language processing with music information retrieval. Five groups from four countries submitted eight runs. The best submission scored Beat Precision 0.713 and Beat Recall 0.904.
There is a series of questions with required answers: Provided Question: • A short noun phrase in English referring to musical features in a score, • A short classical music score in MusicXML.
Required Answer: • The location(s) in the score of the requested musical feature.
Figure 1 shows a score extract and a corresponding question+answer. The type of the query is followed_by, which in this case requires G# to be followed by B. There are four answer passages, the first being [ 4/4, 4, 2:5-2:6 ].
The music for the task was chosen from works by wellknown composers active in the Renaissance and Baroque periods. The MusicXML format was chosen because it is widely used, it is relatively simple and it can capture most important aspects of a score.
For the test collection there were twenty scores, with ten questions being set for each. Scores were on one, two, three, four or five staves according to a prescribed distribution. Instrumentation was typically voices (e.g. SATB, SSA etc), Harpsichord, Lute, Violin and Harpsichord etc.
We adapted the well-known Precision and Recall metrics of Cyril Cleverdon which are universally used in NLP and IR. We say a passage is Beat Correct if it starts in the correct bar (measure) and at the right beat offset and it ends in the correct bar and at the right beat offset. Conversely a passage is Measure Correct if it starts in the correct bar and ends in the correct bar.
We define Beat Precision as the number of beat-correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Similarly, Beat Recall is the number of beat-correct passages returned by a system divided by the total number of answer passages in the Gold Standard.
On the other hand, Measure Precision is the number of measure-correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Measure Recall is the number of measure-correct passages returned by a system divided by the total number of answer passages.
200 test queries were drawn up, based on twenty scores with ten questions being asked on each. American terminology (e.g. quarter note) was used for ten scores and English terminology (e.g. crotchet) for ten scores. Queries were devised in twelve different types according to a prescribed distribution as shown in Table 1 which also shows examples of each type. The Gold Standard answers were drawn up by the first author and then each file was carefully checked by one of the other authors. worked with Python and Music21 while others adapted their own pre-existing systems in C++ and Common Lisp.
This was a new task at MediaEval and indeed we know of no other work combining NLP and MIR in this way. Many technical details had to be solved which sometimes took us to the limits of western classical music notation. A lot was learned from the exercise both about evaluation (e.g in devising versions of P and R to use) and about music (e.g. where does a cadence begin and end). A future task could tackle a wider range of questions involving more complicated natural language structures, as well as addressing some loose ends in the task design.
Type simple_pitch simple_length pitch_and_length perf_spec stave_spec word_spec followed_by melodic_interval cadence_spec triad_spec texture_spec
D# crotchet
D sharp trill D4 in the right hand word "Se" on an A flat
melodic octave harmonic major sixth perfect cadence tonic triad polyphony crotchet followed by semibreve
Five groups from four countries (Table 2) submitted eight runs (Table 3) which were evaluated automatically using Beat Precision (BP), Beat Recall (BR), Measure Precision (MP) and Measure Recall (MR). BP and BR are much stricter, since the exact passage must be specified. However, MP and MR are also included because in practical contexts it is often sufficient to know the bar numbers - the required feature can usually be spotted very quickly by an expert.
Results were generally very good. The best was CLAS01
with Beat Precision 0.713 and Beat Recall 0.904. However,
almost all runs beat the baseline run LACG01 which was prepared
with the Baseline System distributed to all participants at the start
[
Most participants used hand crafted dictionaries and string processing to analyse the queries, rather than parsing. Generally people converted a score into feature information, extracted the required features from the query and then matched the two. Some
University University of Limerick
Tata Consultancy Services
BP 0.713 0.372 0.380 0.440 0.135