Cl@ssical Music Extraction of Relevant Aspects by Text Analysis (C@merata) is a shared evaluation held at MediaEval and this is the third time the task has run. The input is a natural language query ('F# in the cello') and the output is a passage in a MusicXML score which contains this note played on the instrument in question. There are 200 such questions each year and evaluation is via modified versions of Precision, Recall and FMeasure. In 2014 and 2015 the best Beat F (BF) scores were 0.797 and 0.620, both attained by CLAS. This year, queries were more difficult and in addition the most experienced groups from previous years were unable to take part. In consequence, the best BF was 0.070. This year, there was progress concerning the development of the queries, many of these being derived from real sources such as exam papers, books and scholarly articles. We are thus converging on our goal of relating musical references in complex natural language texts to passages in music scores.
The aim of the Cl@ssical Music Extraction of Relevant Aspects by Text Analysis (C@merata) evaluations is to advance our knowledge of how musical references in texts relate to actual passages within symbolic music scores. The evaluation takes place at MediaEval each year and consists of a series of 200 questions. Each question consists of a short string making reference to a musical feature (A4 sung to the word 'bow') and a score in MusicXML. Participants have to build a system which can answer each question by specifying a set of one or more passages which exactly demarcate the feature in question. In the example above, we wish to know the exact beginning and end of a note of unspecified length which has pitch A4 and which is being sung to the word specified (‘bow’).
The general organisation of the task has remained the same
for three years; this includes the XML format of the question file,
the format of the answer file, the type of music scores used
(MusicXML) and the means of evaluating answers (Beat
Recall/Precision and Measure Recall/Precision). Detailed
descriptions of all these can be found in the 2014 [
Paticipants have to build a system which will take as input an XML file containing 200 questions and produce as output a file containing one or more answers for each of them. Questions are grouped into twenty sets of ten, each corresponding to a particular MusicXML score. There are thus twenty scores in total and each year these are different. The scores for each year are listed in Table 2 (at the end of the paper).
Scores are chosen from public sources, are in MusicXML [
Concerning an answer, we use the concept of a passage which starts at a particular beat in a bar and ends at another beat in a bar. Bar numbers are taken from the MusicXML. Beats are measured in divisions, a concept taken from the MusicXML standard. A division is an integer and a value of one means we are counting in crotchets; a value of two indicates quavers and so on. Where necessary, a high divisions value (e.g. 12) can be used where we wish to beat in crotchets, quavers, crotchet triplets or quaver triplets for difference answers to the same question. In C@merata, the question always specifies the divisions value to be used for all answers to a particular question. In summary, a passage such as [4/4,1,1:1-2:4] means we are in 4/4/ time, divisions is set to one (crotchets) the passage starts before the first crotchet beat of bar one (1:1) and the passage ends after the fourth crotchet beat of bar two (2:4).
In evaluation, a passage is beat correct if it starts at the correct beat in the correct start bar and it ends at the correct beat in the correct end bar. So, if the correct answer is a crotchet, the passage must start immediately before the crotchet in question and it must end immediately after it. Similarly, a passage is measure correct if it starts in the correct start bar and ends in the correct end bar. Based on beat correct and measure correct, we can define the following:
Beat Precision (BP) as the number of beat-correct passages returned by a system, in answer to a question, divided by the number of passages (correct or incorrect) returned.
Beat Recall (BR) is the number of beat-correct passages returned by a system divided by the total number of answer passages known to exist.
Beat F-Score (BF) is the harmonic mean of BP and BR.
Measure Precision (MP) is the number of measure-correct passages returned by a system divided by the number of passages (correct or incorrect) returned.
Measure Recall (MR) is the number of measure-correct passages returned by a system divided by the total number of answer passages known to exist.
Measure F-Score (MF) is the harmonic mean of MP and
For the first year of the task [
For the second year [
This year, questions were chosen from the Renaissance, Baroque, Classical and Early Romantic periods. The twenty MusicXML scores were selected from kern.ccarh.org and from musescore.com. The former scores are from Stanford and have been prepared from various public domain and out-of-copyright sources. They are created in the kern format and are also available in a conversion to MusicXML.
The distribution of scores in terms of staves can be seen in Table 4. It is similar to 2015 except that there are now five scores with eight or more staves: two on eight staves and one each on ten, thirteen and eighteen staves.
The scores themselves can be seen in Table 3, which shows the work, number of staves and scoring (i.e. the istruments used). Composers this year were Bach, Beethoven, Bennet, Chopin, Handel, Morley, Mozart, Palestrina, Scarlatti, Schubert, Vivaldi and Weelkes. There were six works for keyboard (three for harpsichord and three for piano), one Schubert song for voice and piano and two string quartet movements; there were three a capella vocal works for SATB and one each for SATTB and SSATB; there were two Vivaldi concertos for strings and continuo and two symphony movements, one by Mozart and the other by Beethoven. Finally there was a movement from Handel’s Messiah for SATB and orchestra.
The distribution of question types is shown in Tables 1 and 2. These tables also show several examples of each type. All 1_melod questions are concerned with one note and can me modified by bar/measure (‘A#1 in bars 44-59’). Thirty-six of the forty can also be modified by perf (‘forte’), instr (‘in the violin’), clef (‘in the bass clef’), time (‘in 3/4’) or key (‘with G major key signature’).
n_melod questions are concerned with a sequence of notes which can be specified exactly (‘D4 D5 A5 D6 in sixteenth notes’) or inexactly (‘two-note dotted rhythm’). They can also be modified by bar/measure, perf etc in the same as 1_melod questions.
1_harm questions deal with single chords (‘whole-note unison E2 E3 E4’) which can be less specific (‘chord of C’ or ‘five-note chord in the bass’). Once again they can be modified as above (‘chord of F#3, D4 and A4 in the lower three parts’, ‘harmonic octave in the bass clef’). Note that we allow two notes to be a chord, including octaves etc. This year, references to inversions (‘Ia chord’) are considered of 1_harm type.
n_harm questions deal with sequences of chords with the usual modifications (‘three consecutive thirds in bars 1-43’). Cadences are also included here, since they are sequences of specific chords (‘plagal cadence in bars 134-138’). There are also some more complex types here (‘A5 pedal in bars 116-138’).
Finally, there are texture questions (‘all three violin parts in unison in measures 1-59’, ‘counterpoint in bars 1-14’). Some more complex forms were added this year (‘imitative texture in bars 1-18’).
Table 2 shows two further forms of question, follow and synch. There are twenty of the former and thirteen of the latter. In a departure from last year, these are not separate types, but range over the queries of type 1_melod, n_melod, 1_harm and n_harm as shown in Table 1. Thus the examples in Table 2 are all within the distribution of query types shown in Table 1. A follow question allows us to specify some passage followed by another passage. Each such passage can be of type 1_melod, n_melod etc. 4. 2016 CAMPAIGN
This year, ten groups registered for the task, the largest number so far in the three years. Unfortunately, however, only four of these were able to return a run. Participants are shown in Table 5. There was one each from England, Ireland, Poland and Russia.
The following are short notes on the various systems this year. Full details can be found in the participant papers in this volume.
This year, the DMUN system was re-written. The group
developed a text query parser that, given a sentence such as a
C@merata question, generates a script for music operations. The
script contains the music concepts and their relations as described
in the query, but in a structured form related to SQL in such a way
that workflows of specific music data operations are formed. A
parser then reads the script and calls the corresponding functions
from a framework created on top of music21 [
The KIAM system is written in PHP and is based on regular expressions. Queries are categorised and analysed using regular expressions; answers are then extracted from raw MusicXML files.
This allows quite complex sequences to be specified (‘D C# in the right hand, then F A G Bb in semiquavers in the left hand’, ‘5 B4s followed by a C5’).
Questions of type synch can link two passages which must occur at the same time. In the simplest case, each passage is of exactly the same length (‘quarter note E5 against a quarter note C#3’). However, this is not necessarily the case (‘C#3 minim and E4 semibreve simultaneously’); here, according to our rules, the whole of the minim must lie somewhere within the duration of the semibreve. The length of the passages need not be specified (‘D3 in the bass at the same time as C5 in soprano 1’, ‘three-note chord in the harpsichord right hand against a two-note chord in the harpsichord left hand in measures 45-52’).
When we reach the follow and synch questions, they are starting to become interesting from a musicological perspective as such musical phenomena as these cannot readily be specified except in a natural language. The key advantage of language here is that it can vary in specificity from the constrained to the open; to interpret the open queries requires considerable musical knowledge. Hence, C@merata starts to become interesting and not merely a simple exercise in finding notes.
We will finish this section by summarising how we derived
the questions. In previous years, we decided the question types
and distribution first; we then selected scores and devised queries
by going through them, trying to find passages against which
queries could be posed. This reverse-logic approach was a simple
development of the one which we had used at CLEF for many
years [
Modes: Dorian, Phrygian, Lydian, Mixolydian, Aeolian, Locrian, Ionian (these would be n_melod queries); Melodic intervals: diminished fifth, augmented fourth (n_melod queries); Harmonic intervals: perfect concords, imperfect concords and discord (1_harm queries); Movement of parts: similar, contrary, oblique and parallel (n_melod against n_melod); Special relationships: false relation of the tritone (1_harm) exposed fifths and octaves (n_harm).
In the event, we managed queries relating to melodic intervals and movement of parts and we plan to investigate the others in future. The second suggestion was to base questions on music exam papers set in English schools at GCSE level (aged sixteen) and A Level (aged eighteen). This had been proposed by DMUN participant Tom Collins and by co-organiser Richard Lewis. DMUN participation was handed over to Andreas Katsiavalos this year, so Tom Collins joined the organisers and did indeed generate some questions based on a study of exam papers.
The third strand of work was concerned with the derivation
of queries from musicological texts. For this campaign, we
revisited some of the texts we studied previously [
The system used is similar to last year and is based on the author’s CPNView system.
The UMFC system was based on recurrent neural networks which is undoubtedly the most original approach.
Overall results are shown in Table 6. The best run was DMUN01 which scored BF 0.070 and MF 0.106. These results are very low. Moreover, they are only based on a small subset of the queries. The best scores in 2014 were BF 0.797 and MF 0.854, and in 2015 were BF 0.620 and MF 0.656. In both previous years, the CLAS system was the best but unfortunately they could not participate this year. The questions were more difficult this year but another factor was that three of the four participants had not taken part before. All systems this year were able to answer simple note questions such as ‘C# crotchet’ but the task has moved on and there are now very few 1_melod questions as simple as that.
Table 6 shows the average results by question type. As expected, 1_melod are the the easiest overall (BF 0.054) followed by n_melod (BF 0.028). After that come 1_harm (BF 0.019) and n_harm (BF 0.013). These questions are progressively more difficult, so this order is to be expected. Texture questions this year could not be answered by any system (BF 0).
Turning to the more complex follow and synch questions (which were a subset of the questions of other types this year), the bottom half of Table 6 shows the average scores for follow (BF 0.078) and synch (BF 0). The follow score is higher than might be expected and this is because KIAM scored well on these questions (see Table 13, KIAM, BF 0.227, the next highest being DMUN with BF 0.044). While BF for synch questions was 0, MF was 0.018, so some systems could at least determine the correct start and end bar, if not the exact beat.
Looking at the results for different systems across question types, the performance of UMFC on 1_harm questions is noteworthy (BF 0.042) this system scored BF 0 on 1_melod and n_melod questions.
Generally, all the systems scored very low on Recall, because they missed out many correct answers. However, they were much better on Precision, because answers returned tended to be correct. Looking at Table 6, average BP over all participants was 0.167 and average MP was 0.447. The best BP was 0.420 and the best MP was 0.640 (both DMUN). These are more respectable looking figures. In fact, looking at MP in Table 6, three out of the four systems scored 0.511 or better. The problem with systems this year was that they were not sophisticated enough to handle complex questions; those that they did answer were often correct.
If we look at the MP figures for the various different questions types (Tables 8-14) we can see the highest values scored on any measure in this year’s campaign: For 1_melod, DMUN scored 0.857 and KIAM scored 0.727; for n_melod, DMUN scored 0.706; for 1_harm OMDN scored 0.8; for n_harm, DMUN scored 0.5. Finally, concerning follow and synch, for follow, KIAM scored 1 and OMDN scored 0.333; for synch, OMDN scored 0.333. Of course, in most of these cases, a system was chancing to match passages for the particularl queries and was not in fact attempting the vast majority of questions, leading to very low MR and MF scores.
The main achievement this year was to develop some very interesting questions against some quite complex scores. Questions were a lot more complicated than those of last year which in turn were more complicated than in the first year when many were quite elementary.
Unfortunately, the systems have not kept up with the task.
They can answer very few of the questions, though answers when returned are often correct, at least for MF. There is some ambiguity concerning the exact beats of certain questions, in particular those of types follow and synch, and this can partly account for the much lower BF scores than MF scores.
Concerning practical aspects of the organisation, as always
there were two problems in finding scores. The first was that there
is a limited choice of public domain works, especially ones which
are of sufficiently high quality. Over the years we have
more-orless exhausted the musescore archive [
However, this leads to the second problem, which is the
quality of the MusicXML. Scores on musescore are produced
mainly by amateurs using many different types of software.
Conversion to MusicXML is carried out at the end of the
encoding process and relies on the extent to which the score
writing software supports it. We have found that many
MusicXML files do not load in the Musescore software [
The situation for the kern scores is rather similar. Conversion
to MusicXML is carried out by a script at the end; MusicXML is
not considered the primary format as it is assumed that kern will
always be used. We have found that some of these conversions are
inaccurate or crash Musescore. In some cases we were able to
reconvert from kern using the latest version of music21 [
Finally, looking to the future, there have now been three
years where the C@merata query and answer format remained the
same: the input was a noun phrase and the output was a passage.
Over the years we have also begun to investigate the relationship
between our queries and actual musicological texts [
Q31: minim on the word "cease" A: [ 4/4, 1, 7:3-7:4 ], [ 4/4, 1, 9:1-9:2 ] Q32: note on the word "mine" against a note on the word "eyes" A: [ 4/4, 1, 3:4-3:4 ] Q33: semibreve tied to a minim in the Bass clef A: [ 4/4, 1, 1:1-2:2 ], [ 4/4, 1, 3:1-4:2 ] Q35: semibreve tied to a minim followed by crotchet, crotchet A: [ 4/4, 1, 1:1-2:4 ], [ 4/4, 1, 2:1-3:4 ] Q36: minor third between Basse and Tenor in bars 1-35 A: [ 4/4, 1, 1:3-2:2 ], [ 4/4, 1, 8:1-8:2 ], [ 4/4, 1, 9:3-9:4 ] Q37: Ia chord in bars 1-10 A: [ 4/4, 1, 2:1-2:2 ], [ 4/4, 1, 5:3-5:4 ], [ 4/4, 1, 8:3-8:4 ], [ 4/4, 1, 9:3-9:4 ] Q40: counterpoint in bars 1-14 A: [ 4/4, 1, 1:3-14:4 ] 1_melod qualified by perf, instr, clef, 36 time, key n_melod
15 n_melod qualified by perf, instr, clef, 45 time, key 1_harm 1_harm possibly qualified by perf, instr, clef, time,
key n_harm n_harm possibly qualified by perf, instr, clef, time,
key texture
All 17 23 25 15 20 flute dotted half note only against strings half note on the tonic in the bass clef
two-note dotted rhythm in measures 1-24 eight note rising passage in quarter notes
repeated Bb4 whole note D4 D5 A5 D6 in sixteenth notes repeated twice
two tied dotted minims in bars 72-88 dotted minims C B A in the Bass clef in bars 70-90
melodic interval of a minor 7th in the voice rising arpeggio in the left hand in measures 1-10 five-note melody in the cello in measures 20-28 whole note rest, quarter note in the Violin 4 in measures 1-103 7th triad in measures 1-3
Ia chord in bars 1-10
chord of C whole-note unison E2 E3 E4
chord III in bars 44-59 chord of F#3, D4 and A4 in the lower three parts
harmonic fifth in the oboe harmonic octave in the bass clef harmonic perfect fourth between the Soprano and Alto in bars 1-9 cello and viola playing dotted minims an octave apart in bars 40-70 interrupted cadence
A5 pedal in bars 116-138 authentic cadence in measures 14-18
plagal cadence in bars 134-138 three consecutive thirds in bars 1-43 consecutive sixths between the Altos and Basses in measures 73-80 flute, oboe and bassoon in unison in measures 1-56
consecutive descending sixths in the left hand alternating fourths and fifths in the Oboe in bars 1-100 Soprano and Alto moving one step down together in measures 1-12 all three violn parts in unison in measures 1-59
polyphony in measures 5-12 homophonic texture in measures 125-138 imitative texture in bars 1-18 counterpoint in bars 1-14 D C# in the right hand, then F A G Bb in semiquavers in the left hand B flat in the cbass followed a quarter note later by B natural in the cbass
quarter note E5 against a quarter note C#3
C#3 minim and E4 semibreve simultaneously
D3 in the bass at the same time as C5 in soprano 1 three-note chord in the harpsichord right hand against a two-note chord in the harpsichord left hand in measures 45-52 A#3 in the piano and F#5 in the voice simultaneously 2 2 2 2 2 2 3 4 4 4 4 4 4 5 5 8 8 10 13 18
Scoring hpd pf pf pf hpd hpd S, pf
2 vn, va, vc
2 vn, va, vc
Eng.
Eng. Eng. Eng.
Eng. Eng. Eng.
4 vn, 2 va, vc,
Amer.
db 3 vn, va, vc, Amer.
db, hpd fl, 2 ob, 2 bn, 2 hn, 2 vn, va, Eng.
vc, db 2 fl, 2 ob, 2 cl, 2 bs, 2 hn, 2 Amer. tpt, timp, 2 vn,
va, vc, db, fl, 2 ob, cl, bs, hn, tbn, tuba, Amer. SATB, hpd, 2 vn, va, vc, db