This paper presents a system that was developed for the C@merata task to perform music information retrieval using textbased queries. The system is built on findings from previous attempts and achieved best results and functionality so far. The C@merata task is split in two modules that handle the queryparsing and music-information retrieval separately. The sub-tasks are connected with a formal-information-request, a dictionary that contains the parsing information. The system is not fully extended but key issues and methods are identified. This paper draws from works in previous C@merata events and studies in music information retrieval generally. The clear distinction of query parsing and music-information retrieval between the C@merata sub-tasks enabled independent developments for each system. In 2015 [3], the focus was on the development of highly-parameterized music-information retrieval functions for high-level musical concepts, such as arpeggios and scales, while the system's text parsing was relying on Collins' Stravinsqi algorithm. The following year [2], the focus shifted to language processing for the development of an automated query parser. The results where promising and key tasks where identified and addressed, however, the connection between the query-parser and the music-information retrieval functions was very poor.
The C@merata task [
The C@merata task is important because it is addressing a fundamental need in music research, that of a simplified contentbased music information retrieval system. Content-based retrieval systems are implemented in fields such as music informatics, with highly specialized applications, and, in general text- and multimedia-based systems in web-search engines. However, there are no user-friendly applications to perform what the C@merata task is challenging. Thus, the development of text-based query systems for music-information retrieval will fill the gap between specialized and non-content-based retrieval services for music.
A service that will satisfy the needs of the C@merata task would be helpful to everyone related with music and especially in higher-level music education where research often requires the identification of diverse and complex musical elements in large corpora. The textual-interface that is suggested from the task is also very practical for novice music enthusiasts that begin to discover the theoretic establishment of tonal music.
The system presented in this paper is a prototype method to connect text parsing and music information retrieval. The C@merata task is handled in two main stages: a) the text parsing, and, b) the music information retrieval. A shell was developed that integrates and connects the above elements, also handling I/O operations. The two stages are operating independently and are connected by the use of a data structure, named Formal Information Request (see below).
Each stage uses custom code that is not dependent on any high-level external libraries for either language processing or music information processing. Concerning language processing, the system is not able to handle completely ‘natural’ language but rather a collection of word constructs where each valid sentence is viewed as a structure of valid terms, types and type combinations. In this prototype system, only selected constructs were implemented for proof of concept; however, the language is easily extensible. While text parsing is carried out completely from scratch, the reading of musicXML files and some dictionaryrelated operations were facilitated by music21.
Two important notions of the system are the Formal Information Request (FIR) and the notion of (musical) ‘durational element’. The FIR is a method to connect the output of the query parsing with the music-information retrieval functions. It basically transfers all the parsing data to a music function selector that further processes the parsing elements to be inputted to the music information retrieval functions. The notion of the durational element is very helpful in chaining input and output between music information retrieval functions.
Overall, as displayed in Figure 1, the system inputs a text query and initializes a query parser object by loading a .json language file, a dictionary with single term types for keys and sets of terms for values. The query-parser converts the text of the query into a Formal Information Request (FIR), another dictionary, by gradually identifying and replacing the terms, term types and compound types of the query with their types found in the language file, until a top-level description of the query is found. The FIR is then sent to the music information retrieval (MIR) module which in turn selects the corresponding information request retrieval function. All the currently possible information requests are implemented as combinations between three core types of MIR functions that find, relate and constrain music-entities such as notes/rests and note-sets (melodies, chords, etc.). Lastly, the output of the MIR functions, which are music elements, are converted into passages. patterns cannot integrate more and since their content, context, and requirements are identified, they are viewed as high-level functions. 3.2
The query parsing module inputs the query phrase and after a sequence of parsing operations it outputs the FIR . The parsing is based on a ‘language’ file that holds all the information that is required to identify the type of the query. Parts of the language file are generated algorithmically.
Since all the questions where converted into combinations of Entities(E), Relations(R), and. Qualifications(Q), the set of valid combinations can be given from the graph shown in Figure 3, starting with an entity (E). Following this graph in text parsing was revealing in what kind of patterns are used and what kind of functions need to be developed.
As shown in Figure 2 from top to bottom, the query parsing process starts with breaking down the query phrase into word tokens (terms) while commas (‘,’) are removed. Next, the TYPE of each TERM is identified based on the language TERMS set. Next, compound types (cTYPE) are identified by searching for the maximal subset of adjacent parsed TYPES. Next, the query is parsed again to check if there are any multi-compound-types (mcTYPE). At this point, the query is viewed as a high-level pattern of musical-entities, relations and qualifications. These The music information retrieval module starts with the formal information request of the query parser and outputs the music elements that satisfy the query question. In general the reverse process of text parsing is followed: while in query parsing the language dictionary was used to find integrations of terms in order to identify the top query description, once the function is identified, the descriptions are broken down into elements but this time removing and combining terms to read values and perform music content searching.
The music information retrieval operations are handled by a simple script that was developed for this reason. The system operates with ‘datapoint’ lists, where notes and rests are the atoms. The music entities that are identified in the text parser as (E)ntities are shown in Figure 4; the MIR functions can currently retrieve the elements from the top three rows. Note that all the combinations between them are possible.
There are generally two extremes in declaring and identifying Entities in queries and each one has different approach in retrieval. An entity may contain the specific constituents of the element, from highly specific e.g. a query ‘C4 E4 G4 chord’, to more abstract e.g. ‘major chord’.
The Entities in Figure 4 are durational entities, meaning that they all have similar attributes such as a starting point and an ending point in time. The system makes use of these generic properties with robust MIR functions that can handle and mix any of them. For example a query ‘G4 followed by minor’ is served by an MIR function that handles ‘Entity-After-Entity’ and not ‘Chord-After-Note’. This is an interesting feature with only partial exploitation. 4
The system found great difficulties with text parsing and for that reason two groups of answers were made: 1. 2.
‘auto’, where the queries were inputted ‘as is’ from the C@merata questions file without any alterations. ‘altered’, where some parts of the query had to be altered to match the parsing capabilities.
Due to the small number of ‘auto’ answers and also to the fact that the alterations that had to be made are considered trivial, the results for the two groups were summed. The alterations are considered trivial because the methods to parse the original queries is known but not implemented. Also, all the answered questions were manually selected so that the MIR functions would be able to run them. This explains the overall low recall and high precision of the results shown in Figure 5 meaning that when the FIR was produced then the MIR was usually successful.
In general, as shown in Figure 5, the overall Beat Recall and Measure Recall did not exceed 0.2 percent (0.155 and 0.172 respectively), and from the total of 200 questions only 30 were answered. The generally high precision (0.833 for beat and 0.924 for measure) is, as stated earlier, due to the manual selection of queries into feasible and not feasible, and to minor alterations to their text. More specifically, the ‘synch’ category was completely excluded and very few ‘follow’ and ‘texture’ queries where tested. Most of the emphasis was given to the ‘melodic’ and ‘harmonic’ queries trying to answer as many as possible, but still with low recall in both.
The current system presents a working paradigm for the complete C@merata task, however as a prototype, it doesn’t reach its potential. Although multi language support was not tested, this can be easily achieved by using a different language file. This way, apart from the differences in terms, different grammar constructs can also be used as the language file is fully customizable allowing the user to add their own grammatical constructs.