<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>SPARQL-DJ: The MIDI Linked Data Mashup Mixer for Your Next Semantic Party</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Albert Meroño-Peñuela</string-name>
          <email>albert.merono@vu.nl</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Rick Meerwaldt</string-name>
          <email>rickmeerwaldt@hotmail.com</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Stefan Schlobach</string-name>
          <email>k.s.schlobach@vu.nl</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Computer Science, Vrije Universiteit Amsterdam</institution>
          ,
          <addr-line>NL</addr-line>
        </aff>
      </contrib-group>
      <abstract>
        <p>Many datasets describing musical resources are published today as Linked Data, mainly focusing on metadata, notation, and audio features. However, the availability of musical data as Linked Data is often not enough for musicians, who need additional layers of software and queries to accomplish their workflows. Mashups are compositions created by blending two or more pre-recorded songs, and it has been shown that they can be generated by using SPARQL on top of MIDI music represented as Linked Data. In this demonstration, we showcase SPARQL-DJ, a web-based application implementing those layers for the workflow of generating MIDI mashups using Linked Data and SPARQL. The demonstration focuses on three parts: (1) coverage of the mashup composition workflow; (2) scoping of input retrieval for beatmaching by broadening text search and leveraging track annotations; and (3) management of off-beat and dissonance.</p>
      </abstract>
      <kwd-group>
        <kwd>MIDI</kwd>
        <kwd>mashups</kwd>
        <kwd>SPARQL</kwd>
        <kwd>MIDI Linked Data</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        With well-known linked datasets like MusicBrainz, BBC Music and MySpace,
music is an important player in the Linked Open Data cloud. Most of these
datasets represent the metadata of music, like bands, musicians, song titles,
albums, firms, etc. Some works in the Semantic Web have investigated formal
ways of dealing with music itself, like chord representation [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] and the Music
Ontology [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Deep learning has been applied to link audio with notation [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
      </p>
      <p>
        More recently, proposals have been made to represent music notation as
Linked Data, such as the MIDI Linked Data cloud, a dataset that connects
308,443 MIDI songs by representing their tracks, events, and notes as 10,215,557,355
RDF triples1 [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. MIDI (Musical Instrument Digital Interface) is a protocol for
the interchange of musical information between musical instruments, which can
be converted to RDF and back to playable MIDI without data loss [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ].
      </p>
      <p>
        This wealth of information could be a great aid to musicians in supporting
many of their creative workflows. Concretely, the activity of remixing existing
music to make new, genuine compositions, also known as mashups, is not
currently supported by any semantically rich knowledge base. AutoMashUpper, for
example, is an “interactive system for creating music mashups by automatically
selecting and mixing multiple songs together” based on their harmonic
similarity [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Mashh! is an online tool where people can find songs and loops and
where they can mix their own mashups [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. The creative process of creating these
mashups could be to a great extent automated and scaled up by leveraging
semantic representations and Linked Data. In previous work, we have proposed
and evaluated a framework for creating mashups using MIDI Linked Data as
input and SPARQL queries as method [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. This demonstration consists of
showcasing SPARQL-DJ, a prototype implementing this workflow, at: (1) covering
a proposed framework for generating mashups; scoping the search of mashable
candidates in two ways: (a) by broadening the search of beatmatching using song
metadata, and (b) by leveraging track annotations; and (3) managing off-beat
and dissonance in output mashups.
2
      </p>
      <p>
        Mashups with MIDI Linked Data
Although the creation of mashups involves a more intricate process [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], in this
demo we focus on covering the workflow presented in Figure 1. After finding the
first pattern, the user searches for its tempo values (step 2). Using the tempo
values of the first pattern, the user can search for the second pattern (step 3).
After two patterns are found, they are combined in step 4. The result of the
query in step 4 can be converted to MIDI, using the MIDI2RDF suite of tools
[
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] (step 5). The result is evaluated in step 6. Even though at this point the songs
are beat-synchronous, it is still possible that they are not in harmony with each
other, or that their beat does not match. Consequently, it is possible to search
for a different second pattern (step 3). It is also possible to search for the tracks
of the songs (step 7) and combine the songs using only certain tracks (step 8).
Filtering out certain tracks that cause the dissonant tones can circumvent the
problem of the songs not being harmonious. Filtering tracks can also cause for
a better mix of instrumentals. After combining the songs using only the desired
tracks, the result of the query in step 8 can be converted to MIDI (step 9). The
result is evaluated in step 10. If the songs still do not match, different tracks can
be combined or filtered out. Contrarily, if the songs do match, a beat-synchronous
mashup has been created (step 11).
We implement this MIDI mashup creation framework with SPARQL-DJ (see
Figure 2), a web-based interface for creating MIDI mashups by reusing Linked
Data from the MIDI Linked Data cloud [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. These MIDI data are both retrieved
and mixed into new compositions by exclusively executing SPARQL queries2
against the MIDI Linked Data cloud SPARQL endpoint3. These queries are
managed by SPARQL-DJ, and their parameters are supplied from the interface.
The resulting MIDI mashups can be played in the browser, or downloaded as
either MIDI or Turtle. The demonstration will cover the following use cases.
      </p>
      <p>Workflow coverage. Here, the demonstration will cover the steps of Figure
1 by completing the forms in the interface of SPARQL-DJ (Figure 2). First, the
user introduces a search string for the first MIDI, getting a number of results
that can be inspected by metadata (hovering their URIs), or by playing their
MIDI conversions in the browser. After this, users select the first MIDI they
wish to mashup. Selecting the second MIDI is analogous, but this time users can
search only metric-matching candidates (same tempo values ), or supply their
own metric values. Finally, the user selects which tracks from both MIDIs she
wants to have in the mashup. Clicking play will create the MIDI mashup and
send it to the browser, while clicking download will retrieve the MIDI or
RDF/Turtle serialization. This will be demoed with Michele (Beatles) + Smells Like
Teen Spirit (Nirvana) and Eine Kleine Nacthmusik (Mozart) + War Pigs (Black
Sabbath).</p>
      <p>Scoping of mashable candidates. We will showcase how the search for
beatmatching songs can be broadened in two different ways. First, we will use the
additional metadata available in the MIDIs to use the bands’ names, their genre,
or the collection they come from. Second, we will narrow the number of tracks
in the final mashup by filtering them based on their track name annotations.</p>
      <p>Managing off-beat and dissonance. In this part, we will show cases in
which off-beat and harmonic dissonance might occur, even if the mashable songs
are metric compatible; and will discuss about how to manage these.</p>
      <p>2See queries at https://github.com/rickmeerwaldt/SPARQLmashup and https://github.
com/rickmeerwaldt/SPARQLmashup/raw/master/example/example.pdf</p>
      <p>3http://virtuoso-midi.amp.ops.labs.vu.nl/sparql
(a) MIDI lookup, play, and metric filtering.</p>
      <p>(b) Selection of tracks for the mahsup.</p>
      <p>In the future, we will extend SPARQL-DJ’s coverage of the mashup creation
workflows, we will leverage external links to other datasets for scoping the search
of mashable candidates, and we will use harmonic features to manage dissonance.
Acknowledgements. We would like to express our gratitude to Ingrid Arcas for her dedication at
testing SPARQL-DJ, her contributed mashups, and her valuable design advice.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Davies</surname>
            ,
            <given-names>M.E.P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hamel</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Yoshii</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Goto</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>AutoMashUpper: An Automatic Multi-Song Mashup System</article-title>
          .
          <source>In: International Conference on Music Information Retrieval (ISMIR)</source>
          . pp.
          <fpage>575</fpage>
          -
          <lpage>580</lpage>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Harte</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sandler</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Abdallah</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gómez</surname>
          </string-name>
          , E.:
          <article-title>Symbolic representation of musical chords: A proposed syntax for text annotations</article-title>
          .
          <source>In: Proceedings of the International Conference on Music Information Retrieval (ISMIR)</source>
          . pp.
          <fpage>66</fpage>
          -
          <lpage>71</lpage>
          (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Meerwaldt</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Meroño-Peñuela</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schlobach</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Mixing Music as Linked Data: SPARQL-based MIDI Mashups</article-title>
          . In: Workshop on Humanities in the Semantic Web - WHiSe
          <string-name>
            <surname>II</surname>
          </string-name>
          (ISWC
          <year>2017</year>
          )
          <article-title>(</article-title>
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Meroño-Peñuela</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hoekstra</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          :
          <article-title>The Song Remains the Same: Lossless Conversion and Streaming of MIDI to RDF and Back</article-title>
          .
          <source>In: 13th Extended Semantic Web Conference</source>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Meroño-Peñuela</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hoekstra</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gangemi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bloem</surname>
          </string-name>
          , P., de Valk, R.,
          <string-name>
            <surname>Stringer</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Janssen</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>de Boer</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Allik</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schlobach</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Page</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>The MIDI Linked Data Cloud</article-title>
          .
          <source>In: 16th International Semantic Web Conference (ISWC</source>
          <year>2017</year>
          )
          <article-title>(</article-title>
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Raimond</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Abdallah</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sandler</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Giasson</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          :
          <article-title>The Music Ontology</article-title>
          .
          <source>In: International Conference on Music Information Retrieval (ISMIR)</source>
          . vol.
          <volume>422</volume>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Thickstun</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Harchaoui</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kakade</surname>
            ,
            <given-names>S.M.</given-names>
          </string-name>
          :
          <article-title>Learning Features of Music from Scratch</article-title>
          .
          <source>In: International Conference on Learning Representations (ICLR)</source>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Tokui</surname>
          </string-name>
          , N.:
          <article-title>Massh!: a web-based collective music mashup system</article-title>
          .
          <source>In: Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts</source>
          . pp.
          <fpage>526</fpage>
          -
          <lpage>527</lpage>
          . ACM (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>