This paper is motivated by the massively increasing structured data on the Web (Data Web), and the need for novel methods to exploit these data to their full potential. Building on the remarkable success of Web 2.0 mashups, this paper regards the internet as a database, where each web data source is seen as a table, and a mashup is seen as a query over these sources. We propose a data mashup language, which allows people to intuitively query and mash up structured and linked data on the web. Unlike existing query methods, the novelty of MashQL is that it allows people to navigate, query, and mash up a data source(s) without any prior knowledge about its schema, vocabulary, or technical details. We even do not assume even that a data source should an online or inline schema. Furthermore, MashQL supports query pipes as a built-in concept, rather than only a visualization of links between modules.
In this short article we propose a data mashup approach in a
graphical and Yahoo Pipes’ style. This research is still a work in
progress, thus please refer to [
This trend of structured and linked data is shifting the focus of web technologies towards new paradigms of structured-data retrieval. Traditional search engines cannot serve such data because their core design is based on keyword-search over Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Challenges: Before a user formulates a query on a data source, she needs to know how the data is structured, and what are the labels of the data elements, i.e., the schema. Web users are not expected to investigate “what is the schema” each time they search or filter structured information. This issue is particularly more difficult in case of RDF and linked data. RDF data may come without a schema\ontology, and if exists, the schema is mixed up with the data. In addition, as RDF data is a graph, one have to manually navigate this graph in order to formulate a query about it. Imagine large and multiple linked data sources, with diverse content and vocabularies, how you would manage to understand the data structure, inter-relationships, namespaces, and the unwieldy labels of the data elements. In short, formulating queries in open environments, where data structures and vocabularies are unknown in advance, is a hard challenge, and may hamper building data mashups by non-IT people. To allow people to query and mash up data sources intuitively, we propose a data mashup language, called MashQL. The main novelty of MashQL is that it allows non IT-skilled people to query and explore one (or multiple) RDF sources without any prior knowledge about the schema, structure, vocabulary, or any technical details of these sources. To be more robust and cover most cases in practice, we even do not assume that a data source should have -an offline or online- schema\ontology at all. In the background, MashQL queries are translated into and executed as SPARQL queries.
Paper organization: Before presenting MashQL, in the next section we overview the art of query formulation, which has been studied by different research communities. We present MashQL in section 3, and in section 4 we introduce the notion of query pipes. The implementation of MashQL and a three use cases are presented in section 5 and 6 respectively. The coverage and the limitations of MashQL and its future directions are discussed in section 7.
Several approaches have been proposed by the DB community to
query structured data sources, such as query-by-example [
Differently from the above Web 2.0 mashup editors, a more
sophisticated editor has been proposed in [
More about query formulations scenarios and (which scenario is
more intuitive to the casual user) can be found in a recent
usability study in [
The main goal of MashQL is to allow people to mash up and fuse
data sources easily. In the background MashQL queries are
automatically translated into and executed as SPARQL queries.
Without prior knowledge about a data source, one can navigate
this source and fuse it with another source easily. To allow people
to build on each other’s results MashQL supports query pipes as a
built-in concept. The example below shows two web data sources
and a SPARQL query to retrieve “the book titles authored by Lara
and published after 2007”. The same query in MashQL is shown
in Figure 2. The first module specifies the query input, and the
second module specifies the query body. The output can be piped
into a third module (not shown here), which renders the results
into a certain format (such as HTML,XML or CSV), or as RDF
input to other queries. Notice that in this way, one can easily build
a query to fuse the content of two sources in a linked manner [
The intuition of MashQL is described as the following: Each query Q is seen as a tree. The root of this tree is called the query subject (e.g. Article), denoted as Q(S), which is the subject matter being inquired. Each branch of the tree is called a restriction R and is used to restrict a certain property of the query subject, Q(S) ؔ
R1 AND … AND Rn. Branches can be expanded to allow sub trees (called query paths), which enable one to navigate the underlying dataset. In this case, the object in the restriction is considered the subject of its sub query. As Figure 3 shows, the query retrieves the title of every article, published after 2005, and written by an author, who has an address, this address has a country called Cyprus.
PREFIX S1: <http://www.example.com> SELECT ?ArticleTitle FORM < http://www.example.com? WHERE { ?X1 rdf:type :Article.
?X1 S1:Title ?ArticleTitle. ?X1 S1:Year ?X2.
FILTER (?X2 > 2005). ?X1 S1:Author ?X3. ?X3 S1:Address ?X4. ?X4 S1:Country ?X5.
FILTER regex(?X5, “Malta”)} Formulating queries in MashQL is designed to be an interactive process, by which the complexity of understanding data structures is moved to the query editor. Users only use drop-down lists to express their queries.
The query subject is selected from a list generated dynamically from, either: (1) the set of the subject-types in the dataset; (2) or the union of all subject and object identifiers in the dataset; users can also choose to (3) introduce their own label; in this case the label is seen as a variable and displayed in italic. The default subject is the variable “Anything”. To add a restriction, the list of properties (e.g., Title, Author) is generated, depending on the chosen subject. Users may then select a filter (e.g., Equals, Contains, Between, etc.), or select an object identifier from a list, which is then generated from the set of the possible objects identifies, depending on the previous selections. Furthermore, users select to expand the tree to declare a query path. The projection symbol 5 can be used before a variable to indicate that it will be returned in the results1. In short, while interacting with the editor, the editor queries the dataset in the background in order to generate the next list depending on the previous selections. In this way, people can navigate a graph without prior knowledge about it.
Similar to SPARQL, all restrictions in MashQL are considered necessary when evaluating a query. However, if a restriction is prefixed with “maybe”, it is considered optional; and, if it is prefixed with “without” is considered unbound (see Figure 3). MashQL supports also union (denoted as “\”) between objects, predicates, subjects, and queries; as well as, a type operator (“Any”), Inverse predicates, datatype and language tags, and many objects filters.
PREFIX a: <http:www.example.nam.com> PREFIX S1: <http:www.example.si.com> SELECT ?SongTitle, ?AlbumName FROM <http:www.example.si.com> WHERE {?Song S1:Title ?SongTitle.
{{?Song S1:Duration ?X1} UNION {?Song a:Length ?X1}}
FILTER (?X1 > 3).
{{?Song S1:Artist S1:Shakira} UNION {?Song S1:Artist S1:AxelleRed}} OPTIONAL{?Song S1:Album ?AlbumName}.
OPTIONAL{?Song S1:Copyright ?X2}.
FILTER (!Bound(?X2)).}
To deploy MashQL in an open world some challenges might be
faced. This section overviews these challenges (from a query
formulation viewpoint) and introduces the notion of query pipes.
As discussed earlier, one may create a mashup and redirect its
output to another mashup. We call the chain of queries that
connect to each other in this way as pipe. Allowing people to
formulate query pipes is not merely a visualization of links
between query modules, but when compiling a pipe (i.e.,
translating it into SPARQL), some issues should be considered.
First: Translating MashQL into SPARQL SELECT statements is
not enough, because the SELECT statement produces the results
1 Some issues are lengthy to illustrate here. For example, when a
user moves the mouse over a restriction, it gets the editing mode
and all other restrictions get the verbalize mode (i.e., all boxes
and lists are made invisible, but the verbalization of their
content is generated and displayed instead). This is not only to
make the readability of the queries closer to natural language,
but also to allow users to validate whether what they did is what
they intended. The editor also detects and normalizes
namespaces: find similar URLs and hide them when necessary.
For example, when two properties originating from different
data sources have the same URL, their namespaces are found
and hided.
in a tabular form. To allow queries to input each other (especially
for producing linked data), the results of a query should be
formed as a graph. In SPARQL, the CONSTRUCT statement
produces a graph, but then one needs to manually specify how
this graph should be produced. To overcome this, we propose the
construct (CONSTRUCT *). This is not part of the standard
SPARQL but has been proposed also by others to be included in
the next version of the standard [
Second: When executing a SPARQL query, all query engines
assume that the queried data is stored locally; otherwise, this data
must be downloaded and stored at the engine-side before the
execution process starts. The time complexity of executing a
query on local data is usually fast2; however, the bottleneck will
be the downloading time. In case the input of a query is an output
to another query (i.e., in case of query pipes) the problem will be
even more difficult, as queries will be calling each other.
Furthermore, it is also possible that users (intentionally or by
mistake) end up with query loops (e.g. Q1→Q2→Q3→Q1), which
may cause computational overheads. To face this challenge,
MashQL allows users to materialize the results of their
queries/pipes and decide their refreshing strategies, as follows:
The results of a query (called derived source) are stored
physically and deployed as a concrete RDF source. Primal input
sources (called base sources) are also cached for performance
purposes. Given a query Q over a set of base or derived sources
{D1,..,Dm}, the results of this query is denoted as D = Q(D1,..,Dm),
and D ∉ {D1,..,Dm}. We define a Pipe as an acyclic chain of
queries, where the result of a query is an input to the next. The
chain of the queries that derives D is denoted as the pipe P(D).
We call the problem of keeping a pipe up-to-date, the pipes
consistency. Let D be the results of a query Q(D1,..,Dm), and T the
latest time the set {D1,..,Dm} has been changed. Then, D is
consistent at T if D=Q(D1,..,Dm). To maintain pipes consistency,
two updating strategies are used: Query auto-refresh and Pipe
auto-refresh. MashQL maintains for each base or derived source
D a timestamp of its last update RDT and an auto-refresh time
interval RDA; and for each query Q a timestamp of its previous
successful execution RQT and an auto-refresh interval RQA.
Query auto-refresh: Each query will be automatically executed if
its auto-refresh interval expires and one of its inputs is updated.
2 A query with medium size complexity over a large dataset takes one or
few seconds [
Let Qi be a query over a set of sources {D1,..,Dm}, and T is a given time. Qi will be re-executed if (RQiT + RQiA) ≤ T and (RQiT < RDjT), where 1 ≤ j ≤ m.
Pipe auto-refresh: Each pipe P(D) is automatically refreshed if RDA expires. This implies re-executing the chain of queries in this pipe. Let P(D) be a pipe, D=Qn(D1,..,Dm), and T is a given time. If (RDT+RDA) ≤ T, then each ith query in P(D) is executed if (RQiT < RDjT ), where 1 ≤ j ≤ m for Qi, and 1 ≤ i ≤ n. Queries in P(D) are executed from the bottom to the topmost, or recursively as P(P(D1),…,P(Dm)).
As argued in the data warehousing literature [
First: we have developed an online mashup editor, which will be publically available next month. Similar to creating feed mashups in Yahoo Pipes, MashQL users can query and fuse data sources and the output of their queries can be redirected as input to other queries. In the background, Oracle 11g is used for storing and querying RDF. When a user specifies a data source(s) as input, it is bulk-loaded to the Oracle’s semantic technology tables. MashQL queries are also translated into Oracle’s SPARQL. While interacting with the editor to formulate a query, the editor performs some background queries through AJAX. Each published query is given a URL. Calling this URL means executing this query and getting its results back.
Second: We started to also develop a Firefox add-on in order to allow people develop mashups at the client side. The opened pages -in the browser tabs- are automatically selected as input sources, and at the left-side panel a mashup can be created. The results are rendered by the browser in a new tab. The idea is to allow web pages that embed RDF triples (i.e., RDFa or microformats) to be queried and mashed up. For example, one will be able to compose his publication list from Google Scholar, DBLP, ACM, and CiteSeer; or, filter all video lectures given by Berners-Lee from YouTube and VedioLectures. Because the mentioned web sites do not support RDFa yet, one can mine/distil the RDF triples, using third party services such as triplr.org, buzzword.org.uk, wandora.org or Dapper.
This is section we present two hypothetical use cases to illustrate using MashQL for developing data mashups.
Fnac is a large retailer of cultural and consumer electronics products. When a new product arrives to Fnac, it has to be entered to the inventory database. This is usually done by scanning the barcode on each product, and then manually filling the product specifications. Furthermore, as Fnac trades in many countries, their product specifications have to be translated into several languages. To save time entering and translating information manually, Fnac decided to reuse the product data specifications (and their translation) that are produced at the factory side. For example, suppose Fnac received three packages from Cannon, Alfred, and IMDB. Fnac would like to scan the barcode of the received products and then get their specifications directly from the online catalogues of those suppliers. In Figure 5 we show samples of online product catalogues of the three suppliers (we assume they are published in RDFa). Figure 6 illustrates a query that Fnac built to look up the multilingual titles of three products. This query is a mashup of three RDF data sources with a userinput of three barcode numbers. The query takes each of these barcodes and finds the English and French titles. Notice that Fnac assumed that short titles provided by Cannon are in English, thus, they are joined with the other titles that are tagged with "@en". See the retrieved results in Figure 8. In this same way, a barcode reader could be connected with user-input module, to retrieve the specifications (which could be stored at the supplier side) each time a product is scanned. http:www.cannon/products/rdf http://www.alfred.com/books _:P1 :ShortName “CanScan 4400F” <:B1> :Type <:Book> _:P1 :FullName “Canon CanoScan <:B1> :Title “The Prophet”@en 4400F Color Image Scanner” <:B1> :Title “Le prophète”@fr _:P1 :Producer “Canon” <:B1> :BCode 8765422097653 _:P1 :ShippingWeight> “4 pounds” <:B1> :Authors “Kahlil Gibran” _:P1 :Barcode 9780133557022 <:B1> :ISBN-10 0394404289 _:P2 :ShortName “PowerShot SD100” <:B3> :Type <:Book> _:P2 :FullName “Canon PowerShot <:B3> :Title “Alfred Nobel”@en
SD10007.1MP Camera 3x Zoom” <:B3> :Title “Alfred Nobel”@fr _:P2 :Producer “Canon” <:B3> :BCode 75639898123 _:P2 :ShippingWeight> “2 pounds” <:B3> :Authors “Kenne Fant” _:P2 :Barcode 9781143557532 <:B3> :ISBN- 0531123286 http://www.imdb.com/movies _:1 rdf:Type <:Movie> _:1 :Title “All about my mother”@en _:1 :Title “Tout sur ma mère”@fr _:1 :ProdCode 3248765355133 _:1 :NumberOfDiscs: 1 _:2 rdf:Type <:Movie> _:2 :Title “Lords of the rings”@en _:2 :Title “Seigneur des anneaux”@fr _:2 : ProdCode 4852834058083 _:2 :NumberOfDiscs: 3
Bob would like to compile the list of articles that cited his articles (excluding what he cited himself). He built a mashup using MashQL to mix his citations retrieved from both Google Scholar and CiteSeer, and then filter out the self-citations. First, he performed a keyword search (“Bob Hacker”) on both Google Scholar and CiteSeer3. Figure 9 shows a sample of the extracted RDF triples. Bob’s MashQL query is shown in Figure 10, and its SPARQL equivalent in Figure 11. In this query, Bob wrote: retrieve every article that has a title (call it CitingArticle), has an 3 Similar to the previous use case, we assume that both Google Scholar’s and CiteSeer’s render their search results in RDFa (i.e. the RDF triples are embedded in HTML), as many companies started to do nowadays. However, Bob can also use a third party’s service (e.g. triplify.org) to extract triples from HTML pages. author that does not contain "Bob Hacker" or "Hacker B.", and cites another article that has a title (call it CitedArticle), and has an author that contains "Bob Hacker" or "Hacker B.". Figure 12 shows the result of this query. http://scholar.google.com/scholar?q=b http://www.citeseer.com/search?s=“Bo ob+Hacker b Hacker” <g:3> :Title “Prostate Cancer” _:1 :Title “Prostate Cancer” <g:3> :Author “Hacker B.,Hacker A.” _:1 :Author “Hacker B., Hacker A.” <g:4> :Title “Best and Worst _:2 :Title “Protocols in Molecular Lifestyles” Biology” <g:4> :Atuhor “Bob Hacker” _:2 :Atuhor “Bob Hacker” <g:4> :Cites <g:3> _:2 :ArticleCited _:1 <g:7> :Title “Protein Categories” _:3 :Title “Cancer Vaccines” <g:7> :Atuhor “Bob Smith” _:3 :Atuhor “Eve Lee, Bob Hacker” <g:7> :Cites <g:3> _:4 :Title “Overview about Systems <g:7> :Cites <g:4> Biology” <g:8> :Title “Cancer Vaccines” _:4 :Atuhor “Tom Lara” <g:8> :Atuhor “Alice Hacker” _:4 :ArticleCited _:1 <g:8> :Cites <g:3> _:4 :ArticleCited _:2 Bob has a PhD in bioinformatics. He is looking for a full-time, well paid, and research-oriented job in some European countries. He spent an enormous amount of time searching different job portals, each time trying many keywords and filters. Instead, Bob used MashQL to find the job that meets his specific preferences. Figure 13 shows Bob’s queries on Google Base and on Jobs.ac.uk. First, he visited Google Base and performed a keyword search (bioinformatics OR "computational biology" OR "systems biology" OR e-health); he copied the link of the retrieved results from Google (which are in rendered in RDFa) into the RDFInput module; and then created a MashQL query on these results. He performed a similar task to query Jobs.ac.uk. The third MashQL module in Figure 13, mixes the results of the above two queries and filters them based on location preferences (provided in the UserInput module). The SPRQAL equivalent to Bob’s MashQL query is shown in Figure 14.
This article proposed a language that allows people to query and mash up structured data without any prior knowledge about the schema, structure, vocabulary, or technical details of this data. Not only non-IT experts can use MashQL, but professionals can also use it to build advanced queries.
MashQL supports all constructs of the W3C standard SPARQL, except the “NAMED GRAPH” construct, which is introduced for advanced use, i.e. switching between different graphs within same query. To be close to user needs and intuition, we defined new constructs (e.g. OneOf, union “\”, Without, Any, reverse “~”, and others). The constructs are not directly supported in SPARQL, but emulated. We plan to include aggregation and grouping functions; especially as they are supported by Oracle’s SPARQL. Yet, MashQL does not support inferencing constructs (such as SubClass, or SubProperty), which are useful indeed for data fusion. As these constructs are expensive to compute (thus lead to bad interactivity of MashQL), we plan replace the Oracle’s semantic technology that we are currently using as an RDF store, with an RDF index that we are developing, for speedy OWL inferencing.
We have downloaded most of the public RDF sources, on which our MashQL editor will be deployed online next month. Not only people will benefit from this, but we will also have the opportunity to better evaluate the usability of MashQL and its contribution to linking and fusing more data bottom-up.
We are indebted to Dr. George Pallis, Dr. Demetris Zeinalipour, and other colleagues for their valuable comments and feedback on the early drafts of this paper. This research is partially supported by the SEARCHiN project (FP6-042467, Marie Curie Actions).