<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Generic knowledge-based analysis of social media for recommendations</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Victor de Graaff</string-name>
          <email>v.degraa @utwente.nl</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Maurice van Keulen</string-name>
          <email>m.vankeulen@utwente.nl</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Anne van de Venis</string-name>
          <email>a.j.vandevenis@student.utwente.nl</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Rolf A. de By</string-name>
          <email>r.a.deby@utwente.nl</email>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>General Terms</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Algorithms</institution>
          ,
          <addr-line>Design, Experimentation</addr-line>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Dept. of Computer Science, University of Twente</institution>
          ,
          <addr-line>Enschede</addr-line>
          ,
          <country country="NL">The Netherlands</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>Fac. of Geo-Information Science, &amp; Earth Observation (ITC), University of Twente</institution>
          ,
          <addr-line>Enschede</addr-line>
          ,
          <country country="NL">The Netherlands</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Recommender systems have been around for decades to help people nd the best matching item in a pre-de ned item set. Knowledge-based recommender systems are used to match users based on information that links the two, but they often focus on a single, speci c application, such as movies to watch or music to listen to. In this paper, we present our Interest-Based Recommender System (IBRS). This knowledge-based recommender system provides recommendations that are generic in three dimensions: IBRS is (1) domain-independent, (2) language-independent, and (3) independent of the used social medium. To match user interests with items, the rst are derived from the user's social media pro le, enriched with a deeper semantic embedding obtained from the generic knowledge base DBpedia. These interests are used to extract personalized recommendations from a tagged item set from any domain, in any language. We also present the results of a validation of IBRS by a test user group of 44 people using two item sets from separate domains: greeting cards and holiday homes.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Recommender systems</kwd>
        <kwd>knowledge-based</kwd>
        <kwd>DBpedia</kwd>
        <kwd>social media</kwd>
        <kwd>domain-independent</kwd>
        <kwd>language-independent</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Categories and Subject Descriptors</title>
      <p>H.4.2 [Information Systems Applications]: Types of
Systems|Decision support</p>
    </sec>
    <sec id="sec-2">
      <title>1. INTRODUCTION</title>
      <p>
        The aim of a recommender system (RS) is to help people
nd the items they are most interested in. A requirement
to provide personalized recommendations is that the RS has
knowledge of the person using it. In 2013, Facebook claimed
to have 1.11 billion active users [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], and the top-100 pages
CBRecSys 2015, September 20, 2015, Vienna, Austria.
      </p>
      <p>
        Copyright remains with the authors and/or original copyright holders
alone currently have a total of 5.87 billion facebook-likes [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ].
The items that people express a preference for on social
media, whether through a like of a Facebook page, a follow on
Twitter, or a tip on the renewed FourSquare, can be taken to
disclose personal traits of interest and the things they want
to be associated with. This vast amount of information is
the starting point for our Interest-Based Recommender
System (IBRS).
      </p>
      <p>
        But what people express their preference for on social media,
cannot always directly be related to commonly used tags or
words in descriptions in an existing item set. These items
are often example instances of broader concepts. For
example: Cristiano Ronaldo has 103 million facebook-likes at the
time of writing, whereas Soccer (66 million) and Football
(46 million) have considerably fewer facebook-likes.1 Tag
sets or descriptions, on the other hand, are more likely to
contain these broader concepts, as for example is the case
in greeting cards, sports equipment, or campsites with
soccer elds. In fact, one of our validation item sets contains
tagged greeting cards with practically only generic terms
such as soccer/football. To bridge this generalization gap in
a domain- and language-independent way, we use the
multilingual, generic knowledge base DBpedia to automatically
detect broader concepts. We call these concepts the user's
interests. In this paper, we validate our hypothesis that
automated user interest detection can also be used to select
preferred items in an item set, independent of the item set
domain, language and used social medium. As a boundary
requirement to our solution, the cold-start problem, as for
example discussed by Bobadilla et al. [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], needs to be
circumvented. The system we propose shall be seen as a feature
of a larger recommender system, either to bootstrap or to
support that system, rather than as a stand-alone system.
In addition to the recommendation approach we propose in
this paper, we also present the results of a validation thereof.
A user group of 44 people tested our RS, using item sets
from two completely di erent domains: greeting cards and
1Synonyms like this one cause problems as well, and are
discussed in more detail in Section 3
holiday homes. Both the recommendation selection, as well
as the explanation interface were validated by these users,
using their own social media pro le.
      </p>
      <p>This paper is further structured as follows: related work is
discussed in Section 2, the motivation behind this research is
discussed in Section 3, the IBRS technology is presented in
Section 4, while the validation approach and results are laid
out in Section 5, and Section 6 nally contains concluding
remarks and hints at future work.</p>
    </sec>
    <sec id="sec-3">
      <title>2. RELATED WORK</title>
      <p>
        The creation of a RS that makes use of social media or
DBpedia is not a new ambition. Social media have especially
received much attention in the eld of content-based
recommender systems. Fijalkowski and Zatoka presented an
architecture of a recommender system for e-commerce based on
Facebook pro les [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. Guy et al. proposed ve recommender
types, based on social media and/or tags [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. In their
approach, they also presented the users with recommendation
explanation. The social media they focus on however, are
not of the mainstream type, but speci c for the Lotus
Connections suite. The system of He et al., on the other hand,
uses common social media [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Whereas they claim to
overcome the cold-start problem, their system appears to still
su er from the new item cold-start problem, as described
by Bobadilla et al. [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
      </p>
      <p>
        The creation of a RS based on DBpedia has also received
quite some attention already, especially in the eld of
music [
        <xref ref-type="bibr" rid="ref7 ref8">7, 8</xref>
        ] and movie [
        <xref ref-type="bibr" rid="ref10 ref11 ref12 ref13 ref9">9, 10, 11, 12, 13</xref>
        ] recommendation. Di
Noia et al. took it a step further and also bene ted from the
integration of DBpedia in the linked open data (LOD)
initiative. Their movie recommendations are not only based on
DBpedia knowledge, but also on Freebase and LinkedMDB.
A more generic approach to create a RS using LOD was done
by Heitmann and Hayes [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], who use also use LOD to
overcome the cold-start problem. Even though their validation
is based on a music dataset, their approach has the
genericity to be used for other applications as well. Our approach
for broader concept detection through DBpedia is a form
of knowledge-based query expansion. Liang et al. already
showed in [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ] that document recommendation based on the
user's interests improves as a result of query expansion, or
semantic-expansion as they call it.
      </p>
      <p>
        What distinguishes our approach from other RS research,
is that we use both social media pro les and DBpedia data
to create a generic RS. Passant and Raimond, for
example, created a RS based on exported social media pro les
and DBpedia data in [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ], but their approach is limited to
the music-speci c relations in DBpedia. To the best of our
knowledge, the only other generic approach is TasteWeights
by Bostandjiev et al. [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. They build a user pro le based on
social media data, and then apply a collaborative
lteringbased approach to select recommendations. This still implies
all of the three cold-start problem categories: new item, new
user, and new community, again as described by Bobadilla et
al. [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. As it is exactly our goal to overcome the cold-start
problem, our approach is a hybrid between content-based
and knowledge-based, according to the RS classi cation by
Burke and Ramezani [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ]. Basile, Lops et al. would classify
our work as a top-down semantics-aware content-based RS
[
        <xref ref-type="bibr" rid="ref18 ref19">18, 19</xref>
        ].
      </p>
      <p>
        Our work is inspired by Shi et al.'s HeteRecom [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ], which is
based on the similarity calculation HeteSim [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ]. Similar to
their work, our ultimate goal is to nd the matching paths
between a user and the item set that carry the most weight.
In this paper however, we focus on the detection of existing
paths.
      </p>
    </sec>
    <sec id="sec-4">
      <title>3. MOTIVATION</title>
      <p>In this work, we aim to extract recommendations that are
generic in three dimensions: the recommendation approach
shall be independent of the item set domain, the item set
language, and the used social medium. As a fourth criterium,
it shall not su er from any of Bobadilla's three cold-start
problem categories. Below, we discuss the motivation for all
of these challenges:</p>
      <sec id="sec-4-1">
        <title>Domain-independence</title>
        <p>As discussed in the previous section, currently most
recommender systems based on knowledge bases and social media
are focused on one speci c domain. Independence of the
item set domain only allows us to reuse the solution and its
future improvements for multiple applications.</p>
      </sec>
      <sec id="sec-4-2">
        <title>Language-independence</title>
        <p>
          Similar to domain-independence as a requirement for
reusability, a language-independent solution improves the RS's
potential to be used in multiple applications. A sub-requirement
of of language-independence is synonym-independence. As
Zanardi and Capra pointed out in [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ], synonyms are a
typical RS problem, especially for tag-based RSs. The example
of people facebook-liking either the Soccer page or the
Football page from Section 1 already showed that people may
facebook-like di erent pages, while referring to the same
concept. Despite recent e orts by Facebook to merge pages
about the same topic from di erent languages into one page,
and improving the search functionality to help people
nding such pages while searching for their name in a di erent
language, still several pages exist to describe similar
concepts.
        </p>
      </sec>
      <sec id="sec-4-3">
        <title>Social medium-independence</title>
        <p>From the rst form of genericity, domain-independence,
follows another requirement. Several social media, such as
Facebook, LinkedIn, Twitter, Instagram, and Pinterest, are
widely used, and each of these has its own focus. When one
decides to create a RS for job vacancies, LinkedIn may be a
more logical social medium to base the recommendations on
than any of the other, while a RS for touristic hotspots will
most likely lead to another choice. Therefore, to create a RS
based on social media content that is domain-independent,
it shall also be independent of the underlying social medium.</p>
      </sec>
      <sec id="sec-4-4">
        <title>Cold-start problem</title>
        <p>
          The cold-start problem has been widely discussed in RS
literature. Bobadilla et al. categorized it into three
subcategories: the new item problem, the new user problem,
and the new community problem [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ]. Knowledge-based RS
have been designed to overcome all of these problems, but
often require domain-speci c knowledge.
        </p>
        <sec id="sec-4-4-1">
          <title>Colosseum</title>
        </sec>
        <sec id="sec-4-4-2">
          <title>Colosseum</title>
        </sec>
        <sec id="sec-4-4-3">
          <title>Vespasian</title>
        </sec>
        <sec id="sec-4-4-4">
          <title>Pizza</title>
        </sec>
        <sec id="sec-4-4-5">
          <title>Francesco</title>
        </sec>
        <sec id="sec-4-4-6">
          <title>Totti</title>
        </sec>
        <sec id="sec-4-4-7">
          <title>Pizza</title>
        </sec>
        <sec id="sec-4-4-8">
          <title>Francesco</title>
        </sec>
        <sec id="sec-4-4-9">
          <title>Totti</title>
        </sec>
        <sec id="sec-4-4-10">
          <title>Calzone A.S.</title>
        </sec>
        <sec id="sec-4-4-11">
          <title>Roma</title>
        </sec>
        <sec id="sec-4-4-12">
          <title>Rome</title>
        </sec>
        <sec id="sec-4-4-13">
          <title>Italy</title>
        </sec>
        <sec id="sec-4-4-14">
          <title>Stadio</title>
        </sec>
        <sec id="sec-4-4-15">
          <title>Olimpico</title>
        </sec>
        <sec id="sec-4-4-16">
          <title>Rome</title>
        </sec>
        <sec id="sec-4-4-17">
          <title>Italy</title>
        </sec>
        <sec id="sec-4-4-18">
          <title>Stadio</title>
        </sec>
        <sec id="sec-4-4-19">
          <title>Olimpico</title>
          <p>Overcoming all of these four challenges at the same time
has motivated us to create IBRS: a domain-independent,
language-independent, social medium-independent,
knowledge-based RS.</p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>4. CONCEPT &amp; TECHNOLOGY</title>
      <p>The foundation of IBRS is the idea that people are more
likely to be interested in items that have a not too distant
relation with things we know they like. Although things
people express a preference for on social media are typically
in a di erent domain than our item set, they may still give
hints towards a person's interests. In IBRS, we link the
preferred items on social media to resources in the DBpedia
Resource Description Framework (RDF) graph. We use this
graph to explore related concepts, which are then matched
with a known tag set, that is used to label the item set. As
a nal step, we rank the item set based on the number of
matched tags. This concept is illustrated, using the
holiday home domain, in Figure 1. In this example, the user
facebook-liked the Colosseum, pizza, and Francesco Totti.
These facebook-likes are mapped onto DBpedia, and the
DBpedia RDF graph is explored to detect the broader
concepts Rome, Italy, and Stadio Olimpico. These items are
mapped onto holiday home tags, to ultimately match the
user with a speci c holiday home.</p>
      <p>The remainder of this section is structured as follows: RDF
graph exploration is discussed in Section 4.1. The data
model of the IBRS abstraction layer is presented in
Section 4.2. Section 4.3 presents a method for automated tag
generation from descriptions. In Section 4.4 the ranking
mechanism and Facebook-DBpedia mapping approach are
presented. Section 4.5, nally, presents a short introduction
of the IBRS prototype.</p>
    </sec>
    <sec id="sec-6">
      <title>4.1 DBpedia graph exploration</title>
      <p>After matching a facebook-like with a DBpedia resource, we
traverse the RDF graph in exactly two steps. Since RDF
tuples have a subject, predicate and object, RDF graphs are
directed. Therefore, there are four possible di erent
direction combinations to travel from node A through node B to
its second neighbor C.2 In Table 1, we show the top-10 of
second neighbors when traversing the DBpedia graph
starting from the Ei el Tower as node A, using all four possible
direction combinations. DBpedia pages in italics also
occur as tags in at least one of our two validation sets, which
are discussed in detail in Section 5. The rst approach,
A ! B ! C, leads to results describing France, in
uential French people, and several other buildings in France.
The second approach, A B ! C, has some overlap with
the rst approach, but also contains several results
unrelated to France, such as Los Angeles and the United States.
The third approach, A B C, shows some remarkable
buildings throughout Europe, but also very unrelated lists
towards the bottom of the top-10. The fourth and nal
approach, A ! B C, results in several famous French
people, especially scientists. Other starting points show similar
results: the third approach, A B C, shows promising
results for single domain recommendations, whereas the rst
approach shows the best results for broader concept
detection. Since our aim is to match these second neighbors with
a tag set, we use the rst approach, A ! B ! C.</p>
    </sec>
    <sec id="sec-7">
      <title>4.2 Abstraction layer data model</title>
      <p>To ensure IBRS genericity, an abstraction layer is used on
top of the underlying data source, such as a product database.
This abstraction layer can consist of physical tables, views,
or a mix thereof, but we will refer to its items as tables from
here on. The abstraction layer contains two entity tables:
abstract items and tags, and one relationship table:
abstract items tags, as depicted in Figure 2.
The abstract items table contains the id and object type
2Depending on the directions of the relationships, and the
existence of bi-directional relationships, node A may be
equal to node C, as can also be seen in Table 1.
of the items in the item set. The object type eld allows
us to use one IBRS instance for the recommendation of
multiple item sets.</p>
      <p>The tags table contains the tag's id, name, and
dbpedia resource id. The name eld can be used in the
language of the item set tags. Since we have one item set that
is tagged in Dutch, and one item set that is tagged in
English, we added the name eng eld for English tags. The
dbpedia resource id is cached in the database for better
performance.</p>
      <p>The abstract items tags table is a regular relation table
containing the abstract item id and tag id. It also
contains the abstract item type for improved join executions.</p>
    </sec>
    <sec id="sec-8">
      <title>4.3 Tag generation</title>
      <p>
        In case an item set is not tagged, but does contain
descriptive texts, tags can be extracted automatically. Natural
language processing algorithms can be used for this purpose,
such as the named entity extraction and disambiguation
approach by Habib et al. [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ]. We used Habib's approach with
a manually trained model to extract named entities from
holiday home descriptions. A drawback of this approach
is that descriptions are often the result of free-text input.
Phrases such as \only a 3 hour ight from Amsterdam" or
\25 kilometers from the border with France" led to correctly
extracted named entities, but semantically not the best tags
to distinguish this object from others. Therefore, we
additionally removed those tags that tagged a holiday home with
another country than the one it is located in. In total, this
approach allowed us to assign 455,777 (non-unique) tags to
42,148 holiday homes, from which 106,430 tags (of which
12,151 unique) could be mapped onto a DBpedia resource.
      </p>
    </sec>
    <sec id="sec-9">
      <title>4.4 Ranking</title>
      <p>The IBRS ranking method consists of four steps: (1)
retrieving preferred items from social media, (2) matching these
items with DBpedia resources, (3) extracting abstracts from
DBpedia, (4) ranking items based on matched tags. For
performance reasons, several items are cached o ine.</p>
      <sec id="sec-9-1">
        <title>Obtaining preferred items from social media</title>
        <p>To map social media items while remaining independent of
the social medium, we must take into account that not all
APIs are the same. Some social medium APIs allow
developers to nd out what a user's friends prefer, while others
limit the developer to information about the logged in user.
Therefore, when using the Facebook Graph API, we
limited ourselves to the name and category elements of each
facebook-liked page.</p>
      </sec>
      <sec id="sec-9-2">
        <title>Matching social media items with DBpedia resources</title>
        <p>Facebook-likes are mapped onto DBpedia resources through
their name. Those facebook-pages that mapped onto
ambiguous terms in DBpedia were ltered out. To create a
more complete mapping, we used the category element to
post x the name of those pages pages for which the
category element was lled with \movie," \tv show," or
\musician/band." In these cases, we also checked if a page
exists with the additional su x \ (movie)," \ (TV series)," or
\ (band)" respectively. This leads to the following SPARQL
query:
PREFIX dbpont: &lt;http://dbpedia.org/ontology/&gt;
PREFIX dbpres: &lt;http://dbpedia.org/resource/&gt;
# We use the prefixed versions here for readability
SELECT ?uri ?label
WHERE {
# Find exact match with category suffix
{ ?uri dbpont:wikiPageID [].</p>
        <p>FILTER(?uri = dbpres:The_Net_(movie)) }
# Or exact match without category suffix
UNION { ?uri dbpont:wikiPageID [].</p>
        <p>FILTER(?uri = dbpres:The_Net) }
# Or the label version
UNION {?uri rdfs:label "The_Net"@en.}
# Check if page has redirect
UNION { dbpres:The_Net_(movie)</p>
        <p>dbpont:wikiPageRedirects ?uri}
UNION { dbpres:The_Net</p>
        <p>dbpont:wikiPageRedirects ?uri}
?uri rdfs:label ?label.
?uri dbpont:wikiPageID ?wikiPageid.</p>
        <p>FILTER (langMatches(lang(?label),"en")).
# Filter out ambiguous terms
FILTER NOT EXISTS { ?uri
dbpont:wikiPageDisambiguates ?disambiguates } .
# Filter out Wikipedia categories
MINUS {?uri rdf:type skos:Concept}
}
LIMIT 1
Using this approach on a test set of 11,674 unique Facebook
pages, obtained from the likes of 309 users, we were able
to match 2,240 (19.2%) Facebook-pages with a DBpedia
resource.</p>
      </sec>
      <sec id="sec-9-3">
        <title>Extracting abstracts from DBpedia</title>
        <p>
          For all matched DBpedia resources, the abstracts are
retrieved from the SPARQL endpoint provided by DBpedia
[
          <xref ref-type="bibr" rid="ref24">24</xref>
          ] using the following query:
PREFIX dbpont: &lt;http://dbpedia.org/ontology/&gt;
PREFIX dbpres: &lt;http://dbpedia.org/resource/&gt;
SELECT DISTINCT
?o3 (count(?o3) as ?count) ?abstract ?label
WHERE {
# UNION concatenation of mapped FB pages
{dbpres:Vienna ?p1 ?o2} UNION
{dbpres:Recommender_system ?p1 ?o2} UNION
{dbpres:Computer_science ?p1 ?o2}
# Neighboring object has Wikipage
?o2 dbpont:wikiPageID ?o2id ;
# Neighboring object has neighbor
        </p>
        <p>?p2 ?o3 .
# Second neighbor object has Wikipage
?o3 dbpont:wikiPageID ?o3id ;
dbpont:abstract ?abstract ;
rdfs:label ?label .
# English is used as an example
FILTER(langMatches(lang(?abstract), 'en')) .</p>
        <p>FILTER(langMatches(lang(?label), 'en')) .
# Second neighbor object must not be a category
MINUS {?o3 rdf:type skos:Concept}
}
# `Only' the 1000 most important abstracts
ORDER BY DESC(?count)
LIMIT 1000</p>
      </sec>
      <sec id="sec-9-4">
        <title>Ranking items based on matched tags</title>
        <p>
          Each tag that (1) has a dbpedia resource id and (2) is
contained in at least one of the downloaded abstracts, is
marked as a matched tag. The item set is then ranked on the
basis of the number of matched tags. As a nal step, those
items that are too close to a higher ranked item, based on a
pre-de ned distance function, are removed from the ranking.
This last step is added to ensure diversity among the
recommended items. For the recommendation of geographic
objects, as for example in a geo-social RS like the one discussed
in [
          <xref ref-type="bibr" rid="ref25">25</xref>
          ], one can think of the Euclidean distance, but for
more generic purposes the cosine similarity (as for example
discussed in [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ]) of the item's tags may be a good starting
point. The tag input makes our RS domain-aware. However,
since the approach can be applied to any tag domain, we
still consider the concept itself domain-independent. This
in contrast to for example music recommenders that rely on
the artist-song relationship.
        </p>
      </sec>
    </sec>
    <sec id="sec-10">
      <title>4.5 Prototype</title>
      <p>For demonstration and validation purposes, we have created
a prototype of IBRS, using the Cake PHP platform. The
prototype can be used with either one's own Facebook
prole, or by manually combining several DBpedia resources.
It can be accessed through http://ibrs.ewi.utwente.nl.</p>
    </sec>
    <sec id="sec-11">
      <title>5. VALIDATION</title>
      <p>To validate our ranking mechanism, as well as to
determine the user perception of recommendations with
explanations, we validated IBRS in a carefully designed user study
with a test user group of 44 people. We used two
product sets from di erent domains to demonstrate its
domainindependence: greeting cards and holiday homes. The
greeting card set contains Dutch tags, while the holiday homes
did not contain any tags, but only descriptions. From the
holiday homes, we used the English descriptions to extract
(English) tags, to emphasize the potential to use IBRS in a
language-independent way.</p>
      <p>This section is further structured as follows: Section 5.1
describes the item set details. In Section 5.2, we present
the approach taken to validate both our ranking mechanism
and the recommendation explanation interface. Section 5.3
nally, discusses the validation results.</p>
    </sec>
    <sec id="sec-12">
      <title>5.1 Item set details</title>
      <p>The rst item set contains greeting cards from the Dutch
company Kaartje2Go (\Card2Go"). People search through
a collection of cards electronically, which are distributed
through regular (non-electronic) mail by Kaartje2Go in name
of the customer. The customers can choose between sending
greeting cards to one or multiple people at once. 75% of the
purchases are of the latter type, for which the preferences of
the sender are more relevant than those of the (potentially
many) recipients. To facilitate the search, users can search
for tags that have been entered manually by the Kaartje2Go
employees. These tags, which are mostly in Dutch, are
inconsistent in their completeness: for example some of the
soccer cards are also tagged using the names of popular
Dutch soccer teams, but not all of them. Less popular teams
are never mentioned as tags. The top-10 of the translated
greeting card tags can be found in Table 2.</p>
      <p>The second item set contains holiday homes from the
holiday home portal EuroCottage. This item set did not
contain tags, but a description in one, two or three languages
(Dutch, English and/or German). We followed the approach
discussed in Section 4.3 to extract mentions of geographic
places from the English holiday home descriptions. The
top-10 of resulting tags can be found in Table 3. The
advantage of extracting geographic places is that these also often
have Wikipedia pages, which makes them suitable for the
requirement that the tags need to have a dbpedia resource id.
Many pages of the holiday home descriptions were in
German, even though they were entered into the system by the
holiday home owners as English descriptions. As a result
thereof, many German words or phrases were extracted as
geographical references, since the model was trained for
English descriptions. However, the impact of these terms was
practically zero, as these extracted tags were not matched
with an English DBpedia resource.3 For the validation, the
holiday homes were plotted on a map that was zoomed in
on Europe, since most holiday homes in the set are located
there. A relatively small subset of homes outside Europe
could therefore not be displayed on the map, and were
removed from the validation set, just as those without a
coordinate pair. This coordinate pair was also used for the
diversity function: all top-10 holiday homes had to be located
at least 250 kilometers away from higher ranked items.</p>
    </sec>
    <sec id="sec-13">
      <title>5.2 Validation approach</title>
      <p>Our test users were requested to participate through
Facebook, and used their own existing Facebook account for the
recommendations. The test users were not aware of what
they were testing, except for the information that they were
testing a RS. Most test users do not have a background in
computer science, and none of them were aware of how IBRS
works. We asked our test users to validate our algorithm
through a total of 30 questions, split up into three batches
of 10. Once a question had been answered, users could not
return to that question. The rst two batches were intended
3Even though the approach can be applied to any language
contained in the knowledge base, the tags are still matched
with knowledge base resources in the tag language.
to validate our ranking mechanism, the third batch was
intended to determine the user perception of recommendations
with explanations, as compared to recommendations
without explanations.</p>
      <p>For the rst ten questions, users were asked to select their
favorite greeting card from a greeting card pair using the
interface of Figure 3. On one side of the screen, an item from
the top-10 greeting cards according to IBRS was shown. On
the other side, a card was shown that was not tagged with
any of the matched tags. We called these recommendations
Inverted IBRS. IBRS and Inverted IBRS were shown on the
left or right side at random.
For the second batch of ten questions, our test users were
presented with the choice between two holiday homes, in a
similar way. Again, IBRS and Inverted IBRS were shown
on the left or right side at random. For each holiday home,
its location was shown on a map, with the name of the
holiday home and the rst 1000 characters of its description, as
shown in Figure 4.
The nal batch of ten questions required the test users to
rate a recommendation. Each of the holiday homes was one
of the top-10 holiday homes according to IBRS. At random,
a user was assigned to the group of users who received
recommendations with an explanation, as shown in Figure 5,
or without an explanation.</p>
      <p>IBRS
(47%)</p>
      <p>Inverted IBRS
(31%)</p>
      <p>In test runs of the validation process, we determined that
in a set-wise comparison of the two systems, users tended
to prefer the set that was spread out over the map, rather
than one that contained clusters of recommendations. Since
Inverted IBRS is extremely spread out, due to the fact that
items had no relation with the users or each other, this
caused a bias in the validation results. Therefore, we
decided to only compare the results item-wise. Furthermore,
we removed tags with a negative connotation, such as \die,"
or \death."</p>
    </sec>
    <sec id="sec-14">
      <title>5.3 Validation results</title>
      <p>The rst two batches of the validation were used to
determine the potential of the IBRS ranking mechanism. The
results are shown in the pie charts of Figure 6. Figure 6a
shows which system was the test user's preferred system,
based on a majority vote between the two systems. Most
users participated in the validation of both the
recommendation of greeting cards and holiday homes. Each batch was
counted separately. 47% of the users preferred IBRS, 22%
voted equally often for both of the systems, and 31% of the
users preferred Inverted IBRS. In the pie chart of Figure 6b,
the results are shown when the results of holiday homes
with the greeting cards are combined per user. Since this
increases the number of votes per user, ties are less common.
In this scenario, 55% of the users preferred the IBRS results,
while 34% preferred Inverted IBRS.</p>
      <p>The nal batch of the validation was used to determine the
usefulness of the proposed recommendation explanation
interface for holiday homes. The results of this batch are
shown in the histograms of Figure 7. Contrary to our
expectations, users preferred to receive recommendations without
explanations. Using the 5-point Likert scale, the users who
were presented with an interface with explanations rated
(a) Split out between (b) Overall (batches
comgreeting cards and holiday bined)
homes (batches counted
separately)
Figure 6: Most frequent choices per user for the rst two
batches of questions
the recommendations with an average score of 3.3772, while
users without recommendation explanation rated the
recommendations with a 3.4709 on average. From this validation,
we can conclude that people that receive recommendations
based on tags that do not describe them well, are more likely
to reject a recommendation with a \strongly disagree," when
they see the rationale behind the recommendation.
Despite satisfying results with respect to the system's
potential to rank recommendations for users, we should not
forget that many aspects play a role in the decision-making
that cannot (yet) be detected from Facebook pro les. When
choosing either a greeting card, a holiday home, or anything
else, one will always look at domain-speci c item
characteristics. For a greeting card, the user looks at colors, style,
and the occasion the card is sent for. Similarly, for a holiday
home, he looks at price, number of beds, the picture of the
home, and the distance to the beach. For this reason, this
approach shall only be used as a feature of a larger system.
(a) With recommendation (b) Without
recommendaexplanation; average rat- tion explanation; average
ing: 3.3772. rating: 3.4709.</p>
      <p>Figure 7: Recommendation ratings split out by
recommendation presentation interface</p>
    </sec>
    <sec id="sec-15">
      <title>6. CONCLUSION</title>
      <p>In this paper, we presented the approach behind IBRS. We
discussed the concept of mapping items marked as preferred
or liked in social media onto a generic knowledge-base, and
query expansion using DBpedia. We presented the
technology, including the abstraction layer, tag generation
approach, and ranking mechanism. We also presented the
validation results of a test user group. As said, we recommend
to use the proposed and validated approach from this
paper as a feature of a larger recommender system. In a more
complete system, one also needs to take domain-speci c
features, as well as item popularity and other collaborative
ltering features, into account. However, these features would
contradict with our objective to create a generic RS that
overcomes the cold-start problem, and therefore were not
taken into account in this work.</p>
      <p>Currently, IBRS uses all paths in the knowledge base graph
as an indication for a useful recommendation. However,
some paths in the graph actually form a reason not to
recommend that item. For example, in the holiday home
domain, a user is less likely to book a home in his own town,
even though there may be many paths between him and
that holiday home based on his local likes. Furthermore,
some nodes are more useful than other for recommendation.
DBpedia nodes like \European Central Time" have a lot of
incoming paths, while it is unlikely that this actually forms
an interest for this user. The next step for IBRS is to
further improve the ranking mechanism by incorporating these
characteristics and explore the possibility to automatically
detect (negative) weights of paths.</p>
    </sec>
    <sec id="sec-16">
      <title>7. ACKNOWLEDGEMENTS</title>
      <p>This publication was supported by the Dutch national
program COMMIT/. We also thank Mena Habib for his
support in the tag generation process.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <surname>Facebook</surname>
          </string-name>
          , \
          <article-title>Facebook j photos." https://www</article-title>
          .facebook.com/facebook,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>S.</given-names>
            <surname>Bakers</surname>
          </string-name>
          , \
          <article-title>Statistics of the top facebook pages." http://www</article-title>
          .socialbakers.com/statistics/ facebook/pages/total/,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>J.</given-names>
            <surname>Bobadilla</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Ortega</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Hernando</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Gutierrez</surname>
          </string-name>
          , \
          <article-title>Recommender systems survey," Knowledge-Based Systems</article-title>
          , vol.
          <volume>46</volume>
          , pp.
          <volume>109</volume>
          {
          <issue>132</issue>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>D.</given-names>
            <surname>Fijalkowski</surname>
          </string-name>
          and
          <string-name>
            <given-names>R.</given-names>
            <surname>Zatoka</surname>
          </string-name>
          , \
          <article-title>An architecture of a web recommender system using social network user pro les for e-commerce,"</article-title>
          <source>in Computer Science and Information Systems (FedCSIS)</source>
          ,
          <source>2011 Federated Conference on</source>
          , pp.
          <volume>287</volume>
          {
          <issue>290</issue>
          , IEEE,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>I.</given-names>
            <surname>Guy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Zwerdling</surname>
          </string-name>
          ,
          <string-name>
            <given-names>I.</given-names>
            <surname>Ronen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Carmel</surname>
          </string-name>
          , and E. Uziel, \
          <article-title>Social media recommendation based on people and tags," in Proc. of the 33rd intern</article-title>
          .
          <source>ACM SIGIR conference on Research and development in information retrieval</source>
          , pp.
          <volume>194</volume>
          {
          <issue>201</issue>
          ,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>J.</given-names>
            <surname>He</surname>
          </string-name>
          and
          <string-name>
            <given-names>W. W.</given-names>
            <surname>Chu</surname>
          </string-name>
          ,
          <article-title>A social network-based recommender system (SNRS</article-title>
          ). Springer,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>A.</given-names>
            <surname>Passant</surname>
          </string-name>
          , \
          <article-title>dbrec - music recommendations using DBpedia,"</article-title>
          <source>in The Semantic Web{ISWC</source>
          <year>2010</year>
          , pp.
          <volume>209</volume>
          {
          <issue>224</issue>
          , Springer,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>A.</given-names>
            <surname>Passant</surname>
          </string-name>
          and
          <string-name>
            <given-names>Y.</given-names>
            <surname>Raimond</surname>
          </string-name>
          , \
          <article-title>Combining social music and semantic web for music-related recommender systems,"</article-title>
          <source>in The 7th International Semantic Web Conference</source>
          , p.
          <fpage>19</fpage>
          ,
          <string-name>
            <surname>Citeseer</surname>
          </string-name>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>R.</given-names>
            <surname>Mirizzi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. Di</given-names>
            <surname>Noia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ragone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V. C.</given-names>
            <surname>Ostuni</surname>
          </string-name>
          , and E. Di Sciascio, \
          <article-title>Movie recommendation with DBpedia,"</article-title>
          <source>in IIR</source>
          , pp.
          <volume>101</volume>
          {
          <issue>112</issue>
          ,
          <string-name>
            <surname>Citeseer</surname>
          </string-name>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>J.</given-names>
            <surname>Golbeck</surname>
          </string-name>
          and
          <string-name>
            <given-names>J.</given-names>
            <surname>Hendler</surname>
          </string-name>
          , \Filmtrust:
          <article-title>Movie recommendations using trust in web-based social networks,"</article-title>
          <source>in Proceedings of the IEEE Consumer communications and networking conference</source>
          , vol.
          <volume>96</volume>
          , pp.
          <volume>282</volume>
          {
          <issue>286</issue>
          , University of Maryland,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>B. N.</given-names>
            <surname>Miller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>I.</given-names>
            <surname>Albert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. K.</given-names>
            <surname>Lam</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. A.</given-names>
            <surname>Konstan</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Riedl</surname>
          </string-name>
          , \
          <article-title>MovieLens unplugged: experiences with an occasionally connected recommender system,"</article-title>
          <source>in Proceedings of the 8th international conference on Intelligent user interfaces</source>
          , pp.
          <volume>263</volume>
          {
          <issue>266</issue>
          ,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          ,
          <year>2003</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>V. C.</given-names>
            <surname>Ostuni</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Di</surname>
          </string-name>
          <string-name>
            <surname>Noia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Mirizzi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Romito</surname>
          </string-name>
          , and E. Di Sciascio, \
          <article-title>Cinemappy: a context-aware mobile app for movie recommendations boosted by DBpedia,"</article-title>
          <source>SeRSy</source>
          , vol.
          <volume>919</volume>
          , pp.
          <volume>37</volume>
          {
          <issue>48</issue>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>P.</given-names>
            <surname>Symeonidis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Nanopoulos</surname>
          </string-name>
          , and
          <string-name>
            <given-names>Y.</given-names>
            <surname>Manolopoulos</surname>
          </string-name>
          , \
          <article-title>Moviexplain: a recommender system with explanations,"</article-title>
          <source>in Proceedings of the third ACM conference on Recommender systems</source>
          , pp.
          <volume>317</volume>
          {
          <issue>320</issue>
          ,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>B.</given-names>
            <surname>Heitmann</surname>
          </string-name>
          and
          <string-name>
            <given-names>C.</given-names>
            <surname>Hayes</surname>
          </string-name>
          , \
          <article-title>Using linked data to build open, collaborative recommender systems</article-title>
          .,
          <article-title>" in AAAI spring symposium: linked data meets arti cial intelligence</article-title>
          , pp.
          <volume>76</volume>
          {
          <issue>81</issue>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>T.-P. Liang</surname>
            ,
            <given-names>Y.-F.</given-names>
          </string-name>
          <string-name>
            <surname>Yang</surname>
            ,
            <given-names>D.-N.</given-names>
          </string-name>
          <string-name>
            <surname>Chen</surname>
          </string-name>
          , and Y.-C. Ku, \
          <article-title>A semantic-expansion approach to personalized knowledge recommendation," Decision Support Systems</article-title>
          , vol.
          <volume>45</volume>
          , no.
          <issue>3</issue>
          , pp.
          <volume>401</volume>
          {
          <issue>412</issue>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>S.</given-names>
            <surname>Bostandjiev</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. O'Donovan</surname>
          </string-name>
          , and T. Hollerer, \
          <article-title>TasteWeights: a visual interactive hybrid recommender system,"</article-title>
          <source>in Proc. of the 6th ACM conf. on Recommender systems</source>
          , pp.
          <volume>35</volume>
          {
          <issue>42</issue>
          ,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>R.</given-names>
            <surname>Burke</surname>
          </string-name>
          , \
          <article-title>Hybrid web recommender systems," in The adaptive web</article-title>
          , pp.
          <volume>377</volume>
          {
          <issue>408</issue>
          , Springer,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>P.</given-names>
            <surname>Lops</surname>
          </string-name>
          , \
          <article-title>Semantics-aware content-based recommender systems</article-title>
          ,"
          <volume>10</volume>
          2014. Keynote at Workshop on New Trends in
          <source>Content-based Recommender Systems.</source>
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>P.</given-names>
            <surname>Basile</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Musto</surname>
          </string-name>
          , M. de Gemmis,
          <string-name>
            <given-names>P.</given-names>
            <surname>Lops</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Narducci</surname>
          </string-name>
          , and G. Semeraro, \
          <article-title>Content-based recommender systems + DBpedia knowledge = semantics-aware recommender systems," in Semantic Web Evaluation Challenge</article-title>
          , pp.
          <volume>163</volume>
          {
          <issue>169</issue>
          , Springer,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>C.</given-names>
            <surname>Shi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Zhou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Kong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. S.</given-names>
            <surname>Yu</surname>
          </string-name>
          , G. Liu, and
          <string-name>
            <given-names>B.</given-names>
            <surname>Wang</surname>
          </string-name>
          , \
          <article-title>HeteRecom: A semantic-based recommendation system in heterogeneous networks,"</article-title>
          <source>in Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining</source>
          , pp.
          <volume>1552</volume>
          {
          <issue>1555</issue>
          ,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>C.</given-names>
            <surname>Shi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Kong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Huang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. Y.</given-names>
            <surname>Philip</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Wu</surname>
          </string-name>
          , \
          <article-title>HeteSim: A general framework for relevance measure in heterogeneous networks," IEEE Transactions on Knowledge &amp; Data Engineering</article-title>
          , no.
          <issue>10</issue>
          , pp.
          <volume>2479</volume>
          {
          <issue>2492</issue>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>V.</given-names>
            <surname>Zanardi</surname>
          </string-name>
          and
          <string-name>
            <given-names>L.</given-names>
            <surname>Capra</surname>
          </string-name>
          , \
          <article-title>Social ranking: uncovering relevant content using tag-based recommender systems,"</article-title>
          <source>in Proceedings of the 2008 ACM conference on Recommender systems</source>
          , pp.
          <volume>51</volume>
          {
          <issue>58</issue>
          ,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <surname>M. B. Habib</surname>
            and
            <given-names>M. van Keulen</given-names>
          </string-name>
          , \
          <article-title>Improving toponym disambiguation by iteratively enhancing certainty of extraction,"</article-title>
          <source>in Proceedings of the 4th International Conference on Knowledge Discovery and Information Retrieval</source>
          ,
          <string-name>
            <surname>KDIR</surname>
          </string-name>
          <year>2012</year>
          , Barcelona, Spain, (Spain), pp.
          <volume>399</volume>
          {
          <issue>410</issue>
          , SciTePress,
          <year>October 2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24] DBpedia, \SPARQL explorer for http://dbpedia.org/sparql." http://dbpedia.org/snorql/,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <surname>V. de Graa</surname>
          </string-name>
          , M. van
          <string-name>
            <surname>Keulen</surname>
          </string-name>
          , and R. A. de By, \
          <article-title>Towards geosocial recommender systems," in 4th Intern</article-title>
          . Workshop on Web Intelligence &amp;
          <string-name>
            <surname>Communities (WI</surname>
          </string-name>
          &amp;C
          <year>2012</year>
          ), Lyon, France, ACM,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>