Information overload on the Internet motivates the need for ltering tools. Recommender systems play a signi cant role in such a scenario, as they provide automatically generated suggestions. In this paper, we propose a novel recommendation approach, based on semantic networks exploration. Given a set of celebrity gossip news articles, our systems leverage both natural language processing text annotation techniques and knowledge bases. Hence, real-world entities detection and cross-document entity relations discovery are enabled. The recommendations are enhanced by detailed explanations to attract end users' attention. An online evaluation with paid workers from crowdsourcing services proves the e ectiveness of our approach.
The amount of publicly available data on the World Wide Web nowadays has
dramatically increased and has led to the problem of information overload.
Recommender systems try to tackle this issue by o ering personalized suggestions.
News recommendation is a real-world application of such systems and is growing
as fast as the online news reading practice: it is estimated that, in May 2010,
57% of U.S. Internet users consumed online news by visiting news portals [
The online reading practice leads to the so-called post-click news recommendation problem: when a user has clicked on a news link and is reading an article, he or she is likely to be interested in other related articles. This is still a typical editor's task, namely an expert who manually looks for relevant content and
html
builds a recommendation set of links, which will be displayed below or next to the
current article. The primary aim is to keep users navigating on the visited portal.
News recommender systems attempt to automate such task. Current strategies
can be clustered into 3 main categories [
In this paper, we propose a novel news recommendation strategy, which leverages both natural language processing techniques and semantically structured data. We show that entity linking tools can be coupled to existing knowledge bases in order to compute unexpected suggestions. Such knowledge bases are used to discover meaningful relations between entities. As a preliminary work to assess the validity of our approach, we focus on a celebrity gossip use case and consume data from the TMZ news portal and the Freebase graph database.2 For instance, given a TMZ article on Michael Jackson, our strategy is able to detect from Freebase that Michael Jackson (a) is a dead celebrity who had drug problems and (b) dated with Brooke Shields, thus suggesting other TMZ articles on Amy Winehouse, Kurt Cobain (other dead celebrities who had drug problems) and Brooke Shields. We investigate if user attention can be attracted via speci c explanations, which clarify why a given recommendation set is proposed. Such explanations are built on top of the entity relations. Finally, we conducted an online evaluation with real users. We outsourced a set of experiments to the community of paid workers from Amazon's Mechanical Turk (AMT) crowdsourcing service.3 The collected results con rm the e ectiveness of our approach.
Our primary aim is to attract the attention of a generic user, since
postclick news recommendation generally relies on a single click user pro le data.
Therefore, we are set apart from most traditional recommender systems with
respect to three main features:
1. User agnosticity : user interests are deduced from user pro le data and
contribute to the quality of recommendations. Collecting explicit feedback is a
costly task, as it requires motivated users. Our approach gives low priority
to user pro les.
2. Unexpectedness : similarity, novelty and coherence are key components for
satisfactory news recommendations [
Content-based recommendation applies to unstructured text, such as news
articles. Document representation with bag-of-words vector space models and the
cosine similarity function still represent a valid starting point to suggest
topicrelated documents [
Recommender systems evaluation frameworks boil down to two main
approaches [
Our strategy merges content-based and knowledge-based approaches and is dened as a hybrid entity-oriented recommendation strategy enhanced by humanreadable explanations. Given a source article from a news portal, we recommend other articles from the portal archive, namely the corpus, by leveraging both entity linking techniques and knowledge extraction from semantically structured knowledge bases. Speci cally, we gathered a celebrity gossip corpus from TMZ and chose Freebase as the knowledge base.
We consider both the corpus and the knowledge base as a unique object, namely a dataspace, which results from heterogeneous data sources integration. Each data source is converted into an RDF graph and becomes an element of the dataspace. Such dataspace can then be queried in order to retrieve sets of recommendations. A semantic recommender exploits SPARQL graph navigation capabilities to output recommendation sets. Each recommender is built on top of a concept, e.g., substance abuse.
The entity linking step in the corpus processing phase enables the detection of both real-world entities and encyclopedic concepts. We compute concept statistics on the whole corpus and assume that the most frequent ones are likely to generate interesting recommendations. A mapping between corpus concepts and meaningful relations of the knowledge base allows the creation of recom
menders. Table 1 shows the TMZ-to-Freebase n-ary concept mapping we manually built. Each Freebase value represents the starting point for the construction of a recommender, while the string after the last dot becomes the name of the recommender, e.g., parents.
Given an entity of the source article, a name of a recommender and an entity contained in the recommendation sets, we are able to construct a speci c explanation. Ultimately, a ranking of all the recommendation sets produces the nal top-N suggestions output.
TMZ
Family Intimate relationship
Dating Ex (relationship)
Net worth Substance abuse
Conviction
Court
Arrest
Legal case Criminal charge
Judge
Death Television program
TMZ Processing Pipeline. Given as input a set of TMZ articles, we output an
RDF graph and load it into the dataspace. Corpus documents are harvested via
a subscription to the TMZ RSS feed. The RSS feed returns semi-structured XML
documents. A cleansing script extracts raw text from each XML document. The
entity linking step exploits The Wiki Machine,5 a state-of-the-art [
6 The full corpus of TMZ RDFa documents is available at http://bit.ly/QLph9B Knowledge base (Freebase)
Corpus (TMZ) the link to the corresponding Wikipedia page and the annotation con dence score are also expressed. RDFa documents are converted into RDF data via the Any23 library.7 RDF data is loaded into a Virtuoso8 triple store instance, which serves the dataspace for querying.
Freebase Processing Pipeline. Freebase provides exhaustive granularity for several domains, especially for celebrities. Given that such knowledge base is large, we avoid loading its complete version, because of severe performance issues we encountered. Consequently, meaningful slices corresponding to the corpus domains, e.g., celebrities, people, are selected. A domain-dependent subset is then produced via a lter written in Java. The dataset is converted into RDF data with logic implemented in Java. Finally, RDF data is loaded into a Virtuoso triple store instance. 4.1
A recommender performs a join between an entity belonging to the TMZ graph and the corresponding entity belonging to the Freebase graph. TMZ entities are identi ed by a DBpedia URI, which di ers from the Freebase one. Therefore, we exploit sameAs links between DBpedia and Freebase URIs. Recommenders are divided in two categories, namely (a) entity-driven and (b) property-driven.9 For each detected entity of the source article, we run Freebase schema inspection queries10 and retrieve its types and properties. Thus, we are able to recognize which recommenders can be triggered for a given entity. Building a recommender 7 http://incubator.apache.org/any23/ 8 http://virtuoso.openlinksw.com/ 9 The full sets are available at http://bit.ly/MWGu06 and http://bit.ly/MWGsW3 10 Available at http://bit.ly/MVGVtE requires (a) knowledge of relevant Freebase schema parts in order to properly browse its graph and (b) a su ciently expressive RDFa model for named entities and link retrieval. The NAM type and the hasLink property provide such expressivity.
Entity-Driven Recommenders. The queries behind entity-driven recommenders contain an %entity% parameter that must be programmatically lled by an entity belonging to the source article. For instance, given an article in which Jessica Simpson is detected and triggers the sexual relationships recommender, we are able to return all the corpus articles (if any) that mention entities who had sexual relationships with her, e.g., John Mayer. To avoid running empty-result recommenders, we built a set of ASK queries,11 which check if recommendation data exists for a given entity. The sexual relationships query follows: PREFIX fb: <http://rdf.freebase.com/ns/> PREFIX twm: <http://thewikimachine.fbk.eu#> SELECT DISTINCT ?had_relationship_with ?link WHERE <http://dbpedia.org/resource/%entity%> owl:sameAs ?fb_entity . ?fb_entity fb:celebrities.celebrity.sexual_relationships ?fb_sexual_rel . ?fb_sexual_rel fb:celebrities.romantic_relationship.celebrity ?fb_celeb . ?fb_celeb fb:type.object.name ?had_relationship_with . ?dbp_celeb owl:sameAs ?fb_celeb ; a twm:NAM ; twm:hasLink ?link ; twm:hasConfidence ?conf . FILTER (?fb_entity != ?fb_celeb) . FILTER (lang(?had_relationship_with)='en') . ORDER BY DESC (?conf) Property-Driven Recommenders. After the schema inspection step, an entity of the source article can directly trigger one of these recommenders if it contains the corresponding property. Property-driven queries return articles that mention entities who share the same property. Hence, they do not require a parameter to be lled. For instance, given an article in which Lindsay Lohan is detected and the property legal entanglements is identi ed during the schema inspection step, we can suggest other articles on people who had legal entanglements, e.g., Britney Spears.
Building Explanations. Speci c explanations are handcrafted from <s, r, o> triples, where s is a subject entity that was extracted from the source article, r is the relation expressed by the triggered recommender and o is an object entity for which the recommendation set is computed. Therefore, we are able to construct di erent explanations depending on the elements we use. For instance, (a) s,r,o yields: Jessica Simpson had sexual relationships with John Mayer. Read more about him. (b) s,r yields: Read more about Jessica Simpson's sexual relationships. (c) r,o yields: Read more about her sexual relationships with John Mayer. 4.2
Since recommendations originate from database queries, they are unranked and
in some cases too many. To overcome the problem, we implemented an
information retrieval ranking algorithm and are able to provide top-N recommendations.
11 Available at http://bit.ly/NDNORH
The bag-of-words (BOW) cosine similarity function is known to perform e
ectively for topic-related suggestions [
The assessment of end user satisfaction has high priority in our work.
According to Hayes et al. [
Our primary aim is to demonstrate that evaluators generally prefer our recommendations. Thus, we need to put our strategy in competition with a baseline. We leveraged the already implemented BOW+LSA information retrieval ranking algorithm. In addition, we set two speci c objectives, related to the speci c explanation and unexpectedness assumptions, as outlined in Section 1: (a) con rm that a speci c explanation better attracts user attention rather than a generic one; (b) check if the recommended items are interesting, although they may appear unrelated and no matter what kind of explanation is provided.
Quality control of the collected judgements is a key factor for the success of the experiments. The essential drawback of crowdsourcing services relies on the cheating risk: workers (from now on called turkers) are generally paid a few cents for tasks which may only need a single click to be completed. Hence, it is highly probable to collect data coming from random choices that can heavily pollute the results. The issue is resolved by adding gold units, namely data for which the requester already knows the answer. If a turker misses too many gold answers within a given threshold, he or she will be agged as untrusted and his or her judgments will be automatically discarded. 5.1
Our evaluation framework is designed as follows: (a) the turker is invited to read a complete news article. (b) A set of recommender systems are displayed 12 http://hlt.fbk.eu/en/technology/jlsi 13 http://crowdflower.com/ below the article. Each system consists of a natural language explanation and a news title recommendation. (c) The turker is asked to give a preference on the most attracting recommendation, namely the one he or she would click on in order to read the suggested article. A single experiment (or job) is composed of multiple data units. A unit contains the text of the article and the set of explanation-recommendation pairs. Figure 2 shows a unit fragment of the actual web page that is given to a turker who accepted one of our evaluation jobs. Both instructions and question texts need to be carefully modeled, as they must mirror the main objective of the task and should not bias turkers' reaction. Since we aim at evaluating user attention attraction, we formulated them as per Figure 2. Table 2 provides an overview of our experimental environment. The parameters we have isolated for a single experiment are presented in Table 2a. On top of the possible variations, we built a set of nine experiments, which are described in Table 2b. We modeled two Q values, namely direct (as per Figure 2) and indirect (Which recommendation do you consider to be more trustworthy?), to monitor a possible alteration of turkers' reaction. Experiments having A = 5 aim at decreasing the probability a turker gets trusted by chance, because he or she accidentally selected correct gold answers. They have an additional F value in the Rec parameter, as we randomly extracted 3 fake recommendations per unit from a le with more than 2 million news titles. However, such an architectural choice generated noisy results, since it occurred that some fake titles were selected.14 Exp is a key parameter, which allows us to check whether the presence or the absence of a speci c explanation represents a discriminating factor. SExp is intended to measure the e ectiveness of a speci c explanation while reducing its complexity.
Each job contains 8 regular + 2 gold units, namely 5 articles proposed twice, in combination with 2 signi cant (and eventually 3 fake) explanationrecommendation pairs. The recommendation titles of the regular units are extracted from the top-2 links of the baseline and the semantic rankings. Gold is created by extracting the title from the last, i.e., less related link of the baseline ranking, the top link of the semantic ranking and assigning the correct answer to the latter. We collected a minimum of 10 valid judgments per unit and set the number of units per page to 3.
Once the results obtained, it frequently occurred that the expected number of judgments was higher: depending on their accuracy in providing answers to gold units, turkers switched from untrusted to trusted, thus adding free extra judgments. The proposed articles come from the TMZ website, which is well known in the United States. Therefore, we decided to gather evaluation data only from American turkers. The total cost of each experiment was 3.66$.
After visiting some news web portals, we chose the following generic explanations and randomly assigned them to both the baseline and the fake recommendations: (a) The most related story selected for you; (b) If you liked 14 See Table 3 for further details. this article, you may also like; (c) Here for you the hottest story from a similar topic; (d) More on this story; (e) People who read this article, also read. 2 regular units were removed from the relation only and the object + relation experiments: it was impossible to build speci c explanations with an implicit subject or object, since the entities that triggered the recommendations di ered from the main entity of the source article. 5.3
Table 3 provides an aggregated view of the results obtained from the Crowd ower platform.15 With respect to the absolute percentage values, we rst observe that our approach always outperformed the baseline. Furthermore, statistical signi cance di erences emerge when a complete <s; r; o> speci c explanation is given. We ran twice, i.e., in two separate days the pilot experiment and noticed an improvement. The indirect experiment only di ers from the pilot in the question parameter and yielded similar results. The 4 generic + 1 speci c experiment has the highest semantic percentage: this translates into an expected behavior, since the presence of a single speci c explanation against four generic ones is likely to bias turkers' reaction towards our approach. As the complexity of the speci c explanation decreases, i.e., in the subject + relation, object + relation and relation only experiments or when only generic explanations are presented, namely in the 5 generic and same explanation experiments, judgments towards our approach tend to decrease too. Hence, we evince the importance of providing speci c explanations in order to attract user attention. 5.4
Experiments containing a speci c explanation aim at assessing its attractive power (assumption 3). If we compare experiments which only di er in the Exp parameter, namely 4 generic + 1 speci c and 5 generic, pilot 1-2 and Same explanation, in the formers turkers prefer our strategy with a statistically sig15 The complete set of full reports is available at http://bit.ly/MOrN30 ni cant di erence. Therefore, speci c explanations are proven to enhance the trustworthiness of the system.
The evaluation of the unexpectedness factor (assumption 2) boils down to check whether turkers privilege the novelty of a recommendation or its similarity to the source article. In experiments including only generic explanations, namely Same explanation and 5 generic, we noticed the following: (a) no statistically signi cant di erences exist between the strategies; (b) when the baseline returns articles that are unrelated to the topic or the entity of the source article, turkers prefer our strategy and vice versa. Hence, we argue that users tend to privilege similarity if they are given a generic explanation. On the other hand, when the baseline strategy suggests a clearly related article and when a speci c explanation is provided, turkers tend to choose our strategy even if it suggests an apparently unrelated article. This is a rst proof of the unexpectedness factor: users are attracted by the speci c explanation and are eager to read an unexpected article rather than another article on the same topic/entity. 6
In this paper, we presented a novel recommendation strategy leveraging entity
linking techniques in unstructured text and knowledge extraction from
structured knowledge bases. On top of it, we build hybrid entity-oriented
recommender systems for news ltering and post-click news recommendation. We
argued that entity relations discovery leads to unexpected suggestions and speci c
explanations, thus attracting user attention. The adopted online evaluation
approach via crowdsourcing services assessed the validity of our systems. A demo
prototype consumes Freebase data to recommend TMZ celebrity gossip articles
and can be viewed at http://spaziodati.eu/widget_recommendation/. For
our future work, we have set the following milestones:
1. Ecological evaluation. AMT allowed us to build fast and cheap online
evaluation experiments. However, the collected judgments may be biased by
the politeness e ect of the economical reward and the turkers' awareness
of performing a question-answering task. Therefore, we intend to set up an
ecological evaluation scenario, which simulates a real-world usage of our
recommender systems and enables natural user reactions. We will adopt the
Google AdWords16 approach proposed by Guerini et al. [
Acknowledgements. This work was supported by the EU project Eurosentiment, contract number 296277.