Recommender systems apply statistical and knowledge discovery techniques to the problem of making recommendations during live user interaction. This paper describes a novel approach of building recommender systems for the Web with the aid of usergenerated content. Recently certain communities of Internet users have engaged in creating high quality peer reviewed content for the Web. In our approach we are planning to extract the semantics of such user-generated content and to use these semantics to make more useful recommendations.
Recommender systems attempt to predict items (web pages, movies, books) that a user may be interested in, given some information about the user's profile.
Collaborative filtering is the most popular approach
to building such systems. Individual users are
automatically joined in groups based on similarity in
their interests or past behaviour and recommendations
are made based on the preferences of their group
members. Another method in generating
recommendations is based on predicting users’ interests
based on his/her past preferences. In the former
approach new content is compared against user’s past
preferences and similar items are recommended. Both
systems suffer from a number of deficiencies: mainly
they both fail to recommend novel interesting topics.
For example, a recommender system for travellers
based on collaborative filtering can assign a user to a
class of people who visit European capitals. However,
once he visits all of the capitals, this system cannot
recommend anything new to this person. More
generally, it has been described in [
In our work, we avoid this difficulty by generating recommendations of Web pages with the aid of semantics, extracted from user-generated content. This allows us to make recommendations based on the relationships between concepts, created and peerreviewed by a large community of users. It is obvious that building a recommender system for Web pages that extracts semantics from all of the content on the Web would be very resource demanding. Instead we picked Wikipedia as our source of user-generated semantics, which is a comprehensive and up-to-date corpus of knowledge and relationships between concepts.
User-generated content refers to various kinds of content that is produced or primarily influenced by endusers. However, average quality of Web content is quite poor, as evidenced by vast amounts of Web spam and unverified information submitted by non-authoritative users. Therefore, instead of using the Web, we chose Wikipedia as our base, since it is a body of usergenerated content that is all encompassing and at the same time of high quality and peer reviewed.
In every article of Wikipedia links guide users to associated articles, often with additional information, and lists of categories for each article organize Wikipedia articles in a taxonomic structure. These links convey important semantic information that we can use to produce high quality recommendations. Furthermore, any Internet user is welcome to add further information, cross-references, or citations, so long as user do so within Wikipedia's editing policies and to an appropriate standard. So after a time semantically rich and high quality peer reviewed content emerges.
The English-language Wikipedia currently (when article was written) contains more then 1,500, 000 articles (6,000,000 when including redirects, discussion pages and portals).
Furthermore, we can consider that Wikipedia is a form of web summarization because of its broad scope, conciseness and ability to quickly reflect new trends. Therefore, Wikipedia can provide us with extremely useful information about articles and Web page relationships.
The main goal of this research is to develop a recommender system for the Web based on userContent processing
Blogs
Runtime system
Wikipedia has its own markup, links to internal and external articles, redirects and list of categories. All of this information would be useful for our research. So far we are only using article titles and internal links. Though this is only a small part of information could be extracted from Wikipedia, we would be able to get results very quickly and rate their quality.
When you analyze the link topology of Wikipedia carefully, you will notice that in many cases an article will contain links to other articles that are completely unrelated. Thus, for example, in article about Moscow there is a link to an article about Fahrenheit temperature scale. Clearly, we should make a distinction between these kind of links and high quality links such as link from “Moscow” to “Capital of Russia”. So we need a mechanism to clean or rank links on the basis of their quality.
For solving link cleaning task it is necessary to
investigate how such low-quality links appear.
Typically, Wikipedia editors carefully insert relevant
links between key concepts of their article to other
articles. Occasionally a rogue user will insert a bunch of
irrelevant links into an otherwise quality article. We
can see a similar pattern with Web spam, where
spammers create large artificial chunks of the Web to
boost the page rank of some specific site. Therefore we
would like to modify and use emerging Web spam
combating algorithms [
Widely known models of the evolution of the Web
[
So, web graph relates to the class of scale-free networks with most distinguishing characteristic are that their degree distribution follows a power law relationship. The second property of this class of networks is self-similarity: a large-enough supporter set should behave similar to the entire Web. Thus we can guess that properties of Wikipedia links graph and its subgraphs would be same to the Web graph.
The basic idea is to analyze rank distribution of some page in its neighborhood. If link distribution in some article neighborhood isn’t power law we have a high probability that page rank is artificially overstating. Therefore emerging spam detection algorithms require the following properties: • Power-law link distribution • Self-similarity
We have analyzed Wikipedia and found that its link structure follows the power-law distribution (figure 2) and it follows that self-similarity holds for Wikipedia. Hence we can use modified Web spam detection algorithms for this task.
A naïve way to produce recommendations is to
recommend blogs associated with nearest neighbors in
the Wikipedia link graph. However there are serious
problems with this method. Researchers [
Semantic extraction and ontology development is well-studied topic. WordNet is the most successful hand crafted semantic lexicon for the English language. It groups English words into sets of synonyms called synsets, provides short, general definitions, and records the various semantic relations between these synonym sets. WordNet distinguishes between nouns, verbs, adjectives and adverbs because they follow different grammatical rules. Every synset contains a group of synonymous words or collocations (a collocation is a sequence of words that go together to form a specific meaning, such as "car pool"); different senses of a word are in different synsets. The meaning of the synsets is further clarified with short defining glosses (Definitions and/or example sentences).
Simple methods are used in IBM research center to
make automatic semantic annotation of WEB pages.
Seeker is a platform for large-scale text analytics,
described in [
In recent works researches have started to extract
semantics from Wikipedia; the most profound work was
done by Kozlova [
In her work article link structure and article structure itself was used for ontology extraction. For example, if analyze the link [[Capital of France | Paris]] we can easily produce synonyms: “Paris” and “Capital of France. Also, if a document is linked under one of the special sections like “see also”, “similar topics” it indicates, that this document has something to do with the topic.
Unlike the previous works we will extract a more semantically rich ontology. For now we will use categories, “see also” links and general links in the articles.
List of categories form a directed graph over the articles of Wikipedia, which can be very useful in pruning irrelevant links when making recommendations. For example, Wikipedia article about Kurchatov contains links to “physics”, “PhysicoTechnical Institute” and other topics that are poor recommendation candidates. With the aid of categories we can prune these links and recommend more relevant topics such as articles about Kurchatov colleagues. This is the most basic use of our ontology; we will investigate more sophisticated methods in our future work.
Now we deal with preprocessing web logs. In order to find blogs most similar to user preference set it’s necessary to extract terms from each blog and correlate these terms with concepts from the ontology. This will enable us to make recommendations based on these concepts.
When we correlate blogs with concepts each blog
will become associated with a large number of
concepts. In order to identify essential concepts we use
a modified tf-idf weighting scheme [
At runtime the recommender system derives top-N
recommendation from the ontology based on users
preference set. Little research has been done on this
topic. An ontology-based information retrieval model
[
Tapestry [
А myriad of other recommender systems exist,
particularly on e-commerce sites. Schafer [
All of these systems are based on collaborative filtering. Correspondingly they have problems as stated above. We avoid these problems by using high quality user-generated content as a foundation for making recommendations.
In this article we propose a novel architecture for building recommendation systems and formulate the major directions of future work. In the future we plan to modify and apply Web spam detection algorithms for cleaning Wikipedia links. We will then evaluate various approaches to making recommendations using the extracted ontology (we have given a basic example of such an approach in Sections 2.2 and 2.4).
Finally, we will implement a complete blog recommender system based on the described techniques and evaluate it on the Internet users.