Collaborative filtering techniques are commonly used in social networking environments for proposing user connections or interesting shared resources. While metrics based on access patterns and user behavior produce interesting results, they do not take into account qualitative information, i.e. the actual opinion of a user that used the resource and whether or not he would propose it for use to other users. This is of particular importance on educational repositories, where the users present significant deviations in goals, needs, interests and expertise level. In this paper, we propose the introduction of sentiment analysis techniques on user comments regarding an educational resource in order to extract the opinion of a user for the quality of the latter and take into account its quality as perceived by the community before proposing the resource to another user.
Recommender Systems are of particular importance within social environments, where users share access to a common set of resources. The variability of crucial user characteristics, like their background, their special interests, their degree of expertise, pose interesting issues in terms of proposing a resource that is interesting, useful and comprehensible to a particular user.
Collaborative filtering approaches based on explicitly given user ratings do not always reflect the differentiation between the various criteria that apply to a resource and the weight that the users give to each criterion. On the other hand, techniques that examine access patterns may suffer from the appearance of stigmergy phenomena. The visibility of a resource, or even more elaborate features like the time spent in a resource, the amount of downloads etc. are not directly connected to its quality or suitability. Hence, the examination of access and use patterns can lead to poor recommendation that will be further propagated due to the users continuing to follow previously defined paths within the repository of available content.
In this context, we propose the exploitation of user generated comments on the resources of a repository of educational content in order to deal with the lack of explicit ratings and discover qualitative information related to a specific resource and the impressions it left to the users that accessed it. To this end, we applied sentiment analysis to comments on educational content and examined the accuracy of the results and the degree to which they reflect user satisfaction.
The rest of the paper is structured as follows: we provide a brief review of the sentiment analysis in
Section 2. We present the four algorithms that we aim to implement and examine for the
Organic.Edunet recommendation system in Section 3. Section 4 describes the experimental setup and the
results for the first of the proposed approaches. We conclude with our conclusions so far and report on
the intended next steps.
Recommender systems, particularly using collaborative filtering techniques, aim to predict the
preferences of an individual (user/ customer) and provide suggestions of further resources or entities (other
users of the same system, resources, products) that are likely to be of interest. The usage of
recommender systems is widely spread in e-commerce environments [
Sentiment analysis regards extracting opinion from texts and classifying it into positive, negative or
neutral valence [
Lexical approaches rely on the creation of appropriate dictionaries. The terms present in the dictionary are tagged with respect to their polarity. Given an input text, the presence of dictionary terms is GDH[LQP DGQ HWK HURDOY QHVWLP RI WHK W[ LV FRSXWHPG EDVHG RQHWK LVF[WHQ RI ³SH´RLWYV DGQ WDHJ´³QLYUVPLWKZQ Despite its simplicity, the lexical approach has produced results significant EHWUDQKFRL³ -WR´V [7, 8, 9]. The way of constructing the lexica that are used for sentiment analysis is the subject of several works. In [10] and [11] the lexicons comprised solely adjective terms. 7HKDJVXRISWLYUGVZONHR³JG´DQEG³´QDHLUWKRDVFQLWKZHDUJWZRGVLDOR frequently met approach. In [9] and [12], the minimum path between each target word and the pivot terms in the WordNet hierarchy was calculated in order to determine the polarity of the term and its inclusion in the dictionary. In [8], the authors executed search queries with the conjunction of the pivot words and the target word given as input. The query that returned the most hits determined the polarity of the given word.
Machine learning techniques focus on the selection of feature vectors and the provision of tagged corpora to a classifier, which will be used for analysing untagged corpora. The most frequent routes for choosing the feature vectors are the inclusion of unigrams or n-grams, counting the number of positive/ negative words, the length of the document etc. The classifiers are usually implemented as a Naive Bayes classifiers or as Support Vector Machines [9, 13]. Their accuracy is dependent on the selection of the aforementioned feature vectors, ranging in the same space as the lexical approaches (63%-82%). 3
Algorithms under Analysis For our experiments, we aim to examine the following sentiment analysis algorithms and evaluate their performance in order to deploy the most suitable for a repository of educational content.
The fact that we are dealing with user generated content drives us to take into account its unstructured nature and the potential unbalanced distribution it may present. This gives rise to the fact that our training set may be unbalanced and therefore learning may not be able to cope with such diversity in the number of instances per class. Hence, these properties require simulating sentiment representations onto which the input text will be mapped, since sentiment prediction calls for predefined knowledge. Therefore, we focus on lexical approaches for capturing the polarity expressed in a comment. Specifically, we produced implementations of the following algorithms. 3.1
Affective TermsFrequency Rather small documents or text chunks that carry a certain kind of sentiment polarity have been found to present that valence throughout the text, or in most parts of it. For example, in tweets we observe DFHVOLN'³RWQ¶X\MYH LWKVFDHU"P,¶J´QKXDSLHFRIGUVWKED\LOI tracing negative terms is rather low.
This observation gives rise to determining the affective term frequency that appears in cases as the above mentioned. In order to capture the overall sentiment expressed in such inputs, we proceed as follows:
The algorithm receives as input the text to be processed and two lists of affective terms, one of positive and one of negative valence.
For every word of the text to be processed, we examine whether it is mapped on either of the two lists [14]. If it matches an entry of in either of them, a corresponding value gets incremented by 1. After having traversed the whole text, we compare the two sums and the one with the highest value is con
KHUZǻLVHWKUFGQLIEHWZKSRLWHVYGDQWHJLYXVP[ i is each instance in the vector representing the positive valence of weight wi, wi is the value of the positive weight of instance x, y is each instance in the vector representing the negative valence of weight wj and wj is the value of the negative weight of instance yi.
If ǻ = 0, the sentiment is neutral.
Else, if ǻ > 0, we calculate:
KHUZǻ 1 is the absolute difference between the positive and neutral sums, zk is each instance in the vector representing the neutral valence of weight wk and wk is the value of the neutral weight of instance zk. If ǻ 1 = 0, the sentiment is neutral, else it is positive.
Else, we calculate: ࢤ ൌ ࢞ ି ୀ ࢤ ൌ ࢞ ࢤ ൌ ݕ
KHUZǻ 2 is the absolute difference between the negative and neutral sums. If ǻ 2 = 0, the sentiment is neutral, else it is negative. 3.3
When we are called to tackle a more specific problem, e.g. a more targeted question in a more concrete domain, it is required that our processing is also more focused on the object, i.e. the entity or entities that represent it, and/or the domain towards which opinion is expressed. This way, we aim at capturing the entity characteristics that affect sentiment rendering. For example, we may come across a HFRWQPOLN,³DGVEW\HKRN,DUGYQ¶XILWHKOVEGRYL really grHDW´ 7HKSVLRUYXZWDSURHVFKGOXZFDVLI\KWFRHQPDVEJHLQDWYVDUHJG the video, while the writer expresses positive opinion about it.
Moreover, comments, in specific, comprise in a lot of cases unstructured chunks and other abnormalitLHV FXVK DV HRWLFPQV FDUWLIO RUGZV HJ H³\´V SFWXQDLR DWLVHPN HJ :DWK³ LV "DWKRUH\V´GQPFLJOYIDX
In consideration of such differentiations, we count the distance between the affective terms and the terms that are involved in the representation of the entity towards which sentiment is expressed. If sentiment is expressed towards more than one entity or entity representations, then all distances are counted recursively.
The algorithm receives as input the text to be processed, two lists of affective terms, one of positive and one of negative valence and the entity/entities towards which sentiment is expressed. In the rest of the paper we will also refer to these latter terms aH³NVRU\GSZOFW´LIDQP
The position of the entity towards which sentiment is expressed is tracked in the text to be processed. The words of the document to be examined are mapped onto the affective terms of the input lists. If the document contains an affective term, then its position in the text is tracked as well and we calculate the distance that separates them.
To be more specific, we detect whether the affective term precedes or follows the entity in question. After having located these two points in the text, we calculate the distance between them, in the substring that separates them [19]. This is based on word count versus character count, because in this approach words are considered to be autonomous semantic meaningful units, unlike alphanumeric strings, regarded as self-contained units in graph-based approaches.
The above mentioned procedure takes place for all affective terms. In particular, for every positive term that appears in the input text, its distance from the entity in question is counted. After all positive terms have been checked, the smallest distance is kept to be compared with the respective smallest distance between the negative terms and the entity in question. If the two values equal to zero, or are equal, neutral sentiment is attributed. Otherwise, the post receives the sentiment represented by the smaller of the two values. If we have more than one entity representation, the same procedure is applied and again the shortest distance is taken as representing the sentiment of the writer.
KHUZǻLVHWKUGIFQRHWKTDLQXGRHWJKLFZRIHWKRZFVUHLKJ[LVHWKQP imum distance between the positive term and the key word and y is the minimum distance between the negative term and the keyword. If ǻ > 0, the sentiment is positive, else if ǻ < 0, the sentiment is negative, else, the sentiment is neutral. 3.4
More formal documents tend to present a more consistent and accurate syntactic and grammatical structure, hence more concrete and concise textual forms. This characteristic restricts the number of alternatives we may have in expressing a certain meaning and therefore facilitates us to capture it. As a result, the better we are able to represent this structure, the closer we get in capturing the semantics it pertains [20].
In our approach, we are interested in detecting the syntactic dependencies between the keywords and the affective targeted term(s). In particular, when the sentiment analysis algorithm accepts a text as input, it accepts it attached to certain categories and/or the description of the educational material in question. As a consequence, our goal is to track the sentiment of the writer in relation to the material we are examining. For this reason, we process every sentence of the particular comment so as to identify whether a reference of the material is attached to an affective term. Syntactic Parsers provide the necessary tools to analyze the input text. An example is illustrated in Figure 1, where in the second line HZFDQWHODKHZQUILJWRHKUODWLRVSQIRWZUGVHZUIWRUGZVWXR³K\& i´DQGWHKLGNQRIUHODWLQ KGELQV HWKPLVE³QMX´OHDP\&L´K³QVWHKVEMXFRIDHUEYGDQ UWRV´³LXZK\ the attribute of the same verb.
On the other hand, if a sentiment-bearing word is tracked, we try to identify whether this sentiment word renders sentiment to the word/words describing the material in question. If the examined text reports the beliefs of another person, the sentence being examined is considered neutral. To identify such cases, we use the respective lists given as input to the algorithm. If the sentence contains a verb also found in the assertions or counterfactuals lists, the process is stopped, the sentence is appointed with a neutral value and the analysis continues for the next sentence. In the case of the existence of verbs denoting counterfactual, the list is employed taking into consideration that we contemplate at segregating secondary if-clauses that are dependent from a question verb, that is when the main clause they depend on regards indirect speech, versus if-FODXHVWKD G¶tRQ depend on question verbs, that is when the main clause they depend on regards counterfactuals [21].
Otherwise, we investigate the existence of valence shifters within the examined sentence. At this moment, we take into account negations and comparisons. In the first case, if a word that discloses negation is present (e.g. R³Q´ R³QW´ HWF GDQ LW LV WDFVQ\LO DRFLWHVG WLKZ WHK found affective term, the latter term is considered to carry the opposite polarity value. In the case of comparisons, the affective term pertains to both the compared entities. We distinguish two general cases: x One entity accepts the actual valence of the affective word and the other one the opposite. In specific, the valence to be accredited is decided in reference with the syntactic relationship between the keyword term under examination and the word that discloses comparRLVHQJW´D³K x Both entities accept the valence of the affective word. Specially, the valence to be accredited is decided in reference with the syntactic relationship between the keyword term under examination and the word that discloses comSDRULVHQ³´J
We first eliminate stop words from the keyword list. Therefore, a new set of keywords is created. For every word of the text to be processed, we examine whether it is mapped on this new set and on the other four lists. For every sentence of the input text, if an entry of the lists in the reporting verbs is met GDQ WHK GHSFQ\ WDK GELQV LW WR WHK WHV[¶ RSLQ n holder is of subject type, neutral sentiment is attributed. Else, for every word of the text to be processed, we examine whether it is mapped on either of the two lists containing the affective terms. If they match an entry in either of them, if no negation or comparison dependencies are met, a corresponding value gets incremented by 1, else, the reverse one.
After having traversed the whole text, we compare the two sums and the one with the highest value is considered to be the dominant one and the respective valence is attributed to the input text. If they are equal or no sentiment is detected, the text is considered to carry neutral sentiment. 4
Results A set of experiments have taken place so as to evaluate the performance of each algorithm. At this moment, we have completed and present here the results for the first of the presented algorithms. Given the fact that our task is a classification one, standard classification metrics from the literature have been used. In specific, we try to detect the precision, recall and accuracy values obtained from the above described input data sets.
We wanted to detect opinion in three classes, i.e., positive, negative and neutral. So, precision will show us for each of the positive class how many of the positive instances found are indeed positive; recall will show how many of the positive instances have been found out of the total number of the positive instances that should have been found are indeed positive. The same measures will be given for the other two classes; finally, accuracy will show for each data set how many instances were correctly classified as far as all three classes are concerned.
For an initial corpus of user generated reviews, we used content from the Merlot1 repository. Merlot is an online repository distributing free access to resources for learning and online teaching. It provides learning material of higher education aiming at promoting access to scientific data and as a result to their manipulation and exploitation by research communities. Reaching its instructional objectives necessitates ensuring that the quality of its content is of high standards. It, therefore, accredits reviews and peer reviews, attending on continuously enhancing their quantity and quality. Our system aims at enabling this procedure by proposing a way of evaluating automatically opinions expressed for the learning materials and thus contributing to enabling the community accessing valuable data and promoting its scientific goals.
Within Merlot, we are interested in the user comments and the expert reviews associated with each educational resource. To be more specific, users and community experts have expressed their opinion in respect of its quality, its orientation and the degree to which it complies with helping the user exploit its potentials. :HUIWRKPFDJV\X³HQ´GWRKODUV³[SH´LYZ The expert reviews provide an evaluation for three distinct subcategories, namely (a) Content quality, (b) potential effectiveness as a teaching tool and (c) ease of use for both students and faculty.
For each category of the corpus we have performed two experiments, as provided by the two sets of lists respectively. Our first category regards the processing of the 6792 user comments stored in the Merlot repository. These comments have been considered as attributing positive opinion with respect to a research material if they have been rated with 5 or 4 stars, neutral if they have been attributed 3 and otherwise negative.
As peer reviewers state their opinions with respect to strengths and concerns in each of the aforementioned subcategories, the neutral class is empty in this context. To be more specific, for each subcaRWHJU\ZHDYKGWVURXWHV¶\PSRDUFIQLJWKZERHWVRIOWLVQHWK VWH[FDU ying sentiment. 4.1
Construction of the Listsof affective Terms For the experiments conducted thus far, two sets of lists from the literature have been tested as input, both of which contain positive and negative terms. No list of neutral terms has been taken into considFHUODWLRVQXG¶SKY
The first set of lists is provided by [22] GDQHZLOUIWRVDHK1(:³$´EWX7 he second is derived from SentiWordNet [23].
Namely, SentiWordNet is a lexical resource for opinion mining. It assigns to each synset (synonym set) of WordNet three sentiment scores, each representing respectively: positivity, negativity, objectivity. In specific, according to WordNet, a synset or synonym set is defined as a set of one or more synonyms that are interchangeable in some context without changing the truth value of the proposition in which they are embedded.
The values of positivity, negativity and objectivity follow the rule: where,:
ݐ݅ݕݒݏ ݕݒ݊݁݃ܽݐ݅ ܾ݆݁ܿݐ݅ݕݒ ൌ ͳሺሻ x positivity describes how positive the terms contained in the synset are, x negativity describes how negative the terms contained in the synset are and x objectivity describes how neutral the terms contained in the synset are.
Our goal was to extract two lists of words, positives and negatives. Due to the fact that both positivity and negativity values are assigned to a word we needed to make sure the word was clearly biased. So, each of the three classes can take values from 0 to 1 and they are complementary, as deduced by the formula.
The rule we've used for extracting the lists is: ݐ݅ݏݒݕ ݒ݊݁݃ܽݐ݅ݕ ͲǤ
Ƭ ܾ݆݁ܿݐ݅ݒݕ ͲǤʹሺሻ
This way we check that the word has a positivity or negativity value above 70% and from the rest of the percentage, at least 20% goes to objectivity leaving only 10% max for the opposite sentiment.
By applying the check defined in (7) we make sure there is a clear bias towards the positivity or negativity and the rest is assigned to objectivity.
Finally, we have created a subset of lists from the two above mentioned subsets, i.e. the ANEW and the SentiWordNet ones. To be more specific, two hash tables have been created one containing the positive ANEW terms and the other the positive SentiWordNet WHUVP,IDHN\RIWKLUVDEOHWQZ¶ also a key entry in the second one, it was added in the new list. Having applied the same de-duplication procedure for the negative terms, we obtained two new lists, containing all terms of the first and the second list with unique entries. 4.2 The respective results of each subcategory are presented in the following tables.
Tables 1 and 2 show the precision and recall achieved by the current system version for user comments and experts reviews respectively. What is of interest is that the User Comments present very high accuracy in the positive class, unlike the negative one. The reason for such results is the unbalanced distribution of instances per class in the specific input set. Moreover, we can tell that, when prior sentiment knowledge is received as input via the ANEW lists, precision and mostly recall is higher than when SentiWordNet or Mixed lists are adopted.
SentiWN Both
SentiWN
Both ANEW SentiWN
Both ANEW SentiWN
Both
Conclusions & Future Work The preliminary results of the sentiment analysis on user comments in the context of a repository of educational resources indicated that there can be valuable qualitative information that can be added to a UHFGRDWLQPVYDGQEHGVXWRDGMVXWHKSUFLYG´DWJ³QRIDHLYJQUVRFXE\DSVHFLI user. The accuracy of the first of the examined algorithms, while satisfactory, leaves room for improvement. We expect that more elaborate techniques that introduce association of entities and contextual information will produce better results. However, it is important to note that sentiment analysis does not suffer much from domain differentiation or variability on user roles (that is, the results for expert reviews and general user comments presented similar success). An interesting remark regarding the linguistic characteristics of the examined content is that the criticism is usually employed using mild terminology, which is in contrast of user-generated reviews for products/ movies etc. This indicated the necessity of repeating the experiments with different thresholds for the restriction employed in (7), as a review considered neutral or even positive by the system is actually negative but the phrasing of the reviewer is not strong enough to provide strong indications of his/ her polarity. Our immediate next step is to measure the performance of the remaining sentiment analysis algorithms and draw conclusions for their suitability in the context of large-scale educational repositories. Following the finalization of the sentiment analysis methodology, we intend to incorporate the results in the recommendation system for the Organic.Edunet platform LQRUGHWSFXDLEOVW³\FRUH´ID user-resource pair or a community-resource pair. Our aim is to define this score in a way that reflects both quantitative (visits, access time, downloads) and qualitative (opinions) characteristics. The foundation of our envisioned approach is the building of a connectivity graph between the sysWH¶VPUXV and communities with respect to their profile similarity and their interests as perceived by their activity. The sentiment analysis module will be used for extracting their opinion on the overall quality of the resources they have reviewed or commented on, as well as more specific characteristics (ease of understanding, innovation) where such features can be recognized by the linguistic analysis of the reviews/ comments. The sentiment score will be incorporated in the calculation of the trust and reputation scores of the users and resources will be proposed to other members of the community based on these scores. The research leading to these results has received funding from the European Union Seventh Framework Programme, in the context of the SYNC3 (ICT-231854) project.
This paper also includes research results from work that has been funded with support of the European Commission, and more specifically the project CIP-ICT-PSP-DQ2UJLF/³X'RHWVP the potential of a multilingual WeERUSWDOIELQHV6XOUFD$JW(LQRHYPODF´GWXRI the ICT Policy Support Programme (ICT PSP).
Acknowledgments
on feren v c e e
O o ri n n C ta o ti m o p n u n d tio
G n a r l d L a i b n il g i u t i y s o n ics , S en t Ne e w ce Br
S u u n bje swi c c ti k t y.
NJ v Un 0 0 O su 2 ) rie
C
A .
I :
In kp en u tation Ber gler o f Blo g : a l a s.
Sen In t n d In ti el M : m li .
I en t Cl gen ce, Ursean ternati f M 06 ) Ar n c e e o i a d
P o u
R
U
C
V N o o n M W e o
Eq gs u a a n l: d n g M G d r ( e I
Di f CW fer e SM n ces -2 00 n ), w A o r A k A f I o r C
M on o d fe r eli en n c g e
I o n f n u A e r n ti c f O al pin In io tel n l s ig a e n n d c S , ru ct V an r o e u i v n e S , o cia BC, J M ul e y
In: 0 7 , P p r p o . ± i c v r a d
D t
M n n ) , p .
1 1 ,
S ,
U
S
A , 2 2 ± 2 ,
A . 9 . 1 0 . 1 1 . 1 2 . 1 3 . 1 4 . 1 5 . 1 6 . 1 7 . 1 8 . 1 9 . 2 0 . 2 1 . 2 2 . 2 3 . e c S y s T E
L Me 0 9 n . c f m
L ? i i n d g = u 1 5 , G ( e C n
O ev
L a I , t e l 4 ), 23 2 , n e d ,
R .
B a r -L +D P Q G ² h c g , U
S
A , 1 3 ± 1 , J l 0 8 S n
S . : 1 3 f r n e n
W e s a d
S c a M d a ( I C
W
S
M 2 0 0 ) , p . h r p p , U
S
A 6 2 8 u a r n a d l t L h a e n g 4 u th ag
I e n t P e r r o n c a e tio ssi n n a g , L S a u n n g t u e e S P n o a e o s e t A e -G VH%D S KURF$ W 2LQ ´ J 0 3 RHIK b e o n
W e b
S e a r c h a d
W
D t
M n n , p .
2 3 1 2 4 0 P o n C fer on en fer c e e n o n ce ( Kno ISW w C C ), r S ( n K ib e C l A Is
P la 2 n 0 0 d , ± n jun Oct ti b n r t
N a t u r
L
P , . 1 8 0 ± 1 8 9 S , 1 , T h e
C e n t e r f o r
R
P , U
F n
T
C , .
D l ,
I , 2 2 2 3
O ,