This work focuses on aspect-based sentiment analysis, a relatively recent task in natural language processing. We present a new dataset for Czech aspect-based sentiment analysis which consists of segments from user reviews of IT products. We also describe our work in progress on the task of aspect term extraction. We believe that this area can be of interest to other workshop participants and that this paper can inspire a fruitful discussion on the topic with researchers from related fields.
Sentiment analysis (or opinion mining) is a field related to natural language processing (NLP) which studies how people express emotions (or opinions, sentiments, evaluations) in language and which develops methods to automatically identify such opinions.
The most typical task of sentiment analysis is to look at some short text (a sentence, paragraph, short review) and determine its polarity – positive, negative or neutral.
Aspect-based sentiment analysis (ABSA) refers to discovering aspects (aspect terms, opinion targets) in text and classifying their polarity. The prototypical scenario are product reviews: we assume that products have several aspects (such as size or battery life for cellphones) and we attempt to identify users’ opinions on these individual aspects.
This is a more fine-grained approach than the standard formulation of sentiment analysis where the goal would be to classify the polarity of entire sentences (or even whole reviews) without regard for internal structure.
Recently, ABSA has been gaining researchers’ interest,
as evidenced e.g. by the two consecutive shared tasks
organized within SemEval in 2014 and 2015 [
ABSA can be roughly divided into two subtasks: (i) identification of aspects (or aspect term extraction) in text, i.e. marking (occurrences of) words which are evaluated; (ii) polarity classification, i.e. deciding whether the opinions about the identified words are positive, negative or neutral.
In this work, we introduce a new Czech dataset of product reviews annotated for ABSA and describe a preliminary method of aspect term identification which combines a rule-based approach and machine learning. 2
We downloaded a number of user product reviews which are publicly available on the website of an established Czech online shop with electronic devices. Each review consists of negative and positive aspects of the product. This setting pushes the customer to rate its important characteristics.
The dataset consists of two parts: (i) random short segments and (ii) longest reviews. The difference in length is reflected also in the use of language.
The first part of this dataset contains 1000 positive and 1000 negative reviews which were selected from source data and their targets were manually tagged. These targets were either aspects of the evaluated product or some general attributes (e.g. price, ease of use). The polarity of each aspect is based on whether the user submitted the segment as negative or positive. These short reviews often contain only the aspect without any evaluative phrase.
The second part of dataset consists of the longest reviews. We chose 100 of them for each polarity. These reviews represent more usual text and they tend to keep proper sentence structure. The longest review has 7057 characters.
The whole dataset provides a consistent view of language used in the on-line environment preserving both specific word forms and language structures. There is also a large amount of domain specific slang due to the origin of the text.
Random, positive Random, negative Longest, positive Longest, negative #targets 640 508 484 353 #reviews 1000 1000 100 100
34.17 39.72 953.35 855.04
The data was annotated by a single annotator. The basic instruction was to mark all aspects or general characteristics of the product. The span of the annotated term should be as small as possible (often a single noun). For evaluation, the span can be expanded e.g. to the immediate dependency subtree of the target. Any part of speech can be marked; e.g. both “funk cˇnost” (“functionality”) and “funk cˇní” (“functional”) should be marked. a-tree zone=en
The fried AuxA Atr DT JJ is .
Pred AuxK
VBZ . rice amazing Sb Pnom
NN JJ
The whole dataset contains 1985 target tags; 1124 of these are positive and 861 are negative. Detailed target statistics are shown in Table 1.
The dataset is freely available for download at the following URL:
http://hdl.handle.net/11234/1-1507. 3
Our work is inspired by the pipeline of [
Unlike [
We use the same rules as [
The rules are designed for opinion target identification, i.e. discovering targets of evaluative statements.1 They are based on syntactic relations with evaluative words, i.e.
1The underlying assumption of this approach is that opinion targets tend to be the sought-after aspects. words listed in a subjectivity lexicon for the given language.
In the example in Figure 1, the rule vbnm_sb_adj is triggered because “amazing” is an evaluative word and it is a predicate adjective – the word “rice”, as the subject of this syntactic construction, is then marked as a likely aspect term.
Originally, the rules were written for English. Their
adaptation to Czech proved very simple. We modified
expressions which involved morphological tags to work with
the Czech positional tagset [
We chose linear-chain conditional random fields (CRFs)
for our work [
A linear-chain CRF is a statistical model. It is related to hidden Markov models (HMMs), however it is a discriminative model, not a generative one – it directly models the conditional probability of the labeling P(y|x). Linearchain CRFs assume that the probability of the current label (B, I or O) only depends on the previous label and on the input words x.
Formally, a linear-chain CRF is the following conditional probability distribution:
P(y|x) =
1 T K Z(x) exp{t∑=1 k∑=1 λk fk(yt , yt−1, t, x)} (1)
Roughly speaking, P(y|x) is the score of the sentence labeling y, exponentiated and normalized.
The score of y corresponds to the sum of scores for labels yt at each position t ∈ {1, . . . , T } in the sentence. The score at position t is the product between the values of feature functions fk(yt , yt−1, t, x) and their associated weights λk, which are estimated in the learning stage.
Feature functions can look at the current label yt , the previous label yt−1 and the whole input sentence x (which is constant).
Z(x) is the normalization function which sums over all possible label sequences:
T K Z(x) = ∑ exp{∑ ∑ λk fk(yt0 , yt0−1, t, x)} y0 t=1 k=1 (2)
To train the model, we require training data, i.e. sentences with the labeling already assigned by a human annotator. During CRF learning, the weights λk are optimized to maximize the likelihood of the observed labeling 2This “BIO” labeling scheme is common for CRFs. In practice, it brings us a consistent slight improvement as opposed to using only binary classification (inside vs. outside an aspect). ID adverb but_opposite coord sub_adj subj_of_pat verb_actant_pat verb_actant_act vbnm_patn vbnm_sb_adj
Actor or patient of a verb with a subjective adverb. Words coordinated with an aspect with “but”. Words coordinated with an aspect are also aspects. Nouns modified by subjective adjectives. Subject of a clause with a subjective patient. Patient of a transitive evaluative verb.
Actor of an intransitive evaluative verb.
Predicative nominal (patient).
Subject of predicative adjectives.
The pizza tastes so good.
The food is outstanding, but everything else sucks.
The excellent mussels, goat cheese and salad.
A very capable kitchen.
The bagel have an outstanding taste.
I liked the beer selection.
Their wine sucks.
Our favourite meal is the sausage.
The fried rice is amazing. in the dataset. Gradient-based optimization techniques are usually applied for learning.
At prediction time, the weights λk are fixed and we are looking for such a labeling yˆ which is the most probable according to the model, i.e.: yˆ = arg max P(y|x) y (3) yˆ can be found efficiently using a variant of the Viterbi algorithm (dynamic programming). In our work, we use the CRF++ toolkit3 both for training and prediction. 3.3
We now describe the various feature sets evaluated in this work.
Surface features. We use the surface forms of the current word, two preceding and two following words as separate features. Additionally, we extract all (four) bigrams and (three) trigrams of surface forms from this window. We also use the CRF++ bigram feature template without any arguments; this simply produces the concatenation of the previous and current label (yt−1, yt ).
Morpho-syntactic features. We extract unigrams, bigrams and trigrams from a limited context window (identical to the above) around the current token but instead of surface forms, we look at: • lemma, • morphological tag, • analytical function.
Analytical functions are assigned by the dependency parser and their values include “Sb” for subject, “Pred” for predicate etc.
Sublex features. We mark all words in the data whose lemma is found in the subjectivity lexicon. For each token in the window of size 4 around the current token (included), we extract a feature indicating whether it was marked as subjective. We also concatenate these indicator features with the surface form of the current token. 3http://taku910.github.io/crfpp/
Rule features. Finally, for each type of rule, we extract features for the current token, the preceding and the following token, indicating whether the rule marked that token. Again, these features have two versions: one standalone and one concatenated with the surface form of the current token. 4
We analyze our data using Treex [
Table 3 shows the obtained precision (P), recall (R) and f-measure (F1) for both parts of the data set. The results in all cases were acquired using 5-fold cross-validation on the training data.
Random segments. The baseline (surface-only) features achieve the best precision but the recall is very low.
Morpho-syntactic features lower the precision by a significant margin but push recall considerably. As the review data come from the “wild”, they are quite noisy; many segments are written without punctuation, reducing the benefit of using morphological analysis, let alone dependency parsing.4
Often, the segments are rather short, such as “Rychlé dodání” (“fast delivery”) or “ Fotky fakt parádní.” (“ Photos really awesome.”). This also considerably limits the benefit that a parser can bring – there is a major domain mismatch both in the text topic and types of sentences between the parser’s training data and this dataset, so we cannot expect parsing accuracy to be high.
Most of the improvement from adding morphosyntactic features thus probably comes from the availability of word lemmas – this allows the CRF to learn which 4This issue could perhaps be addressed by using a spell-checker, we leave that to future work.
surface +morpho-syntactic +sublex +rules words are frequently marked as aspects in this domain and to generalize this information beyond their current inflected form.
Adding the information from the sentiment lexicon further improves performance, though not as much as we would expect. We could possibly further increase its impact through more careful feature engineering – so far, the features only capture whether a subjective term is present in a small linear context. For example, the lemma of the evaluative word could be included in the feature.5
Finally, adding the output of syntactic rules further improves the results. Due to the uncommon syntactic structure of the segments, most rules were not active very often, so the space for improvement is quite limited. Yet the results show that when the rules do trigger, their output can be a useful signal for the CRF.
The observed improvement in recall at the slight
expense of precision is in line with the results of [
Long reviews. It is immediately apparent that the long reviews are a much more difficult dataset than review segments – the best f-measure achieved on the short segments is 65.79 while here it is only 30.27. This can be explained by the lower density of aspect terms compared to random review segments and a much higher sentence length – after sentence segmentation, the average sentence length is over 29 words, compared to only 6 words for the random segments.
When using only the baseline features, the recall is extremely low. Adding morpho-syntactic features has a similar effect as for the random segments – precision is lowered but recall nearly triples.
Interestingly, adding features from the subjectivity lexicon changes the picture considerably. This feature set obtains the highest precision but recall is lower compared to both +morpho-syntactic and +rules. It may be that due to the high sentence length, sublex features help identify aspects within the short window but their presence pushes the model to ignore the more distant ones. A more thorough manual evaluation would be required to confirm this.
Finally, the addition of syntactic rules leads to the highest f-measure, even though neither recall nor precision are the best. In this dataset, possibly again thanks to the length 5CRF++ feature templates do not offer a simple way to achieve this without also generating a large number of uninformative feature types. of sentences, the rules are trigged much more often than for the random segments. Rule features can therefore have a more prominent effect on the model. 5
In terms of using rules for ABSA, our work is inspired
by [
For Czech, a dataset for ABSA was published by [
We have presented a new dataset for ABSA in the Czech language and we have described a baseline system for the subtask of aspect term extraction.
The dataset consists of segments from user reviews of IT products with the annotation of aspects and their polarity.
The system for aspect term extraction is based on linear-chain CRFs and uses a number of surface and linguistically-informed features. On top of these features, we have shown that task-specific syntactic rules can provide useful input to the model.
Utility of the syntactic rules could be further evaluated on other domains (such as the Czech restaurant reviews) or languages (e.g. using the official SemEval data sets) and the impact of individual rules could be thoroughly analyzed across these data sets.
This research was supported by the grant GA15-06894S of the Grant Agency of the Czech Republic and by the SVV project number 260 224. This work has been using language resources developed, stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (project LM2010013).