By focusing on the online-reviews domain, this study aims to provide a complete solution to the sentiment-analysis task consisting o its three constituent components: opinion holder, polarity of the underlying sentiment and target. For the purposes of this research, several challenges and issues related to the nature of the problem are addressed such as class imbalance and the need for meaningful linguistic data-augmentation techniques to increase the size of the training set and make the use of Long Short-Term Memory models (LSTMs) possible. For both of them, new e ective approaches are proposed and evaluated. As a means of quantifying class imbalance, the Minority-to-Majority Ratio (M2MR) is introduced. The two sub tasks of target and polarity detection are tackled using machine-learning means. To support the training process, a new data set, which combined sentences from two different review-based corpora, was constructed. In our research, the bestperforming LSTM-based models make use of the context-sensitive BERT embeddings and yield F1-Scores of 0.9263 and 0.8911 over all possible classes for the polarity and target components respectively.
The aim of this study is to introduce a complete solution to the sentiment-mining problem used for social media analysis, by abstracting it as a sequence-tagging task. The ultimate goal will be to label each individual word in a sequence, based on whether it belongs to any of the sentiment components. For example, given the sentence \I, really, enjoyed watching `Bohemian Rhapsody' last night." as input, the proposed system will have to detect \I" as the opinion holder, "Bohemian Rhapsody" as the target and determine that the sentiment polarity is \Positive". These three elements together form the so-called \sentiment triple".
In addition to the primary goals of this research, some elementary issues will be addressed. Firstly, an attempt will be made to overcome the problem of data set shortage by following a process of target-oriented annotation on a data set comprised of reviews about products and services. This new data Copyright c 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). set was named ZyLABSent and can be made available upon request. Secondly, the issue of class imbalance and its e ects on this particular context will be thoroughly investigated and a number of e ective solutions for mitigating it will be presented and assessed. In order to be able to evaluate them, a quantitative measure of class balance will be introduced with di erent adaptations for use in the sentence- and data set-level. Novel approaches of linguistic augmentation functions will be proposed that can be employed for not only the construction of a more balanced data set, but also for increasing the number of useful training samples. Finally, BERT and air, two types of contextual embeddings will be used as part of this research. 2
Yang and Cardie [
A recent study on target-based sentiment analysis is presented by Li et al. [
In addition to the above, Li et al.'s architecture is enriched by three custommade components: Boundary Guidance (BG), Sentiment Consistency (SC) and Opinion-Enhanced (OE) Target-Word Detection. BG utilises information about the target boundaries by modeling all the constraint transitions from target boundaries to target polarities explicitly, in the form of a transition matrix which is used for determining their proportions based on the underlying con dence of the target boundary tagger later on. SC makes use of a gate mechanism which combines the features extracted from the current word with its predecessor ones in the sequence, so that it is less likely for tokens belonging to the same target to end up being allocated with di erent sentiments. Finally, OE attempts to improve the quality of the boundary prediction for the opinion targets, by looking for words that indicate the expression of an opinion in a certain window around each token. Subsequently, it trains an elementary classi er which distinguishes between target and non-target words based on distant supervision.
To evaluate the performance of their model, Li et al. merge the training
sets from the data sets provided by the SemEval ABSA challenge [
The advances in the implementation of contextual word embeddings has proved to be of particular importance for tasks in which context really matters such as sentiment analysis. In sentiment analysis, words may be associated with a di erent meaning based on the context they appear in. Also, not only can there be sudden changes of meaning and sentiment due to di erent word arrangements, but word-based and syntactical negations can also cause additional levels of ambiguity.
Thus, a natural approach was to employ context-sensitive word embeddings
as features in order to be able to capture intrinsic properties and internal
relationships between the words that consist the data set. Where Word2Vec [
According to the work by Huang et al. [
Akbik et al. [
In our initial experiments, di erent class imbalance handling techniques, various combinations of word embeddings as well as some tweaks on the con guration of the model architecture (i.e. the inclusion of the CRF layer or not) were applied and compared in order to determine the optimal con guration for the purposes of this task. We will discuss this in more detail in the results section. 4
The main focus of this research is on the creation of a data set that was
sufciently large to allow for the use of deep-learning models. The only
publiclyavailable2 data set that includes target-oriented annotations is the \Restaurant"
domain of the SemEval-2016 Task 5: \Aspect-Based Sentiment Analysis" data
set [
A research team from Johns Hopkins University collected Amazon Reviews from several product categories (four domains in the rst release3 and twentyve in the second one4): the Multi-Domain Sentiment Data set. The reviews are accompanied by their star rating (from one up to ve stars). Reviews with less than three stars are labelled as \Negative" whilst the ones that contain more than three are marked as \Positive". Three stars indicate a neutral review.
Unfortunately, the Multi-Domain data set is characterised by two important limitations: (i) the scope of sentiment annotations is provided in a per-review and not in a per-sentence manner. So, when considering each sentence individually, it is highly possible to come across sentences whose polarity disagrees with the overall review polarity. For example, in a negative review, there can be a sentence which expresses a positive opinion, in contrast with the rest. This may become even more acute with the use of cynical language. Thus, it is not safe to deduce that the polarity of each sentence matches the global polarity of the review it belongs to. (ii) The data set states the product for which a review is about, but the same does not hold for each individual sentence. It is plausible that a sentence may include an opinion related to a di erent entity than the rest of the review.
Despite the fact that the aforementioned data sets were not suitable as standalone solutions on their own, their union could be useful for training an e ective model. In particular, SemEval-2016-Task-5 and Multi-Domain are pretty similar since they are both comprised of reviews. On the other hand, they originate from di erent domains, which means that they could introduce some variety to
the models during training. Unlike any social media postings which are written in a more quick and informal way, reviews tend to resemble the formal way of writing to a higher degree due to the fact that their authors aim to associate their writing with more credibility.
Therefore, based on the above factors, a new data set (thus, referred to as ZyLAB-Targeted-Sentiment-Reviews or ZyLABSent ) was compiled, which consists of subsets from these two existing data sets. In order to overcome the restrictions a ecting the Multi-Domain one, a manual annotation process was initiated on a set of 4000 sentences extracted from the Multi-Domain corpus. For the given sentences, the annotators were asked to detect the entities that appear to be targets of sentiment; either positive or negative. Additionally, they were informed on the polarity of the review from which a sentence originated, as an indication of the overall context. It was stressed that the \entities to-bedetected" should appear as tokens in the sentence. If a sentence was associated with a target entity of a speci c sentiment that could be inferred from the context, but did not appear explicitly in the sentence, the corresponding section should have been marked by the \[Unknown]" token. Also, in case a sentence did not include an element of a speci c polarity, this section should have been lled in with the \[None]" token. The Kappa measure for inter-judge agreement between the two annotators was equal to 0.7245. 5 5.1
Approaching the task as a sequence-tagging problem and representing the input accordingly resulted to some interesting e ects related to the distribution of the di erent class labels. Originating from the speci cs of the problem domain itself, the fact that the focus of the di erent sub tasks is only on certain sentence tokens (i.e. words that represent the target entities) means that the vast majority of tokens are expected to be classi ed as \Non-Token" or \Neutral". A serious consequence of this, from a machine-learning point-of-view, is class imbalance which is evident by the abundance of these two classes over the rest. This could possibly result to models that are biased towards predicting these majority classes o ering them an unfair advantage.
However, before proceeding with following any approaches for tackling class imbalance, it was important to assess its severity on this speci c case and estimate any direct impacts that it could have on the performance of the prediction processes. The rst step was to ensure whether the data set was indeed imbalanced and to which degree, whilst the latter could con rm whether the possible class imbalance would hinder the classi cation procedures signi cantly and determine whether it was worth investing time handling it.
Since, by the time this research was conducted, there was no standard way to measure class imbalance for sequence-tagging tasks in the literature, a new metric was de ned aiming to quantify this phenomenon: Let s be a sentence originating from a text corpus c. c contains m sentences. The index of each sentence in the corpus is denoted by i. Each of these sentences consists of word tokens associated with one out of l labels corresponding to the possible output classes of a sequence-tagging task. majs is the set of sentence tokens which are associated with a label belonging to the majority class and mins is the union of tokens assigned with labels originating from the other l 1 classes that together consist the minority classes. We introduce the Minority-to-Majority Ratio (M2MR) measure of a sentence (M 2M Rs), which is de ned as the ratio of the size of the latter over the size of the former (1). To construct a global metric that quanti es the overall balance of a text corpus, the M 2M Rc is also de ned which contains an additional multiplicative parameter corresponding to the inverse of m (2).
M 2M Rs = ( jminsj ; majs >= mins > 0 jmajsj 0; otherwise M 2M Rc = m 1 X M 2M Rci = 1 m m 1
X M 2M Rci i=0 i=0
By de nition, the size of the majority class should be greater than or equal to the size of the union set of the minority classes. The equal case corresponds to the scenario of a balanced sentence that would yield an M 2M Rs of one. In the case when mins is equal to zero, the M 2M Rs becomes zero immediately due to the fact that the complete absence of minority-class tokens from a sentence signi es a state of full imbalance. If majs is zero, it means that mins is also zero which is only possible when an empty sentence with no tokens is considered. In this scenario the M 2M Rs becomes zero, too.
The use of the parameter in the corpus-wide version guarantees that a perfectly-balanced data set (i.e. a data set that consists solely balanced sentences) would receive an M 2M Rc score of one which is the maximum value this formula can take. Therefore, the optimisation objective when trying to construct a balanced data set can be thought of as the attempt to maximise the score of M 2M Rs for each individual sentence or the M 2M Rc for the entire data set.
M 2M Rs=M 2M Rc ! 0 Highly Imbalanced Sentence/Corpus M 2M Rs=M 2M Rc ! 1 Highly Balanced Sentence/Corpus (1) (2) 5.2
A sequence-tagging data set consists of several sentences containing a varying amount of tokens (i.e. words) each. These tokens, being annotated for a particular purpose, are highly likely to be associated with a class that signi cantly outnumbers the rest. However, using the M 2M R metric, it is possible to select the slice of this sentence which o ers the most balanced con guration using the available tokens. Attempting to balance each sentence individually can help mitigating the overall class-imbalance issue which characterises the entire data set.
In order to nd the optimal slice for each sentence, all possible ordered slices with sizes ranging from two up to the length of the entire sentence were generated. Slices of size one were not considered due to the fact that such slice is basically a word and thus it no longer corresponds to a proper sequence. Subsequently, the M 2M Rs value of each slice was calculated and, after sorting the slices in descending order based on it, a ranking was constructed. To provide an additional boost to slices which were both balanced, but also retained the largest amount of original tokens possible, the slices were grouped based on their M 2M Rs score and internally re-sorted based on their length. The slice that made it to the top of this re ned ranking bore the highest M 2M Rs of all and it was chosen to represent the sentence in the corpus. This process was repeated for all the sentences resulting to a balanced version of the original data set.
However, this approach could be associated with an important limitation. Since, it aimed to arrange the class distribution in the most optimal way, it could demonstrate a tendency towards building a data set which was ideally balanced, but not realistic anymore. For example, if most of the sentences originally contained a single token that was allocated with a minority-class label then a direct consequence of this would be that most of the sentences would end up being replaced by a slice of size two. Thus, if this data set was used for any training purposes, it might result in a biased model which would demonstrate an aptitude towards labelling unseen sentences of this length, but not on sentences of di erent sizes. 6 6.1
To mitigate the abovementioned issue of unrealistic distribution, a generalised version of the ideally-balanced data set construction could be proved e cient. Instead of constantly favouring the slices with the highest M 2M Rs, these slices were simply associated with higher probabilities of being drawn whilst the selection was kept completely random. We named this Data Con guration 2 (DC2).
The steps up to and including the ranking process that were described in the previous balancing technique were also followed for this approach. Then, the ranking was split into four leagues based on the sorting order and the members of each league were replicated in the data set based on them. More speci cally, samples belonging to the top league were copied four times, tripled in the second league, doubled in the third and remained unchanged in the last league.
Finally, a random slice (or a number of slices) was chosen from the list. This approach still bene ted the balanced slices by giving them more chances of being selected while it introduced the probabilistic element which would lead to a more \realistic" setting. A new version of the data set would be created that is more likely to be balanced than the original one, but still with higher resemblance to a real data set than the ideally-balanced one. 6.2
The second approach was based on the creation of new samples by means of data synthesis. To accomplish this, all sentences were iterated over, and their tokens which were annotated as targets of sentiment (i.e. assigned with labels from the minority classes), were detected. Based on the idea that a synonym is, most of the times, capable of conveying the same meaning as the original word without altering the quality of the overall sentence, new copies of the sentences were fabricated by simply replacing the target tokens with their synonyms. One could argue that due to the use of word embeddings, adding additional training data based on synonyms would not add su cient additional information for the machine-learning process. For this reason, we did not only use strict synonyms, but we also experimented with other methods to generate linguistically and semantically valid training samples.
To be able to identify all the possible synonyms for the di erent target words, the WordNet lexical database for English 5 was utilised. The target tokens were looked up in the aforementioned resource for synonyms. However, there was some variety on the lemmas returned for a few words. In the English language, words such as \work" can be used in either verb or noun form. In order to tackle this, extracted lemmas that were associated with a di erent Part-of-Speech (PoS) tag than the original target token were discarded. For their identi cation, the spaCy PoS-tagging model 6 was employed. The remaining lemmas belonged to synonyms matching the syntactic role of the initial word. As an additional clearance measure, any synonyms which appeared in a di erent form than the original word (i.e. synonyms in plural while the word itself was in singular and the converse) were disposed to avoid the inclusion of meaningless information in the training data set. Subsequently, the selected synonyms were incorporated on the original sentences and allocated with the same labels as their predecessors, resulting in Data Con guration 3 (DC3).
It is worth noting that the outsourced lemmas are not always strict synonyms of the original words. WordNet also looks up for any hypernyms, hyponyms and older forms of a word. These words albeit quite similar, they do not always have an identical meaning. For example, for the word \ lm", WordNet returns both \movie" and \picture". However, the use of the latter will result to a sentence with a completely new meaning which can be interpreted in a very di erent way. This was expected to introduce new content and supply the trained classi ers with more variability.
In a similar manner, it was also possible to fabricate new sentences by inverting the meaning of the existing ones. The main di erence with the abovementioned synonym-based approach was that the tokens of interest were not the target tokens themselves, but any adjectives or adverbs that were related to them. Their selection was based on the rationale that these are the ones which actually bear the sentiment towards the target. They were identi ed by employing dependency parsing to resolve any relationships between the target and other sentence tokens characterised by these two PoS tags. The Dependency Parser 7 which is part of spaCy processing pipeline was utilised for this purpose. Two di erent ways of using antonym-based augmentation were investigated. These approaches will be explained in the next subsections.
Not-Based Inversion (DC4) The rst and simplest approach involved the inversion of meaning with the use of the adverb \not". In order to change the polarity of sentiment, it is usually su cient to ip the polarity of the aforementioned sentiment-bearing words with the use of \not". Equivalently, for the ones which were already preceded by \not", this special negation token was removed to cause the change of sentiment. Due to the aptitude of context-sensitive embeddings in appreciating the surroundings of words, the scope of negation would be appropriately interpreted and handled during the encoding.
Lexical-Antonyms Replacement (DC5) For the more complicated method of inverting the sentiment of these words with their lexical antonyms, an approach quite similar to the synonym-based augmentation was followed. Again, the WordNet lexical database was utilised to look up these words for all their possible lemmas. After the appropriate check for matching PoS tags between the original word and its antonyms, the existing sentences were replicated with the chosen antonyms replacing the sentiment-bearing words.
In a similar manner as its synonym counterpart, the augmentation of the data set with antonyms and the subsequent inversion of polarity introduced new examples of possible sentimental relationships. In this case, in particular, a novel category of training data was created since the di erent antonyms exploit previously unseen instances of text that express the opposite polarity than their source sentence.
An example that illustrates the results from the application of the di erent augmentation processes on an actual sentence is provided below: { Original Sentence: \This lm is terrible." { Synonym-Based: \This movie is terrible.", \This motion picture is terrible.", \This picture is terrible." { Antonym-Not-Based: \This movie is not terrible." { Antonym-Lexical-Based: \This movie is wonderful."
Extraction of the opinion holder was not the main topic of this research project. However, in order to provide a complete solution for the sentiment-extraction problem, we also decided to address this aspect of the problem. In contrast to the detection of the sentiment target and polarity, the identi cation of the opinion holder can be thought as a fairly simple process in the context of online reviews. After an exploratory analysis of the data set samples, it was possible to deduce a few distinct types of opinion-holder occurrences in the sentences. Therefore, instead of following a supervised approach as with the other sentiment constituents, a rule-based heuristic procedure was followed with the expectation of receiving results of high quality.
The rst type of opinion-holder occurrence, concerned the scenario when a sentence lacks a direct reference to the opinion holder. For example, in the sentence \This lm was simply amazing.", the entity that expresses this positive opinion is not mentioned explicitly anywhere. Therefore, it is safe to infer that the opinion holder here is/are the author(s) of this sentence. In our model, the generic \(Author)" token implies either a single or multiple authors.
In other cases, the opinion holders may be explicitly referenced. These situations can be generally broken down into two categories: rst-person thoughts or experiences and other-people views or perceptions. An instance of the rst class is a sentence in which an opinion or experience is described by the speakers themselves. Here, the identity of the speaker is evident by the appearance of rstperson pronouns (e.g. I, We). The second class includes all sentences in which opinions or perceptions of other people are quoted by their author. Therefore, the opinion holders (or \subjects") in this scenario are these third-party entities.
In order to handle the aforementioned scenarios, a set of rules was de ned
which, when satis ed, they classi ed each sentence to the most suitable
opinionholder category and treated it appropriately in order to extract the entity of
interest. Before this, any personal pronouns were resolved to their subjective
form (e.g. my ! I). Starting o with the most general scenario of
considering the \(Author)" as the opinion holder, dependency parsing was applied to
detect whether the opinion holder corresponded to any of the more speci c
categories. This was accomplished by investigating the structure of the sentence
and exploiting any noun phrase - thinking/perceiving verb dependencies which
were expected to indicate the expression of an opinion by the rst constituent of
these syntactic relationships. Here, the detected subject was pronounced as the
opinion holder. Appropriate re nements to these rules were made when the
entities of interest contained multiple tokens or when they expressed a conjugation
of many entities. Lastly, as a nal re nement step, co-reference resolution was
applied to uncover the existence of any co-referring clusters. The Neuralcoref
module 8 of spaCy NLP engine was used for this which performs co-reference
resolution by employing a scoring model based on a neural network proposed by
Clark and Manning [
clusters, it was replaced by its head. For clarity, the entire process is presented below.
Due to the classi cation nature of the di erent sub tasks, F1-Score was used as the evaluation metric. This was also the metric used by most of the previous studies. However, it was hard to directly compare them with this experimental iteration due to the fact that they tackled the sentiment-analysis problem in di erent ways (and not as a sequence-tagging problem) and made use of di erent annotated data. Thus, for the overall performance of each approach, MicroAveraged F1-Scores (F 1 ) were calculated because, by de nition, this measure takes class imbalance into consideration and re ects it accordingly on the results. Additionally, this metric was used as the loss function for training all the LSTMbased models.
It is important to note that, although F 1 considers the class distribution, it can overshadow the poor performance of smaller classes completely, if the majority class performs signi cantly well. Of course, this is unfair to the minority classes since their low individual scores would be masked by a high overall metric and thus they would end up being overlooked during the optimisation of a model.
Therefore, a more fair, Weighted F1-Score (F 1w) was additionally considered for the two machine-learning-based components which assigned weights to every class. These weights were inversely proportional to their frequency in the data set. This means that the smaller the number of instances from a given class appearing in the data set was, the larger the weight that was associated with it would be. 8.2
In order to compare the e ectiveness of the di erent augmentative approaches described earlier, each of them was applied on 80% of the ZyLABSent dataset (Data Con guration 0 - DC0), which was subsequently used for training the two models. The additional 20% was set apart for testing and was not augmented to avoid any side e ects from the augmentation on the training results. Apart from the individual approaches described in the previous sections, two additional Data Con gurations (DC) were applied as well: Combined Antonym Augmentation (DC6) and Combined Synonym and Antonym Augmentation (DC7).
In addition to F 1 and F 1w, the M 2M Rc score of the training corpus was also calculated to uncover possible correlations between the data set balance and the classi cation performance. Along with them, the total number of data set sentences per experiment (NoS), after the application of each approach was also counted.
As it can be seen on Table 1, the ideally-balanced scenario resulted in the worst performance. This can be explained by the fact that this con guration is not realistic, as it was explained earlier. The classi er was trained to label balanced sentences, but this optimal distribution did not characterise the test split which was still highly imbalanced (M 2M Rc = 0:1541).
When it came to the di erent data augmentation techniques, they all led to a more balanced state when compared to the original data set. However, higher balance did not directly correlate with better performance. The Proportion-Based approach, albeit the most competent in terms of balancing the data set, did not result to the best classi cation performance for either models. On the contrary, two approaches that involved the generation of new sentences using antonym and synonym replacement, were proved the most e ective. The combination of the two antonym-based techniques was the best-performing one for the \Target" model (F 1w = 0:6478) whilst the synonym-based augmentation outperformed the rest (F 1w = 0:7808) for the \Polarity" one.
Target
Polarity DC# NoS M 2M Rc F 1
F 1w
NoS M 2M Rc F 1
F 1w
Between the two types of contextual embeddings namely BERT and air, the former resulted to higher F 1w than the latter for both components (0.6478 and 0.7808, respectively). Additionally, in order to prove the superiority of BERT over context-insensitive word embeddings, one con guration which employed GloVe was also used. Indeed, the BERT-based models outperformed it by more than 6% in terms of F 1w for both models. The next experiment aimed to investigate whether the inclusion of the extra CRF layer on top of the Bidirectional LSTM would result to any additional performance gains for the classi cation process. Without the use of this layer, the nal classi cation was performed using softmax. Otherwise, the CRF unit undertook the classi cation process itself. As the experiments demonstrated, the con guration which included a CRF component resulted to an increased performance for the \Target" model, but not for the \Polarity" one.
In order to be able to evaluate the performance of the opinion holder extraction, a set of 100 randomly sampled sentences from ZyLABSent was manually annotated with respect to the person who expressed the opinions they included. To get an idea of its performance in a per-sentence manner, the ground-truth values along with the predicted ones were compared one by one. For 96 out of the 100 sentences (i.e. a success rate of 96%), these two values matched completely.
Furthermore, the performance was evaluated in a per-token basis. A new representation of the sentences was constructed where each word was tagged with either \H" or \NH" depending on whether it was part of the detected opinion-holding-entity string or not. Finally, the labelled sentence was compared with the ground-truth version of it. The per-token results showed F1-Scores of 0.9200 and 0.9970 for the \H" and \NH" classes, respectively, and an F 1 of 0.9940.
An example that illustrates the results from the application of the complete sentiment-extraction process on an actual sentence is provided below:
This study aimed to provide a complete solution to the challenging task of sentiment analysis by breaking it down to its three fundamental components: opinion holder, sentiment's polarity and target. Due to the straightforward character of the opinion holder extraction, this could be tackled using a simple rule-based method. The last two were approached by means of supervised learning. The result from the application of full sentiment analysis on a sentence, was a sentiment triple consisting of the aforementioned elements.
An amalgamation of two publicly-available review-based data sets from various domains was used as the development corpus. 4000 sentences from it were manually annotated by di erent individuals. The Kappa measure of this e ort was 0.7450, which shows an almost strong agreement between the annotators.
Di erent class-balancing approaches were introduced and explained thoroughly in this study and subsequently evaluated via an experimental procedure. In addition to class balancing, these techniques could also be used for augmenting a data set with new samples and introducing additional variability to the models. Antonym-Based and Synonym-Based Augmentation were the best performing approaches that resulted to both a more balanced data set con guration, but also a signi cant increase in the classi cation performance.
The incorporation of synthetic data enriched the data set with more useful information. The described experiments have proved that these new training samples acted as an improvement to the already well-performing contextual embeddings. Even though such embeddings are capable of capturing the semantic relationship between words with the same meaning, these new augmentative approaches introduced more variety by constructing similar sentences, but with slightly di erent or completely opposite meaning. These solutions were accompanied with the de nition of several metrics that aimed to quantify and evaluate class imbalance. By using these methods, we were able to triple the size of a labelled set of 4000 sentences. We believe that there are other linguistic transformation functions that will allow us to add even more useful sentences. This will be a subject of future research for us.
Apart from the di erent data set con gurations mentioned earlier, many experiments were conducted involving various types of contextual and contextinsensitive embeddings and the inclusion of a CRF decoding layer on top of the existing architecture. As far as the di erent contextual embeddings are concerned, BERT and air were used and compared with each other for the very rst time (as far as it is known) for sentiment-analysis purposes on the token level since their publication. The best-performing models for the \Target" and \Polarity" subtasks have reached a Micro-Averaged F1-Score of 0.9263 and 0.8911, respectively, and made use of the BERT embeddings. The inclusion of a CRF layer yielded better results only for the \Target" model.
Although this was not the primary focus of our research, to complete our solution for the sentiment-extraction problem, the opinion-holder sub task was tackled by devising a rule-based approach that leveraged syntactical and grammatical patterns which appear in opinionated sentences frequently. The model managed to identify the opinion holder correctly in 96% of the sentences. This high score is expected considering that the detection of the opinion holder is a relatively easy task compared to the other two components and can be mapped to a small number of scenarios and handled appropriately.
We would like to thank ZyLAB for their support throughout the course of this research. This included, but was not limited to, help during the data set annotation, provision of computational units for executing the conducted experiments and valuable advice.