=Paper= {{Paper |id=Vol-2253/paper18 |storemode=property |title=Integrating Terminology Extraction and Word Embedding for Unsupervised Aspect Based Sentiment Analysis |pdfUrl=https://ceur-ws.org/Vol-2253/paper18.pdf |volume=Vol-2253 |authors=Luca Dini,Paolo Curtoni,Elena Melnikova |dblpUrl=https://dblp.org/rec/conf/clic-it/DiniCM18 }} ==Integrating Terminology Extraction and Word Embedding for Unsupervised Aspect Based Sentiment Analysis== https://ceur-ws.org/Vol-2253/paper18.pdf
 Integrating Terminology Extraction and Word Embedding for Unsu-
              pervised Aspect Based Sentiment Analysis

         Luca Dini                          Paolo Curtoni                        Elena Melnikova
        Innoradiant                          Innoradiant                            Innoradiant
      Grenoble, France                     Grenoble, France                      Grenoble, France
luca.dini@innoradiant.com         paolo.curtoni@innoradiant.com         elena.melnikova@innoradiant.com

                                                      section 4 we summarize our previous approach
                    Abstract                          in order to provide a context for our
                                                      experimentation. In section 5 we prove the
    English. In this paper we explore the ad-         benefit of the integration of unsupervised
    vantages that unsupervised terminology ex-        terminology extraction with ABSA, whereas in 6
    traction can bring to unsupervised Aspect         we provide hints for further investigation.
    Based Sentiment Analysis methods based on
    word embedding expansion techniques. We           2    Background
    prove that the gain in terms of F-measure is
    in the order of 3%.                               ABSA is a task which is central to a number of
                                                      industrial applications, ranging from e-
    Italiano. Nel presente articolo analizziamo       reputation, crisis management, customer
    l’interazione tra syistemi di estrazione          satisfaction assessment etc. Here we focus on a
    “classica” terminologica e systemi basati su
                                                      specific and novel application, i.e. capturing the
    techniche di “word embedding” nel contesto
    dell’analisi delle opinioni. Domostreremo         voice of the customer in new product
    che l’integrazione di terminogie porta un         development (NPD). It is a well-known fact that
    guadagno in F-measure pari al 3% sul              the high rate of failure (76%, according to
    dataset francese di Semeval 2016.                 Nielsen France, 2014) in launching new products
                                                      on the market is due to a low consideration of
1    Introduction                                     perspective users’ needs and desires. In order to
                                                      account for this deficiency a number of methods
The goal of this paper is to bring a contribution     have been proposed ranging from traditional
on the advantage of exploiting terminology            methods such as KANO (Wittel et al., 2013) to
extraction systems coupled with word                  recent lean based NPD strategies (Olsen, 2015).
embedding techniques. The experimentation is          All are invariantly based on the idea of collecting
based on the corpus of Semeval 2016. In a             user needs with tools such as questionnaire,
previous work, summarized in section 4, we            interviews and focus groups. However with the
reported the results of a system for Aspect Based     development of social networks, reviews sites,
Sentiment Analysis (ABSA) based on the                forums, blogs etc. there is another important
assumption that in real applications a domain         source for capturing user insights for NPD: users
dependent gold standard is systematically absent.     of products (in a wide sense) are indeed talking
We showed that by adopting domain dependent           about them, about the way they use them, about
word embedding techniques a reasonable level of       the emotions they raise. Here it is where ABSA
quality (i.e. acceptable for a proof of concept) in   becomes central: whereas for applications such
terms of entity detection could be achieved by        as e-reputation or brand monitoring capturing
providing two seed words for each targeted            just the sentiment is largely enough for the
entity. In this paper we explore the hypothesis       specific purpose, for NPD it is crucial to capture
that unsupervised terminology extraction              the entity an opinion is referring to and the
approaches could further improve the quality of       specific feature under judgment.
the results in entity extraction.                        ABSA for NPD is a novel technique and as
   The paper is organized as follows: In section 2    such it might trigger doubts on its adoption:
we enumerate the goal of the research and the         given the investments on NPD (198 000 M€ only
industrial background justifying it. In section 3     in the cosmetics sector) it is normal to find a
we provide a state of the art of ABSA                 certain reluctance in abandoning traditional
particularly focused towards unsupervised ABSA        methodologies for voice of the customer
and its relationship to terminology extraction. In
collection in favor of social network based AB-            an Attribute. Entity and attributes, chosen from a
ABSA. In order to contrast this reluctance, two            special inventory of entity types (e.g.
conditions need to be satisfied. On the one hand,          “restaurant”, “food”, etc.) and attribute labels
one must prove that ABSA is feasible and                   (e.g. “general”, “prices”, etc.) are the pairs
effective in a specific domain (Proof of Concept,          towards which an opinion is expressed in a given
POC); on the other hand the costs of a high                sentence. Each E#A can be referred to a linguistic
quality in-production system must be affordable            expression (OTE) and be assigned one polarity
and comparable with traditional methodologies              label.
                                                               The evaluation assesses whether a system
(according to Eurostat the spending of European
                                                           correctly identifies the aspect categories towards
PME in the manufacturing sector for NPD will
                                                           which an opinion is expressed. The categories
be about 350,005.00 M€ in 2020, and PME                    returned by a system are compared to the
usually have limited budget in terms of “voice of          corresponding gold annotations and evaluated
the customer” spending).                                   according to different measures (precision (P),
   If we consider the fact that the range of               recall (R) and F-1 scores). System performance
product/services which are possible objects of             for all slots is compared to baseline score.
ABSA studies is immense 1 , it is clear that we            Baseline System selects categories and polarity
must rely on almost completely unsupervised                values using Support Vector Machine (SVM)
technologies for ABSA, which translates in the             based on bag-of-words features (Apidianaki et al.,
capability of performing the task without a                2016).
learning corpus.
                                                           3.2   Related works on unsupervised
3       State of the Art                                         ABSA
3.1      Semeval2016’s overview                            Unsupervised ABSA. Traditionally, in ABSA
                                                           context, one problematic aspect is represented by
SemEval is “ an ongoing series of evaluations of           the fact that, given the non-negligible effort of
computational semantic analysis systems” 2 ,               annotation, learning corpora are not as large as
organized since 1998. Its purpose is to evaluate           needed, especially for languages other than
semantic analysis systems. ABSA (Aspect Based              English. This fact, as well as extension to
Sentiment Analysis) was one of the tasks of this
                                                           “unseen” domains, pushed some researchers to
event introduced in 2014. This type of analysis
                                                           explore unsupervised methods. Giannakopoulos
provides information about consumer opinions on
products and services which can help companies             et al. (2017) explore new architectures that can
to evaluate the satisfaction and improve their             be used as feature extractors and classifiers for
business strategies. A generic ABSA task consists          Aspect terms unsupervised detection.
to analyze a corpus of unstructured texts and to              Such unsupervised systems can be based on
extract fine-grained information from the user             syntactic rules for automatic aspect terms
reviews. The goal of the ABSA task within                  detection (Hercig et al., 2106), or graph
SemEval is to directly compare different datasets,         representations (García-Pablos et al., 2017) of
approaches and methods to extract such                     interactions between aspect terms and opinions,
information (Pontiki et al., 2016).                        but the vast majority exploits resources derived
    In 2016, ABSA provided 39 training and                 from      distributional  semantic      principles
testing datasets for 8 languages and 7 domains.            (concretely, word embedding).
Most datasets come from customer reviews                      The benefits of word embedding used for
(especially for the domains of restaurants,                ABSA were successfully shown in (Xenos et al.,
laptops, mobile phones, digital camera, hotels and         2016). This approach, which is nevertheless
museums), only one dataset (telecommunication              supervised, characterizes an unconstrained
domain) comes from tweets. The subtasks of the             system (in the Semeval jargon a system
sentence-level ABSA, were intended to identify             accessing information not included in the
all the opinion tuples encoding three types of
                                                           training set) for detecting Aspect Category,
information: Aspect category, Opinion Target
                                                           Opinion Target expression and Polarity. The
Expression (OTE) and Sentiment polarity. Aspect
is in turn a pair (E#A) composed of an Entity and          used vectors were produced using the skip-gram
                                                           model with 200 dimensions and were based on
    1                                                      multiple ensembles, one for each E#A
      The site of UNSPC reports more than 40,000 catego-
    ries of products (https://www.unspsc.org).
                                                           combination. Each ensemble returns the
    2
      https://aclweb.org/aclwiki/SemEval_Portal, seen on   combinations of the scores of constrained and
    05/24/2018                                             unconstrained systems. For Opinion Target
expression, word embedding based features ex-              Word2vec's methods (such as skip-gram and
tend the constrained system. The resulting scores      CBOW) are also used to improve the extraction
reveal, in general, rather high rating position of     of terms and their identification. This is done by
the unconstrained system based on word embed-          the composed filtering of Local-global vectors
ding. Concerning the advantages derived from           (Amjadian et al., 2016). The global vectors were
the use of pre-trained in domain vectors, they are     trained on the general corpus with GloVe (Pen-
also described in (Kim, 2014), who makes use of        nington et al., 2014), and the local vectors on the
convolutional neural networks trained on top of        specific corpus with CBOW and Skip-gram. This
pre-trained word vectors and shows good per-           filter has been made to preserve both specific-
formances for sentence-level tasks, and especial-      domain and general-domain information that the
ly for sentiment analysis                              words may contain. This filter greatly improves
   Some other systems represent a compromise           the output of ATE tools for a unigram term ex-
between supervised and unsupervised ABSA, i.e.         traction.
semi-supervised ABSA systems, such an almost               The W2V method seems useful for the task of
unsupervised system based on topic modelling           categorizing terms using the concepts of an on-
and W2V (Hercig et al., 2016), and W2VLDA              tology (Ferré, 2017). The terms (from medical
(García-Pablos et al., 2017). The former uses          texts) were first annotated. For each term an ini-
human annotated datasets for training, but enrich      tial vector was generated. These term vectors,
the feature space by exploiting large unlabeled        embedded into the ontology vector space, were
corpora. The latter combines different unsuper-        compared with the ontology concept vectors. The
vised approaches, like word embedding and La-          calculated closest distance determines the onto-
tent Dirichlet Allocation (LDA, Blei et al., 2003)     logical labeling of the terms.
to classify the aspect terms into three Semeval            Word2vec method is used also to emulate a
categories. The only supervision required by the       simple ontology learning system to execute term
user is a single seed word per desired aspect and      and taxonomy extraction from text (Wohlgen-
polarity. Because of that, the system can be ap-       annt and Minic, 2016). The researchers apply the
plied to datasets of different languages and do-       built-in word2vec similarity function to get terms
mains with almost no adaptation.                       related to the seed terms. But the minus-side of
   Relationship with Term Extraction. Auto-            the results shows that the candidates suggested
matic Terminology Extraction (ATE) is an im-           by word2vec are too similar terms, as plural
portant task in NLP, because it provides a clear       forms or near synonyms. On the other hand, the
footprint of domain-related information. All ATE       evaluation of word2vec for taxonomy building
methods can be classified into linguistic, statisti-   gave the accuracy of around 50% on taxonomic
cal and hybrid (Cabré-Castellvi et al., 2001).         relation suggestion. Being not very impressive,
   The relationship between word embedding and         the system will be improved by parameter set-
ATE method is successfully explored for tasks of       tings and bigger corpora.
term disambiguation in technical specification             In the experiments described in this paper we
documents (Merdy et al., 2016). The distribu-          exploit only the Skip-gram approach based on
tional neighbors of the 16 seed words were eval-       the word2vec implementation. It is important to
uated on the basis of the three corpora of differ-     notice that this choice is not due to a principled
ent size: small (200,000 words), medium (2 M           decision but to not functional constraints related
words) and large (more than 200 M words). The          the fact that that algorithm has a java implemen-
results of this study show that the identification     tation, is reasonably fast and it is already inte-
of generic terms is more relevant in the large         grated with Innoradiant NLP pipeline.
sized corpora, since the phenomenon is very
widespread over the contexts. For specified            4    Previous Investigations
terms, medium and large sized corpora are com-
                                                       The experiments described in Dini et al. (under
plementary. The specialized medium corpora
                                                       review), have been performed by using Innoradi-
brings a gain value by guaranteeing the most rel-
                                                       ant’s Architecture for Language Analytics
evant terms. As for the small corpora, it does not
                                                       (henceforth IALA). The platform implements a
seem to give usable results, whatever the term.
                                                       standard pipelined architecture composed of
Thus, the authors conclude that word2vec is an
                                                       classical NLP modules: Sentence Splitting →
ideal technique to constitute semi-automatically
                                                       Tokenization → POS tagging → lexicon access
term lexicon from very large corpora, without
                                                       → Dependency Parsing → Feature identification
being limited to a domain.
→ Attitude analysis. Inspired by Dini et al.                   measure: 0.6425726). These numbers are im-
(2017) and Dini and Bittar (2016), senti-                      portant because in our approach a positive match
ment/attitude analysis in IALA is mainly sym-                  is always given by a positive match of polarity
bolic. The basic idea is that dependency repre-                and a correct entity identification (in other words
sentations are an optimal input for rules compu-               a perfect entity detection system could achieve a
ting sentiments. The rule language formalism is                maximum of 0.64 precision).
inspired by Valenzuela-Escárcega et al. (2015)
and thanks to its template filling capability, in              4.1     Entity Matching
several cases, the grammar is able to identify the             Entity matching was achieved by manually asso-
perceiver of a sentiment and, most importantly                 ciating two seed words to each Semeval entity
the cause, of the sentiment, represented by a                  (RESTAURANT, FOOD, DRINK, etc.) and then
word in an appropriate syntactic dependency                    applying the following algorithm:
with the sentiment-bearing lexical item. For                    • Associate each entity to the average vector
instance the representation of the opinion in I                     of the seed words (e-vect. E.g.
hate black coffee. would be something such as:                      evect(FOOD)=avg(vect(cuisine),vect(pizza)
                                                                    ).
.
                                                                • If a syntactic cause is found by the grammar
                                                                    (as in “I liked the meal”)assign it the entity
(where integers represent position of words in a                    associated to the closest e-vect.
CONLL like structure).                                          • Otherwise compute the average vector of n
   By default entities (which are normally prod-                    words surrounding the opinion trigger and
ucts and services under analysis) are identified                    assign the entity associated to the closest e-
since early processing phases by means of regu-                     vect.
lar expressions. This choice is rooted in the fact                With n=35 we obtain precision= 0.47914252,
that by acting at this level multiword entities                recall= 0.4888 and F-measure=0.3998.
(such as hydrating cream) are captured as single
words since early stages.                                      5      Integrating terminology
   The goal of the Dini et al. (2018) work was to              A possible path to improve results in entity as-
minimize the domain configuration overhead by                  signment can be found in the usage of “syno-
i) expanding automatically the polarity lexicon to             nyms” in the computation of the set of e-vect.
increase polarity recall and ii) to perform entity             These can again be obtained from W2VR by se-
recognition by providing only two words (seeds)                lecting the n closest world to the average of the
for each target entity.                                        seeds and using them in the computation of the
   Both goals were achieved by exploiting a                    e-vect. Expectedly, the value of n can influence
much larger corpus than Semeval, obtained by                   the result as shown in Figure 1.
automatically scraping restaurant review from
TripAdvisor. The final corpus was composed of
3,834,240 sentence and 65,088,072 lemmas.
From this corpus we obtain a word2vec resource
by using the DL4j library (skip-gram). The re-
source (W2VR, henceforth) was obtained by us-
ing lemma rather than surface forms. Relevant
training parameters for reproducing the model
are described in that paper.
   We skip here the description of i) (polarity ex-
pansion) as in the context of the present work we                    Figure 1. Changes of measures accoring to
kept polarity exactly as it was in Dini & al.                          different top N closest worlds (without
(2018)3. We just mention the achieved results on                                    terminology).
polarity only detection which were a precision of
0.78185594 and a recall of 0.54541063 (F-
                                                               We notice that best results are achieved by using
3
  Some previous works on unsupervised polarity lexicon         a set of closest world around 10: after that
acquisition for sentiment analysis were done in (Castellucci   threshold the noise caused by “false synonyms”
et al., 2016; Basili et al., 2017)                             or associated common words causes a decay in
the results. We also notice that overall the results    6    Conclusions
are better than the original seed-only method, as
now we obtain precision: 0.51 recall: 0.35 F-           Many improvements can be conceived to the
measure: 0.42. Here the positive fact is not only       method presented here, especially concerning the
a global raise of the f-measure, but the fact that      computation of the vector associated to the opin-
this is mainly caused by an increased precision,        ionated windows, both in terms of size, direc-
which according to Dini et al. (2018) is the            tionality and consideration of finer grained fea-
crucial point in POC level applications.                tures (e.g. indicators of a switch of topic). How-
   As a way to remedy to the noise caused by an         ever our future investigation will rather be ori-
unselective use of the n closest words coming           ented towards full-fledged ABSA, i.e. taking into
from W2VR we decide to explore an approach              account not only Entities, but also Attributes.
that filters them according to the words appear-        Indeed, if we consider that the 45% F measure is
ing as terms in a terminology obtained from un-         obtained on a corpus where only 66% sentences
supervised terminology extraction system. To            were correctly classified according to the senti-
this purpose we adopted the software TermSuite          ment and if we put ourselves in a Semeval per-
(Cram & Daille, 2016) which implements a clas-          spective where entity evaluation is provided with
sic two steps model of identification of term can-      respect to a “gold sentiment standard” we
didates and their ranking. In particular TermSuite      achieve a F-score of 68%, which is fully ac-
is based on two main components, a UIMA To-             ceptable for an almost unsupervised system.
kens Regex for defining terms and variant pat-
terns over word annotations, and a grouping             References
component for clustering terms and variants that        Ehsan Amjadian, Diana Inkpen, T.Sima Paribakht and
works both at morphological and syntactic levels          Farahnaz Faez. 2016. Local-Global Vectors to Im-
(for more details cf. Cram & Daille, 2016). The           prove Unigram Terminology Extraction. Proceed-
interest of using this resource for filtering results     ings of the 5th International Workshop on Compu-
from W2VR is that “quality word” lists are ob-            tational Terminology, Osaka, Japan, Dec 12, 2016,
tained with the adoption of methods fundamen-             2-11.
tally different from W2V approach and heavily           Marianna Apidianaki, Xavier Tannier and Cécile
based on language dependent syntactic patterns.           Richart. 2016. Datasets for Aspect-Based Senti-
   We performed the same experiments as                   ment Analysis in French. Proceedings of the 10th
W2VR expansion for the computation of e-vect,             International Conference on Language Resources
with the only difference that now the top n must          and Evaluation (LREC 2016). Portorož, Slovenia.
appear as closest terms in W2VR and as terms in         Roberto Basili, Danilo Croce and Giuseppe
the terminology (The W2VR parameters, includ-             Castellucci. 2017. Dynamic polarity lexicon
ing corpus are described in section 4; the termi-         acquisition for advanced Social Media analytics.
nology was obtained from the same corpus about            International Journal of Engineering Business
restaurants). The results are detailed in Figure 2.       Management, Volume 9, 1-18.
                                                        David M. Blei, Andrew Y. Ng and Michael I. Jordan.
                                                          2003. Latent Dirichlet Allocation. The Journal of
                                                          machine Learning research, Volume 3, 993–1022.
                                                        M. Teresa Cabré Castellví, Rosa Estopà Bagot, Jordi
                                                          Vivaldi Palatresi. 2001. Automatic term detection:
                                                          a review of current systems. In Bourigault, D.
                                                          Jacquemin, C. L’Homme, M-C. 2001. Recent
                                                          Advances in Computational Terminology, 53-88.

  Figure 2. Different measures with top N words         Giuseppe Castellucci, Danilo Croce and Roberto
                                                          Basili. 2016. A Language Independent Method for
           filtered with terminology.
                                                          Generating Large Scale Polarity Lexicons. Pro-
                                                          ceedings of the 10th International Conference on
We notice that all scores increase significantly.
                                                          Language Resources and Evaluation (LREC'16),
In particular at top n=10 we obtain                       Portoroz, Slovenia, 38-45.
P=0.550233483,          R=0.381750288          and
F=0.450762752, which represents a 5% increase           Damien Cram and Béatrice Daille. 2016. TermSuite:
(in F-measure) w.r.t. the results presented in Dini       Terminology Extraction with Term Variant Detec-
et al. (2018).                                            tion. Proceedings of the 54th Annual Meeting of
  the Association for Computational Linguistics—        Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey
  System Demonstrations, Berlin, Germany, 13–18.          Dean. 2013a. Efficient estimation of word repre-
                                                          sentations in vector space. arXiv preprint
Luca Dini, Paolo Curtoni and Elena Melnikova. 2018.
                                                          arXiv:1301.3781
  Portability of Aspect Based Sentiment Analysis:
  Thirty Minutes for a Proof of Concept. Submitted      Tomas Mikolov, Wen-tau Yih and Geoffrey Zweig.
  to: The 5th IEEE International Conference on Data       2013b. Linguistic regularities in continuous space
  Science and Advanced Analytics. DSAA 2018, Tu-          word representations. Proceedings of NAACL-HLT
  rin.                                                    2013, 746–751.
Luca Dini, André Bittar, Cécile Robin, Frédérique       Dan R. Olsen. 2015. The Lean Product Playbook:
  Segond and M. Montaner. 2017. SOMA: The                 How to Innovate with Minimum Viable Products
  Smart Social Customer Relationship Management.          and Rapid Customer Feedback. John Wiley &
  Sentiment Analysis in Social networks. Chapter 13.      Sons Inc: New York, United States.
  197-209.    DOI:      10.1016/B978-0-12-804412-
                                                        Jeffrey Pennington, Richard Socher and Christopher
  4.00013-9.
                                                           D. Manning. 2014. GloVe: Global Vectors for
Luca Dini and André Bittar. 2016. Emotion Analysis         Word Representation. Proceedings of the 2014
  on Twitter: The Hidden Challenge. Proceedings of         Conference on Empirical Methods in Natural Lan-
  the Tenth International Conference on Language           guage Processing (EMNLP), 1532-1543.
  Resources and Evaluation LREC 2016, Portorož,
                                                        Maria Pontiki, Dimitrios Galanis, Haris Papageor-
  Slovenia, 2016.
                                                          giou, Ion Androutsopoulos, Suresh Manandhar,
Arnaud Ferré. 2017. Représentation de termes com-         Mohammad Al-Smadi, Mahmoud Al-Ayyoub,
  plexes dans un espace vectoriel relié à une ontolo-     Yanyan Zhao, Bing Qin, Orphée De Clecq, Vér-
  gie pour une tâche de catégorisation. Rencontres        onique Hoste, Marianna Apidianaki, Xavier Tanni-
  des Jeunes Chercheurs en Intelligence Artifcielle       er, Natalia Loukachevitch, Evgeny Kotelnikov,
  (RJCIA 2017), Jul 2017, Caen, France.                   Nuria Bel, Salud M. Jiménez-Zafra, Gülşen Ery-
                                                          iğit. 2016. SemEval-2016 Task 5: Aspect Based
Aitor García-Pablos, Montse Cuadros and German
                                                          Sentiment Analysis. Proceedings of the 10th Inter-
   Rigau. 2017. W2VLDA: Almost unsupervised sys-
                                                          national Workshop on Semantic Evaluation
   tem for Aspect Based Sentiment Analysis. Expert
                                                          (SemEval 2016). San Diego, USA.
   Systems with Applications, (91): 127-137.
   arXiv:1705.07687v2 [cs.CL], 18 jul 2017.             Marco A. Valenzuela-Escárcega, Gustave V. Hahn-
                                                          Powell and Mihai Surdeanu. 2015. Description of
Athanasios Giannakopoulos, Diego Antognini, Clau-
                                                          the Odin Event Extraction Framework and Rule
  diu Musat, Andreea Hossmann and Michael
                                                          Language. arXiv:1509.07513v1 [cs.CL], 24 Sep
  Baeriswyl. 2017. Dataset Construction via
                                                          2015, version 1.0, 2015.
  Attention for Aspect Term Extraction with Distant
  Supervision. 2017 IEEE International Conference       Lars Witell, Martin Löfgren and Jens J. Dahlgaard.
  on Data Mining Workshops (ICDMW)                        2013. Theory of attractive quality and the Kano
                                                          methodology – the past, the present, and the future.
Tomáš Hercig, Tomáš Brychcín, Lukáš Svoboda,
                                                          Total Quality Management & Business Excellence,
  Michal Konkol and Josef Steinberger. 2016. Unsu-
                                                          (24), 11-12:1241-1252.
  pervised Methods to Improve Aspect-Based Sen-
  timent Analysis in Czech. Computación y Sistemas,     Gerhard Wohlgenannt, Filip Minic. 2016. Using
  vol. 20, No. 3, 365-375.                                word2vec to Build a Simple Ontology Learning
                                                          System. Proceedings of the ISWC 2016 co-located
David Jurgens and Keith Stevens. 2010. The S-Space
                                                          with 15th International Semantic Web Conference
  package: An open source package for word space
                                                          (ISWC 2016). Vol-1690. Kobe, Japan, October 19,
  models. Proceedings of the ACL 2010 Systel
                                                          2016
  Demonstrations, 30-35.
                                                        Dionysios Xenos, Panagiotis Theodorakakos, John
Noriaki Kano, Nobuhiku Seraku, Fumio Takahashi,
                                                          Pavlopoulos, Prodromos Malakasiotis, Ion An-
  Shinichi Tsuji, 1984. Attractive Quality and Must-
                                                          droutsopoulos. 2016. AUEB-ABSA at SemEval-
  Be Quality. Hinshitsu: The Journal of the Japanese
                                                          2016 Task 5: Ensembles of Classifiers and Embed-
  Society for Quality Control, 14(2) : 39-48.
                                                          dings for Aspect Based Sentiment Analysis. Pro-
Emilie Merdy, Juyeon Kang and Ludovic Tanguy.             ceedings of SemEval-2016, San Diego, California,
  2016. Identification de termes flous et génériques      312–317.
  dans la documentation technique : expérimentation
                                                        Yoon Kim. 2014. Convolutional Neural Networks for
  avec l’analyse distributionnelle automatique. Actes
                                                          Sentence Classification. Proceedings of the 2014
  de l'atelier "Risque et TAL" dans le cadre de la
                                                          Conference on Empirical Methods in Natural Lan-
  conférence TALN.
                                                          guage    Processing      (EMNLP),    1746–1751.
                                                          arXiv:1408.5882