Not just about size - A Study on the Role of Distributed Word Representations in the Analysis of Scienti c Publications

Abstract

The emergence of knowledge graphs in the scholarly communication domain and recent advances in arti cial intelligence and natural language processing bring us closer to a scenario where intelligent systems can assist scientists over a range of knowledge-intensive tasks. In this paper we present experimental results about the generation of word embeddings from scholarly publications for the intelligent processing of scienti c texts extracted from SciGraph. We compare the performance of domain-speci c embeddings with existing pre-trained vectors generated from very large and general purpose corpora. Our results suggest that there is a trade-o between corpus speci city and volume. Embeddings from domain-speci c scienti c corpora e ectively capture the semantics of the domain. On the other hand, obtaining comparable results through general corpora can also be achieved, but only in the presence of very large corpora of well formed text. Furthermore, we also show that the degree of overlapping between knowledge areas is directly related to the performance of embeddings in domain evaluation tasks.

Keywords

• Word Embeddings
• Knowledge Graphs
• Scholarly publications
• Convolutional Neural Networks

-

In 2017 Springer Nature released the rst version of SciGraph1, an open linked data graph about publications from the editorial group and cooperating partners. This graph connects funders, research projects, grants, conferences, a liations, and publications, and in the future it is planned to add citations, patents, and clinical trials. This initiative is a step forward to bring semantics to scholarly publications that contributes to the vision where software agents assist scientists in their research endeavors [ 12, 7 ]. Nevertheless, publication content is still largely text-based, and hence the limitations for the automatic understanding of content, reproducibility of experiments [ 1 ], and knowledge reuse [ 6 ] remain.

We envision a scenario where knowledge graphs about Scholarly communications contain semantic metadata about the content of the publication beyond the 1 SciGraph homepage: https://www.springernature.com/gp/researchers/scigraph traditional descriptors used, including keywords and taxonomy categories where articles are placed by authors or editors. This content-based metadata could describe the work hypothesis, conclusions and the approach followed, among others. Therefore, natural language processing (NLP) is a key enabler to extract structured data from scholarly publications that can semantically enrich and shed light on the publication content.

Recently, distributed word representations in the form of dense vectors, known as word embeddings, have been used with great success in NLP tasks such as part-of-speech tagging, chunking, named entity recognition, semantic role labeling and synonym detection [ 3 ]. Vectors can be learned from large corpora using shallow neural networks [ 18 ] or following count-based approaches that perform matrix factorization [ 19, 14, 22 ]. Mikolov et al. [ 18 ] showed that embeddings capture semantic relations between words, for example between man and woman, or cities and countries, and syntactic relations based on tenses, singular and plurals, comparatives, superlatives, to name a few, that can be mapped to basic vector operations.

In this paper we explore the use of word embeddings in the scholarly communications domain through an empirical study. Our goal was to understand whether learning embeddings from a corpus of scienti c publications yields better results than using public, pre-trained embeddings generated from very large and general corpora. We learned word embeddings from the publications described in SciGraph. Then, since intrinsic evaluation strategies like word analogies were of limited utility in this case, we used the available metadata contained in the knowledge graph to perform a task-based evaluation consisting of classifying publications along SciGraph's categories. Classi ers were learned through neural networks, including Convolutionals [ 3 ] (CNN), which have shown good performance in text classi cation tasks [ 11 ]. In the paper, we also re ect on the ability of CNNs to learn features for the task at hand, which has been proved in computer vision [ 24 ] but still is a matter of debate in text understanding.

Evaluation results show a trade o between the knowledge speci city of the corpus used to train the word embeddings and its size. In our evaluation task, embeddings from a scienti c publication corpus consistently generate classi ers with a top performance that is only matched by classi ers learned from embeddings from very large document corpora such as Common Crawl (http://commoncrawl.org), with 42 billion tokens, or a mix of Wikipedia, news and the UMBC web corpus [ 10 ], with 16 billions tokens. Nevertheless, corpus size seems to lose relevance as the amount of short and informal language it contains increases, e.g. as in Twitter. In addition, we found that embeddings from very speci c knowledge elds that are conceptually close tend to perform better in our evaluation task than embeddings from knowledge elds with less overlap.

The rest of the paper is structured as follows. In section 2 we brie y describe SciGraph content and ontologies. Next, section 3 presents an overview of approaches to generate word embeddings, focusing on FastText and GloVe, that we use in our experiments. In section 4 we introduce the text classi cation problem and convolutional neural networks to cope with limitations of traditional linear algorithms. In Section 5 we describe our experiments and discuss the results, and nally, in section 6 we present our conclusions. 2

A Knowledge Graph for Scholarly Publications

SciGraph is a linked open data platform for the scienti c domain. It comprises information from the complete research process: research projects, conferences, authors and publications, among others. It contains metadata for millions of entities stored in triples. Currently the knowledge graph contains 1 billion facts (triples) about objects of interest to the scholarly domain, distributed over some 85 million entities described using 50 classes and more than 250 properties [ 9 ]. Currently, most of the knowledge graph is available under CC BY 4.0 License (i.e., attribution) with the exception of abstracts and grant metadata, which are available under CC BY-NC 4.0 License (i.e., attribution and non-comercial)

A core ontology expressed in OWL encodes the semantics of the data in the knowledge graph consisting of 47 classes and 253 properties. Nevertheless, the semantic metadata regarding the publication content is scarce. According to the ontology, just two predicates provide some information at a very high level of abstraction about the article content: i) sg:hasFieldOfResearchCode property relates an Australian and New Zealand Standard Research Classi cation (ANZSRC) Field of Research (FOR) code to a publication, and ii) sg:hasSubject property relates a publication to a subject term which describes one of the main topics the publication is about. In addition, text content of publication is limited to titles and abstracts of research articles and book chapters. 3

Word Embeddings

Distributed word representations are based on the distributional hypothesis where words that co-occurr in similar context are considered to have similar (or related) meaning. Word embedding algorithms yield dense vectors so that words with similar distributional context appear in the same region in the embedding space [ 22 ]. Two main families of algorithms to generate embeddings have been identi ed [ 19, 15 ]: global matrix factorization (count-based) [ 14, 19, 22 ], and local context window methods (prediction) [ 18 ]. Nevertheless, Levy and Goldberg [ 14 ] blurred that distinction, showing that local context window methods like the one proposed by Mikolov et al. [ 18 ] are implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the corresponding word and context pairs, shifted by a global constant. 3.1

Word2Vec: Distributed Word Representations

The work by Mikolov et al. [ 18 ] brought back the research interest to word embeddings in NLP since their approach reduced the computational complexity required to generate word vectors, based on negative sampling and other optimizations. This allowed training with much larger corpora than previous architectures and their evaluation results showed that the vectors encoded semantic and syntactic relations between words that could be calculated with vector operations. They proposed two model architectures to compute continuous word vectors from large datasets: Skip-gram and Continuous Bag-of-Words Model (CBOW). The models were evaluated using semantic and syntactic similarity test sets, with results showing that the Skip-gram model signi cantly outperforms other architectures, specially in terms of semantic evaluation. On the other hand, Levy et al. [ 15 ] showed that much of the performance gains of word embeddings generated through these approaches are due to hyperparameter optimizations and design choices, instead of the embedding algorithms themselves. They also argued that these modi cations can be implemented in matrix factorization approaches generating similar performance gains. 3.2

FastText: Enriching Vectors with Subword Information

Bojanowski et al. [ 2 ] proposed an evolution of the Skip-Gram model that takes into account n-grams at the character level. The motivation behind it is improving the representation of rare words and taking advantage of the words structure, specially important in morphologically rich languages. This model combines words and subwords in the form of character n-grams. They evaluate their model with nine languages and di erent word similarity and analogies datasets. The model outperforms Skip-gram and CBOW on almost every dataset. The results also show that computing vectors for out-of-vocabulary words, by summing their n-gram vectors, always obtains equal or better scores than not doing it. 3.3

GloVe: Global Vectors for Word Representation

Pennington et al. [ 19 ] proposed a model to generate word vectors by training on the nonzero elements of a word coocurrence matrix, instead of training on context windows for each word in the corpus. Their model performs more efciently by grouping together coocurrence probabilities instead of training in an online manner over the corpus. The model uses a weighted least squares objective. The computational complexity is substantially reduced by training only over the nonzero elements of the coocurrence matrix. Their experiments show that GloVe outperforms Skip-gram and CBOW, among other approaches, while substantially reducing the training time. 4

Text Classi cation

Classifying text documents in one or more classes is a main problem in NLP with applications in e.g. sentiment analysis, spam detection, email sorting and topicbased vertical search engines [ 16 ]. Naive Bayes (NB), their multinomial version (MNB), and Support Vector Machines (SVM) models are frequently used for text classi cation, depending on the problem to be addressed. For example, Wang et al. [ 23 ] showed that NB performs better in sentiment analysis tasks with short documents, while SVM obtains better results for longer texts. These algorithms work in a highly dimensional space and therefore feature selection is required to improve e ciency and accuracy. However, the choice of features is an empirical task, often following a trial and error approach. 4.1

Convolutional Neural Networks

Convolutional neural networks (CNN) were originally proposed for image recognition following a connectionist approach inspired on the visual cortex of the brain. A rst layer of simple cells activates with simple elements, like edges or corners, and subsequent layers contain more and more complex cells, which combine simple cells outputs to detect certain shapes, regardless of their position. CNNs are based on convolutional layers [ 13 ] that slide lters (or kernels) across the input data and return the dot products of the elements of the lter and each fragment of the input. In doing so, CNNs automatically learn features from the data, alleviating the manual selection required in traditional approaches. Stacking several convolutional layers allows feature composition, increasing the level of abstraction as we go from the initial layers to the output.

Single layer CNNs, consisting of a single convolutional layer, pooling, and fully connected neural layers, have been proposed for natural language processing applications. Collobert and Weston [ 3 ] used a single layer CNN in various NLP-related tasks such as part of speech tagging, named entity recognition, and chuncking, and reached state of the art performance without the need of handcrafted features. Similarly Kim [ 11 ] used this architecture for text classi cation and his results improved over the state of the art according to existing benchmarks for sentiment analysis at di erent granularity levels, detecting subjective and objective sentences, and question classi cation.

Multi layer CNN, with more than one convolutional layer, pooling and fully connected neural layers, have been proposed to include information at the character level as a complement to word level information. Dos Santos et al. [ 20 ] proposed a multi layer CNN with two convolutional layers to analyse sentiments in short texts. Similarly, Zhang et al. [ 25 ] presented a multi layer convolutional neural network with up to six convolutional layers. Their experiments showed that character-level CNN is an e ective method, however its performance depends on many factors, such as dataset size and texts quality among other variables. Convolutional neural networks applied to text classi cation use word embeddings as input. In some approaches, the CNN architecture learns the embeddings as part of the neural network training [ 3 ], while others use pre-trained embeddings [ 20 ]. 5

Experiments

So as to produce word embeddings from SciGraph publications we need to gather publication text from the knowledge graph. To this purpose, we query SciGraph for nodes of type sg:Article and sg:BookChapter, identify research articles and book chapters, and lter them according to publication date in the range 2001 to 2017. For each node, we query its title (rdfs:label ) and abstract (sg:abstract ) and keep only publications written in English. In total our dataset consists roughly of 3.2 million publications, 1 million distinct words, and 886 million tokens. In terms of size, it is similar to the United Nations corpus [ 26 ] (around 600 million tokens), which on the other hand is very general.

We use FastText with the Skip-gram algorithm and GloVe to generate embeddings from our dataset. In both cases we generate embeddings with 300 dimensions, truncate the vocabulary at a minimum count of 5 (word frequency), and set the context window size equal to 10. We have faced some issues with the implementation of GloVe when we use the default number of iterations, since some embeddings where produced with null values. In order to address these issues, we decreased the iteration number from 15 to 12. Bear in mind that a lower number of iterations may in uence the evaluation of the resulting embeddings.

On the other hand, to compare SciGraph embeddings we use pre-trained embeddings generated with GloVe and FastText learned from very large and general corpora. FastText embeddings where generated from Common Crawl and Wikipedia [ 8 ], and from Wikipedia exclusively [ 2 ], while GloVe embeddings were learned from Wikipedia, Gigaword, Common Crawl, and Twitter [ 19 ]. 5.1

Analogical Reasoning and Word Similarity

We initially evaluate our word embeddings through intrinsic evaluation methods [ 21 ], such as the analogy task [ 17 ] and word similarity. The goal of the analogy task is to nd x such that the relation x:y resembles a sample relation a:b by operating on the corresponding word vectors. The analogy dataset2 contains 19,544 question pairs (8,869 semantic and 10,675 syntactic questions) and 14 types of relations (9 syntactic and 5 semantic). The word similarity evaluation is based on the WordSim353 dataset3, which contains 353 word pairs with similarity scores assigned by humans that we compare with similarity based on the word embeddings.

Table 1 reports the accuracy values for SciGraph and pre-trained embeddings in the analogy task and Spearmans's rank correlation coe cient in word similarity. As expected, given the rather small size of the SciGraph corpus (886M tokens) compared to the other sources (number of tokens between 3B and 42B) and the fact that SciGraph focuses on the scienti c domain, the performance obtained was signi cantly lower. A quick look at those benchmarks clearly shows that the SciGraph corpus does not cover all the vocabulary and semantic and syntactic relations that are evaluated in such tasks. However, as we will see below, the fact that SciGraph embeddings do not perform well in this task does not mean that they are not suitable for other tasks focused on the domain from which they were learned. Since word analogies and similarity are not t, with the existing benchmarks, to evaluate SciGraph embeddings, we propose an extrinsic evaluation method in the form of a classi cation task. 2 https://aclweb.org/aclwiki/Google analogy test set (State of the art) 3 https://aclweb.org/aclwiki/WordSimilarity-353 Test Collection (State of the art) Algorithm Dimensions Corpus Analogy

Word Sim.

Sem. Synt. Total Spearman's rho GloVe FastText GloVe FastText 300 Wiki+Giga (6B) 77.4 67.0 71.7 300 Common Crawl (42B) 81.9 69.3 75.0 300 Wikipedia (3B*) 77.8 74.9 76.2 300 SciGraph (886M) 8.1 1.7 4.6 300 SciGraph (886M) 17.1 48.5 34.3 Table 1. Results from the word analogy and similarity tasks In SciGraph, each publication has one or more eld of research codes that classify the documents in 22 categories such as Mathematical Sciences, Engineering or Medical and Health Sciences. Based on this classi cation scheme, we de ne a multi-label classi cation problem to predict one or more categories for each publication through neural networks. The design and con guration of a particular neural network architecture is a complex task that falls out of the scope of this paper. Several approaches try to assist [ 5 ] data scientist in this task, simplifying the process and helping to select the optimal [ 27 ] con guration. In our case, we use two types of neural networks: a regular, fully connected network and a convolutional one. The neural network is composed of an input layer, a fully connected 50-unit neural layer with a ReLU activation function, and an output sigmoid layer. The convolutional network was an out-of-the-box implementation available in Keras with 3 convolutional layers with 128 lters and a 5-element window size, each followed by a max pooling layer, a fully-connected 128-unit ReLU layer and a sigmoid output.

To evaluate the classi ers we select articles published in 2011 and use ten-fold cross-validation. As baseline, we train a classi er that learns from embeddings generated randomly following a normal distribution. As upper bound we learn a classi er model that learns a new set of word embeddings during training of either the neural and the convolutional networks. The performance of the classi ers is shown in table 2.

The results of the regular neural network show that for this architecture the best classi er is produced from FastText SciGraph embeddings and FastText Wiki+Web+News, although the f-score is far from the upper bound, meaning that there is still room to get better embeddings for this learning algorithm. Looking at the results produced by the convolutional network, we see that all the classi ers have increased their performance to a similar level and the fmeasure variation is very close to the upper-bound of 0.79. Also note that the baseline classi er learned from random embeddings has risen its performance to 0.72 and now is closer to the upper bound.

The fact that, regardless of the corpora used to generate the word embeddings, the convolutional network systematically obtains a performance similar Dim.

F-Score

NN CNN 300 0.13 0.72 300 0.78 0.79 Algorithm

Dataset Random Normal Dist.

Classi er GloVe FastText GloVe FastText

Wiki+Giga (6B) 300 0.67 0.77 Common Crawl (42B) 300 0.67 0.77 Twitter (27B) 200 0.61 0.75 Wiki+Web+News (16B) 300 0.69 0.78 Wiki (3B) 300 0.68 0.77 Scigraph (886M) Scigraph (886M) 300 0.67 0.76 300 0.69 0.78 to the best results produced by the upper bound is interesting and shows evidence of the bene ts derived from automatic feature selection and composition in text classi cation. The convolutional network trained on Twitter embeddings is the only case that achieved lower results. This is partially due to the informal language, both vocabulary and grammar, as well as the shorter text found in Twitter, which does not have a signi cant overlap with the rather formal and speci c scienti c vocabulary presented in our dataset. 5.3

Fine-Grained Classi cation

We further the evaluation of SciGraph embeddings through a ne-grained classi cation task where we aim at predicting second-level categories in three elds of knowledge: Computer Science, Mathematics, and Chemistry. In addition to the previous embeddings, we generate embeddings for each speci c eld. We widen the dataset and include publications between 2011 and 2012 so that each second-level category counts on more samples. Finally, we evaluate the multilabel classi ers using ten-fold cross validation and focus on CNNs due to the superior performance showed in the previous experiment.

Evaluation results (table 3) show that embeddings generated from the document corpora of each knowledge eld in SciGraph consistently lead to the best classi ers and their performance is very similar to the upper bound. FastText (Wiki+Web+News) embeddings and GloVe (Wiki+Giga) achieve similar f-scores. In this ne-grained classi cation, we observe that embeddings from the SciGraph general corpus produce average results and do not completely discriminate second-level categories across knowledge elds. This is partially due to di erences in corpora size between the general and eld-speci c cases but also in the semantics captured for each speci c word in their corresponding vectors. In the case of the general corpus, the latter is in uenced by all the di erent contexts where a particular word can appear across elds of knowledge. As a consequence, Algorithm

Dataset Random Normal Classi er GloVe FastText

Wiki+Giga (6B) Common Crawl (42B) Twitter (27B) Wiki+Web+News (16B) Wiki

Dim.

300 300 300 300 200 300 300 the location in the vector space of the point associated to a particular word is shifted compared to embeddings generated locally to each eld.

Table 4 compares the performance of the classi ers learned, using only embeddings generated from each knowledge eld. In general, embeddings generated from a eld produce the best classi er for such eld. In addition, we observe that the conceptual proximity between knowledge elds also in uences the results. For example, embeddings obtained from Mathematics perform well in Computer Science and Chemistry, which are both related to Mathematics. The other way around also holds: embeddings from Computer Science and Chemistry perform well in Mathematics. However, Chemistry embeddings obtain worse results in Computer Science and Computer Science embeddings perform worse in the Chemistry category. This could be related to the fact that these knowledge areas have less similarities with each other than with Mathematics. Curiously, Computer Science and Chemistry embeddings seem to work even better for Mathematics than for their originating elds, which will need to be further researched.

Embeddings

F-Score

Computer Science Mathematics Chemistry

Conclusions and Future Work

This paper presents experimental results related to the generation and evaluation of word embeddings from scholarly communications in the scienti c domain, leveraging SciGraph content and metadata. We compare the resulting, domainspeci c embeddings with pre-trained vectors generated from large and general purpose corpora over a selection of intrinsic and extrinsic tasks.

We show that intrinsic evaluation methods like word analogy and word similarity are not a reliable benchmark for embeddings learned from scienti c corpora, the reason being the mismatch in terms of vocabulary and domain coverage between the scienti c corpus and the evaluation dataset. Our ndings should raise community awareness about the need for larger (or domain-speci c) intrinsic evaluation benchmarks for word embeddings. We followed on to conduct an extrinsic evaluation in the form of a domain-speci c classi cation task for scienti c publications at di erent granularity levels. The evaluation shows that our classi ers make the most of embeddings generated through FastText both from a corpus of scienti c publications (886 million tokens) and from a much larger mix of Wikipedia, Web content, and News (16 billions tokens).

Our results raise an interesting discussion about corpus size vs. speci city for domain-dependent tasks. For scholarly communications and in the optimal con gurations of the evaluation systems, the embeddings learned from focused corpora produce similar results to those generated from much larger and general corpora with many billions tokens. We also observed that all the classi ers learned through convolutional neural networks were closer to the upper bound, indicating that the features learned by the convolutional network are more expressive than pre-trained word embeddings. In both cases, the features learned during the training of the convolutional nets seem to improve over the knowledge captured in the input word vectors, making the overall system more resilient to the quality of such vectors for the task at hand. In addition, we showed that embeddings generated from speci c knowledge elds perform well in classi cation tasks over related knowledge elds, such as Computer Science and Mathematics, and not so well where the knowledge elds are not so close, such as Computer Science and Chemistry.

We expect that this work lays a foundation for the future use of word embeddings in NLP tasks applied to scienti c publications. Applications include, e.g. the curation of existing knowledge graphs, such as SciGraph itself, with metadata about the publication content so that not only accessory, but also content-wise structured metadata is available for software applications. Related work in this direction involves merging embeddings and Expert System's Cogito knowledge graph 4 in a joint word-concept vector space [ 4 ], which showed improvements in word similarity evaluations with respect to traditional word embeddings and other NLP downstream tasks. Finally, interesting work lays ahead in trying to better understand the features learnt by convolutional neural networks in NLP and their representation for human inspection. 4 http://www.expertsystem.com/products/cogito-cognitive-technology/

Acknowledgments

We gratefully acknowledge the EU Horizon 2020 program for research and innovation under grant EVER-EST-674907. We also thank Constantino Roman for his contributions to the experimental evaluation of this work.

References

1. Bechhofer , S. , Buchan , I. , Roure , D.D. , Missier , P. , Ainsworth , J. , Bhagat , J. , Couch , P. , Cruickshank , D. , Delder eld, M., Dunlop , I. , Gamble , M. , Michaelides , D. , Owen , S. , Newman , D. , Su , S. , Goble , C. : Why linked data is not enough for scientists . Future Generation Computer Systems 29 ( 2 ), 599 { 611 ( 2013 ), special section: Recent advances in e-Science

2. Bojanowski , P. , Grave , E. , Joulin , A. , Mikolov , T. : Enriching word vectors with subword information . arXiv preprint arXiv:1607.04606 ( 2016 )

3. Collobert , R. , Weston , J.: A uni ed architecture for natural language processing: Deep neural networks with multitask learning . In: Proceedings of the 25th International Conference on Machine Learning . pp. 160 { 167 . ICML '08, ACM , New York, NY, USA ( 2008 ), http://doi.acm. org/10 .1145/1390156.1390177

4. Denaux , R. , Gomez-Perez , J.M. : Towards a vecsigrafo: Portable semantics in knowledge-based text analytics . In: Joint Proceedings of the International Workshops on Hybrid Statistical Semantic Understanding and Emerging Semantics, and Semantic Statistics co-located with 16th International Semantic Web Conference, HybridSemStats@ISWC 2017 , Vienna, Austria October 22nd, 2017 ( 2017 ), http://ceur-ws. org/ Vol-1923/article-04.pdf

5. Desell , T. : Developing a volunteer computing project to evolve convolutional neural networks and their hyperparameters . In: 2017 IEEE 13th International Conference on e-Science (e-Science) . pp. 19 { 28 (Oct 2017 )

6. Gomez-Perez , J.M. , Palma , R. , Garcia-Silva , A. : Towards a human-machine scienti c partnership based on semantically rich research objects . In: 2017 IEEE 13th International Conference on e-Science (e-Science) . pp. 266 { 275 (Oct 2017 )

7. Gomez-Perez , J.M. , Denaux , R. , Garcia-Silva , A. , Palma , R.: A holistic approach to scienti c reasoning based on hybrid knowledge representations and research objects . In: Proceedings of Workshops and Tutorials of the 9th International Conference on Knowledge Capture (K-CAP2017) . pp. 47 { 49 ( 2017 )

8. Grave , E. , Bojanowski , P. , Gupta , P. , Joulin , A. , Mikolov , T. : Learning word vectors for 157 languages . In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018 ) ( 2018 )

9. Hammond , T. , Pasin , M. , Theodoridis , E. : Data integration and disintegration: Managing springer nature scigraph with shacl and owl . In: Nikitina, N. , Song , D. , Fokoue , A. , Haase , P. (eds.) International Semantic Web Conference ( Posters, Demos and Industry Tracks) . CEUR Workshop Proceedings , vol. 1963 . CEUR-WS.org ( 2017 ), http://dblp.uni-trier.de/db/conf/semweb/iswc2017p. html#HammondPT17

10. Han , L. , Kashyap , A.L. , Finin , T. , May eld, J., Weese , J.: UMBC EBIQUITYCORE : Semantic Textual Similarity Systems . In: Proceedings of the Second Joint Conference on Lexical and Computational Semantics . Association for Computational Linguistics ( June 2013 )

11. Kim , Y. : Convolutional neural networks for sentence classi cation . In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29 , 2014 , Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL . pp. 1746 { 1751 ( 2014 ), http://aclweb.org/anthology/ D/D14/D14-1181.pdf

12. Kitano , H.: Arti cial intelligence to win the nobel prize and beyond: Creating the engine for scienti c discovery . AI Magazine 37 ( 1 ), 39 { 49 ( 2016 ), http://www.aaai. org/ojs/index.php/aimagazine/article/view/2642

13. LeCun , Y., Bottou , L. , Bengio , Y. , Ha ner , P.: Gradient-based learning applied to document recognition . Proceedings of the IEEE 86 ( 11 ), 2278 { 2324 ( 1998 )

14. Levy , O. , Goldberg , Y. : Neural word embedding as implicit matrix factorization . In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 . pp. 2177 { 2185 . NIPS'14, MIT Press, Cambridge, MA, USA ( 2014 ), http://dl.acm.org/citation.cfm?id= 2969033 . 2969070

15. Levy , O. , Goldberg , Y. , Dagan , I. : Improving Distributional Similarity with Lessons Learned from Word Embeddings . Transactions of the ACL 3 ( 0 ), 211 { 225 ( 2015 )

16. Manning , C.D. , Raghavan , P. , Schutze, H.: Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA ( 2008 )

17. Mikolov , T. , Sutskever , I. , Chen , K. , Corrado , G. , Dean , J. : Distributed Representations of Words and Phrases and their Compositionality . In: NIPS ( 2013 )

18. Mikolov , T. , Chen , K. , Corrado , G. , Dean , J.: E cient estimation of word representations in vector space . CoRR abs/1301 .3781 ( 2013 ), http://arxiv.org/abs/ 1301.3781

19. Pennington , J. , Socher , R. , Manning , C.D.: Glove: Global vectors for word representation . In: EMNLP . vol. 14 , pp. 1532 { 1543 ( 2014 )

20. dos Santos, C. , Gatti , M. : Deep convolutional neural networks for sentiment analysis of short texts . In: Proceedings of COLING 2014 , the 25th International Conference on Computational Linguistics: Technical Papers . pp. 69 { 78 ( 2014 )

21. Schnabel , T. , Labutov , I. , Mimno , D. , Joachims , T. : Evaluation methods for unsupervised word embeddings . In: EMNLP . pp. 298 { 307 . ACL ( 2015 )

22. Shazeer , N. , Doherty , R. , Evans , C. , Waterson , C. : Swivel: Improving Embeddings by Noticing What's Missing. arXiv preprint ( 2016 )

23. Wang , S. , Manning , C.D.: Baselines and bigrams: Simple, good sentiment and topic classi cation . In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2 . pp. 90 { 94 . ACL ' 12 , Association for Computational Linguistics, Stroudsburg, PA, USA ( 2012 ), http: //dl.acm.org/citation.cfm?id= 2390665 . 2390688

24. Zeiler , M.D. , Fergus , R.: Visualizing and understanding convolutional networks . In: Fleet, D. , Pajdla , T. , Schiele , B. , Tuytelaars , T. (eds.) Computer Vision { ECCV 2014 . pp. 818 { 833 . Springer International Publishing, Cham ( 2014 )

25. Zhang , X. , Zhao , J. , LeCun, Y.: Character-level convolutional networks for text classi cation . In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 . pp. 649 { 657 . NIPS'15, MIT Press, Cambridge, MA, USA ( 2015 ), http://dl.acm.org/citation.cfm?id= 2969239 . 2969312

26. Ziemski , M. , Junczys-Dowmunt , M. , Pouliquen , B. : The united nations parallel corpus v1. 0 . In: Language Resource and Evaluation ( 2016 )

27. Zoph , B. , Le , Q.V. : Neural architecture search with reinforcement learning . CoRR abs/1611 .01578 ( 2016 ), http://arxiv.org/abs/1611.01578