Introduction

Distributional Semantic Models (DSM) that represent words as vectors of weights over a high dimensional feature space [13], have proved very effective in representing semantic or syntactic aspects of lexicon. Incorporating such representations has allowed improving many natural language tasks. They also reduce the burden of feature selection since these models can be learned through unsupervised techniques from plain text. Deep learning algorithms for NLP tasks exploit distributional representation of words. In tagging applications such as POS tagging, NER tagging and Semantic Role Labeling (SRL), this has proved quite effective in reaching state of art accuracy and reducing reliance on manually engineered feature selection [8].

Word embeddings have been exploited also in constituency parsing [8] and dependency parsing [4]. Blanco et al. [3] exploit word embeddings for identifying entities in web search queries. This paper presents DeepNL, an NLP pipeline based on a common Deep Learning architecture: it consists of tools for creating embeddings, and tools that exploit word embeddings as features. The current release includes a POS tagger, a NER, an SRL tagger and a dependency parser.

Two methods are supported for creating embeddings: an approach that uses neural network and one using Hellinger PCA [15].

Building Word Embeddings

Word embeddings provide a low dimensional dense vector space representation for words, where values in each dimension may represent syntactic or semantic properties.

DeepNL provides two methods for building embeddings, one is based on the use of a neural language model, as proposed by [26,8,18] and one based on spectral method as proposed by Lebret and Collobert [15].

The neural language method can be hard to train and the process is often quite time consuming, since several iterations are required over the whole training set. Some researcher provide precomputed embeddings for English1 . The Polyglot project [1] makes available embeddings for several languages, built from the plain text of Wikipedia in the respective language, and the Python code for computing them2 , that supports GPU computations by means of Theano3 .

Mikolov et al. [20] developed an alternative solution for computing word embeddings, which significantly reduces the computational costs. They propose two loglinear models, called bag of words and skip-gram model. The bag-of-word approach is similar to a feed-forward neural network language model and learns to classify the current word in a given context, except that instead of concatenating the vectors of the words in the context window of each token, it just averages them, eliminating a network layer and reducing the data dimensions. The skip-gram model tries instead to estimate context words based on the current word. Further speed up in the computation is obtained by exploiting a mini-batch Asynchronous Stochastic Gradient Descent algorithm, splitting the training corpus into partitions and assigning them to multiple threads. An optimistic approach is also exploited to avoid synchronization costs: updates to the current weight matrix are performed concurrently, without any locking, assuming that updates to the same rows of the matrix will be infrequent and will not harm convergence.

The authors published single-machine multi-threaded C++ code for computing the word vectors 4 . A reimplementation of the algorithm in Python is included in the Genism library [23]. In order to obtain comparable speed to the C++ version, they use Cython for interfacing to a coding in C of the core function for training the network on a single sentence, which in turn exploits the BLAS library for algebraic computations.

2.1

Word Embeddings through Hellinger PCA Lebret and Collobert [15] have shown that embeddings can be efficiently computed from word co-occurrence counts, applying Principal Component Analysis (PCA) to reduce dimensionality while optimizing the Hellinger similarity distance.

Levy and Goldberg [16] have shown similarly that the skip-gram model by Mikolov et al. [20] can be interpreted as implicitly factorizing a word-context matrix, whose values are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant.

DeepNL provides an implementation of the Hellinger PCA algorithm using Cython and the LAPACK library SYSEVR.

Co-occurrence frequencies are computed by counting the number of times each context word w  D occurs after a sequence of T words:

𝑝(𝑤|𝑇) = 𝑝(𝑤, 𝑇) 𝑝(𝑇) = 𝑛(𝑤, 𝑇) ∑ 𝑛(𝑤, 𝑇) 𝑛

where n(w, T) is the number of times word w occurs after a sequence of T words. The set D of context word is normally chosen as the subset of the top most frequent words in the vocabulary V.

The word co-occurrence matrix C of size |V||D| is built. The coefficients of C are square rooted and then its transpose is multiplied by it to obtain a symmetric square matrix of size |V||V|, to which PCA is applied for obtaining the desired dimensionality reduction.

Context Sensitive Word Embeddings

The meaning of words often depends on their context. Our approach for learning word embeddings in context is inspired by the method for learning paragraph vectors [14]. We improve on their approach, avoiding the cost of computing at query time an embedding for the paragraph of the query. Our solution bears some resemblance to the approach in [12]. We add padding at sentence boundaries and substitute <UNK> for OOV words.

The prediction task is performed by a neural network with a softmax layer:

𝑝(𝑤 𝑡 |𝑤 𝑡−𝑘 , … , 𝑤 𝑡+𝑘 ) = 𝑒 𝑦 𝑖 ∑ 𝑒 𝑦 𝑖 𝑖

Each y i is the score for output word i, computed as: 𝑦 = 𝑏 + 𝑈ℎ(𝑤 𝑡−𝑘 , … , 𝑤 𝑡+𝑘 ; 𝑊, 𝐷) where b, U are network parameters, W and D are the weight matrixes for words and paragraphs respectively. h concatenates the vectors of each word extracted from W and of their sum multiplied by D:

ℎ(𝑖 1 , … , 𝑖 𝑛 ; 𝑊, 𝐷) = 𝑊 1 || … || 𝑊 𝑛 ||𝐷 ∑ 𝑊 𝑖 𝑛 𝑖

The combination of word vector and paragraph vector can be used for word sense disambiguation. A word i k within paragraph i 1 ,…,i n is represented by the concatenation

𝑊 𝑖 𝑘 ||𝐷 ∑ 𝑊 𝑖 𝑛 𝑖

Sentiment Specific Word Embeddings

For the task of sentiment analysis, semantic similarity is not appropriate, since antonyms end up at close distance in the embeddings space. One needs to learn a vector representation where words of opposite polarity are far apart. Tang et al. [25] propose an approach for learning sentiment specific word embeddings, by incorporating supervised knowledge of polarity in the loss function of the learning algorithm. The original hinge loss function in the algorithm by Collobert et al. [6] is:

L CW (x, x c ) = max(0, 1  f  (x) + f  (x c ))

where x is an ngram and x c is the same ngram corrupted by changing the target word with a randomly chosen one, f  (•) is the feature function computed by the neural network with parameters θ. The sentiment specific network outputs a vector of two dimensions, one for modeling the generic syntactic/semantic aspects of words and the second for modeling polarity.

A second loss function is introduced as objective for minimization:

L SS (x, x c ) = max(0, 1   s (x) f  (x) 1 +  s (x) f  (x c ) 1 )

where the subscript in f  (x) 1 refers to the second element of the vector and  s (x) is an indicator function reflecting the sentiment polarity of a sentence, whose value is 1 if the sentiment polarity of x is positive and -1 if it is negative. The overall hinge loss is a linear combination of the two:

L(x, x c ) = L CW (x, x c ) + (1 -) L SS (x, x c )

DeepNL provides an algorithm for training polarized embeddings, performing gradient descent using an adaptive learning rate according to the AdaGrad method. The algorithm requires a training set consisting of sentences annotated with their polarity, for example a corpus of tweets. The algorithm builds embeddings for both unigrams and ngrams at the same time, by performing variations on a training sentence replacing not just a single word, but a sequence of words with either another word or another ngram.

Deep Learning Architecture

DeepNL adopts a multi-layer neural network architecture, as proposed in [6], consisting of five layers: a lookup layer, a linear layer, an activation layer (e.g. hardtanh), a second linear layer and a softmax layer. Overall, the network computes the following function:

f(x) = softmax(M 2 a(M 1 x + b 1 ) + b 2 )

where

M 1  R hd , b 1  R d , M 2  R oh , b 2  R o ,

Lookup layer

The first layer of the network transforms the input into a feature vector representation. Individual words are represented by a vector of features, which is trained by backpropagation.

For each word w  D, an internal d-dimensional feature vector representation is given by the lookup table layer LTW(•):

𝐿𝑇 𝑊 (𝑤) = 〈𝑊〉 𝑤 1

where 𝑊 ∈ ℝ 𝑑×|𝒟| is a matrix of parameters to be learned, 〈𝑊〉 𝑤 1 ∈ 𝒟 is the w th column of W and d is the word vector size (a hyper-parameter to be chosen by the user).

Discrete Features

Besides word representations, a number of discrete features can be used. Each feature has its own lookup table 𝐿𝑇 𝑊 𝑘 (•) with parameters 𝑊 𝑘 ∈ ℝ 𝑑 𝑘 ×|𝒟 𝑘 | , where D k is the dictionary for the k-th feature and d k is a user specified vector size. The input to the network becomes the concatenation of the vectors for all features:

𝐿𝑇 𝑊 1 (𝑤)𝐿𝑇 𝑊 2 (𝑤) ⋯ 𝐿𝑇 𝑊 𝐾 (𝑤)

Sequence Taggers

For sequence tagging, two approaches were proposed in [6], a window approach and a sentence approach. The window approach assumes that the tag of a word depends mainly on the neighboring words, and is suitable for tasks like POS and NE tagging.

The sentence approach assumes that the whole sentence must be taken into account by adding a convolution layer after the first lookup layer and is more suitable for tasks like SRL. We can train a neural network to maximize the log-likelihood over the training data. Denoting by  the trainable parameters of the network, we want to maximize the following log-likelihood with respect to :

∑ log 𝑝(𝑡 𝑖 |𝑐 𝑖 , 𝜃) 𝑖

The score s(w, t, ) of a sequence of tags t for a sentence w, with parameters , is given by the sum of the transition scores and the tree scores: 𝑠(𝑥, 𝑡, 𝜃) = ∑(𝑇(𝑡 𝑖−1 , 𝑡 𝑖 ) + 𝑓 𝜃 (𝑥 𝑖 , 𝑡 𝑖 ))

𝑛 𝑖=1

where T(i, j) is the score for the transition from tag i to tag j, and f  (x i , t i ) is the output of the network at word x i with tag t,. The probability of a sequence y for sentence x can be expressed as: 𝑝(𝑦|𝑥, 𝜃) = 𝑒 𝑠(𝑥,𝑦,𝜃) ∑ 𝑒 𝑠(𝑥,𝑡,𝜃) 𝑡

Experiments

We tested the DeepNL sequence tagger on the CoNLL 2003 challenge 5 , a NER benchmark based on Reuters data. The tagger was trained with three types of features: word embeddings from SENNA, a "caps" feature telling whether a word is in lowercase, uppercase, title case, or had at least one non-initial capital letter, and a gazetteer feature, based on the list provided by the organizers. The window size was set to 5, 300 hidden variables were used and training was iterated for 40 epochs. In the following table we report the scores compared with the system by Ando et al. [2] which uses a semi-supervised approach and with the results by the released version of SENNA The slight difference with SENNA is possibly due to the use of different suffixes.

Software Architecture

The DeepNL implementation is written in Cython and uses C++ code which exploits the Eigen7 library for efficient parallel linear algebra computations. Data is exchanged between Numpy arrays in Python and Eigen matrices by means of Eigen Map types. On the Cython side, a pointer to the location where the data of a Numpy array is stored is obtained with a call like: <FLOAT_t*>np.PyArray_DATA(self.nn.hidden_weights)

and passed to a C++ method. On the C++ side this is turned into an Eigen matrix, with no computational costs due to conversion or allocation, with the code:

Map<Matrix> hidden_weights(hidden_weights, numHidden, numInput)

which interprets the pointer to a double as a matrix with numHidden rows and nu-mInput columns.

Feature Extractors

The library has a modular architecture that allows customizing a network for specific tasks, in particular its first layer, by supplying extractors for various types of features. An extractor is defined as a class that inherits from an abstract class with the following interface: class Extractor(object): def extract(self, tokens) def lookup(self, feature) def save(self, file) def load(self, file)

Method extract, applied to a list of tokens, extracts features from each token and returns a list of IDs for those features. Method lookup returns the vector of weights for a given feature. Methods save/load allow saving and reloading the Extractor data to/from disk. Extractors currently include an Embeddings extractor, implementing the word lookup feature, a Caps, Prefix and Postfix extractor for dealing with capitaliza-tion and prefix/postfix features, a Gazetteer extractor for dealing with the gazetteers typically used in a NER, and a customizable AttributeFeature extractor that extracts features from the state of a Shift/Reduce dependency parser, i.e. from the tokens in the stack or buffer as described for example in [21].

Parallel gradient computation

The computation of the gradients during network training requires computing the conditional probability over all possible sequences of tags, which grow exponentially with the length of the sequence. They can however be computed in linear time by accumulating them in a matrix, and then the matrix computation can be parallelized, as in the following code:

delta = scores delta[0] += transitions[-1] tr = transitions[:-1] for i in xrange(1, len(delta)): # sum by columns logadd = logsumexp(delta[i-1][:,newaxis] + tr, 0) delta[token] += logadd

The array scores[i, j] contains the output of the neural network for the i-th element of the sequence and for tag j, delta[i, j] represents the sum of all scores ending at the i-th token with tag j; transitions[i, j] contains the current estimate of the probability of a transition from tag i to tag j. The computation can be optimized and parallelized using suitable linear algebra libraries. We implemented two versions of the network trainer, one in Python using NumPy 8 and one in C++ using Eigen 9 .

Identification of Idiomatic Multiword Expressions

As an application of word embeddings, we experiment on the identification of idiomatic multiword expressions. Multiword expressions are combinations of two or more words which can be syntactically and/or semantically idiosyncratic in nature. There are many varieties of multiword expressions: we concentrate on non-decomposable idioms, i.e. those idioms in which the meaning cannot be assigned to the parts of the MWE.

MWE identification is typically split into two phases: candidate identification and filtering.

For identifying potential candidates for MWE one can exploit the technique for discovering collocations [5] based on Pointwise Mutual Information.

We adopt the simple variant proposed by Mikolov et al. (2013a), of computing a score for the likelihood of forming a collocation, using the unigram and bigram counts:

𝑠𝑐𝑜𝑟𝑒(𝑤 𝑖 , 𝑤 𝑗 ) = 𝑐𝑜𝑢𝑛𝑡(𝑤 𝑖 , 𝑤 𝑗 ) − 𝛿 𝑐𝑜𝑢𝑛𝑡(𝑤 𝑖 ) • 𝑐𝑜𝑢𝑛𝑡(𝑤 𝑗 )

The bigrams with score above a chosen threshold are then used as phrases. The  is a discounting coefficient and prevents generating too many phrases consisting of very infrequent words. We also apply a cutoff on the frequency of bigrams, to avoid depending too much on the frequency of the individual words and in particular to limit the tendency of assigning higher scores to lower frequency words.

The process is repeated a few times by replacing the bigrams with a single token and decreasing the threshold value, in order to extract longer phrases.

As many have noted [17], just relying on statistical measures of frequency for identifying MWEs does not achieve very satisfactory results, since idiomatic phrases are not that much frequent in texts, hence data is sparse; therefore some sort of semantic knowledge is required.

Srivastava and Hovy [24] introduce a segmentation model for partitioning a sentence into linear constituents, called motifs, which is learned though semi-supervised learning. They then build embeddings for such motifs using the Hellinger PCA technique of Lebret and Collobert [15].

For deciding whether a candidate collocation is indeed a phraseme, we rely on the distinctive properties of idiomatic expressions: non-composability, i.e. their meaning is not obtainable as a composition of the meaning of its part; non-substitutivity, i.e. replacing near-synonyms for the parts of a phrase would produce something weird or nonsensical.

We assume that these two aspects should be fairly evident to the reader, otherwise he would not be able to distinguish a phraseme from a normal phrasal combination. Therefore, if we replace some of the words in the expression with similar words, we should end up with an apparently weird combination.

The basic idea in our experiments is to select replacement words that are similar according to their distance in the word embedding space. As a criterion for deciding if a phrase is unusual, we check first if no variant occurs in the corpus, otherwise we check whether the LM probability of all variants is below a given threshold.

Experiments

We carried out experiments on the corpus consisting of the plain text extracted from the English Wikipedia, for a total of 1,096,243,235 tokens, 4,456,972 distinct. We created word embeddings on the corpus obtained by performing token combinations as described above, using a threshold of 500 on the first iteration and 300 on the following ones. We used a cutoff of 80 on the first iteration and 40 on the following ones. The vocabulary for this corpus consists of 225,000 words or phrases.

For evaluating our model, we used the WikiMwe corpus [10], which includes a gold evaluation set consisting of 2,500 expressions, annotated in four categories: noncompositional, collocation, regular natural language phrase and ungrammatical. Table 2 shows the results of our experiments. An online demo of a similar system for the identification of Italian idiomatic phrases is available at: http://tanl.di.unipi.it/embeddings/mwe. A potential application of the technique is the identification of chunks in search queries or in AdWords queries, in order to recognize expression whose intended meaning does not correspond to the combination of the individual words in the query.

Type

Conclusions

We have presented the architecture of DeepNL, a library for building NLP applications based on a deep learning architecture. The toolkit includes various methods for creating embedding, either generic embeddings and sentiment specific or context sensitive embeddings.

As an example of the effectiveness of the embeddings, we have explored their use in the identification of idiomatic word expressions.

The implementation is written in Python/Cython and uses C++ linear algebra libraries for efficiency and scalability, exploiting multithreading or GPUs where available. The code is available for download from: https://github.com/attardi/deepnl.

Table 1. Results on the identification of idiomatic expressions.

There are several potential applications for the library, in particular sentiment specific word embeddings might be applied to other classification tasks, for example detecting tweets that signal dangers or disasters.

Context sensitive word embeddings can be exploited in artificial tasks like word sense disambiguation or word sense similarity. Hopefully they should provide also benefits for more relevant tasks such as relation extraction, negation identification, data linking, and ontology creation. Context-aware embeddings have been indeed applied effectively to matching ad words to queries by Grbovic et al. [10].

We hope that the availability of the code will encourage exploring their use in further applications.