Topic identi cation as a speci c case of text classi cation is one of the primary steps toward knowledge extraction from the raw textual data. In such tasks, words are dealt with as a set of features. Due to high dimensionality and sparseness of feature vector result from traditional feature selection methods, most of the proposed text classi cation methods for this purpose lack performance and accuracy. In dealing with tweets which are limited in the number of words the aforementioned problems are re ected more than ever. In order to alleviate such issues, we have proposed a new topic identi cation method for Spanish tweets based on the deep representation of Spanish words. In the proposed method, words are represented as multi-dimensional vectors, in other words, words are replaced with their equivalent vectors which are calculated based on some transformation of raw text data. Average aggregation technique is used to transform the word vectors into tweet representation. Our model is trained based on deep vectorized representation of the tweets and an ensemble of different classi ers is used for Spanish tweet classi cation. The best result obtained by a fully connected multi-layer neural network with three hidden layers. The experimental results demonstrate the feasibility and scalability of the proposed method.
Topic identi cation is one of the primary steps toward text understanding. This
process has to be done automatically due to the large amount of existing texts.
A few number of topic identi cation applications are as follow [
People use social medias to state their idea about social events. People's
tweets capture researchers' attention on different issues and they have been used
for a variety of purposes, such as marketing communication, education and
politics [
There are a lot of challenges regarding classifying tweets includes
colloquialism in tweets, spelling variation, use of special characters, violating regular
grammar rules etc [
In order to work with textual data, it should be described numerically to
enable computers to process that. In traditional approaches, words are considered
as distinct features for representing textual data. Those representations suffer
from scarcity and disability to detect synonyms [
In this paper, we used deep text representation to classify Spanish tweets.
We applied the method provided in our previous work [
The structure of this paper is as follows: rst, related works are studied in Section 2, deep representation of text is described in Section 3. An introduction to tweet classi cation using deep representation is given in Section 4. Section 5 deals with an introduction to the evaluation metric, and results of the proposed approach over the provided data sets. Finally, Section 6 ends the paper with conclusion and some insights. 2
In this section, some well-known methods for text classi cation are described and speci cally some of the recently presented tweet classi cation approaches are discussed and reviewed.
In the work of Ghavidel et al.[
Deep learning tries to nd more abstract features using deep multiple layer
graph. Each layer has linear or non-linear function to transform data into more
abstract ones [
In application of deep representation in natural language processing, each word
is described by the surrounding context. This vector which is generated by a
deep neural networks and contain semantic and syntactic information about the
word. In distributed word representation, generally known as word-embedding,
the similar words have the similar vectors [
A composition function should be provided for combining word vectors to
represent text. Paragraph Vector is an unsupervised algorithm that used the idea of
word vector training and considered a matrix for each piece of text. This matrix
also update during language modeling task. Paragraph vector outperforms other
methods such as bag-of-words models for many applications [
In this section, we describe our approach for Spanish tweet classi cation. These steps include pre-processing that describe the text re nement process, then composition function to combine the word embedding to construct and represent each tweet is illustrated. Then the classi cation algorithms applied to classi ed the tweets into the aforementioned categories. Figure 1 shows the steps of tweet topic identi cation procedure.
Spanish Tweets
Preprocessing (remove stopwords, ...) Calculate sentence
vector Deep representation for training tweets train classifier
Model
Deep representation
for test tweets Classify test tweets
by model As the rst step of the processing, the following steps are done as pre-processing of every block of text, i.e tweet. Such as elimination of special characters as: \&", \(", \)", \#", and removal of all numbers.
Other most common pre-processing functions is removing the stop-words. In this regard, we simply omit list of stop words available on-line1. This list includes 178 stop words like: \una", \es", \soy", \vamos", etc.
4.2
We rst retrieve Spanish 300-dimensional word vectors on-line2. Then Spanish stop words eliminated while text pre-processing. After that, for each sentence an average of all word vectors is calculated as in equation (1).
F 1macro = L1 ∑ F1(y1; y^1)
l2L
Si = ∑n i=1 wi n Where S is the vector representation for tweet and wi is the word vector for i th word of the sentences and n is the number of words in that sentence.
We represent each tweet in train and test corpus by this approach. For each tweet in training, validation and test corpus we have a 300-dimensional text representation. These vectors considered as feature vectors which are required to classify the tweets. 5
In this section, at rst, dataset used in the Coset Shared task will be described. Then evaluation metrics for tweet classi cation are de ned. 5.1
In COSET Shared task, provided dataset consist of 5 issues: political issues, related to the most abstract electoral confrontation; policy issues, about sectoral policies; personal issues, on the life and activities of the candidates; campaign issues, related with the evolution of the campaign; and other issues. The tweets are written in Spanish and they talk about the 2015 Spanish General Election. The training set consists 2242 tweets. The development contains 250 tweets to help participants to train their model and test it on 624 test tweets3. 5.2
The metric used for evaluating the participating systems was the F1 macro measure. This metric considers the precision and the recall of the systems predictions, combining them using the harmonic mean. Provided that the classes are not balanced, the Coset committee proposed using the macro-averaging method for preventing systems biased towards the most populated classes. (1) (2) 5.3
The results of applying the proposed method over training Spanish Tweet corpus
is presented in table 1, The nal results as well as rankings are reported in [
Vectorized tweets are trained and tested via different learning algorithms such as random forest, support vector machine with linear kernel, naive bayse, and logistic regression. Table 1 shows the achieved results of different learning algorithms on the Spanish corpus.
Furthermore, since neural network structures usually grasp a better intuition of deeply extracted set of features from the datasets, vectorized sentences are feed into a multi layer perception neural network with three hidden layers inside. table 2 shows results of the neural network and its characteristic over the given dataset.
Based on the achieved experimental results of studied algorithms, it is clear that neural network structure is a splendid choice to deal with vectorized sentences for performing Twitter topic classi cation task. In this regard and with numbers of trials, a multi layer perceptron neural network with 3 hidden layers each containing 500, 240, and 100 neurons is selected for the Coset shared task. 6
In this paper, we proposed a method for topic identi cation as a typical case of
text categorization on Spanish tweets by using the deep representation of words.
Results from experimental evaluations showed the feasibility of our approach.
With no hand engineering feature and simple composition function, we achieved
55(%) based on F 1macro score. The best method reported 64(%) of F 1macro [
In order to improve the proposed method, we can consider the following ideas: In this work, we used words which are included in Spanish word vectors, while it is clear that incorporating stemming would help us to nd all words in the tweet on that list. We can also provide methods to deal with out of vocabulary words. In addition, considering the unknown topic in the proposed method is another issue in this area.
Furthermore, given its admirable runtime, our proposed method is scalable and is applicable for a large number of documents and can be used for practical purpose. As a future work, we are going to apply other composition functions and also try word-by-word vector comparison in order to eliminate drawbacks of the current method.
The authors would like to thank the reviewers for providing helpful comments and recommendations which improve the paper signi cantly.