=Paper=
{{Paper
|id=Vol-2328/session4_paper7
|storemode=property
|title=CL-Aff Deep semisupervised clustering
|pdfUrl=https://ceur-ws.org/Vol-2328/4_5_paper_30.pdf
|volume=Vol-2328
|authors=Johnny Torres,Carmen Vaca
|dblpUrl=https://dblp.org/rec/conf/aaai/TorresV19
}}
==CL-Aff Deep semisupervised clustering==
https://ceur-ws.org/Vol-2328/4_5_paper_30.pdf
Neural semi-supervised learning
for multi-labeled short-texts
Johnny Torres and Carmen Vaca
ESPOL Polytechnic University
Department of Electrical and Computer Engineering (FIEC)
jomatorr,cvaca@espol.edu.ec
Abstract. The massive data generated by users in online platforms, such as so-
cial networks, create challenges for text classification tasks based on supervised
learning. Supervised learning often requires a lot of feature engineering or a sig-
nificant amount of annotated data to achieve good results. However, the scarcity
of annotated data is a critical issue, and manual annotation can be both costly and
time-consuming. Semi-supervised learning requires far less annotated data and
achieve similar performance as supervised approaches. In this paper, we intro-
duce a semi-supervised neural architecture for muti-label settings, that combines
deep learning representation and k-means clustering. The results show that the
semi-supervised approach can leverage large-scale unlabeled data and achieve
better results compared to baseline unsupervised as well as supervised methods.
1 Introduction
The classification or grouping of short texts is critical in various tasks in text mining
and the retrieval of information in the context of social networks or data generated by
users on the web. Specifically, these tasks aim to categorize or group similar texts, so
that texts with the same label or group are similar to each other and different from texts
in other categories or groups. Traditional classification or grouping models often use a
sparse representation for text data, such as the bag of words (BOW) or TF-IDF [12].
However, the characteristics of the short texts create some problems for both con-
ventional unsupervised and supervised models. Usually, the number of unique words in
each short text is small (90% of the texts instances in the HappyDB dataset have less
than 23 words), and as a result, the problem of lexical shortage generally leads to poor
grouping quality [8].
An alternative to address lexical shortages is to enrich text representations by ex-
tracting characteristics and relationships with sources such as Wikipedia [4] or ontolo-
gies [9]; however, this approach requires written knowledge, which also depends on
the language. Other alternative is to code texts in distributed dense vectors [14] with
neural networks [18].
Another problem is the definition of the labels for a specific task and the number of
manually annotated instances for each label. Unsupervised methods learn the categories
from the data, but the resulting groupings may not be related to the expected labels.
Supervised methods have predefined labels but often require a considerable number of
labeled instances to learn to categorize. Semi-supervised approaches offer an alternative
2 Torres et al.
to solve these problems by using a small amount of labeled data according to predefined
classes, at the same time take advantage of the massive unlabelled data availability [3].
This paper investigates the research question: How can a semi-supervised approach
learn to categorize short texts in a multi-label taxonomy using a small set of labeled
data and leveraging the availability of large amounts of unlabeled data? To that end,
we build upon neural semi-supervised k-means clustering that modifies the conven-
tional objective function and adds a penalty term for labeled data [17]. We extended
the neural semi-supervised clustering and applied to multi-label settings. The results
show that semi-supervised k-means outperform other baseline unsupervised models for
multi-label classification tasks.
The rest of the paper is structured as follows: a) we review related work and the
k-means clustering, b) we describe the neural semi-supervised clustering for multi-label
setting, c) we analysis the experimental results, and d) finally, we outline the conclu-
sions and future work.
2 Related Work
Previous work on semi-supervised clustering methods analyzes methods based on: con-
straints and representation. The constraint-based approaches use a small percentage of
labeled data to restrict the clustering process [7]. Instead, the representation-based
methods first learn a data representation model that satisfies the labeled data, and then
use it to group both labeled and unlabeled data [3].
The hybrid approaches try to integrate both methods in a unified framework [5];
However, the use of linear projection for learning by representation has limitations to
achieve good performance. Recent methods use deep neural architectures to learn text
representations that overcome the limitation of linear models [18]. However, the sepa-
ration of the learning process of the data representation model and the clustering model
restrict the benefits and is more similar to the techniques representation-based. In this
work, the proposed model builds on an approach that combines into an integrated frame-
work both the representation of deep learning and the clustering method [17].
3 Unsupervised Learning
In unsupervised learning, k-means is an algorithm for clustering data used in many ap-
plications, including text mining tasks [6]. The k-means algorithm divides the data into
a K number of clusters, so that minimizes the distance of each point to the centroids of
the clusters, assigning it to the nearest cluster. The input to the clustering model are the
set of short texts {s1 , s2 , s3 , ..., sN } represented by the data points {x1 , x2 , x3 , ..., xN },
where xi is a sparse or dense vector.
The k-means algorithm defines a set of binary variables rnk ∈ {0, 1} for each data
point xn , where k ∈ {1, ..., K} specifies the cluster assigned. For example, rnk = 1 if
xn is assigned to cluster k, and rnj = 0 for j 6= k. The objective function in k-means is
defined as:
Neural semi-supervised learning for multi-labeled short-texts 3
N X
X K
Junsup = rnk kxn − µk k2 (1)
n=1 k=1
where µk is the centroid of the k-th cluster. The k-means algorithm learns the values
of {rnk } and {µk } such that optimizes Junsup . To minimize the objective function, k-
means utilizes the gradient descent approach with an iterative procedure [16].
Each iteration involves two steps: assign clusters and estimate centroids. In the
assign clusters step, k-means minimizes Junsup with respect to {rnk } by keeping fixed
{µk }. In this case, Junsup is a linear function for {rnk }, so that we optimize each data
point separately by merely assigning the n-th data point to the closest cluster centroid.
In the estimate centroids step, k-means minimizes Junsup with respect to {µk } by
keeping {rnk } fixed. In this case, Junsup is quadratic function of {µk }, and we mini-
mize it by setting to zero the derivative for {µk }, as follows:
N
∂Junsup X
=2 rnk (xn − µk ) = 0 (2)
∂µk n=1
Then, we can solve {µk } as
PN
n=1 rnk xn
µk = PN
(3)
n=1 rnk
Thus, µk corresponds to the mean of all the data points assigned to the cluster k.
4 Neural Semi-supervised Clustering
The classical k-means algorithm uses unlabeled data to solve the clustering problem
based on an unsupervised learning approach; however, the clustering results may not be
consistent with the expected labels. We extend the semi-supervised approach in [17],
which injects some supervised information into the learning process to produce useful
and coherent clusters. Similar to the classic k-means algorithm, the training steps for
the neural semi-supervised k-means are:
1. Initialize {µk } and f (·).
2. Repeat until convergence:
(a) assign clusters: Assigns each short-text to its nearest cluster centroid based on
its neural representation.
(b) estimate centroids: Estimates the clusters’ centroid based on the cluster assign-
ments from previous step.
(c) update parameters: Updates the neural networks parameters according to the
objective function by keeping fixed the centroids and cluster assignments.
4 Torres et al.
4.1 Representation Learning
We represent each short text entry si as a sequence of word indices and together with the
initial centroids form the input data to the semi-supervised neuronal clustering model.
Then, the embedding layer maps each word in the sequence as a dense vector x = f (s),
using word embeddings initialized randomly or from pre-trained embeddings [14, 13].
In this approach, rather than training the text representation model independently, the
semi-supervised clustering integrates it with the k-means algorithm training process.
4.2 Objective Function
The neural semi-supervised clustering uses a small number of labeled instances to guide
the clustering process and minimizes the objective function defined as:
C
X N X
X K
Jsemi = {α rnk kf (sn ) − µk k2
c=1 n=1 k=1
L
(4)
X
+ (1 − α) {kf (sn ) − µgn k2 +
n=1
X
[l + kf (sn ) − µgn k2 − kf (sn ) − µj k2 ]+ }}
j6=gn
where {(s1 , y1 ), (s2 , y2 ), ..., (sL , yL )} denote the labeled data, and the unlabeled
data is {sL+1 , sL+2 , ..., sN }. The label yi specify the cluster for each short-text si . The
outer sum iterates over the number of labels C defined in the taxonomy; thus, extending
the original objective function in [17]. The objective function contains two terms:
1. The first term is the objective function in the classic k-means algorithm (Eq. 1),
and the second term penalizes depending how far are the predicted clusters from
the ground-truth clusters for labeled data. The factor α ∈ [0, 1] is used to tune the
importance of unlabeled data.
2. The second term contains two sub-terms:
(a) The first sub-term penalizes depending on the distance between each labeled
instance and its correct cluster centroid, where gn = G(yn ) indicates the clus-
ter ID given by the label yn . The mapping function G(·) uses the Hungarian
algorithm [15].
(b) The second sub-term specifies a hinge loss function with a margin l, where
[x]+ = max(x, 0). This term incurs in some loss if the distance to the ground
truth centroid is larger (by a margin l) than the distances to the wrong centroids.
4.3 Model Training
The parameters in Jsemi are: the clusters’ assignment for each text {rnk }, the clusters’
centroids {µk }, and the neural network weights f (·). The goal is to find the values of
Neural semi-supervised learning for multi-labeled short-texts 5
{rnk }, {µk }, and parameters in f (·) that minimizes Jsemi . Based on the k-means al-
gorithm, the semi-supervised model iteratively minimizes Jsemi with respect to {rnk },
{µk }, and parameters in f (·).
First, the model initializes the clusters’ centroids {µk } with the k-means method [1],
and also initializes randomly the parameters in the neural network. Then, the model it-
eratively carry out three steps (assigns clusters, estimates the centroids, and updates the
parameters) until Jsemi converges.
The assign clusters step minimizes Jsemi with respect to {rnk } by keeping fixed
f (·) and {µk } to assign a cluster ID for each data point. The second term in Eq. (4) has
no relation with {rnk }. Thus, the model only needs to minimize the first term, by setting
the nearest cluster centroid to each text, i.e., is identical to the assign clusters step in
the k-means algorithm. In this step, the model also calculates the mappings between the
ground-truth clusters specified by {yi } and the cluster assignments for the labeled data.
The estimate centroids step minimizes Jsemi with respect to {µk } by keeping {rnk }
and f (·) fixed, which corresponds to the estimate centroids step in the classic k-means
algorithm. It aims to estimate the cluster centroids {µk } based on the cluster assign-
ments {rnk } from the assign cluster step. In the Eq. 4, the second term considers each
labeled instance in the process of estimating cluster centroids. By solving ∂Jsemi /∂µk =
0, we get
PN PL
n=1 αrnk f (sn ) + n=1 wnk f (sn )
µk = PN PL (5)
n=1 αrnk + n=1 wnk
0 X 00 X 000
wnk = (1 − α)(Ink + Inkj − Inkj )
j6=gn j6=gn
0
Ink = δ(k, gn )
00 0 (6)
Inkj = δ(k, j) · δnj
000 0
Inkj = (1 − δ(k, j)) · δnj
0
δnj = δ(l + kf (sn ) − µgn k2 − kf (sn ) − µj k2 > 0)
where δ(x1 , x2 )=1 if x1 is equal to x2 , otherwise δ(x1 , x2 )=0; and δ(x) = 1 if
x is true, otherwise δ(x) = 0. In the numerator of Eq. 5, the first term represents the
contributions from all data points, and the weight of sn for µk is αrnk . The second term
represents labeled data, and wnk is the weight of an instance sn for µk .
The update parameters step minimizes Jsemi with respect to f (·) by keeping {rnk }
and {µk } fixed, with no counterpart in the k-means algorithm. The primary goal is to
learn the parameters of the text representation model. The training uses Jsemi as the
loss function and employs Adam algorithm to optimize it [11].
5 Experiments
5.1 Experimental Setting
We evaluate the models on the HappyDB dataset [2], comprising individual accounts of
happy moments. The aim is to predict agency and social labels that indicate the context
6 Torres et al.
of happy moments. For training, we use a small subset of labeled data and a large subset
unlabeled dataset [10]. The table 1 summarizes the number of labeled and unlabeled
text instances for training, as well as the number of text instances in the test set. For
the experiments, the splitting strategy is to randomly sample 80% of labeled instances
for training (training set) and remaining 20% instances for validation (validation set).
The unsupervised and semisupervised models use the unlabeled instances for training
(unlabeled set). We train the models using k-fold cross-validation (k = 10) on the
training set and report the results for the validation set using the metric F1 .
Table 1. Statistics for the HappyDB dataset
Dataset Labeled Unlabeled Test Total
HappyDB 10,560 72,324 17,215 100,099
5.2 Model Hyperparameters
Neural architectures introduce several hyper-parameters like the output dimension of
the text representation models, while semi-supervised k-means clustering has α in Eq.
4. This subsection analyzes the impact of some of the hyper-parameters and determines
the configuration for further experiments.
Embeddings dimension To evaluate the effectiveness of the output dimension in text
representation models, we perform experiments with embeddings size of {50, 100, 200,
300, 500, 1000}, while maintaining the all other parameters fixed. Figure 1 show that
the score F1 drops if the size is ≤ 100 and the curve falls if the size is ≥ 500. Based on
the results, we use 300 as the embeddings size.
cnn
lstm
0.7 bilstm
cnn + lstm
F1 Score
0.6
100 200 300 400 500
Embeddings dimension
Fig. 1. Influence of the dimensionality of the text learning representation.
Neural semi-supervised learning for multi-labeled short-texts 7
Alpha We evaluate the effect of α in Eq. 4, which indicates the importance of unlabeled
data in the performance of the model. We test α with values of: {0.00001, 0.0001,
0.001, 0.01, 0.1}, and maintain the other parameters fixed. The figure 2 shows that
the performance decay for small α values. By increasing the value of α, we notice
progressive improvements and reach a peak F1 score with α = 0.1. Further experiments
use the value of α = 0.1 as it maximizes F1 .
cnn
lstm
0.8 bilstm
cnn + lstm
F1 Score
0.7
0.6
10−5 10−4 10−3 10−2 10−1
α
Fig. 2. Influence of unlabeled data, where the x-axis is α in Eq. (4).
Labeled set size This parameter controls the influence of the size of the labeled data.
We evaluate the ratio of labeled data for training between [1%, 10%], and kept the other
parameters fixed. The figure 3 illustrates the performance improvement as the size of
labeled data increases and confirms the importance of labeled data for training.
cnn
lstm
0.8 bilstm
cnn + lstm
F1 Score
0.7
0.6
10−4 10−3 10−2 10−1 100
Fig. 3. Influence of the size of labeled data used for training.
8 Torres et al.
Pre-training This option measures the effect of the pre-training embeddings for neural
architectures. We use pre-trained embeddings in the models as a classification task with
labeled data, and we then use the weights (excluding the top layer) to initialize semi-
supervised clustering. We evaluate several pre-trained embeddings such as Word2Vec,
Glove, FastText. Figure 4 shows that pre-trained embeddings achieve superior perfor-
mance compared to random embeddings; for further experiments we use FastText.
cnn
lstm
0.7 bilstm
cnn + lstm
F1 Score
0.65
0.6
0.55
no word2vec fasttext glove
Fig. 4. Influence using pre-training embeddings in neural models.
5.3 Models Benchmarking
This subsection compares the proposed semi-supervised approach with unsupervised
and supervised models.
Unsupervised learning: All unsupervised models use k-means for clustering. We clus-
ter with k = 2 to map the values for each label (0, 1). For learning representation, we
use the following methods:
– BOW: represents each short-text as sparse vector based on term frequency (TF).
– TF-IDF: similar to BOW, uses a sparse vector to represent each short-text based on
term frequency-inverse document frequency.
– AVG-EMB: uses word embeddings vectors to represent each short-text and then
calculate the average.
Supervised learning: We evaluate several supervised models for the classification
task, the representation learning used depends on each model as described next:
– LR: uses sparse vector representation that feeds a logistic regression classifier.
– FastText: uses dense word vectors representation (embeddings layer), followed by
a Global Average Pooling layer, which averages the word embeddings, and then
uses a Dense layer with sigmoid activation to predict the labels.
Neural semi-supervised learning for multi-labeled short-texts 9
– CNN: uses dense word vectors representation (word embeddings layer) followed
by a Dropout layer, then a convolutional layer, and an output layer with sigmoid
activation.
– LSTM: similar to CNN, but the word embeddings layer feeds a recurrent LSTM
layer, which is more suitable for sequence modeling such as texts.
– BiLSTM: uses two LSTM networks to model the texts sequences in both directions,
followed by a Dropout layer with rate 0.5, and then a dense layer with sigmoid
activation.
– CNN-LSTM: leverage the advantage of CNN layer to capture salient features and
sequence modeling capability of LSTM.
Table 2 summarizes the scores of the models on the test set. The models fall into
three categories (type): unsupervised, supervised, and semi-supervised. The metrics are
precision, recall, and F1 . We report the scores for each label: agency and social. The last
three columns show the total weighted score of the metrics for each model. The results
show that the supervised systems outperform unsupervised models by a large margin,
which highlights the importance of labeled data. Among the supervised learning, deep
neural models perform better than the baseline method (LR); however only with a small
margin. Finally, the semi-supervised model shows promising results, as it achieves score
close to the supervised models.
Table 2. Performance of the models.
Agency Social Total (Weighted)
Type Model Prec. Recall F1 Prec. Recall F1 Prec. Recall F1
BOW 72.77 38.51 50.37 54.46 74.32 62.86 63.61 56.42 56.61
unsupervised TF-IDF 72.70 84.69 78.24 54.02 52.96 53.49 63.36 68.83 65.86
EMB-AVG 73.73 97.79 84.07 53.59 54.16 53.87 63.66 75.97 68.97
LR 81.06 94.30 87.18 93.00 82.30 87.32 87.03 88.30 87.25
FastText 85.74 93.98 89.67 90.99 86.73 88.81 88.37 90.35 89.24
CNN 90.92 88.53 89.71 89.34 90.53 89.93 90.13 89.53 89.82
supervised LSTM 89.19 89.81 89.50 91.57 87.52 89.50 90.38 88.67 89.50
BiLSTM 89.18 90.84 90.00 92.84 84.96 88.72 91.01 87.90 89.36
CNN-LSTM 89.38 88.92 89.15 89.70 87.88 88.78 89.54 88.40 88.96
semi-supervised CNN 89.10 92.18 90.62 89.56 89.56 89.56 89.33 90.87 90.09
6 Conclusions
This work builds on the neural semi-supervised clustering that integrates a neural repre-
sentation learning for short-texts, and the k-means clustering into a unified framework.
To that end, the model utilizes a small percentage of labeled data to guide the intention
for clustering. We extended the model to use it in muti-labeled clustering of short-texts.
The results show that the proposed neural semi-supervised clustering is more effective
than baselines unsupervised, and it close to the supervised models. Therefore, the re-
sults show the potential to overcome critical issues, such as of scarcity of labeled data,
and leverage the availability of massive unlabeled data.
10 Torres et al.
References
1. Arthur, D., Vassilvitskii, S.: k-means++: The advantages of careful seeding. In: Proceedings
of the eighteenth annual ACM-SIAM symposium on Discrete algorithms. pp. 1027–1035.
Society for Industrial and Applied Mathematics (2007)
2. Asai, A., Evensen, S., Golshan, B., Halevy, A., Li, V., Lopatenko, A., Stepanov, D., Suhara,
Y., Tan, W.C., Xu, Y.: Happydb: A corpus of 100,000 crowdsourced happy moments. In:
Proceedings of LREC 2018. European Language Resources Association (ELRA), Miyazaki,
Japan (May 2018)
3. Bair, E.: Semi-supervised clustering methods. Wiley Interdisciplinary Reviews: Computa-
tional Statistics 5(5), 349–361 (2013)
4. Banerjee, S., Ramanathan, K., Gupta, A.: Clustering short texts using wikipedia. In: Proceed-
ings of the 30th annual international ACM SIGIR conference on Research and development
in information retrieval. pp. 787–788. ACM (2007)
5. Bilenko, M., Basu, S., Mooney, R.J.: Integrating constraints and metric learning in semi-
supervised clustering. In: Proceedings of the twenty-first international conference on Ma-
chine learning. p. 11. ACM (2004)
6. Christopher, D.M., Prabhakar, R., Hinrich, S.: Introduction to information retrieval. An In-
troduction To Information Retrieval 151(177), 5 (2008)
7. Davidson, I., Basu, S.: A survey of clustering with instance level. Constraints 1, 2 (2007)
8. Dhillon, I.S., Guan, Y.: Information theoretic clustering of sparse cooccurrence data. In: Data
Mining, 2003. ICDM 2003. Third IEEE International Conference on. pp. 517–520. IEEE
(2003)
9. Fodeh, S., Punch, B., Tan, P.N.: On ontology-driven document clustering using core semantic
features. Knowledge and information systems 28(2), 395–421 (2011)
10. Jaidka, K., Mumick, S., Chhaya, N., Ungar, L.: The CL-Aff Happiness Shared Task: Results
and Key Insights. In: Proceedings of the 2nd Workshop on Affective Content Analysis @
AAAI (AffCon2019). Honolulu, Hawaii (January 2019)
11. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 (2014)
12. Manning, C.D., Raghavan, P., Schütze, H.: Scoring, term weighting and the vector space
model. Introduction to information retrieval 100, 2–4 (2008)
13. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training dis-
tributed word representations. In: Proceedings of the International Conference on Language
Resources and Evaluation (LREC 2018) (2018)
14. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of
words and phrases and their compositionality. In: Advances in neural information processing
systems. pp. 3111–3119 (2013)
15. Munkres, J.: Algorithms for the assignment and transportation problems. Journal of the so-
ciety for industrial and applied mathematics 5(1), 32–38 (1957)
16. Nasrabadi, N.M.: Pattern recognition and machine learning. Journal of electronic imaging
16(4), 049901 (2007)
17. Wang, Z., Mi, H., Ittycheriah, A.: Semi-supervised clustering for short text via deep repre-
sentation learning. arXiv preprint arXiv:1602.06797 (2016)
18. Xu, J., Wang, P., Tian, G., Xu, B., Zhao, J., Wang, F., Hao, H.: Short text clustering via
convolutional neural networks. In: VS@ HLT-NAACL. pp. 62–69 (2015)