The application of conventional autoencoders for textual data often leads to learning trivial and redundant representations due to the high dimensional nature of the text, sparsity, and following power-law word distribution. In order to address these challenges, we introduce a new autoencoder, termed CSCAT (Coherence-based Second Chance Autoencoder for T ext), which uses competitive learning to select k winning neurons in the bottleneck layer that becomes specialized in recognizing specific patterns-leading to learning semantically significant representations of the text. learning based on a measure of consistency to eliminate incoherent features. Our experiments demonstrate that CSCAT achieves outstanding performance on several tasks, including classification, topic modeling, and document visualization Woodstock'21: Symposium on the irreproducible science, June 07-11, [11]. Adding to the aforementioned challenges, autoen- order to better understand textual data and learn more se-
including K-sparse [19] and KATE (K-competitive Autoencoder for TExt) [11] depicted in Figure 1.
The primary principle behind -competitive autoencoders is to select the top ”winner” neurons that conquer
Deep neural networks [1] have revolutionized many domains, especially unstructured data, including computer vision [2], speech recognition [3], and text classification [4] to name a few. While most current neural network applications use supervised learning, unsupervised learning has also presented significant advances in extracting patterns in unlabeled data with reasonable eficiency. For example, unsupervised models have been used to aid information retrieval [5], discover patterns in medical datasets [6, 7], and video prediction [8].
One of the most popular unsupervised deep learning algorithms are autoencoders [9, 10]. An autoencoder is a neural network that learns data representations by reconstructing the input data at the output layer (i.e., () = () ), where () is the network’s output (prediction) for the () input sample. Thus, the main objective for autoencoders is to learn the important features of the input data by constraining the size of the middle layer named bottleneck, often by reducing its dimension less than the input layer.
While autoencoders have demonstrated significant results in several domains, most notably visual applications such as image compression [12] and denoising images [13]; it has been challenging to use autoencoders for textual data due to the text high-dimensionality and sparsity nEvelop-O 2. RELATED WORK the activation values (a.k.a. power ) from the loser neurons. By incorporating competition among the neurons of the hidden layers, these methods aim to specialize the winner neurons in learning meaningful representations of the text. The top winners are chosen based on some competition criteria.
For instance, K-sparse focuses on maintaining sparsity by preserving the highest activations during training and the highest activations during testing, where is a hyperparameter. Similarly, KATE selects winners made up of the ⌈/2⌉ largest positive activations and the ⌊/2⌋ largest absolute negative activations. Those winner neurons then acquire the total energy of the loser neurons, which become inactive, i.e., set to zero. is a hyperparameter representing the desired number of neurons to compete, and it is strongly related to the number of topics.
K-sparse is vulnerable to the ”dead hidden neurons” problem caused by adding too much sparsity (low values), and therefore some neurons can never be updated in the back-propagation process. While this issue can be addressed by incorporating a sparsity scheduling technique, this solution adds significant overhead during the learning process. In contrast, KATE was built on top of K-sparse and solved the dead hidden neurons problem.
However, its competition considers the largest positive and negative activations (the weakest neurons are loser neurons) only–leading to ignoring some essential knowledge preserved in low signal neurons that are never selected as winners. Indeed, our research proves that some of the neurons that maintain weak signals during early training cycles might hold important information on representative features.
To this end, we present CSCAT (Coherence-based SCAT), a novel autoencoder that builds on earlier work in k-competitive learning called SCAT [20]. CSCAT achieves two main innovations over the previous kcompetitive learning methods. First, it provides a second chance for the weakest neurons to reveal their potential, i.e., important topics that would otherwise be ignored.
Second, a coherence-based filtration technique that removes non-coherent neurons from the competition process. Our extensive evaluation demonstrates that these two innovations can lead to better results compared to the prior work in this domain. To summarize, our work contributes the following: For topic modeling of document collections, Latent Dirichlet Allocation (LDA) has gained prominence. By constructing a probability distribution across words, the model seeks to reveal the hidden structure of documents as a combination of topics. Non-parametric learning [23], sparsity [24, 25] and eficient inference [ 26]are only a few of the LDA versions that have been developed. The fundamental flaw in the LDA is that the order of words was not taken into account because of the underlying use of ”bag of words” [27]. To solve this issue, the Topic Keyword Model (TKM) was created, which takes into account the position word in a context [28]. TKM fully utilized the critical idea of a joint probability × from the aspect model [29] to highlight certain aspects of the topics in the documents. TKM conceives the main ideas of the aspect model, but in text documentations, the position of a word was also taken into consideration. A word’s context was taken into account. This means that if a word appears repeatedly in the same document but with diferent neighboring words, each occurrence may have a diferent probability. In [ 30] , a new version of LDA called ProdLDA was released. This topic model substitutes the mixture model used in LDA with a product of expert distribution across particular words. In terms of topic coherence score and qualitative assessment, ProdLDA creates better topics than regular LDA. When the model was tested based on accuracy, however, the results were not similar, as shown in table 3.
Even with ideal reconstructions, autoencoders often only extract simple representations of text data; however, by adding proper regularization to the models, more meaningful representations can be generated. Many autoencoder versions have lately been proposed based on this premise [19, 31, 32]. K-competitive autoencoders, such as KATE, are recent autoencoders that perform well on text classification tasks. KATE (K-competitive Autoen• A novel idea of a coherence-based criterion for coder for TExt) builds on k-sparse for learning meaningifltering neurons that are eligible to enter the ful representations by introducing competition among learning competition produced by the SCAT hidden layer neurons. KATE’s approach is to select winlayer. This process prevents neurons from a ner neurons composed of /2 largest positive activalow-coherence score to more than /2 other neu- tions and /2 largest absolute negative activations, rons entering the competition. We hypothesize which then gain the energy of loser neurons. that eliminating not coherent features during the Overall, CSCAT’s technique is fairly similar to that training phase will result in better topic represen- of traditional k-competitive autoencoders. However, we choose the winners from among the strongest and weakest positive and negative neurons, guaranteeing more equal competition and giving the weakest negative and positive neurons a second chance. Second, before starting the competitive process, we ofer a filtration mechanism that filters out incoherent neurons. This guarantees that the winning neurons are distinctive and coherent.
Unsupervised learning has seen a lot of success with generative models for learning from unlabeled data. Deep Belief Networks (DBN) are a type of deep generative model in which the input data is reconstructed using a deep autoencoder based on the top two layers of a direct acyclic graph [33]. Maaloe et al. [16] introduced a topic modeling approach based on DBN. The neural variational inference (NVI) approach makes the deep generative framework, such as variational autoencoders, suitable for topic modeling [17]. Neural Variational Document Model (NVDM) is a variational autoencoder based neural network for document modeling [17]. One disadvantage of NVDM is that it ignores the correlation between the topics. Liu et al. [22] presented the Neural Variational Correlated Topic Model (NVCTM), a centralized transformation mechanism that reshapes topic distributions to express links between topics. NVCTM consists of two components: the inference network with a centralized transformation flow and a multinomial softmax generative model. NVCTM’s eficiency in capturing perplexity, topic coherence, and document categorization tasks has been proven through rigorous testing. Although this model frequently earns a high coherence score, its classiifcation performance is inferior to that of other similar models.
Autoencoders draw their technical advantage from constraining a bottleneck layer, often by reducing its dimensions, to force the neural network to learn representative features from the data, and then used to reconstruct the data at the output layer. However, latent representation layers usually learn the minimal set of trivial, redundant features required to reconstruct the input data. When it comes to topic modeling, features are frequently chosen based on the most common words based on power-law word distributions, which might hinder the whole process and lead to ignoring important topics linked to less frequent terms. Thus, we propose a competitive learning approach that not only encourages the competition among the most significant activation values but also (1) gives a second chance to the neurons with the weakest activations and (2) inactivates the neurons with the lowest coherence during training phase. Figure 2 illustrates a toy example of the training process in CSCAT.
The competition criterion in our study is based on a unique finding in the Neuroscience area that has already spawned numerous novel deep learning approaches. Mingorance et al. [34] discovered that the kinase JNK (c-Jun N-terminal protein kinase) gives the weaker neurons a second chance before choosing the neurite that best meets the criteria to produce an Axon. Weak neurons will never have a chance to form an Axon unless there is a fair allocation of power. Without this fair redistribution of power, weak neurons will never receive a chance to form an Axon. Using this analogy, we designed our -competitive learning approach to provide the weakest activations a second chance and then selecting the neurons that activate after energy redistribution. Otherwise, neurons with low power will never make into the autoencoder’s latent features.
Our experiments reflect the findings of [ 35] from the Neuroscience domain into the deep learning domain and prove the correctness of our initial hypothesis–that some essential features might be buried in neurons with low activation values that never receive a chance to appear in the fully-trained network due to initialization randomness or initial low frequency of important words. Based on this idea, we suggested SCAT in a prior work and then extended it with unique coherence-based filtering mechanism in the CSCAT, which we present in this paper. We explain the approach of CSCAT in the following.
We define CSCAT as a neural network accepting an input vector ∈ ℝ with -dimensions, and ∈ ℝ × is the weight matrix, and ℎ1, ℎ2, …, ℎ are the hidden layers, and ∈̂ ℝ is the output vector. The activation values at the hidden layers are calculated as = ( + ) , where represents the activation function and is the bias at the encoder side. We use ℎ() = 22 +−11 as the activation function for the hidden neurons and () = 1+1−1 as the activation function for the output neurons. The output neurons are defined as ̂= ( + ) , where is the weight matrix obtained by weight tying– sharing–and is the bias at the decoder side. We use the binary cross-entropy loss function, (, )̂ , as defined in Equation 1, where is the vocabulary of the dataset. (, )̂ = − ∑ ( ̂ ) + (1 − )(1 − ̂ ) ∈ (1)
Given a vocabulary and the number of times, , a word is mentioned, the input vectors, , are calculated as given in Equation 2.
=
log(1 + ) max∈ log(1 + )
Given our model definition, the CSCAT approach goes through the following steps during the training phase at ∈ (2)
One of the major issues in clustering particularly in topic modeling is that the final topic words are not coherent. In fact, the association among the top words per topic could be a good indication of the highly correlated words. In training phase, we want to ensure that the words learned by the model are logically consistent per topic.
Point-wise mutual information [36] is one measure of the statistical independence of observing two words in close proximity. Given a learned , the practice to extract top-N most probable words for each topic is to take the most positive entries in each column. We define the topic coherence metric NPMI [37] in Equation 3 as follows: (
−1 ) = ∑ ∑ log (
, ) ( )( ) =2 =1 − log ( , ) where and
are the topic word and in the sets of ifltered topics. is the list of top-N words for a topic. For a model generating topics, the overall npmi score is an average over all topics. However, since we incorporate the coherence score into the training phase, we consider the coherence of each topic separately. Thus, top-N word of each topic will get a coherence score. This score will be compared with the mean of scores, we refer to it as , and the topics that have coherence score less than the mean will get inactivated during training phase. This process helps in eliminating topics that may not lead to a coherent topic. (3) (4) = ( ∑ =1 ) where is number of topics. Thus, we compare each ( having coherence greater or equal the average of coher ) with and select those that meet our condition; ence across all topics.
Algorithm 1: Approach of Training Phase procedure Training Phase: for e in epochs do = ℎ( + ) = = =̂ =̂ ( = (, )̂ _ ( , _ () _ (, ) _ (, )
+̂ ) function cscat_layer( ): for each
in do ← { | ( ) > function scat_layer( , ): = get_strongest_positive(/4, ) = get_strongest_negative(/4, ) = get_weakest_positive(/4, ) = get_weakest_negative(/4, ) ← [ , , , ] function power_aggregation( , ): for each ∈
_ for each ∈ E( E( ) = 0 ) +=
and ∉H do += E(
) do _ , ) }
After filtering the neurons using their coherence scores obtained in the previous step, we select the top strongest and weakest, positive and negative activations per dimension among the eligible vectors. The selected top neurons, referred to as winners, will gain the activation values of the loser neurons.
In particular, we select /2 top strongest activation values (positive and negative) and /2 weakest activation values (positive and negative) neurons. Selecting the neurons with the weakest activations in our approach plays a critical role in identifying features that otherwise are buried in weak signals. This is mainly supported by the fact that weak activation values might be caused by, especially in early training epochs: (1) initialization randomness and (2) representing rare (less frequent) words with small values in the vector space. To ensure that weakest activations have a real potential to become representative features, we track their behavior over training cycles and only keep the neurons that illustrate improvement over time. Otherwise, they are removed from the competition process. For example, lets assume that = { 1 in the previous iteration. We will re-evaluate the values , 2, … , } is the set of weakest selected neurons of these activations, after being considered winners and aggregated new power, to only keep those that grow in power during subsequent iterations, as follows: | | ≤ | +1 | ∈ (5)
After winner neurons are selected, they add the total activation values from all loser neurons to their current activation value (we refer to this process as neuron power aggregation). Loser neurons are then inactivated (i.e., assigned the value 0). Algorithm 1, _ function defines this step; where it first calculates the total energy of the loser neurons, assign them to 0, and ifnally adds this total energy to the winner neurons. Note that the base case scenarios are not included in the algorithm for simplicity.
We evaluate the performance of CSCAT on several tasks compared to the current state-of-the-art models. First, we briefly discuss the used datasets and the relevant baseline models. All experiments were performed using Nvidia Titan RTX GPU with 64G RAM. We implemented our models using Keras version 2.2.4 [38] with TensorFlow 1.13 backend [39]. We used an internal model management tool, called ModelKB, to keep track of our experiments [40, 41]. We used three datasets in our experiments: 20 Newsgroups [42], Reuters [43], and Wiki10+ [44]. The details about the datasets are listed in Table 1.
The results of our CSCAT model are compared to the following models: 1. LDA [21]: a probabilistic topic model that uses the bag-of-words technique to model a topic and a mixture of topics to model a document. 2. K-sparse [19]: an autoencoder that enforces sparsity in the hidden layers by keeping the highest activities in the training phase and the highest activities in the testing phase. k-sparse uses linear activation functions, while the non-linearity in the model derives from the selection of highest activities. 3. NVCTM [22]: a novel model that proposes the idea of centralized transformation flow to capture the correlations among topics by reshaping topic distributions. The implementation of this model is not available, so we compared our results to the results reported in their original paper. 4. KATE [11]: a shallow autoencoder model with a competitive hidden layer selects the k largest positive neurons and largest absolute negative neurons. Moreover, KATE uses an additional hyperparameter to amplify the energy value. 5. ProdLDA [30]: a new topic model that replaces the mixture model in LDA with a product of expert. the coherence comparison among topic vectors) for all the experiments is set to 15. Changing from 10 to 50 had little diferences in the model’s performance, so we kept = 15 , which achieved best results. Also, note that we did not run the experiment on the NVCTM model, rather we obtained these results from its research paper.
It is obvious from the table that competition-based autoencoders achieve better results than conventional models, such as LDA. For example, KATE achieves 70% for all three measurements outperforming NVCTM, Ksparse, and LDA, but CSCAT outperform all listed models achieving 0.71 scores on all three measures.
This task included training a simple softmax multi-class classifier with a cross-entropy loss function on the 20 Newsgroups dataset. The classification precision, recall, and F1 scores are listed under the 20 Newsgroups column in Table 3. We set the number of topics to 50, and (number of highest positive activations to consider for 4.2. Quantitative Analysis we implemented a multi-label logistic regression classifier with a cross-entropy loss function to evaluate the models In this section, we analyze the performance of CSCAT on the multi-label classification task using Wiki10+ and compared to the models mentioned above on two tasks: Reuters datasets. The precision, recall, and F1 scores of multi-class classification using the dataset of 20 News- these experiments are also listed in Table 3. Note that due groups and multi-label classification using the Wiki10+ to the missing implementation of the NVCTM model, we and Reuters datasets. The results of both tasks are re- could not reproduce the results reported in the original ported in Table 3. We also compare and report the topic paper. coherence scores of these models. We observe from the table that there are some inconsistencies among the results of this task. We believe 4.2.1. Multi-class classification that this is because those two datasets, i.e., Wiki10+ and Reuters, are highly-imbalanced. Thus, there exist some diferences among the precision on the one hand and the recall on the other hand. KATE wins the precision accuracy in the Wiki10+ task while CSCAT wins both the recall and F1 scores. We also observe that CSCAT significantly outperform the rest of the models.
We used a topic coherence measurement that is known to have a human-level judgment, called Normalized Point- ( ) = (6) wise Mutual Information (NPMI) [45]. We evaluated NPMI across all the models using the 20 Newsgroups. We extracted the top-10 words per topic and then computed 4.3. Qualitative Analysis the NPMI scores as illustrated in Equation (6), using topic In this section, we illustrate that our models can learn numbers, = 25, 50 . semantically meaningful representations from textual
The results of the NPMI are illustrated in Table 4. We data. We compare our results to the baseline models, notice that CSCAT achieves scores of 0.151 for 25 top- including LDA, K-sparse, and KATE and ProdLDA using ics and 0.118 for 50 topics compared to NVCTM, which the 20 Newsgroups dataset, with the number of topics scores of 0.180 and 0.176 for 25 and 50 topics, consecu- set equal to 25. The results are listed in figure 3, 4 and 5 tively. However, this higher coherence score in NVCTM for religion, politics and sports. comes with a lower classification accuracy compared to We can observe from the figures that our CSCAT model both SCAT and CSCAT, as explained in the previous sub- generates the most semantically meaningful topics. Here, section in addition to lower performance at the document we show three topics. The top 10 words learned from visualization task, as we explain in the following subsec- the Religion category: “resurrection”, “doctrine”, “scription. Overall, CSCAT achieves the second-best coherence ture”, “testament”, “holy”, “jesus”, “spirit”, “christ” and score results among the results of the models. “faith” are strongly related to Religion. CSCAT also learns meaningful representation for the Sport category, including words like “players”, “baseball”, “playofs”, “leafs”, −1 ∑ ∑ =2 =1 − log ( , ) log ( , ) ( )( ) “scoring”, “league”, and “scored” and under Politics topic electronic health records, BMC medical informatics words like “congress”, “senate”, “clinton”, “president”, and decision making 19 (2019) 149. “secretary”, “administration” which represent the most [7] S. Goudarzvand, J. S. Sauver, M. M. Mielke, P. Y. meaningful representations among the rest of the words Takahashi, S. Sohn, Analyzing early signals of older generated by other models. adult cognitive impairment in electronic health records, in: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, 5. Conclusions 2018, pp. 1636–1640. [8] M. Hosseini, A. S. Maida, M. Hosseini, G. Raju, InCSCAT is a new autoencoder for textual data based on the ception lstm for next-frame video prediction (stuconcept of competitive learning, in which only neurons dent abstract), in: Proceedings of the AAAI Conof the bottleneck layer engage in the learning process ference on Artificial Intelligence, volume 34, 2020, while the rest are inactivated. Those winning neurons be- pp. 13809–13810. come highly specialized in learning specific properties as [9] Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, a result of the competition. Unlike prior techniques that Greedy layer-wise training of deep networks, in: introduced competition between the strongest positive Advances in neural information processing systems, and negative neurons, our method removes extremely 2007, pp. 153–160. incoherent neurons first and then adds a competition for [10] I. Goodfellow, Y. Bengio, A. Courville, Deep learnthe highest and lowest positive and negative neurons in ing, MIT press, 2016. the autoencoder’s bottleneck layer. [11] Y. Chen, M. J. Zaki, Kate: K-competitive autoen
Our thorough experiments showed that our method coder for text, in: Proceedings of the 23rd ACM delivers very close or higher performance on a variety SIGKDD International Conference on Knowledge of textual data applications, such as classification and Discovery and Data Mining, ACM, 2017, pp. 85–94. topic modeling. Furthermore, compared to the baseline [12] F. Huszar, L. Theis, W. Shi, A. Cunningham, Lossy models we examined in this paper, our model returns image compression with compressive autoencoders more semantically meaningful topics. Our approach can (2020). also be used to reduce the dimensionality of textual data. [13] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, Stacked denoising autoencoders: LearnReferences ing useful representations in a deep network with a local denoising criterion, Journal of machine learn[1] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, ing research 11 (2010) 3371–3408.
nature 521 (2015) 436. [14] S. Zhai, Z. M. Zhang, Semisupervised autoencoder [2] N. O’Mahony, S. Campbell, A. Carvalho, S. Hara- for sentiment analysis, in: Thirtieth AAAI Conferpanahalli, G. V. Hernandez, L. Krpalkova, D. Rior- ence on Artificial Intelligence, 2016. dan, J. Walsh, Deep learning vs. traditional com- [15] H. Larochelle, S. Lauly, A neural autoregressive puter vision, in: Science and Information Confer- topic model, in: Advances in Neural Information ence, Springer, 2019, pp. 128–144. Processing Systems, 2012, pp. 2708–2716. [3] Z. Zhang, J. Geiger, J. . Pohjalainen, A. E.-D. Mousa, [16] L. Maaloe, M. Arngren, O. Winther, Deep belief nets W. Jin, B. Schuller, Deep learning for environmen- for topic modeling, arXiv preprint arXiv:1501.04325 tally robust speech recognition: An overview of (2015). recent developments, ACM Transactions on Intelli- [17] Y. Miao, L. Yu, P. Blunsom, Neural variational ingent Systems and Technology (TIST) 9 (2018) 1–28. ference for text processing, in: International con[4] J. Liu, W.-C. Chang, Y. Wu, Y. Yang, Deep learn- ference on machine learning, 2016, pp. 1727–1736. ing for extreme multi-label text classification, in: [18] C. Zhang, J. Butepage, H. Kjellstrom, S. Mandt, AdProceedings of the 40th International ACM SIGIR vances in variational inference, IEEE transactions Conference on Research and Development in Infor- on pattern analysis and machine intelligence (2018). mation Retrieval, 2017, pp. 115–124. [19] A. Makhzani, B. Frey, K-sparse autoencoders, arXiv [5] X. Wei, W. B. Croft, Lda-based document models for preprint arXiv:1312.5663 (2013). ad-hoc retrieval, in: Proceedings of the 29th annual [20] S. Goudarzvand, G. Gharibi, Y. Lee, Scat: Second international ACM SIGIR conference on Research chance autoencoder for textual data, arXiv preprint and development in information retrieval, 2006, pp. arXiv:2005.06632 (2020).
178–185. [21] D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet [6] S. Goudarzvand, J. S. Sauver, M. M. Mielke, P. Y. allocation, Journal of machine Learning research 3 Takahashi, Y. Lee, S. Sohn, Early temporal charac- (2003) 993–1022. teristics of elderly patient cognitive impairment in [22] L. Liu, H. Huang, Y. Gao, Y. Zhang, X. Wei, Neural variational correlated topic modeling, in: The S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, World Wide Web Conference, ACM, 2019, pp. B. Steiner, I. Sutskever, K. Talwar, P. Tucker, 1142–1152. V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, [23] D. M. Blei, T. L. Grifiths, M. I. Jordan, The nested P. Warden, M. Wattenberg, M. Wicke, Y. Yu, chinese restaurant process and bayesian nonpara- X. Zheng, TensorFlow: Large-scale machine learnmetric inference of topic hierarchies, Journal of the ing on heterogeneous systems, 2015. URL: https: ACM (JACM) 57 (2010) 7. //www.tensorflow.org/, software available from [24] J. Zhu, E. P. Xing, Sparse topical coding, arXiv tensorflow.org.
preprint arXiv:1202.3778 (2012). [40] G. Gharibi, V. Walunj, R. Alanazi, S. Rella, Y. Lee, [25] J. Eisenstein, A. Ahmed, E. P. Xing, Sparse additive Automated management of deep learning experigenerative models of text (2011). ments, in: Proceedings of the 3rd International [26] K. Canini, L. Shi, T. Grifiths, Online inference of Workshop on Data Management for End-to-End topics with latent dirichlet allocation, in: Artificial Machine Learning, 2019, pp. 1–4.
Intelligence and Statistics, 2009, pp. 65–72. [41] G. Gharibi, V. Walunj, R. Nekadi, R. Marri, Y. Lee, [27] M. Bahrani, H. Sameti, A new bigram-plsa language Automated end-to-end management of the modelmodel for speech recognition, EURASIP Journal on ing lifecycle in deep learning, Empirical Software Advances in Signal Processing 2010 (2010) 308437. Engineering 26 (2021) 1–33. [28] J. Schneider, M. Vlachos, Topic modeling based on [42] K. Lang, Newsweeder: Learning to filter netnews, keywords and context, in: Proceedings of the 2018 in: Machine Learning Proceedings 1995, Elsevier, SIAM International Conference on Data Mining, 1995, pp. 331–339.
SIAM, 2018, pp. 369–377. [43] D. D. Lewis, Y. Yang, T. G. Rose, F. Li, Rcv1: A [29] T. Hofmann, Unsupervised learning by probabilistic new benchmark collection for text categorization latent semantic analysis, Machine learning 42 (2001) research, Journal of machine learning research 5 177–196. (2004) 361–397. [30] A. Srivastava, C. Sutton, Autoencoding varia- [44] A. Zubiaga, Enhancing navigation on wikipedia tional inference for topic models, arXiv preprint with social tags, arXiv preprint arXiv:1202.5469 arXiv:1703.01488 (2017). (2012). [31] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, [45] J. H. Lau, D. Newman, T. Baldwin, Machine reading B. Frey, Adversarial autoencoders, arXiv preprint tea leaves: Automatically evaluating topic coherarXiv:1511.05644 (2015). ence and topic model quality, in: Proceedings of [32] A. Makhzani, B. J. Frey, Winner-take-all autoen- the 14th Conference of the European Chapter of the coders, Advances in neural information processing Association for Computational Linguistics, 2014, pp. systems 28 (2015) 2791–2799. 530–539. [33] Y. Bengio, Learning deep architectures for AI, Now
Publishers Inc, 2009. [34] A. Mingorance-Le Meur, Jnk gives axons a second chance, Journal of Neuroscience 26 (2006) 12104–12105. [35] H. Jiang, Y. Rao, Axon formation: fate versus
growth, Nature neuroscience 8 (2005) 544–546. [36] G. Bouma, Normalized (pointwise) mutual information in collocation extraction, Proceedings of GSCL 30 (2009) 31–40. [37] N. Aletras, M. Stevenson, Evaluating topic coherence using distributional semantics, in: Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013)–Long Papers, 2013, pp. 13–22. [38] F. Chollet, et al., Keras, 2015. URL: https://
github.com/fchollet/keras. [39] M. Abadi, A. Agarwal, P. Barham, E. Brevdo,
Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga,