Being able to extract targeted topics from text can be a useful tool for understanding the large amount of textual data that exists in various domains. Many methods have surfaced for building frameworks that can successfully extract this topic data. However, it is often the case that a large number of training samples must be labeled properly, which requires both time and domain knowledge. This paper introduces new alignment-based methods for predicting topics within textual data that minimizes the dependence upon a large, properly-labeled training set. Leveraging Word2Vec word embeddings trained using unlabeled data in a semi-supervised approach, we are able to reduce the amount of labeled data necessary during the text annotation process. This allows for higher prediction levels to be attained in a more time-efficient manner with a smaller sample size. Our method is evaluated on both a publicly available Twitter sentiment classification dataset and on a real estate text classification dataset with 30 topics.
Finding specific topics within textual data is an important task for many
domains. A multitude of approaches for achieving this task have appeared in recent
years [
In all of these cases, once an appropriate number of training samples have
been acquired, many different machine learning algorithms have successfully
been used to automatically identify topics in new examples [
This paper describes a novel method and application for predicting topics at the sentence level in order to reach a high level of accuracy with a limited number of training samples. We evaluate our algorithm on its ability to predict over 30 different sentence-level topic labels within real estate data. We also provide an evaluation of our algorithm on a standard and publicly available twitter sentiment-based prediction dataset. In the real-estate domain, expert knowledge is required for the annotation of the important topics of interest. The topics were not available or known a priori, further limiting the number of possible expert annotators. We show that our algorithm achieved a higher prediction accuracy with a very small number of examples, resulting in a significant improvement over standard methods for topic identification with small sample sizes. Finally, our method maintains its ability to predict well in small sample sizes, even in the absence of negative training examples which would further reduce the burden on the domain expert. The rest of this paper is structured as follows: in Section 2 we present a review of related work; in Section 3 we provide descriptions of relevant machine learning algorithms and introduce our architecture and novel methods; and we conclude with Section 6 where we present our experimental results and discussion. 2
Kim et al. proposes a method for categorizing text at the sentence and document
level using word vector distances [
Topic and feature extraction is popular in the world of data mining, and there
has been a lot of work for making this process more efficient and more accurate.
[
Nguyen et al. introduces an algorithm that combines preprocessing, pattern
recognition, iterative model development, and active learning to annotate and
classify features found within clinical text records [
Wang et al. offers a new approach to modeling targeted subtopics within
text. Instead of extracting all larger topics within a corpus, they search for more
specific subtopics using a targeted topic model (TTM) [
Finally, several attempts have been made to create architectures that can be
used for new domains. [
We will now present brief descriptions of several widely applied methods used for
target identification. This section is broken up into two additional subsections:
feature extraction and machine learning methods. All of the methods described
in this work rely on one of two word embedding methods: bag-of-words (BOW)
or Word2Vec [
Feature Extraction In its simplest form, BOW is an orderless representation
of word frequencies in a document. In the context of this sentence-level
target identification problem, the word counts from each sentence are normalized
into a word frequency matrix prior to classification. The Python natural
language toolkit (nltk) and native Python String library [
Word2Vec is an NLP system that utilizes neural networks in order to create a
distributed representation of words in a corpus [
For the purposes of evaluation in this paper, two Word2Vec models were used.
The first was a model trained on the real estate corpus, and the second was a
publicly available Twitter Word2Vec pre-trained model available at http://yuca.
test.iminds.be:8900/fgodin/downloads/word2vec_twitter_model.tar.gz.
Machine Learning Methods Three standard machine learning methods that
are often used in topic identification were selected for comparison: Na¨ıve Bayes,
SVM, Random Forest (RF), and K-Nearest Neighbor (KNN). Standard
implementations of these algorithms are available in scikit-learn. Na¨ıve Bayes was
applied to the BOW features [
Two novel approaches were developed and evaluated in this work that stem from an alignment-based distance metric. The first approach is a standard KNN classifier utilizing the novel alignment-based distance metric in place of traditional BOW distance metrics. To classify a new unknown sample, the alignment-based KNN calculates the distance between the target sample and all labeled data. The k-nearest neighbors are then found and the majority class is returned. The second approach is an alignment-based threshold classifier that only requires positively labeled data and a predefined threshold. This threshold-based classifier measures the distance from a target unknown sample to only the positively annotated samples. If the score is above the threshold, a positive class prediction is returned. The success of both methods is dependent on the alignment-based distance metric described below.
Alignment Distance Metric All alignment-based approaches were
implemented using the NeedlemanWunsch algorithm that has been made famous for
its use in aligning biological sequences [
Standard Distance Metrics In addition to these two novel approaches, we implemented and tested two standard Word2Vec distance metrics. The first metric was the maximum cosine-similarity between pairs of words in annotated and unknown sentences. The second was the average cosine-similarity score between pairs of scores. 4 4.1
The domain we designed our original system for involved property descriptions from real estate data. This data was created by realtors and contains information
Annotated: — master bedroom — ——- downstairs
Unknown: two master bedrooms are located downstairs about various real estate listings. Each listing has a large amount of metadata (e.g., location, images, basic features); however, the main piece of information we use in our system is the description written for each listing. This description is usually no more than a short paragraph and is created by the real estate agent to summarize the listing as a whole. It can include any information that the agent wishes to convey to the potential buyer, such as property features, kitchen appliances, etc. By analyzing a listing’s description, we attempt to extract specific features about the listing itself.
We created a web application using the Angular JavaScript framework. The main interface for this application provides a way to easily and quickly annotate real estate listings. This interface pulls listings that have not yet been annotated from our server, along with the sentence and word data associated with that listing. It also retrieves the topic information so that the listing can be properly annotated.
Once a listing has been retrieved, the annotator can select a sentence from the listing description to annotate. They can then toggle the individual words in the sentence that match a specific topic. If they believe that a pattern should be mapped to a topic that does not currently exist, then they have the option to create it. Once a topic has been defined, it can then be used by future annotators. When an annotator is satisfied with an annotation, they simply click submit and it is stored inside of the annotations database to be used in future alignment predictions.
Along with this annotation interface, we also included other pages within the web application that provide statistical information regarding the database. Some of these pages display simple information, such as the specific progress of different annotators. Other pages give annotators more control over the database itself, allowing them to take actions such as correcting mistakes in previous annotations or modifying topic names. 4.2
Twitter has grown quite large in the past decade, along with the amount of
textual data it has created. This data has proven to be a popular source for
testing and implementing text-based machine learning algorithms [
In order to annotate this dataset, we created a simple interactive widget that runs inside of a Python Jupyter Notebook. This publicly available widget is very similar to the annotation interface we created for the real estate data and can be seen in supplemental Figure 4. We store the tweets in a text file, which the interface uses to select tweets at random. Once a tweet is selected, the annotator can toggle the words in the tweet that they believe match the sentiment that was predicted. For example, if a tweet was labeled as having a positive sentiment, then only words that are relevant to that sentiment will be selected and stored in the annotation. This allows us to store annotations in a very similar format to the ones that were stored for our real estate data. 5
Each classifier was subjected to iterative cross-validation as a function of the training size for each category. This procedure is summarized in Figure 1. The average F1 score across all iterations for each training set size was calculated and plotted as a function of training set size. Two examples are shown in Figure 2. The area under this curve was then calculated to measure the ability of each classifier to perform well for small as well as larger training set sizes. The 95% confidence interval was calculated for the area under the F1 curve as a function of the training set size. 6
We compared our alignment-based algorithms to bag-of-words derived SVM, Na¨ıve Bayes, random forest, and KNN classifiers on 30 real estate categories (a) C. Fenced Backyard (b) D. Eat-in Kitchen and positive/negative sentiment tweet prediction. Four algorithms based on Word2Vec word embedding were evaluated. Two standard distance metrics (mean and max) were evaluated as baseline classifiers in addition to two novel alignmentbased classifiers (Alignment KNN and Alignment Threshold). These results are summarized in Table 2 which shows the iterative cross-validation confidence intervals of the area under the F1 curve versus training set size. The table is sorted by the average score for the Alignment Threshold method. The first column in the table represents a summary of how this method performed in comparison to the others. If the Alignment Threshold method had a higher and non-overlapping confidence interval when compared to all other methods, this is indicated with a W+. If the lower bound on the Alignment Threshold method was higher than any other method and does overlap, this is indicated with a W. A T was used if the Alignment Threshold confidence interval was not higher than any other method but still overlapped the best method. All other cases are indicated with an L. In total, there are 29% W+, 35% W, 19% T, and 16% L, meaning that the Alignment Threshold method was as good or better in 84% of the classes. These results demonstrate how the alignment-based algorithms are able to perform better on fewer samples or equivalent to standard approaches. This is significant even in the case of equivalent accuracy as the alignment-based threshold algorithm does not require negative training samples which reduces the number of samples an expert must annotate. Further, all alignment-based methods are built upon a semi-supervised learning approach where large amounts of unlabeled data is used to reduce the need for labeled data.
It was our desire to test our algorithm against the multi-level kernel system
developed by Kim et al. discussed in section 2; however, we were unable to find a
public implementation of this algorithm, and our attempts to reach the authors
were not successful. In their paper, the authors evaluated their algorithm using
the Twitter dataset also used in this paper [
Further inspection of Table 2 shows that in some cases the Alignment Threshold classifier performs poorly while the Alignment KNN classifier, which uses both positive and negative training examples, performs better or equivalent to standard approaches. We believe this is due to the underlying word embedding vectors for the specific targets. Words such as those found in positive and negative tweets are relatively ubiquitous, while those found when mentioning a walk-in closet are relatively rare. It is reasonable to assume that hard coded rules could also be developed in some cases, but that the approach presented in this paper would be preferred as it is easily extended to additional categories without the need to maintain a rule management system.
The original annotation system was researched and built specifically for a company specializing in real estate technology. Because this data is proprietary, our algorithms are also tested our algorithm on a publicly available Twitter sentiment prediction dataset. The original Javascript Angular based system is proprietary, but we have implemented the core functionality of our system using Jupyter Notebook widgets. The widget loads directly inside of the Notebook and displays a tweet’s text, its overall sentiment value, and buttons that represent each word in the tweet. These buttons can be toggled to create the annotation pattern. All of this data is stored in a Pandas DataFrame and serialized to its own file, which can be used to train and evaluate our algorithm. We have made this version of the annotation interface open source in addition to all alignment-based implementations and evaluation code (https://github.com/ Anderson-Lab/sentence-annotation). 7
Being able to extract topics from text is an ever-growing problem for many domains given the large amount of textual data that is constantly being created. Therefore, it is necessary to minimize the amount of annotating needed to achieve high levels of accuracy. To accomplish this, we introduced a novel topic prediction algorithm that requires only a small amount of human annotation. Our results show that this approach can provide significant performance benefits when the target labels are not known a priori or when the sample size is small. Future directions of this work include experiments to determine if these alignment-based distance metrics continue to provide non-redundant benefits as the sample size grows significantly. For community and reproducibility purposes, our methods are available in a public repository that includes a Jupyter Notebook annotation widget that allows for annotation to easily be carried out on other real world datasets. g re ra trsop ilappA lan g ilroonF tse irckB ck tsn anL :reanduA teeavyhb reogy rceeeodnhP illtsceokn itteeounC lltsseeeSn ii-tcenhnK rceaahb roonm lreoonFP iiltseaanV iloodnnwm iftreaonP ftrseaaakB iil-ItabnnC itrssounC l ebudnT ittrcenuU -ceaSuD ililteeganudC iregaonpSph itceenhpnK irseaogpLF trrseeooadB ilttrgaahuL treoaonwwD itttrrcenuuU cceeaykndB segaanR tsee trrseaaauR llifsseaooyn le2 red taC rcS aWranG itaS taE eN uS pO uD rC eH rB uB enT ooP raG truS l-C V N O G M N N S F G tw eN rP
Our work was directly motivated to reduce the burden of interaction with human supervisors, and therefore, we present an overview of our architecture that can be broken up into four distinct phases: preprocessing, annotation, learning, and evaluation. Figure 3 illustrates these phases and the steps that are taken in each. Preprocessing & Cleaning The first step in our pipeline involves processing the raw listing description data and converting it into a consistent format. Each description is stripped of any extra whitespace, transformed into all lowercase letters, and decoded using the UTF-8 character encoding. Once this has been done, the description is separated into its constituent sentences. These sentences are then broken up and become lists of words that can be used in our alignment algorithm.
Annotation In order to expedite the annotation process, we made two simple annotation interfaces; one for each of the datasets that we tested our algorithm on. A screenshot of the Jupyter widget is shown in Figure 4.