For tutorial dialogue systems, classifying the dialogue act (such as questions, requests for feedback, and statements) of student natural language utterances is a central challenge. Recently, unsupervised machine learning approaches are showing great promise; however, these models still have much room for improvement in terms of accuracy. To address this challenge, this paper presents a new unsupervised dialogue act modeling approach that leverages non-cognitive factors of gender and selfefficacy to better model students' utterances during tutorial dialogue. The experimental findings show that for females, leveraging learner characteristics within dialogue act classification significantly improves performance of the models, producing better accuracy. This line of investigation will inform the design of next-generation tutorial dialogue systems, which leverage machine-learned models to adapt to their users with the help of non-cognitive factors.
Tutorial dialogue is a highly effective form of instruction, and
much of its benefit is thought to be gained from the rich natural
language dialogue exchanged between tutor and student [
Today’s tutorial dialogue systems utilize a variety of dialogue act
classification strategies, some rule-based and some statistical [
intensive due to the manual annotation and handcrafted dialogue
act taxonomies that are usually domain-specific. To overcome
these challenges, unsupervised dialogue act modeling techniques
including hidden Markov models [
This paper investigates whether the performance of an unsupervised dialogue act classifier can be improved by taking these factors into account. Because non-cognitive factors are shown to affect language, we believe that training dialogue act classifiers tailored to specific learner characteristics can help tutorial dialogue systems to understand students better. We utilize two learner characteristics: gender, as self-reported by students on a survey and domain-specific self-efficacy, as measured by a validated instrument for determining a student’s confidence in her own abilities. Specifically, we train unsupervised dialogue act models that are tailored to students of specific gender and selfefficacy level, and we compare those models to corresponding ones trained without restricting by that learner characteristic. This unsupervised training is conducted entirely without the use of manual tags. We then test all of the models on held-out test sets within leave-one-student-out cross validation, and compare the resulting classification accuracy according to their previously applied manual tags. The results show that for female students, utilizing learner characteristics statistically significantly improves dialogue act classification models. For self-efficacy groups, improvement is observed but not at a statistically reliable level. This paper constitutes the first research toward incorporating noncognitive factors into unsupervised dialogue act classifiers for tutorial dialogue with the overarching goal of providing personalized learning for students. We first administered a survey to collect these characteristics via self-report, and then learned a dialogue act classifier tailored to those characteristics. These results can inform the way that next-generation tutorial dialogue systems conduct their real-time dialogue act classification and language adaptation.
Dialogue act modeling is an important level of representation
within dialogue systems. Following theories proposed several
decades ago within philosophy and linguistics [
Another important difference between the current work and prior
research is in the features used, namely the non-cognitive
characteristics of gender and self-efficacy. Prior work has used a
variety of features for performing supervised dialogue act
classification, including prosodic and acoustic features which
involve the profile of the sound signal itself [
The corpus used in this study consists of student-tutor interactions
in an introductory computer science programming task [
Students were given a pre-survey that included survey items on
computer science self-efficacy, such as ‘I am sure I can learn
programming’. This self-efficacy scale was adapted directly from
the Domain-specific Self-Efficacy Scale [5], with five items
measured on a Likert scale from 1-5 (1 being lowest self-efficacy,
5 being highest). Students also completed a demographic
questionnaire from which gender was obtained. For self-efficacy,
students were divided into classes based on the median score
across all students on that scale. Along with gender, this produces
two partitions of the 42 students: females (12) and males (30), low
(24) and high self-efficacy students (18).
The corpus containing 1640 student utterances was manually
annotated with dialogue act tags in previous work [
Example yeah I'm ready!
Alright i am taking basic fortran right now never seen literal before what does that mean? better? *html messing haha Distribution 39.95% 21.31% 21.20% 15.15% 0.98% 0.79% 0.61%
We hypothesize that dialogue act models built using unsupervised machine learning will perform substantially better when customized to specific learner groups. Specifically, we investigate whether by training a model only on students of a particular learner characteristic, that model would perform significantly better at predicting the dialogue acts of unseen students with the same learner characteristic compared to a model that was trained on students of all learner characteristics.
We note that because the same corpus is being partitioned in two different ways, the same student will occur in one of the gender groups and in one of the self-efficacy groups. This choice to partition in 2-way splits rather than 2n-way splits where n is the number of learner characteristics is because of issues that arise with sparsity. This interdependence between partitions is a limitation to note; however, as discussed in Section 5, this interdependence can be taken into account for making decisions within a tutorial dialogue system by employing a suite of classifiers within a voting scheme.
For gender and self-efficacy, we will test whether an unsupervised dialogue act classifier trained only on students with that characteristic outperforms a classifier that is not specialized by this characteristic. In order to gather accuracy data across these characteristics, we conduct leave-one-student-out training and testing folds. The testing set for each of the n folds (where n varies depending on which learner group is being considered) consists of all of a single student’s dialogue utterances and the model is trained on the remaining n-1 students. The average number of utterances per student in the corpus is 36.8 (σ=12.07; min=16; max=64). These are therefore the average, minimum, and maximum number of utterances across the leave-one-student-out test sets.
We compute the average test set performance of the model across all folds for each non-cognitive characteristic partition. The performance metric utilized in this study is accuracy compared to the manually labeled dialogue acts described in the previous section, where accuracy is computed as the number of utterances in the test set that were classified according to their manual label, divided by the number of utterances total in the test set. As described in 4.2, the process of labeling via unsupervised classification involves taking the majority vote within each cluster.
For constructing the folds, we take an approach to balance the sample size available to model training. This balancing approach is needed to ensure that each model is trained on a similar size of data. Consider, for example, the partition of gender. Without a balanced sampling approach the leave-one-student-out testing folds for the un-specialized classifier for female students would include nfemale=12 test folds but the available data for each training fold would be ntotal-1 = 41. In contrast, the specialized classifier trained only on female students would still include nfemale=12 test points but the available data for each training fold would be nfemale-1 = 11. Therefore, each un-specialized classifier was trained on a randomly selected subset of the corpus. In the case of females, each of the 12 testing folds will utilize a model trained on 11 data points. The specialized classifier will use 11 female data points, and the un-specialized classifier will use 11 randomly selected data points. In this way, we investigate how well a model predicts dialogue acts of a student with and without utilizing learner characteristic information.
Our unsupervised dialogue act classification approach leverages
the k-medoids clustering technique [
The k-medoids algorithm requires seeding clusters at the beginning of each training fold and then proceeds by distributing data points to clusters according to their closest centroids until convergence upon the model. In the standard k-medoids algorithm, the seeds are randomly selected. However, we employ a greedy seed selection approach intended to mitigate the effects of the unbalanced distribution of dialogue acts in the corpus [2]. Within this greedy seed selection, an initial seed is randomly selected and then each of the subsequent seeds are selected by choosing the point that maximizes its distance from the alreadyselected seeds. The goal in using this approach is to select the seeds from diverse utterances so the algorithm produces better clusters, and our initial experiments indicated that it substantially improves the model.
In addition to its seeding approach, the k-medoids approach requires the number of clusters k to be set prior to model training. To discover the number of clusters, we experimented with XMeans and Expectation Maximization clustering, both of which attempt to identify the optimal number of clusters. Both of these algorithms converged at four clusters as the optimal choice, so we proceed with k=4. However, perhaps in part due to the benefit of the greedy seed selection made possible by k-medoids, these models performed with substantially worse overall accuracy than k-medoids.
The utterances were represented as vectors with each column matching a token (punctuation and words) in the corpus and each row matching an utterance. There were a total of 877 distinct tokens.
With these parameters in place, first the clusters were formed using each training set, and then for each utterance of the student held out within the leave-one-student-out fold, we computed the closest cluster to that utterance as indicated by average cosine distance to each point in the cluster. The closest cluster was selected as the cluster to which the test utterance belongs, and the majority vote of the cluster was assigned to the test utterance as its dialogue act label. For each leave-one-student-out testing fold, the accuracy was computed by comparing these cluster-assigned labels to the manual dialogue act tags.
This section presents experimental results for unsupervised dialogue act classification based on learner characteristics. We compare each model built separately by gender and self-efficacy level to the models that are built using utterances from randomly selected students, i.e. not utilizing learner characteristic information. Each comparison in this section is conducted with a one-tailed t-test with a post-hoc Bonferroni correction. The threshold for statistical reliability after the correction has been taken as α=0.05.
Gender. As shown in Figure 1, the average leave-one-student-out cross-validation accuracy for the model built using female students’ utterances (nfemale=12) is higher than the model built on randomly selected students. In each test run, all of one female’s utterances were left out to be used as the test set, and the dialogue act model was built on the remaining eleven female students’ utterances. This process was repeated for each female student. Note that for each of the eleven students, all utterances from that student were considered. Average test set accuracy for the model with randomly selected students was 0.41 (σ=0.2), whereas the average test set accuracy for the dialogue act classification model that was built utilizing female students’ utterances only was 0.56 (σ=0.19). After a Bonferroni correction this difference was statistically significant (pBonf<0.05).
For male students (nmale=30), the average accuracy is only slightly higher with the models tailored to males 0.43 (σ =0.13) than the models learned for randomly selected students 0.40 (σ =0.12), and this difference is not statistically significant (Figure 1). Looking more closely at the results, we find that for eight of the thirty males within the corpus, a tailored model outperformed the random model (with five of these seeing more than 10% increase in accuracy), while twenty-two of the cases saw no difference in accuracy between the random and tailored conditions. Two of the males saw a decrease in accuracy for the tailored condition.
Test Set Accuracies For Gender 0.9 0.8 0.7 cy0.6 rau0.5 c cA0.4 0.3 0.2 0.1
Female test Female test Male test random train female train random train Male test male train Self-efficacy. Models built using the self-efficacy learner characteristic predict the unseen utterances’ dialogue acts marginally more successfully than models that do not use this information, though these differences are not statistically reliable. For students with low self-efficacy (nlowEff=24) the average test set accuracy for dialogue act models that selected students randomly is 0.38 (σ=0.16) and it increases to 0.43 (σ=0.17) with dialogue act models that learn only from low-self-efficacy students’ utterances (Figure 2). In fifteen out of twenty-four cases the dialogue act models tailored to low self-efficacy groups outperform models that are trained on randomly selected students (eight of the cases with more than a 10% increase), while in seven of the cases the performance is decreased by utilizing the learner characteristic (five of them by more than a 5%) and in two of the cases the accuracy remains the same.
The improvement obtained by utilizing learner characteristics in dialogue act classification task is also marginal for high-selfefficacy students, where nhighEff=18. The average performance for the random model is 0.41 (σ=0.14) whereas the model achieves 0.47 (σ=0.11) accuracy when trained only on utterances of highself-efficacy students. This improvement was statistically significant before Bonferroni correction but not afterward. In seven out of eighteen cases, models trained on utterances of high self-efficacy students improved test set accuracy (five of them above 15% improvement) and in two of the cases the learner characteristic decreases the performance (both of them below 5% decrease). Nine of the cases remained unaffected in their dialogue act classification accuracy.
The average accuracies over the leave-one-student-out crossvalidation folds can be found in Table 3. Models tailored to learner groups uniformly outperform their counterpart, and the improvement is statistically significant for females.
Test Set Accuracies For Self-Efficacy
Dialogue act classification is a central task for tutorial dialogue systems. Without accurate dialogue act classification, systems cannot adapt and respond appropriately. Unsupervised machine learning approaches to dialogue act classification are a highly promising new area of study, and we have presented the first unsupervised dialogue act classifier tailored to learner characteristics. The experimental results demonstrated that dialogue act classifiers that leverage the non-cognitive factors of gender and self-efficacy outperform those that do not, and in the case of female students the improvement was statistically significant. This section presents some examples of the learned dialogue act clusters and discusses the implications of this work for tutorial dialogue systems.
First, we examine clusters from the gender-tailored unsupervised
dialogue act classifier. Table 4 displays a selection of utterances
that were clustered together during the unsupervised training of
the model, and afterward the clusters were labeled for testing
purposes using the manual tags that comprise the majority of each
cluster. For those in Table 4 the clusters were labeled as
Acknowledgments and Questions. By examining the structure of
these clusters we gain some intuition as to the types of regularities
that help the tailored models to perform significantly better. We
see females in this study tended to use acknowledgment phrases
such as, “oh I see” and “makes sense,” while males tended to use
the phrasing, “got it” more frequently. Within the cluster labeled
as questions, we observe that females tended to request more
feedback, an observation that also emerged in prior work within a
different corpus in the same domain collected approximately six
years earlier [
In addition, we observe some example clusters from the models based on self-efficacy in Table 5. Students with high self-efficacy tend to use more confident utterances such as “absolutely” compared to “ok” used by low-self efficacy students. We note that questions in the low self-efficacy group often make an implicit request for reassurance within their task-based questions, such as, “and that is it?”. In contrast, students in the high self-efficacy group more often ask contentful questions. Limitations. The present work has several notable limitations. First, as mentioned previously, the partitions of the corpus are not independent; that is, the same student, and associated utterances, are present within one gender group and one self-efficacy group. Because these partitions are not independent, care must be taken when interpreting the findings. Furthermore, it is possible that the self-efficacy of students can change in the course of tutoring, which would not be handled by a classifier built using a one-time self-report. However, we believe that the current approach holds great promise for real-time tutorial dialogue classification. By building separate classifiers by learner characteristic, a suite of classifiers (each smaller and faster than one built on the entire corpus) can be run in parallel and can vote for the classification of a given students’ utterance. However, as is the case with the work presented here, splitting the corpus results in a substantially reduced sample size on which to train, which partially explains the lack of statistically reliable results observed here. Our work has begun to explore the use of intrinsic metrics for accuracy (rather than relying on manual tags), which has the potential to dramatically increase the available data to any dialogue act classifier and mitigate issues of sparsity that arise when splitting by learner characteristics.
More accurately understanding student natural language within intelligent tutoring systems is a critical line of investigation for tutorial dialogue systems researchers. The field has only begun to explore unsupervised approaches and to investigate the range of features that are beneficial within this paradigm. We have presented a first attempt to leverage non-cognitive factors within such a dialogue act classification model, achieving statistically significant improvements in dialogue act modeling for female students, and increasing the models’ performance by small margins for the self-efficacy groups.
Building upon these first steps, there are several promising future directions. First, while sample size prohibited exploring some other learner characteristics here, other characteristics are likely highly influential and should be investigated. These may include ethnicity, personality, and other non-cognitive factors. Additionally, while the current work focused on analyzing dialogue, another aspect of the tutorial interaction that presents challenges in understanding is the task model. Models that aim to understand students’ problem-solving activities and infer their goals or plans may benefit substantially from leveraging learner characteristics. It is hoped that the research community can continue to build richer models of natural language understanding for students of all learner characteristics in order to improve the student experience and enhance learning by adaptation.
The authors wish to thank the members of the Center for Educational Informatics at North Carolina State University for their helpful input. This work is supported in part by the National Science Foundation through Grant DRL-1007962 and the STARS Alliance, CNS-1042468. Any opinions, findings, conclusions, or recommendations expressed in this report are those of the participants, and do not necessarily represent the official views, opinions, or policy of the National Science Foundation.