Many staff in higher education have a sense that useful information is buried within their datathat they are unsure how to access, or even what questions it can answer. This is particularly so with survey text responses from large student cohorts. This paper examines valid and repeatable methods to analyze such data while seeking to minimize computational and analyst workload by maximizing machine learning to accommodate the large volume of data.We evaluate clustering and topic modelling as methods to analyze one year's data from a national student survey in Ireland, an anonymized dataset with more than 44,700 respondents. The primary focus was on free text responses to two questions, namely those seeking to identify the best aspects of students' reported experiences, and those identifying aspects that need improvement. K-means and Latent Dirichlet Allocation unsupervised learners were used to identify key themes emerging from the text data. K-means proved computationally expensive and failed to usefully categorize significant minorities of the data. In contrast, topic modelling had relatively low overheads and effectively categorized more than 97% of the sample data into themes which could be usefully considered in the business domain. From this research, topic modelling provided an effective method to analyze such text data once careful consideration was given to determining the appropriate initial number of topics for configuring the algorithm.
Surveys of students’ experiences have become very common in many higher education systems as
the collection and utilization of data for a variety of practical and policy based purposes has been
steadily increasing [
This research sought to address the apparent lack of both understanding and capacity to efficiently analyse qualitative data generated by the survey. This was done by exploring multiple analytical approaches to identify effective methods which could, in due course, be shared with stakeholders to encourage and promote such analysis more widely. Therefore, the research question asked what is the most efficient method to analyse text data from open survey responses? The main objective set at the outset was to identify a valid and repeatable method which sought to minimize analyst and computational workloads to inform dissemination and promotion to survey partners. A selection of methods identified during background research were assessed.
This paper provides an overview of the dataset, which was relatively large in the context of Irish higher education, and reports on the research carried out to respond to the research question and objective through an iterative process of experimentation and model implementation. Section 2 provides an overview of background research undertaken. Section 3 describes the dataset involved and explains the methods selected. Section 4 presents headline results achieved from a series of experimental models. Section 5 offers analysis of those results and notes a number of possible areas for future research. Finally, Section 6 presents the conclusions.
Despite the “bewildering variety of strategies” for text mining available to the analyst [
Clustering techniques featured as feasible methods to analyse survey text data in a number of
publications. For example, [
While classification has been successfully used to label students’ open survey responses, such as
in [
The dataset used originated from the 2020 iteration of fieldwork for the national survey of students’
engagement in higher education. Responses were collected in February and March 2020, and so
predominantly reflect perspectives prior to COVID restrictions [
This represented 31% of the target student population across a broad range of disciplines from a variety of higher education institutions, including all traditional universities, technological universities, institutes of technology, colleges of education and a number of private institutions.
Attributes could be grouped into 3 broad categories: demographic data (such as year group, ISCED
field of study, mode of study, etc); actual question responses; and attributes calculated from the prior
two groups. The results from analysis of quantitative responses at national level varied little from
year to year since the survey was first administered in 2014, with greatest variation in results within
individual institutions [
Informed by the background research, efforts to address the research question were necessarily explorative in nature, taking a number of feasible approaches adopted elsewhere, to explore and analyse the data and evaluate the effectiveness of the approaches taken up to that stage, given the overall business context. The published papers reviewed tended to each focus on a small number of analysis methods and to report on their findings, rather than compare different approaches. It appeared that little research had been undertaken to compare different methods in order to determine which may offer efficient ways to analyse text data and, therefore, that this study may provide new information to others working with text data, particularly text data generated from student surveys. 3.1.
The priority focus for this research was on responses to two specific survey questions. These asked “What does your institution do best to engage students in learning?” and “What could your institution do to improve students’ engagement in learning?”. Throughout this paper hereafter, these are referred to as Best Aspects (BA) and Needs Improvement (NI). In the original dataset, 18,494 rows contained values for BA data and 20,205 rows contained values for NI data. An overview of these data after removal of non-letters, single letters and blanks is provided in Table 1. As indicated by the differences between mean and median lengths, a large proportion of responses were short – with, for example, 10,216 BA responses and 7,784NI responses each containing less than 50 characters. 3.2.
The data was converted to individual text documents with filenames which included a key identifier and a label indicating best aspects, BA, or needs improvement, NI. Based on consensus from background research, the data was transformed to lower case and tokenized (using non-letters). Shorter tokens which contained meaning in the business context were replaced by full titles to ensure that such information was retained when shorter tokens were subsequently filtered. Examples of these replacements included SU (student union), CA (continuous assessment), and MCQ (multiple choice quiz). A bespoke dictionary was used to list terms to be filtered out from the text data. These included acronyms and other terms which clearly identified individual institutions. Tokens were also filtered by length to remove those with less than 4 characters. Document vectors used TF-IDF to count term frequencies. Data sets were pruned to remove infrequent words (in less than 3% of comments) and frequent words (in more than 30% of comments). These values were chosen as a result of exploring the effect of different values to prune the dataset sufficiently to bring computational reductions without notable loss of potential information value. Experimenting with bigrams resulted in computation times of multiples of hours (>44) but failed to create meaningful clusters. A variety of levels of term pruning and numbers of clusters resulted in >85% of comments allocated to one large cluster in all cases. Therefore, bigrams were deemed unfeasible.
Unigram datasets were prepared with and without Porter stemming, and with retention and removal of stop words to accommodate exploration of differing findings from background research. Data was also divided into 70% training and 30% testing to support evaluation of models developed. This was done by repeatedly taking arbitrary groups from the file in order to reduce the risk of any unknown sequencing impacting future results. In hindsight, random sampling may have been a better approach to increase the level of automation for deployment.
The research question asked about the most efficient method to analyze the text data responses. Based
on background research, clustering and topic modelling were examined in some detail with the
objective of comparing their effectiveness, as had been undertaken to some extent with scientific
documents by [
Background research identified the potential of clustering to identify key themes, both hierarchical
[
Initial experiments with agglomerative, x-means and k-means cluster models using the full dataset
identified that computational cost was a major factor to consider in order to meet the business
objective of achieving an “efficient” analysis process. Therefore, a “fast” version of k-means that used
triangle inequality to accelerate (standard) k-means was chosen for more detailed evaluation of
clustering as a method [
The frequently referenced Latent Dirichlet Allocation (LDA) algorithm was used for topic modelling.
As was the case for clustering, a key question to be considered was the initial parameter setting for
number of topics. Similarly to k-means above, this was explored by executing multiple iterations of
the LDA operator over the same range of number of topics. Perplexity was reported in multiple papers
as a commonly used measure for evaluation of topics. It is calculated as the inverse of the mean
perword likelihood. However, background research had identified that perplexity and human judgement
were not necessarily aligned so coherence was also considered [
There are several possible approaches to minimize computational overheads and many of these require specific expertise for efficient use. This research used RapidMiner Studio (version 9.9), an open-source application with a readily understood graphical user interface which made it suitable for this research. A number of datasets were prepared for modelling from the same core data, as discussed in Section 3.2. These TF-IDF vectors were created using multiple configurations of the Process Documents from Files operator i.e., with various combinations of RapidMiner operators to: Transform Cases to lower case; Tokenise using non-letters; Filter Stopwords (or not); and apply Porter Stemming (or not), as shown in Figure 1. For topic modelling as discussed in Section 3.3.1, the Optimize Parameter operator was used to explore potentially optimal numbers of topics for further use in analytical models, as shown in Figure 2. This enabled testing different values for ‘number of topics’ in the embedded Extract Topics from Data (LDA) operator. Resulting perplexity and coherence values informed the selection of a limited number of topics for further analysis. This was done as a series of single models applied to specific prepared datasets as shown in Figure 3. In each case, the saved models which appeared to offer better performance were applied to the test dataset.
Acknowledging the risk of identifying only local minima for Davies-Bouldin, a series of clustering models were developed using the suggested “good” values for the number of clusters from 10% of the data, namely k=11, 26, 27, 29 and 34. These were applied to the full training dataset(s) with / without stemming and with / without stop words. The models were initially evaluated by setting the proposed cluster names as labels and using a Naïve Bayes classifier to predict class labels. Results presented in Table 2 represent the optimal results based on Naïve Bayes accuracy for both BA and NI. The most accurate clustering models were achieved without stemming and with removal of stop words for 27 clusters of BA and for 29 clusters of NI data. The accuracy as reported for Naïve Bayes of applying these models to unseen (test) data was remarkably high, 99.22% for BA and 85.66% for NI data.
Further evaluation of the content of these clusters is discussed in Section 4.3. The distribution of examples to clusters is discussed in Section 5. 4.2.
Table 3 demonstrates that the most accurate LDA models were found with stemming, matching background research. The best performing models had relatively low numbers of topics, 8 topics for BA data and 10 topics for NI data. This may reflect that topic modelling allowed for multiple topics whereas clustering sought to identify themes for individual clusters and, so, the best performing clustering models involved notably higher numbers of clusters. The three best “performing” models for BA and for NI datasets from Table 3 were then applied to unseen (test) data for BA and for NI, which resulted in a notable drop in accuracy as illustrated in Table 4. As noted in Section 3.3.2, a limitation of classification model accuracy is that it is based on predicting one topic per statement. *Actual numbers for each cluster reflect a 60:40 data split to train Naïve Bayes and estimate model accuracy. The decimal percentage is most telling for relative distribution. *Actual numbers for clusters reflect a 60:40 data split to train Naïve Bayes and estimate model accuracy.
Results described thus far reflected the priority focus on machine learning. Combining machine learning with other techniques was a frequent feature of background research. Table 5 presents a small number of examples of complete student responses for two of the larger clusters for BA data. These were selected as being representative of the predicted cluster based on manually reading the data. However, this represented a potentially significant change of approach from machine learning to human analysis and there are few robust methods to validate how representative these examples may have been. The two clusters outlined in Table 5 accounted for 11% of examples analysed. Similarly, Table 6 presents a sample of responses to two of the larger clusters for NI data. These two clusters accounted for 9.3% of examples analysed. · By making the tutorials compulsory in some of my modules, this really forces me toremain engaged
· Offer more helpful services to help struggling students · Be more involved with students · Maybe organise study groups for students who are struggling and aren't confidentenough to ask for help themselves from their peers · Listen to students on how they learn individually · I believe better engagement between students and lecturers via email or in-person could significantly improve morale amongst students as many of us become frustrated · when communication is poor/our worries are unattended to. · More interesting lectures · More interactive lectures
· Encourage people to attend lectures more · Record the lectures and put them on Moodle after lectures · Not having the lectures so spaced out
An equivalent process of human reading was undertaken for topic modelling. Unlike the data for clustering presented in Tables 5 and 6, topic modelling used the ten most frequently occurring words to describe the “core” theme for each topic. It is acknowledged that stemmed attributes informed allocation of clusters but that variants of individual tokens would remain present in values of the text attribute and, therefore, potentially in the most frequently occurring words in proposed topics. Table 7 illustrates examples of responses to the two largest topics for BA data. These two topics accounted for almost half (49.7%) of all examples categorised. This is in stark contrast to clustering where the two largest clusters contained only circa 10% of examples. Table 8 presents examples from the two largest topics for NI data. These topics included 53.2% of all examples categorized.
Many clusters, as illustrated by Tables 4 and 5, appeared to make intuitive sense. However, analysis of all clusters indicated that some example responses could have been allocated to different clusters. This reflected the fact that documents could belong only to a single cluster whereas responses may refer to multiple issues. However, the distribution of examples to clusters presented a larger problem. As noted, the clusters presented in Tables 4 and 5 represented only 11% and 9.3% of BA and NI responses, respectively. In each case, the largest proposed clusters contained a large proportion of the data with 39.4% of BA examples and 37.2% of NI examples allocated to the largest cluster. These clusters contained, in effect, the examples that had not been allocated to other clusters and did not form coherent themes in themselves but, rather, contained examples where TF-IDF values for all attributes were close to zero. This limitation meant that, while the remaining clusters offered some insights into the data, clustering models effectively did not categorise almost 40% of documents which would significantly limit their usefulness in the business domain. This would particularly be the case when high computational costs are factored in.
Many topics, as illustrated in Tables 6 and 7, featured multiple issues which, in general, appeared to relate quite well to form coherent themes. The top ten most frequently occurring words in each topic provided useful insights into examples contained therein. Unlike clustering, the least intuitive topics accounted for only a small minority of examples at 2.1% of BA documents and 2.7% of NI documents. The largest topics were intuitively feasible and made sense in the business domain. This fact, accompanied by acceptably low computational costs, meant that topic modelling proved to be the most effective method for analysis of these text data, in response to the research question. Some experimental iterations were required to determine optimal numbers of topics but these did not require excessive analyst time.
A number of areas for possible future research were also identified. These include the use of different clustering algorithms to confirm the extent to which difficulties may be associated only with k-means or some other learners. Further research could also seek to categorise the data subset provisionally allocated to the largest clusters, which were found to be uninformative in this research. It may also be informative to disaggregate the data to identify themes or issues that are reported to a greater or lesser extent by particular student cohorts.
A structured series of iterations of clustering and topic modelling experiments were undertaken on prepared student responses to prompts on questions about Best Aspects of their educational experience, and what Needs Improvement. Data had been prepared with and without stemming and with the retention and removal of stop words. The best performing k- means clustering models identified allocated a significant minority of examples to single large clusters for both BA data (39.4%) and for NI data (37.2%) which would be problematic in the business domain. However, other clusters were found to be intuitively feasible and should not automatically be discounted. Computational cost for clustering was also regarded as excessive without sufficient benefits to justify that cost. In contrast, topic modelling using Latent Dirichlet Allocation proved to be a computationally efficient method to categorise documents into feasible topics which appeared to be intuitively coherent in the business domain. More than 97% of examples appeared to be appropriately categorised, acknowledging the key distinction, compared to clustering, that examples were assumed to contain multiple topics. Care was needed to determine initial parameter settings and, in particular, to set the number of topics in advance. The use of local maxima for topic coherence values proved effective to inform those choices, whereas perplexity consistently offered apparently optimal values at the maximum number of topics chosen over class students classes lectures questions Small small tutorials groups discussion · Small class sizes · Small classes so it’s less intimidating to ask questions or speak up in
class · Lecturers engage with the students in class by having discussions on
topics relevant to the module · Have small groups in classrooms · In my opinion, by incorporating tutorials alongside lectures, itprovides an opportunity for students and lecturers to engage and have discussions relating to course topics · Smaller classes allow for discussion and opinions to be said, there’s a lot
of emphasis on continuous assessment · Group learning Interactive games and quizzes as part of assignments.
Presentations of material. Carry out practical course material in class · Having diverse modules and a blend of assignments, real life projects
and exams work · Group work Group · Lab work group projects · Practical courses with a hands-on approach assignments · Various group work and assignments to keep up to date
· Continuous assessment · More interesting lectures · Have smaller tutorial classes and lectures! · Try and make the lectures and practicals more interactive · More active, hands on lectures, more tutorials where we can
work in smaller numbers and have more meaningful discussions, · To have more interaction between lectures and students for all modules · More involvement in lectures · More interactive activities in class
assessment Continuous learning practical lectures class classes tutorials students Make Smaller know interactive activities students feedback · lectures could be more involved by allowing more time to meet · students assignments · More feedback. More emphasis on deadlines. Less readings as is Provide it difficult to balance all academic assignments. better lecturers · Provide more feedback and direction on future career options course · More support from lecturers support Give academic · A chance to get feedback from academic staff/fellow students on assignments prior to submission. One of our sessions dedicated to this would help.
· Give students their exams back when they’re corrected multiple iterations with incrementally increasing numbers of proposed topics. This analysis concurred with background research that human judgement should be used alongside intrinsic measures to determine the optimal initial number of topics.
From experimentation undertaken, topic modelling with stemming proved the most effective method to adopt in future. The key themes contained in responses as identified from analysis were:
• Smaller classes and tutorials facilitating greater discussions and interactions • Group / lab work, with practical or “real life” aspects • Individual engagement with helpful, approachable lecturers • Listening to students, and various combinations of the aspects listed above
• More interaction in lectures; increased smaller group activities • More feedback • Greater focus on individual students • Improved study facilities; more online materials