Supporting mentoring processes in higher education is a relevant and challenging aim. Meta-cognitive, emotional and motivational aspects play a crucial role here. Big data can help to recognize the affects of mentees, to react accordingly and to make the mentoring support scalable. In our study, we processed data from university discussion forums utilizing text and sentiment analysis. The results suggest that this approach can raise mentors' awareness of the activities in discussion forums, but limitations need to be considered. Evaluations with real users can help to develop these approaches further.
Good learning should be individualized and personalized. This goal was already addressed with Intelligent Tutoring Systems, Adaptive and Personalized Learning Environments. These systems mainly aim at the cognitive aspects of the learning process. Intelligent Mentoring Systems (IMS) are going one step further by including metacognitive, emotional and motivational elements in the learning process.
This leads to the following question: What should concepts for designing
learning and teaching look like to make the quality of individual mentoring
scalable for the acquisition of target competences? Compared to coaching and
tutoring, the mentoring process is more spontaneous, more holistic, based on the
needs and interests of the mentee and focusing on psychological support. The
relationship is more complex, two-way and based on emotions [
In the following we rst very brie y mention selected related work. Then
we introduce the analyzed data and the methods applied. In the main part the
results are presented and discussed. Finally we summarize the paper and outline
next steps.
Sentiment analysis (SA) aims to analyse people's opinions and emotions from
written language. It is widely studied in data mining, Web mining, and text
mining to better understand human behaviours [
There is a lack of opinion mining systems in non-English languages. Moreover,
cross-domain SA is still a signi cant challenge, including issues like the di erence
in sentiment vocabularies across di erent domains and an objective assignment
of a strength marker to each sentiment word [
For this research the data from OPAL discussion forums at the Technical University of Dresden between the years 2005 and 2009 have been employed. The dataset contains 16,614 messages from 123 forums exchanged between 1490 users (students and teachers). Each forum, message and user have a unique identi er.
The data is in anonymised form. Messages contain the plain text with the HTML tags and contain a collection of these emoticons: angel, blushing, confused, cool, devil, grin, kiss, ohoh, sad, smile, tongue, ugly and wink.
The analysis focused on the data from 5 forums containing the highest number of messages. Tab. 1 shows the statistics of the selected forums. To uncover information in the data, we applied text mining methods on the selected messages. In the following, the data preprocessing is explained and then each method is introduced.
The text corpus was preprocessed in the following way:
1. Extraction of emoticons: At the beginning, all emoticons presented in the
text as HTML tags "IMG" with class "emoji" were extracted.
2. Removal of HTML formatting: All messages were stripped from the HTML
tags to get the clean text messages.
3. Tokenization: All messages were divided into separate words (tokens),
keeping the information to which message each word belongs. The unnest tokens
algorithm from tidytext R1 package [
The analysis of word frequencies is the most common way to approach text corpus. The purpose is to uncover the most common words re ecting the text content. At rst, the word counts for each forum were analysed by merely counting the number of word occurrences. The analysis showed the most common words in each forum.
To quantify what are the discussion forums about the term frequency - inverse document frequency (t df) measure was used. It measures how each word is important to the forum in the collection. The t df of word i in the document j is product of two measures: tf idfi;j = tfi;j idfi where term frequency tfi;j = Pnkin;jk;j is number of word occurrences (ni;j) divided by document length jDj (Pk nk;j) and inverse document frequency idfi = log jj:ti2Djj is the logarithm of number of documents (jDj) divided by number of documents in which the word is presented (jj : ti 2 Djj). 4.3
For SA, the SentimentWortschatz sentiment lexicon [
Sentiment Trajectory The sentiment of the messages in chronological order was visualised. The information can be interpreted as sentiment trajectory during the whole forum lifetime. This visual interpretation can uncover the general sentiment trend as well as outliers from the overall sentiment. Outliers are messages "too" positive or negative compared to the others.
Sentiment Wordcloud The Wordcloud visualisation showing the most common words can be used in combination with the sentiment. The "cloud" is divided into two halves. One half of the cloud represents words with a positive attitude, and the second half those with the negative. The size of halves, in this case, is irrelevant. What is important are the terms themselves. They represent the most common negative and positive words in the text.
Correlation of Sentiment and Emojis The last analysis answers the
question of whether the emoticons used within the messages somehow correspond
to the sentiment of the message. We assigned the sentiment values to the
emojis (sentiment value is in brackets): angel (0), blushing (0.4), confused (-0.2),
cool (0.8), devil (0), grin (0.8), kiss (0.4), ohoh (-0.8), sad (-0.8), smile (0.4),
tongue (0.6), ugly (-0.8) and wink (0.4). Then the emojis and corresponding
text sentiment were compared using Pearson's product-moment correlation test
[
The previously presented methods have been applied to the data, and the corresponding results are presented within this section. 5.1
Fig. 1 presents the results of word count analysis for top 5 forums. Every chart represents the top words used in the discussion forum. We can observe that one of the most used words is "aufgab", which is the root form of the word "Aufgabe", representing the assignment within the course. Other terms such as "frag", "klausur" or "dank" are also standard within the selected discussion forums. Thus one can assume that most of the message content are questions about course assignments and exams. The content is not surprising since that is why forums exist in many educational settings.
Fig. 2 shows the most representative words for each forum. One can observe that forums 447053831, 220528647 and 320634883 discuss mathematical issues in their courses. There are words like "hilbert", "algebra", "logit", which are representatives of the mathematical terms. The other two forums cover topics in economics containing the terms like "frank", "gmbh" and "gemeinkost". Based on the analysis of word count and t df, one can assume that the forums focus on questions related to the course assessments. 5.3
Our analysis focused on the sentiment within the discussion forums. Fig. 3 shows the sentiment trajectory of each forum. One can observe that most discussion forums tend to be slightly negative, and there are several negative outlier values. For example, forum 447053831 has multiple negative peaks, suggesting that these messages may be worth analysing and answering by a mentor. One can also observe that the trajectories have lowering sentiment values over time, which suggests that in the end, the messages were more urgent. The forum educational focus can explain the overall negativity of the messages. One can expect that the huge portion of words used in communication between students and their mentors will be of neutral sentiment. Still, if the student has di culty, the sentiment tends to be more negative.
Another way to analyse sentiment is sentiment wordcloud, as shown in Fig. 4. One can observe important positive and negative words. The typical representatives of positive words are: "einfach", "genau" and "verstand". These words re ect the understanding of assignments and student's success. Words representing negative sentiments are for example "falsch", "frag" and "nicht".
Finally, we also compute correlation between emojis and sentiment of the corresponding text. The resulting Pearson's product moment correlation test estimates correlation of 0.04 with p-value 4.5 thus we can conclude, that there is only small correlation between sentiment in the text and emoji used. Mentoring is of crucial importance for university students if provided timely and with good quality. As the capacity of experienced mentors is limited, scalable solutions are needed to make their work e cient. The available technology can analyze the emotions of students from the big data, revealing the learning progress and the need to intervene.
This work in progress deals with the text and sentiment analysis of university discussion forums, which may help mentors to notice critical points where their help is required. As mentioned earlier, there are many challenges in this research area, including a lack of opinion mining systems in non-English languages and cross-domain sentiment analysis. The sentiment vocabularies with their assignments of words are of signi cant importance.
Using our approach, we made several observations. In one case, the exceptionally high sentiment positivity was caused by the commercial interests of the author. Several other posts in this category were not related to learning, but rather to various celebrations. Apparently, for mentoring purposes, additional pre-processing and ne-tuning would be helpful. On the other side, a post with high sentiment negativity explained privacy and data protection rules. In another post, the indicated negativity was a context issue. E.g. the short text "das Risiko des Verlusts der Paritatsinfos geringer ist" contains three strongly negative words, but it just describes a fact. One critical post deals with the information architecture of an educational web site. So it seems for mentors, especially posts with an increased negative sentiment, are relevant to consider. The sentiment trajectory indicates that the messages after a certain point of time may become more urgent.
Of course, we realize various limitations of this study, in addition to the general challenges mentioned above, regarding the domain and context-dependency.
Also, the tf-idf method does not capture the contextual information, assuming the complete independence among all the words.
Nevertheless, despite these limitations, text and sentiment analysis of discussion forums can undoubtedly help to make the work of mentors more e ective and e cient. Even if not every pointed post turns to be urgent, such noti cations should be valid in a longer time frame, if proper tools are deployed. To justify their usefulness evaluations with real mentors need to be performed. Relevant functionalities will be integrated in the infrastructure of the tech4comp project (https://tech4comp.de/).
The project underlying this report is funded by the German Federal Ministry of Education and Research under the funding code 16DHB2102. The presented work was partially inspired by discussions with Cathleen Stuetzer and Ralf Klamma. Responsibility for the content of this publication lies with the authors.