We explore how the emotions and sentiments of users of Stack Overflow evolve over time based on their reputation scores. In this initial exploratory study, we compute four dimensions of emotions and sentiments: analytic, clout, authentic and tone for the question and answer posts of users in our dataset. Our results indicate that Stack Overflow users, experts, and non-experts, become more analytic and less authentic over time in both question and answer posts irrespective of their reputation score. In addition, while the clout of non-experts decreased over time in both question and answer posts, the clout of experts only decreased in their answer posts and not in their question posts. These results, though preliminary, could throw some light on the evolution of the emotional state of learners in online learning communities such as Stack Overflow.
and
Human-centered computing computing systems and tools
With the advent of Web 2.0, more people are seeking for answers to their computer programing questions on online forums such as Stack Overflow1. Stack Overflow is a question and answer site, where users can ask and answer specific computer programing questions. Active and helpful members of this community get rewarded for their participation in the form of reputation points and privileges2. Despite the popularity of Stack Overflow and the reward system of the community, only a few users actively participate in the network [13].
Research [11] suggests that rewards can change the attitude and behavior of people in different ways. Because Stack Overflow is hugely based on rewards, it is important to investigate if people who have earned several rewards over time have changed their behavior in the community. It is equally important to understand the evolution of the behavior of users who have earned a few rewards in the community over time. Therefore, we carried out a preliminary study of the emotions and sentiments of some users in the community over a five-year period. Emotion and sentiments are central to cognition in learning because they can shape learners’ engagement and also influence their overall learning experiences [6], [8], [4]. In online learning environments such as Stack Overflow, where there are no dedicated instructors, there is need to design the consciousness of users’ emotions and sentiments into the system, in order to arouse their feelings of security and self-confidence, which are important to encourage their continued participation. Research has measured emotions in online learning environments [5], but the trends and changes in their emotions and sentiments have not been measured, particularly in the context of lifelong learning. Therefore, we measured the emotions and sentiments of Stack Overflow users with the aim of determining if the emotions and sentiments of learners change over time for users who have earned a high reputation, and also for users, who have not earned a high reputation. In addition, we wanted to determine if there were any changes in emotions and sentiments of these groups of users in their question and answer posts. 2 2.1
Stack Overflow3 is a community question and answer platform where users can ask and answer specific IT related questions. Authors of questions can earn reputation and rewards when their posts get upvoted. The upvotes and downvotes are used to 1 https://stackoverflow.com 2 https://stackoverflow.com/help/whats-reputation Copyright held by the author(s). Use permitted under the CC-BY license CreativeCommons.org/licenses/by/4.0/ compute the final score of a post and are also a means through which users earn reputation. By posting high-quality questions and useful answers, users can gain reputation. The higher the reputation score, the more privileges the user can earn. Privileges control what users can do in Stack Overflow. Stack Overflow currently has over 5 million users with over 11 million questions. We chose Stack Overflow because it is a lifelong learning platform where professionals and programming enthusiasts can post questions and answers to support their continuous professional development. In addition, Stack Overflow data are readily available to be queried for the purpose of our research. 2.2
In this paper, we identify the sentiments and emotions of users in Stack overflow using the Linguistic Inquiry and Word Count tool (LIWC) [12], [16]. The LIWC tool reads text and determines what percentage of words in the text reflect various dimensions of sentiments and emotions of the writer based on its built-in dictionary of over 6,400 words. Although LIWC computes several dimensions of emotions and sentiments, being an exploratory study, we are only interested in four dimensions in this study: analytic, clout, authentic and tone.
According to the LIWC tool4, the dimension analytic represents the extent to which people use formal words, and how logical and hierarchical their thinking patterns are. People low in analytical thinking typically write in more narrative ways, use less of formal logic and rely on knowledge gained from personal experiences [10]. On the other hand, people high in analytical thinking use formal logic, are more detailed in their explanations and avoid contradiction [10].
The dimension clout, as defined by the LIWC tool, is an indication of the social status, confidence or leadership displayed by an individual through their writing or speaking. People with higher clout typically use more first-person plural (such as “we”) and second-person singular pronouns (such as “you”). In addition, they use fewer first-person singular pronouns ( such as “I”) [7]. People in this category tend to focus their attention outwards, towards the people they are interacting with. On the other hand, people low in clout are more self-focused and use more firstperson singular pronouns (such as “I”) [7].
People that are high in the dimension authentic, according to the LIWC tool, reveal themselves to others (through their writing) in a more honest way. Such people are more personal, humble and vulnerable. On the other hand, people that are lower in 3 https://stackoverflow.com/ 4 http://liwc.wpengine.com/interpreting-liwc-output/ authenticity use words that show lower cognitive complexity and more negative emotion words [9].
Tone, according to the LIWC tool, describes the emotion of the author. It summarizes negative and positive emotion dimensions into one variable. The higher the number, the more positive the tone.
We chose the LIWC tool because it has been used extensively in research for analyzing user-generated content in online systems [1, 2], [3], [16], [7], [15]. In addition, while several sentiment analysis libraries and packages detect positive, negative and neutral sentiments, the LIWC tool detects sentiments in addition to other traits such as emotions and personality traits [16]. 3
To carry out this study, we used data from Stack Overflow’s data explorer5. Stack Overflow’s data explorer enables one to directly query Stack Overflow’s publicly available dataset. We were interested in the emotions and sentiments displayed in the last five years by users who are currently active in the system and who joined the community about the same time, just before the fiveyear period. In addition, we were interested in both question and answer posts. Furthermore, we also wanted to compare the users who earned high reputation to those who do not have high reputation in the five-year period. To meet these criteria, we selected posts of users based on the following: 1. Question and answer posts of users who were created in
January 2013 2. Question and answer posts of users who meet the above criteria and have been active in 2019 To split the posts of users into two groups based on reputation, we computed the average reputation score of users who met the criteria stated above. The average reputation is 551. We thus split our users’ posts into two groups; posts created by users with a reputation score of at least 551, expert group and those created by users with reputation score less than 551, non-expert group 6 . Table 1 summarizes the data that was collected from Stack Overflow for this study. 5 https://data.stackexchange.com/stackoverflow/query/new 6 These users are not necessarily experts or non-experts in the community. The terminology was only used to differentiate between the groups.
Using the LIWC tool, the dimensions analytic, clout, authentic, and tone were computed for each post for each year from 2014 to 2018. This was done for the expert and non-expert groups and for question and answer posts. The average score of each dimension for each year was then computed for each group of posts. We computed the Z-score of each year’s average to standardize the scores across the four dimensions of analytic, clout, authentic, and tone. 4
In this section, we present the results of our analysis of the emotions and sentiments expressed by both the expert and nonexpert users in their question and answer posts. 4.1
Expert Users’ Group Our results in figures 1a and b suggest that the expert users became more analytical in writing question and answer posts over time. According to the LIWC tool 7 , the dimension analytic suggests the extent to which people use formal words, and how logical and hierarchical their thinking patterns are. While people with low analytical thinking typically write in more narrative ways and use less of formal logic [10], people high in analytical thinking use formal logic and are more detailed in their posts [10]. Stack Overflow is a question and answer forum for computer programing questions. It is therefore not surprising that the users who have been active in the network over time have become more analytical in their writing.
Our results in figures 1 and b also suggest that expert users become less authentic over time in both questions and answer posts. According to the LIWC tool12, people who reveal themselves to others (through their writing) in an authentic or honest way are more personal, humble and vulnerable. On the other hand, people that are lower in authenticity use words that show lower cognitive complexity and more negative emotion words [9]. This result was unexpected because the increase in analytic writing which we described in the previous paragraph does not suggest lower cognitive complexity. Therefore, we plan to explore this finding in future work.
The LIWC tool12 defines clout as the social status, confidence and leadership people display through their writing. As shown in figures 1a and b, while clout decreases over time for answer posts, it increases for question posts. People with higher clout typically use more first-person plural (such as “we”) and second-person 7 http://liwc.wpengine.com/interpreting-liwc-output/ singular pronouns (such as “you”). In addition, they use fewer first-person singular pronouns ( such as “I”) [7]. Our results could be an indication that over time when asking questions, experts have displayed more confidence and leadership and have used more of “we”s and “you”s and less of “I”s compared to when answering questions. We intend to explore this further in future research.
The tone of users, as indicated in figures 1a and b, increase over time in the answer posts and decrease in the question posts. The LIWC tool describes the tone as the emotions expressed by users in their posts. In the future, we will investigate this difference in emotions. 4.2
Figure 2a. Sentiments and emotions of answers posted by nonexpert users in our data set over 5 years Similar to the posts of experts described in figures 1a and b, the analytical thinking of non-experts also increased over time for both question and answer posts as shown in figures 2a and b. This is expected since Stack Overflow is a computer programing question and answer platform where learning occurs over time [14]. Because computer programing is analytical, writing more analytical posts over time could occur for users who use the platform often. Since our dataset includes users who despite joining the platform in 2013, are still active in 2019, the result is not unexpected.
Clout decreased over time in both the question and answer posts of non-experts. Clout in LIWC refers to confidence and leadership. Our results could infer that the users with lower reputation scores are not confident in their posts, possibly because they are still learning.
The results presented here are our initial exploratory findings. More analyses have to be carried out to determine if, for example, the difference in sentiments and emotions over the five years are significant or not. We plan to do that in the future. 5
We explored the five-year trend of sentiments and emotions of users on Stack Overflow. The results of our sentiment analysis using the Linguistic Inquiry and Word Count tool suggests that irrespective of their reputation score, users are more analytical and less authentic over time. While the clout of non-experts decreased over time in both question and answer posts, the clout of experts only decreased in their answer posts and not in their question posts.
These results, though preliminary, could shed some light on the evolution of the emotional state and sentiments of learners in online learning environments such as Stack Overflow, based on their reputation. More analyses are being carried out to determine the significance of the differences in the sentiments and emotions of the five-year period.