Question and Answering systems and crowd learning are becoming an increasingly popular way of organising and exchanging expert knowledge in speci c domains. Since they are expected to have a signi cant impact on online education [14], we will investigate to which degree the necessary conditions for collaborative learning emerge in open Q&A platforms like Stack Exchange, in which communities grow organically and learning is not guided by a central authority or curriculum, unlike MOOCs. Starting from a pedagogical perspective, this paper mines for circumstantial evidence to support or contradict the pedagogical criteria for collaborative learning. It is observed that although there are technically no hindrances towards true collaborative learning, the nature and dynamics of the communities are not favourable for collaborative learning. The ndings in this paper illustrate how the collaborative nature of feedback can be measured in online platforms, and how users can be identi ed that need to be encouraged to participate in collaborative activities. In this context, remarks and suggestions are formulated to pave the way for a more collaborative and pedagogically sound platform of knowledge sharing.
Computer-assisted instruction (CAI) is one of the hottest
topics in education research [
This however does not mean that online learning isn't
advancing in many interesting directions: Kahn's academy
emerged more or less organically when Salman Kahn started
teaching his cousin mathematics using short videos. When
Salman realized a lot more children could bene t from these
lessons, he started distributing them on YouTube. Today,
Kahn Academy reaches 10 million students per month,
according to Wikipedia. Wikipedia itself has become an
integral part of traditional education too. Some researchers
expect that learning in general will evolve from an
individual task centred around the teacher-student dichotomy, to
a collaborative social activity, in which online knowledge
bases like Wikipedia, forums, social networks and Question
& Answering systems are playing an ever more important
role [
This paper is structured as follows. First, the
pedagogical background of collaborative learning is set out, based
upon the work of Dillenbourg [
Existing de nitions of collaborative learning in the academic
elds of psychology, education and computer science, di er
signi cantly and are often vague or subject to interpretation.
We thus needed a theory that uni ed the di erent theories
and was applicable to the online, computerised world as well.
Not the least, it had to be easily operationalisable. A
review of the literature brought us to the work done by Pierre
Dillenbourg [
This means that collaborative learning can not be reduced to
one single mechanism: just like people do not learn because
they are individual but rather because the activities they
perform trigger learning mechanisms, people don't learn
collaboratively because they are together. Rather, the
interactions between the peers create activities (explanation,
mutual regulation,...) that trigger cognitive learning
mechanisms (elicitation, internalisation, ...) [
For these processes to be e ective, some requirements need to be ful lled. A subset was extracted that could be measured numerically, albeit indirectly, using the information available in our data set (summarized in Table 1). In the next section we will have a closer look at these indicators.
Dillenbourg discriminates three important aspects for collaborative learning to be e ective and characterises situations, interactions and processes as collaborative if they ful l the following criteria:
Peers are more or less at the same level, have a common goal and work together ; Peers communicate interactively, in a synchronous and negotiable manner ; Peers apply mechanisms like internalisation, appropriation and mutual modelling.
These high-level criteria have been re ned by Dillenbourg
into more detailed conditions for collaborative learning, of
which a subset has been summarised in Table 1. Each
corresponding indicator provides indirect circumstantial evidence
for each criterion, as our analysis was limited by the data
available in the Stack Exchange. Nevertheless, as we will
see, they give useful insight in the formation and dynamics
of open online collaborative communities for learning.
participation and interaction between students [
Stack Exchange can be considered as a distant-learning autodidact platform in which communities are formed organically and learning is not guided by a curriculum or some central authority, but exclusively by the members of the community, in contrast with MOOCs. This paper aims at answering the question whether the necessary conditions for collaborative learning emerge spontaneously in these platforms. As the work is done in the context of the PRAISE project2, a social media platform for music learning, the Music Stack Exchange data set was chosen.
Stack Exchange provides an open API, from which all data can be exported. The data set consisted of 2400 questions, 1500 active members and 1.7 million page views The platform is basically a forum in which anyone can ask and reply to questions. As a means of quality control, users can give up- and down votes to questions, and answers. People can also comment on questions and answers which is actually some kind of meta-discussion in which feedback on relevance, terminology, etc... is given. In the following paragraphs, the criteria listed in Table 1 will be studied in more detail.
Symmetry of action expresses the extent to which the same range of actions is allowed by the di erent users. Stack Exchange employs a system of so-called privileges, attributed according to your reputation3. These privileges are generally connected to moderation rights, rather than with the actions of asking and replying to questions { unless you have a negative reputation. The fact that users can exert the same actions, does not imply that this also actually the case. An analysis of the distribution of the ratio of answers over the number of questions, reveals that we can roughly discriminate three kinds of users, based upon their activity pro le: Silent users (62% of the registered users) that never answer, e.g. users that don't register or register but do not ask questions nor reply to them; Regular users (37% of registered users) that give roughly as much as answers as they ask questions, that is, two on average; Super-users (<1% of the registered users), these are 'hubs' that give at least 40x more answers than they ask questions.
The largest part (96%) of regular users, ask less than ve questions, and 76% even asks only one question: there are no 'parasite' users between the regular users that ask question but do not answer. From the other side, only 8 'expert' super-users (0.5% of the community) were responsible for answering 25% of the questions. Above ndings indicate that the symmetry in action is highly skewed because of a small group of 'super-users' and a large group of 'silent users'.
The research in this paper can be seen as an extension of previous research in Educational Data Mining, that measured 2http://www.iiia.csic.es/praise/ 3http://stackover ow.com/help/privileges Aspect
Scale-freeness of the user interaction graph Distribution of reputation within the community
Stack Exchange employs a reputation system by which members get rewarded or punished if a peer up- or down votes your answer or question, when your answer gets 'accepted', etc...
We would expect a "healthy" collaborative community to have a strong correlation between reputation and the time a user has been around on the platform: as users spend more time on the platform, their reputation builds up. An inquiry into the Stack Exchange music data set, however, reveals only a correlation of 0.23 between reputation and "time around". We could thus conclude that there is some odd kind of symmetry, in the sense that no one really builds up reputation.
Traditionally, these reputation systems are believed to make a good indicator for the knowledge a user possesses. However, there are some problems with this reasoning: Knowledge is not a uni-dimensional measure, but is connected to a (sub) domain of expertise; Someone's reputation keeps on increasing, even without activity: there is a bias towards old posts and members; There is a bias towards "easy answerable questions". Figuring the knowledge of the members directly is quite an impossible task to perform, especially in a broad and openended domain like music. To assess symmetry of knowledge, however, one could argue that if everyone in the Stack Exchange music learner's community has more or less the same expertise, then, on average, anyone would answer questions asked by anyone.
In other words, there would be no particular hierarchy in
answering, rather the network of interaction would be
"random" and not scale-free. Another way to put this, is to state
that no hubs of people would exist that answer signi cantly
more questions than others. A network is called scale-free if
the degree distribution follows a power law[
P (k) k (1) with P (k) being the fraction of nodes that have a degree k, and a constant typically between 2 and 3. Figure 1 reveals a power-law relationship, with exception this special group of "super-users". Above ndings therefore suggest that symmetry of knowledge is not observed.
As pointed out before, a small group of super users answer vastly more questions than they ask: a group of 21 users answered half the questions. This is clearly not a balanced situation in which the total labour of answering questions, is equally distributed. Figure 2 shows the relative timing of when users ask and respond to questions over their lifetime. Users tend to ask questions in the beginning (a visit to the site probably triggered by an urgent need to get a question resolved), but start answering more uniformly after a while. The graph also indicates that engagement is largest in the beginning. This information is relevant when developing platforms with a pedagogical purposes: users probably need to be "bootstrapped", allowing them to give lesser answers and ask more questions in the beginning, so they get "locked into" the platform. Note that a relative plot was preferred, in which the x-axis indicates the % of the lifetime, 0% being the moment of signing up, and 100% the date the data set was obtained. It allowed us to grasp the details of both users that had just signed up, as well as users that have been active for a long time (especially as the rate of signing up is probably not constant but increases with time).
To keep people engaged in an activity, according to the
"theory of ow" [
Based upon the analysis done in the previous section, some critical remarks and suggestions are o ered to improve the pedagogical nature and collaborative learning
4.1.1 Limited to no instructional design The data set on Stack Exchange music's forum, is an amalgam of questions (1) with di erent levels of granularity, typically with a small scope, (2) on a wide range of topics, for learners (3) with di erent learning goals and (4) different levels of expertise. The activities are not designed to elicit collaborative learning, and as the data is unstructured, without su cient sca olding of the learning content (e.g. through hyper-linking), it is no natural t for learning but rather provides ad-hoc answers to appease shortterm narrow personal learning goals.
Above remarks wouldn't be so problematic for collaborative learning, if pro cient communities existed within the Stack Exchange platform that had more or less the same goals, expertise and engagement. In the current case, there's a risk of frustration and boredom in expert users that don't see their questions answered and who have to answer straightforward questions. For novice members, on the other hand, their learning remains limited because they do not get sufcient guidance and do not really construct knowledge. Although the group of super-users makes sure that questions get answered quickly and perform the largest part of moderation, they are potentially harmful to collaborative learning as they distort the natural formation and dynamics of collaborative communities. From the other side, their interventions may bootstrap "young" forums.
The dataset revealed a very strong preference for voting up rather than down: only two users gave more down votes than up votes and of all the people that have ever cast a down vote (72 users out of the roughly 1500 active users), 80% gave more than ve times as much up-votes in return. 80% of the questions had no down vote, compared to less than 10% without up-vote. Figure 3 shows the distribution of up- and down-votes. This e ect was even more pronounced in the answers: the number of down-votes is typically zero or very small, whereas the up-votes reach a maximum at about 3 upvotes, then slowly attenuates. A further analysis of questions with more down than up-votes, revealed that these questions where either o -topic (40%), too vague, broad or speci c (35%), not real questions (10%) or Duplicate questions (8%).
4.2.1 Sub-communities
Allowing users to organise themselves in smaller active
subcommunities with common or similar learning goals, may
prove an elegant solution to manage or exploit the variety
in expertise of the users. Also, the concept of reputation
would make more sense. A similar idea was proposed by
Santos [
Good feedback should provoke critical thinking by asking sensible questions, provide a clue to "what's next" and allow to construct knowledge through sca olding and coupling back to acquired knowledge. Though the concept of freely asking questions is very accessible, the content stays rather ad-hoc and unstructured. A way to organise and link different questions in order to guide learners would be very useful.
In the modern ages of web technology, users could bene t from a collaborative interface in which knowledge is constructed together, in a way similar to for example Google Docs where one single entity is shared by all users. So, rather than preserving the strict question/answer or learner/teacher dichotomy, one would go for a situation in which knowledge { not only answers but also questions { is constructed live in an interactive way.
In this paper, the case for collaborative learning in openended auto-didact Q&A environments like Stack Exchange is investigated. Based upon the criteria put forward by Dillenbourg, we can state that though there are technically no hindrances towards collaborative learning, the nature and dynamics of the community that organically form on Stack Exchange, do not support the case for collaborative learning. It was observed that the symmetry of action was distorted due to a small group of "super-users" that answered the majority of questions and a large group of "silent users" that do not really interact with the platform. Inspection of the degree distribution of the user interactions reveals that the community network is scale-free, which means that symmetry of knowledge is very unlikely. The reputation system seems insu cient as a measure of expertise and a strange kind of symmetry of status is observed, in the sense that no one really builds up reputation, except for a small group of users.
Lastly, the limited possibilities to instructional design, elicits short-term narrow and personal learning goals. Also, the very heterogeneous nature of the community is not favourable for learning. Suggestions were made to adapt these interesting and popular platforms to learning, like creating subcommunities with common learning goals, extend the possibilities for organising and structuring the content and employ collaborative interfaces.
As future work, these results should be validated by means of other communities on Stack Exchange as well, and on di erent modes of feedback, rather than only text-based.
This research has been supported by the EU FP7 PRAISE project #318770.