=Paper=
{{Paper
|id=Vol-1183/ffmi_paper04
|storemode=property
|title= Mining for Evidence of Collaborative Learning in Question & Answering Systems
|pdfUrl=https://ceur-ws.org/Vol-1183/ffmi_paper04.pdf
|volume=Vol-1183
|dblpUrl=https://dblp.org/rec/conf/edm/Loeckx14
}}
== Mining for Evidence of Collaborative Learning in Question & Answering Systems==
https://ceur-ws.org/Vol-1183/ffmi_paper04.pdf
Mining for Evidence of Collaborative Learning
in Question & Answering Systems
Johan Loeckx
Artificial Intelligence Lab
Vrije Universiteit Brussel
Pleinlaan 2, 1050 Brussel
jloeckx@ai.vub.ac.be
ABSTRACT
Question and Answering systems and crowd learning are
becoming an increasingly popular way of organising and ex-
changing expert knowledge in specific domains. Since they
are expected to have a significant impact on online educa-
tion [14], we will investigate to which degree the necessary
conditions for collaborative learning emerge in open Q&A
platforms like Stack Exchange, in which communities grow
organically and learning is not guided by a central authority
or curriculum, unlike MOOCs. Starting from a pedagogical
perspective, this paper mines for circumstantial evidence to
support or contradict the pedagogical criteria for collabora-
tive learning. It is observed that although there are techni-
cally no hindrances towards true collaborative learning, the
nature and dynamics of the communities are not favourable
for collaborative learning.
The findings in this paper illustrate how the collaborative Figure 1: The degree distribution shows that the
nature of feedback can be measured in online platforms, and network of user-interaction is scale-free, which sup-
how users can be identified that need to be encouraged to ports the hypothesis that there is no symmetry of
participate in collaborative activities. In this context, re- knowledge.
marks and suggestions are formulated to pave the way for
a more collaborative and pedagogically sound platform of
knowledge sharing. vancing in many interesting directions: Kahn’s academy
emerged more or less organically when Salman Kahn started
1. INTRODUCTION teaching his cousin mathematics using short videos. When
Computer-assisted instruction (CAI) is one of the hottest Salman realized a lot more children could benefit from these
topics in education research [9] and often claimed to rev- lessons, he started distributing them on YouTube. Today,
olutionise how we teach and learn [6]. Massive Open On- Kahn Academy reaches 10 million students per month, ac-
line Courses or MOOCs are the newest manifestation of this cording to Wikipedia. Wikipedia itself has become an in-
phenomenon. However, while 2012 was being praised as tegral part of traditional education too. Some researchers
”the year of the MOOC”, more and more critical voices were expect that learning in general will evolve from an individ-
heard during the last year and MOOCs are under increasing ual task centred around the teacher-student dichotomy, to
pressure to finally live up to their promise. Spoken in terms a collaborative social activity, in which online knowledge
of of Gartner’s Hype Cycle [8], we could say that we’re either bases like Wikipedia, forums, social networks and Question
at the peak of inflated expectations, or already entering the & Answering systems are playing an ever more important
through of disillusionment [3, 15, 10]. role [4]. In this paper, we will try to find evidence of the
claimed collaborative properties of Q&A systems, more in
This however does not mean that online learning isn’t ad- particular the music forum site of Stack Exchange1 . Though
the analysis is based on text-based feedback, it is expected
that the dynamics of feedback in collaborative activities also
hold in multi-modal situations.
This paper is structured as follows. First, the pedagogi-
cal background of collaborative learning is set out, based
upon the work of Dillenbourg [7] and conditions for and
indicators of collaborative learners are introduced. Next,
1
http://music.stackexchange.com
educational data mining techniques are applied [12] to find participation and interaction between students [11] and the
evidence of collaborative learning in crowd learning systems, successful formation of learner’s communities [1, 13].
more specifically Question and Answering systems like Stack
Exchange. Lastly, a critical discussion is performed and sug- 3. QUANTITATIVE ANALYSIS
gestions towards more collaborative Q&A systems are pro- Stack Exchange can be considered as a distant-learning auto-
posed, to end with conclusions. didact platform in which communities are formed organi-
cally and learning is not guided by a curriculum or some
2. COLLABORATIVE LEARNING central authority, but exclusively by the members of the
2.1 Pedagogical approach community, in contrast with MOOCs. This paper aims at
Existing definitions of collaborative learning in the academic answering the question whether the necessary conditions for
fields of psychology, education and computer science, differ collaborative learning emerge spontaneously in these plat-
significantly and are often vague or subject to interpretation. forms. As the work is done in the context of the PRAISE
We thus needed a theory that unified the different theories project2 , a social media platform for music learning, the
and was applicable to the online, computerised world as well. Music Stack Exchange data set was chosen.
Not the least, it had to be easily operationalisable. A re-
view of the literature brought us to the work done by Pierre Stack Exchange provides an open API, from which all data
Dillenbourg [7] that perfectly suited our requirements. Dil- can be exported. The data set consisted of 2400 questions,
lenbourg takes a broad view on the subject and argues that 1500 active members and 1.7 million page views The plat-
collaborative learning is a situation in which two or more form is basically a forum in which anyone can ask and reply
people learn through interactions. to questions. As a means of quality control, users can give
up- and down votes to questions, and answers. People can
This means that collaborative learning can not be reduced to also comment on questions and answers which is actually
one single mechanism: just like people do not learn because some kind of meta-discussion in which feedback on relevance,
they are individual but rather because the activities they terminology, etc... is given. In the following paragraphs, the
perform trigger learning mechanisms, people don’t learn col- criteria listed in Table 1 will be studied in more detail.
laboratively because they are together. Rather, the interac-
tions between the peers create activities (explanation, mu- 3.1 Symmetry of action
tual regulation,...) that trigger cognitive learning mecha- Symmetry of action expresses the extent to which the same
nisms (elicitation, internalisation, ...) [7]. range of actions is allowed by the different users. Stack Ex-
change employs a system of so-called privileges, attributed
For these processes to be effective, some requirements need according to your reputation3 . These privileges are generally
to be fulfilled. A subset was extracted that could be mea- connected to moderation rights, rather than with the actions
sured numerically, albeit indirectly, using the information of asking and replying to questions – unless you have a neg-
available in our data set (summarized in Table 1). In the ative reputation. The fact that users can exert the same
next section we will have a closer look at these indicators. actions, does not imply that this also actually the case. An
analysis of the distribution of the ratio of answers over the
2.2 Indicators number of questions, reveals that we can roughly discrimi-
Dillenbourg discriminates three important aspects for col- nate three kinds of users, based upon their activity profile:
laborative learning to be effective and characterises situa-
tions, interactions and processes as collaborative if they fulfil • Silent users (62% of the registered users) that never
the following criteria: answer, e.g. users that don’t register or register but
do not ask questions nor reply to them;
• Peers are more or less at the same level, have a common • Regular users (37% of registered users) that give roughly
goal and work together ; as much as answers as they ask questions, that is, two
• Peers communicate interactively, in a synchronous and on average;
negotiable manner ; • Super-users (<1% of the registered users), these are
• Peers apply mechanisms like internalisation, appropri- ’hubs’ that give at least 40x more answers than they
ation and mutual modelling. ask questions.
The largest part (96%) of regular users, ask less than five
These high-level criteria have been refined by Dillenbourg
questions, and 76% even asks only one question: there are no
into more detailed conditions for collaborative learning, of
’parasite’ users between the regular users that ask question
which a subset has been summarised in Table 1. Each corre-
but do not answer. From the other side, only 8 ’expert’
sponding indicator provides indirect circumstantial evidence
super-users (0.5% of the community) were responsible for
for each criterion, as our analysis was limited by the data
answering 25% of the questions. Above findings indicate
available in the Stack Exchange. Nevertheless, as we will
that the symmetry in action is highly skewed because
see, they give useful insight in the formation and dynamics
of a small group of ’super-users’ and a large group
of open online collaborative communities for learning.
of ’silent users’.
2
The research in this paper can be seen as an extension of pre- http://www.iiia.csic.es/praise/
3
vious research in Educational Data Mining, that measured http://stackoverflow.com/help/privileges
Aspect Criterion Indicator
Situation Symmetry of action Ratio of answers and questions per user
Symmetry of knowledge Scale-freeness of the user interaction graph
Symmetry of status Distribution of reputation within the community
Interactions Synchronous Response times of answering to questions
Division of labour Distribution of questions and answers in the community
Table 1: Criteria of collaborative learning according to Dillenbourg, with corresponding indicators. The
indirect nature of the indicators stems from the fact that only meta data was available from the Stack
Exchange data set, and that the criteria in general are very hard to measure quantitatively.
Figuring the knowledge of the members directly is quite an
impossible task to perform, especially in a broad and open-
ended domain like music. To assess symmetry of knowledge,
however, one could argue that if everyone in the Stack Ex-
change music learner’s community has more or less the same
expertise, then, on average, anyone would answer questions
asked by anyone.
In other words, there would be no particular hierarchy in
answering, rather the network of interaction would be ”ran-
dom” and not scale-free. Another way to put this, is to state
that no hubs of people would exist that answer significantly
more questions than others. A network is called scale-free if
the degree distribution follows a power law[2]:
P (k) ∼ k−γ (1)
with P (k) being the fraction of nodes that have a degree k,
Figure 2: Users tend to ask more questions in the and γ a constant typically between 2 and 3. Figure 1 reveals
beginning when signing up, and start answering as a power-law relationship, with exception this special group
they have been around some time. of ”super-users”. Above findings therefore suggest that sym-
metry of knowledge is not observed.
3.2 Symmetry of status 3.4 Division of labour
Stack Exchange employs a reputation system by which mem- As pointed out before, a small group of super users answer
bers get rewarded or punished if a peer up- or down votes vastly more questions than they ask: a group of 21 users
your answer or question, when your answer gets ’accepted’, answered half the questions. This is clearly not a balanced
etc... situation in which the total labour of answering questions,
is equally distributed. Figure 2 shows the relative timing of
We would expect a ”healthy” collaborative community to when users ask and respond to questions over their lifetime.
have a strong correlation between reputation and the time
a user has been around on the platform: as users spend Users tend to ask questions in the beginning (a visit to the
more time on the platform, their reputation builds up. An site probably triggered by an urgent need to get a question
inquiry into the Stack Exchange music data set, however, resolved), but start answering more uniformly after a while.
reveals only a correlation of 0.23 between reputation and The graph also indicates that engagement is largest in the
”time around”. We could thus conclude that there is some beginning. This information is relevant when developing
odd kind of symmetry, in the sense that no one really platforms with a pedagogical purposes: users probably
builds up reputation. need to be ”bootstrapped”, allowing them to give
lesser answers and ask more questions in the begin-
3.3 Symmetry of knowledge ning, so they get ”locked into” the platform.
Traditionally, these reputation systems are believed to make
a good indicator for the knowledge a user possesses. How- Note that a relative plot was preferred, in which the x-axis
ever, there are some problems with this reasoning: indicates the % of the lifetime, 0% being the moment of
signing up, and 100% the date the data set was obtained. It
allowed us to grasp the details of both users that had just
• Knowledge is not a uni-dimensional measure, but is signed up, as well as users that have been active for a long
connected to a (sub) domain of expertise; time (especially as the rate of signing up is probably not
constant but increases with time).
• Someone’s reputation keeps on increasing, even with-
out activity: there is a bias towards old posts and
members; 3.5 Synchronous feedback
To keep people engaged in an activity, according to the ”the-
• There is a bias towards ”easy answerable questions”. ory of flow” [5], immediate feedback is necessary. In the case
collaborative communities. From the other side, their inter-
ventions may bootstrap ”young” forums.
4.1.3 Strong preference for "liking"
The dataset revealed a very strong preference for voting up
rather than down: only two users gave more down votes than
up votes and of all the people that have ever cast a down vote
(72 users out of the roughly 1500 active users), 80% gave
more than five times as much up-votes in return. 80% of
the questions had no down vote, compared to less than 10%
without up-vote. Figure 3 shows the distribution of up- and
down-votes. This effect was even more pronounced in the
answers: the number of down-votes is typically zero or very
small, whereas the up-votes reach a maximum at about 3 up-
votes, then slowly attenuates. A further analysis of questions
with more down than up-votes, revealed that these questions
where either off-topic (40%), too vague, broad or specific
Figure 3: Users tend to give much more up-votes (35%), not real questions (10%) or Duplicate questions (8%).
than down-votes to questions. Generally speaking,
down-voting is only used to remove off-topic, dupli-
cate questions or questions that are either too spe- 4.2 Suggestions
cific or broad. 4.2.1 Sub-communities
Allowing users to organise themselves in smaller active sub-
of the music Stack Exchange platform, 68% of the questions communities with common or similar learning goals, may
received an answer within the day, and 20% even within the prove an elegant solution to manage or exploit the variety
hour. This may seem odd, but closer inspection reveals that in expertise of the users. Also, the concept of reputation
– once again – this is due to the small-group of ”super-users” would make more sense. A similar idea was proposed by
that are very engaged. Santos [13].
4. CRITICAL DISCUSSION 4.2.2 Knowledge construction
Based upon the analysis done in the previous section, some Good feedback should provoke critical thinking by asking
critical remarks and suggestions are offered to improve the sensible questions, provide a clue to ”what’s next” and al-
pedagogical nature and collaborative learning low to construct knowledge through scaffolding and coupling
back to acquired knowledge. Though the concept of freely
4.1 Remarks asking questions is very accessible, the content stays rather
ad-hoc and unstructured. A way to organise and link dif-
4.1.1 Limited to no instructional design ferent questions in order to guide learners would be very
The data set on Stack Exchange music’s forum, is an amal- useful.
gam of questions (1) with different levels of granularity, typ-
ically with a small scope, (2) on a wide range of topics,
for learners (3) with different learning goals and (4) dif- 4.2.3 Collaborative interfaces
ferent levels of expertise. The activities are not designed In the modern ages of web technology, users could benefit
to elicit collaborative learning, and as the data is unstruc- from a collaborative interface in which knowledge is con-
tured, without sufficient scaffolding of the learning content structed together, in a way similar to for example Google
(e.g. through hyper-linking), it is no natural fit for learning Docs where one single entity is shared by all users. So, rather
but rather provides ad-hoc answers to appease short- than preserving the strict question/answer or learner/teacher
term narrow personal learning goals. dichotomy, one would go for a situation in which knowledge
– not only answers but also questions – is constructed live
4.1.2 A heterogeneous community in an interactive way.
Above remarks wouldn’t be so problematic for collaborative
learning, if proficient communities existed within the Stack 5. CONCLUSIONS
Exchange platform that had more or less the same goals, ex- In this paper, the case for collaborative learning in open-
pertise and engagement. In the current case, there’s a risk ended auto-didact Q&A environments like Stack Exchange
of frustration and boredom in expert users that don’t see is investigated. Based upon the criteria put forward by Dil-
their questions answered and who have to answer straight- lenbourg, we can state that though there are technically no
forward questions. For novice members, on the other hand, hindrances towards collaborative learning, the nature and dy-
their learning remains limited because they do not get suf- namics of the community that organically form on Stack Ex-
ficient guidance and do not really construct knowledge. change, do not support the case for collaborative learning.
Although the group of super-users makes sure that questions It was observed that the symmetry of action was distorted
get answered quickly and perform the largest part of mod- due to a small group of ”super-users” that answered the ma-
eration, they are potentially harmful to collaborative learn- jority of questions and a large group of ”silent users” that
ing as they distort the natural formation and dynamics of do not really interact with the platform. Inspection of the
degree distribution of the user interactions reveals that the 2004.
community network is scale-free, which means that symme- [12] C. Romero and S. Ventura. Educational data mining:
try of knowledge is very unlikely. The reputation system A survey from 1995 to 2005. Expert Systems with
seems insufficient as a measure of expertise and a strange Applications, 33(1):135–146, 2007.
kind of symmetry of status is observed, in the sense that no [13] O. C. Santos, A. Rodrı́guez, E. Gaudioso, and J. G.
one really builds up reputation, except for a small group of Boticario. Helping the tutor to manage a collaborative
users. task in a web-based learning environment. In
AIED2003 Supplementary Proceedings, volume 4,
Lastly, the limited possibilities to instructional design, elic- pages 153–162, 2003.
its short-term narrow and personal learning goals. Also, the [14] M. Sharples, P. McAndrew, M. Weller, R. Ferguson,
very heterogeneous nature of the community is not favourable E. FitzGerald, T. Hirst, and M. Gaved. Open
for learning. Suggestions were made to adapt these inter- university: Innovating pedagogy. 2013.
esting and popular platforms to learning, like creating sub- [15] V. Strauss. Are MOOCs already over?
communities with common learning goals, extend the possi- http://www.washingtonpost.com/blogs/answer-
bilities for organising and structuring the content and em- sheet/wp/2013/12/12/are-moocs-already-over/,
ploy collaborative interfaces. (December 12), 2013.
As future work, these results should be validated by means
of other communities on Stack Exchange as well, and on
different modes of feedback, rather than only text-based.
6. ACKNOWLEDGEMENTS
This research has been supported by the EU FP7 PRAISE
project #318770.
7. REFERENCES
[1] A. R. Anaya and J. G. Boticario. A data mining
approach to reveal representative collaboration
indicators in open collaboration frameworks.
International Working Group on Educational Data
Mining, 2009.
[2] A.-L. Barabási and R. Albert. Emergence of scaling in
random networks. Science, 286(5439):509–512, 1999.
[3] C. G. Brinton, C. Mung, S. Jain, H. Lam, Z. Liu, and
F. Ming Fai Wong. Learning about social learning in
MOOCs: From statistical analysis to generative
model. arxiv.org, abs/1312.2159, 2013.
[4] M. A. Chatti, M. Jarke, and D. Frosch-Wilke. The
future of e-learning: a shift to knowledge networking
and social software. International journal of knowledge
and learning, 3(4):404–420, 2007.
[5] M. Csikszentmihalyi. The Evolving Self: A Psychology
for the Third Millennium. Harper Collins, New York,
1993.
[6] L. Cuban and L. Cuban. Oversold and underused:
Computers in the classroom. Harvard University
Press, 2009.
[7] P. Dillenbourg et al. Collaborative-learning: Cognitive
and computational approaches. Technical report,
Elsevier, 1999.
[8] J. Fenn and M. Raskino. Mastering the Hype Cycle:
How to Choose the Right Innovation at the Right
Time. Harvard Business Press, 2008.
[9] J. Hattie. Visible learning: A synthesis of over 800
meta-analyses relating to achievement. Routledge,
2009.
[10] T. Lewin. After setbacks, online courses are rethought.
New York Times, (December 11), 2013.
[11] T. P. Padilha, L. M. Almeida, and J. B. Alves. Mining
techniques for models of collaborative learning. In
Designing Computational Models of Collaborative
Learning Interaction, workshop at. ITS, pages 89–94,