An Appraisal of a Collaboration-Metric Model based on Text Discourse

Abstract

This paper presents a more in-depth analysis based on discourse of the collaboration-metric model, Word-Count/Gini-coefficient measure of symmetry (WC/GCMS) which was introduced in [3]. We discuss the validity of the model in regards to how well it represents what happens in the groups' discourse content. We discuss the application and implication of WC/GCMS based on the goal to incorporate collaborative learning and its cognitive advantages to E-Learning environments.

Keywords

• Group discourse
• online group
• E-learning
• collaboration-metrics

Introduction and related work

Online learning provides access to education for millions of learners through many environments offered by Universities and other organizations world-wide (e.g. Mass Open Online Courses). This motivates Computer Supported Collaborative Learning (CSCL) research towards leveraging the cognitive advantages of collaboration [ 5, 16, 24, 26, 32 ] for online learning, as it is preeminent in traditional classroom settings. Online collaboration however has two major concerns: (i) media richness- the degree to which a virtual medium conveys the immediacy of face-to-face (F2F) conversation [ 28 ] and (ii) social presence- communication that fosters immediate interaction/feedback and permits people to communicate with multiple senses (e.g. verbal and visual clues) [ 28 ]. Media richness and online social presence are inter-dependent; the richer a media, the more social presence it conveys during online collaboration. For example, there is more social presence in teleconferencing which conveys both the verbal and real-time image of collaborators compared to email exchange or other text-based conversation media. However, implementing robust media that conveys both verbal and visual clues for online programs comes with costs and complexity of deployment, which may inhibit the integration of group learning. Also, a group that is media enabled with verbal and visual interaction is most times synchronous; this excludes the time flexibility to participate, to think, and to search for extra information, and to contribute in a group discussion, which comes with on-line collaboration (e.g. in asynchronous text-based media) [ 11, 25, 27 ]. Text-based group media is cost efficient and prospectively effective for online collaborative learning; De Wever et al. [ 11 ] posit that, textbased discussion makes individual contributions more explicit and provides a better reflection of the process of collaboration for both researchers and instructors. It is a good data source to evaluate both collaboration and individual participation within group [ 18, 23 ].

Online learners who interact via a text-based environment strive to maximize the social presence in the media [ 28 ]; a comparative study between text-based & F2F verbal discourse attests to similarities in both, despite a lack of facial expressions and gestures in the former [ 7 ]. Features such as frequency of agreement or disagreement, use of negative affect terms and frequency of punctuation use in text contributions reveal emotions of discussants, which is similar to facial expressions and gestures in F2F verbal discussions [ 14 ].

Online discussions provide evidence of collaboration as seen in F2F, although it has different representations in both; text or verbal information containing the same content will provide the same emotional or cognitive effect although processed differently [ 10 ]. Soller [ 31 ] corroborated this position stating that learners pose the natural inclination to adapt and maximize social presence when they use text-based media to interact; she however suggested that CSCL research needs to design a new adaptive method to support interaction in this environment. 1.1

Measure of Collaboration with Text Discourse

The instructors' view of collaboration via textual interaction had depended on a review of the transcripts of the groups' discourse [ 12 ]; analysis about how well groups have collaborated is possible only after the Joint Problem Solving (JPS) process has ended and any feedback from such analysis is useful to moderate future group work. In order to accord online groups the kind of real-time support obtainable in F2F groups, we require a real-time approach to view what goes on during online JPS.

Schwarz and Asterhan [ 29 ] explored this objective and presented a real-time view of group interaction using the social network of the connections between the activities within the group (see Fig. 1 a); the measure of participation by members was visualized with a bar charts, each bar representing different variables of activities involved in the task, for each group member (see Fig. 1 b).

Our model contributes to existing knowledge by providing a simpler, scalable and generically adaptable computational mechanism that informs the level of collaboration during online JPS; applicable in real-time. In the following sections, we assess submissions from existing work about indicators and metrics of collaboration. The ideas from these studies are aggregated and extrapolated for text-based online interaction. Additionally, we present the rationale and mathematical relation that inform the Wordcount/Gini-coefficient measure of symmetry (WC/GCMS) model [ 3 ]. Finally, we discuss the method we used to validate this model by triangulating qualitative assessment of the groups' discourse transcript, with the output of the WC/GCMS model. We conclude with a discussion on the implications of the model in regard to a design framework for sustainable and effective online group learning environments. (a) Group collaboration measure with Social network Fig. 1: E-moderation of online group collaboration, Schwarz and Asterhan [ 28 ] (b) Individual group members' activities 1.2

Indicators and Metrics of Collaboration within Groups

Much work has been done to identify indicators of collaboration during group JPS; more of these studies explored F2F or co-located groups. For example, Martinez et al. [ 19 ] mined the frequent sequential pattern of the log trace of groups' JPS activities around a table-top application to categorize groups into high achieving and low achieving. In a similar study, Martinez et al. [ 20 ] proposed an approach to automatically distinguish between groups that engaged in a collaborative or non-collaborative activity during JPS.

Meier et al. [ 22 ] presented a rating scheme to quantify collaboration, Cukurova et al. [ 9 ] explored how group synchrony and individual accountability, equality and intraindividual variability informs good collaboration. The consensus found in these existing studies in regards to indicators of collaboration during JPS are: (i) Symmetry of contribution (ii) Volume of contribution (iii) Connectivity/links between contributions of different group members and (iv) the quality of contributions with respect to context of JPS. In the next section, we will discuss how this informed the WC/GCMS collaboration metric model. 2

Word-count/gini-coefficient measure of symmetry

The components of the WC/GCMS presented in [ 3 ] are given by: WC/GCMS metric of collaboration is given by: = ( )

(1) tion level i.e. : measures the collaboration within a group.

.

assuming that this volume informs the quality of the JPS process [ 21 ]. ( ): represents the volume of activities/texts that the group generate during JPS; : represents the symmetry of the activities within the group and is based on the ginicoefficient measure of symmetry. It ranges from 0-1; 0 being perfect symmetry and 1 asymmetry. Assuming that symmetry of JPS activities is an indication of group collaboration, the numerical value of the is inversely proportional to the group collaborathus given by equation 2.

Volume of group activities: A member i within a group contributes textual Statements ⃗⃗⃗ , ⃗⃗⃗2, … , ⃗⃗⃗⃗⃗ , at time intervals during JPS. All text contributions by member i is a collection of statements, ⃗⃗⃗ . The measure of contribution during JPS by member i, is = ∑ =1 ⃗⃗⃗ , ℎ = ⃗⃗⃗ (2)

Hence, within a group of 4 members, we have contributions 1 , 2 , 3 , 4 . Considering that a non-collaborating member may contribute very little and an extrovert may provide an excessively high text contribution, we represent the group activity volume measure, ( ) with the median in the group: ( ) = ( 1 , 2 , 3 , 4 ) (3) Symmetry of activity within group: This is based on the gini-coefficient measure of symmetry adapted from [ 20 ]. Firstly, we compute the mean number of contributions by group members (equation 4a), then the symmetry of contributions within the group (equation 4b): = 1

∑ =1 | |

= ∑ =1 ∑ =1 | − | 2 2 (4a) (4b) Next, we describe the output of WC/GCMS with data from 5 groups. The study procedure, a brief discussion about the model and findings was presented in [ 3 ]. Here we provide an expanded and more exploratory discussion on the validity of WC/GCMS for quantifying collaboration with text-based discourse.

Text-based discourse data source

The text-based discourse of 5 groups was collected in a study by Adeniran et al. [ 3 ]. The groups were formed from a convenience sample of undergraduate/postgraduate students. Each group had 4 members: (Group 1) 3 male, 1 female, all aged 18-25; (Group 2) 3 male, 1 non-disclosed; all 18-25; (Group 3) 2 male, 2 female; all 18-25; (Group 4) 4 male, all 26-35; (Group 5) 4 male, 3 26-35, 1 36-45. In the study, the groups solved a joint task, the task [ 1 ], is an open-ended problem without clear cut answers as recommended by [ 8 ] for group work. JPS was via a text-based chatroom designed for the study [ 3 ]. Discourse is collected in a database; contributions are time-stamped, and uniquely but anonymously identified with the contributor. This data serves as input for our WC/GCMS model, which tells how well the groups have collaborated relatively. 3.1

Visualization with WC/GCMS metric

Figure 2a shows the relative measure of collaboration between groups based on total discourse, Figure 3 simulates a real-time view of this measure during JPS. Figures 2 and 3 can inform a remote teacher about which group is collaborating less well. We did not define a measure for a collaborative or non-collaborative group; WC/GCMS depends on the comparison between the groups to determine which group needs attention most, at a given time during JPS.

The measure of individuals' participation within the group (shown in Figure 2 b) provides a hint about non-participating members; for example, M3 in group 1 or M4 in group 4. The components of WC/GCMS i.e. ( ) & , are viewed in real-time as shown in Figure 4; this provides information about the groups' JPS process as discussed below. Figure 4a visualizes ( ), we can observe a higher ripple in the line representing Groups 3 and 5, showing that the symmetry of contribution within the group changes more rapidly during JPS. It is a sign of high frequency of contribution within the groups which can be hypothesized as an indication of members' interest in the discussion or a relatively higher knowledge about the task (i.e. the members have more to contribute). On the contrary, the lines representing Groups 1 and 2 are smoother and the Group 4 line the smoothest, indicating that the participation rate in these groups is lower.

From Figure 4 b, which visualizes , we can observe that the volume of text contribution in Groups 3 and 5 is higher and increases steadily during their JPS discussion, corroborating that if the contribution rate is higher, then the contribution volume will be higher. This also confirms the position of Maldonado [ 19 ], that a high verbal activity is an indication of collaboration; in our context: high textual contribution indicates collaboration in a text-based discourse. 4

Validation: WC/GCMS output versus Qualitative Assessment of Discourse

To validate the WC/GCMS's visualizations, we use the groups' discourse transcripts to make a comparative analysis with the inferences from the visualization. Contributions that aid collaboration were conjectured to assume one of the following activity-states: task coordination, acknowledgement, request, inform, argue and motivate [ 2, 30 ]. We assess groups' discourse to determine how much evidence of these collaborative activity-states exist therein and compare these between the groups. Firstly, at the group discourse start, there is evidence of initial coordination within Groups 3 and 5, indicating an interest and enthusiasm to participate; contrary to our observation in the discourse of Groups 1, 2, and 4. Participants in the latter groups did not make any effort to familiarize with the task nor with group members; they went ahead to give suggested solutions (See Table 1).

Secondly, there is evidence of informed argument and planning in Groups 1,2, and 4, where the contributions were mostly erroneous. These groups suggested solutions with blind acceptance and acknowledgement. Most contributions from Groups 1,2, and 4 are similar to what Webb [ 33 ] refers to as “giving and receiving non-elaborated help” (i.e. unexplained solutions to the JPS task). Such contributions during group learning provide no cognitive benefit to the giver of the information nor to other members. The extract from Group 5 discourse particularly contains cognitive elaboration, which is posited to be an evidence of collaboration [ 33 ]. The relative level of collaboration between groups shown by WC/GCMS (as shown in Figures 2a and 3) is thus justified. (a) Group measure (b) Individual Activity Measure within group

Fig. 3: Simulated Real-time view of collaboration level

1 measure between groups (a)

(b) ( ) measure between groups The discourse transcript shows that member "charis" in Group 3 and "unknown" in Group 4 did not participate relatively well within their respective groups. This explains the low bar for M3 in Group 2 and M4 in Group 4 as shown in 2 and further validates the WC/GCMS output.

Quality of contribution and knowledge about task: Assessment of discourse of

Groups 2 and 5, shows evidence of information sharing, of new knowledge, and suggestions based on logical reasoning about the task. Their discussion conveyed knowledge of context (the moon environment) and transfer of knowledge (see Table 2). This kind of elaborated discussion indicates participants' socializing during small group discussion as posited by [ 15 ]. On the contrary, the discourse of Groups 1,2, and 3 lacks such knowledge-based interaction; this inhibits socialization within the groups [ 15 ]. In line with the Vygotskian perspective as mentioned in [ 33 ] that collaboration provides cognitive benefits when “a more expert member helps less-expert ones”. Studies have also shown that there is a knowledge level threshold for a task that can foster optimum collaboration within groups; below it, a group will not attempt a solution at all or suggest unexplained erroneous solutions which hinders collaboration and cognition [ 4 ]. Group 1 The major contributions of this paper are: first, based on literature, we argue that a textbased media is efficient and can be optimized to maximize social presence within an online group [ 7, 11, 25, 27, 28 ]. Second, existing studies proposed measures of collaboration that use the text discourse transcript, providing an analysis after the discourse has been completed [ 6, 13, 17 ], whilst WC/GCMS is intended to be used in a real-time group monitoring dashboard for a remote teacher. Third, we present an explicit comparative analysis of the WC/GCMS metric output with an assessment of the groups' discourse, to validate the model's sensitivity in regards to quantifying text-based group collaboration. We posit that WC/GCMS can provide simple, easily interpretable graphical outputs is upgradeable (to capture verbal and visual clues when using richer interaction media) and generic (can be extrapolated to the collaboration context).

Whilst the indicators of collaboration exceed the characteristics of the text discourse content used in this paper, WC/GCMS is sensitive enough to serve as a proxy-effective metric of collaboration and participation within online groups. We plan to run a larger scale study to further investigate the indicators, factors and models presented. We will also investigate the use of our metrics and visualizations to provide real-time feedback to learners to scaffold collaboration, and measure both quantitatively and qualitatively the effect of such feedback on JPS. We further aim to develop algorithms for a computer agent (taking our models as input) to stimulate participation and consequently scaffold collaboration. 6 References

References

1. Moon exercise. humber.ca/centreforteachingandlearning/assets/files/pdfs/MoonExercise.pdf.

2. Adeniran , A. Investigating feedback support to enhance collaboration within groups in computer supported collaborative learning . In AIED Conference , pp. 487 - 492 . Springer. ( 2018 ).

3. Adeniran , A. , Mathoff , J. , Beacham , N. Model-based characterisation of discourse content: An evaluation of collaboration within online groups . AIED , ( 2019 ).

4. Adeniran , A. , Masthoff , J. , Beacham , N. Analyzing groups problem-solving process to characterize collaboration within groups . In CEUR Workshop Proceedings . ( 2018 ).

5. Alfonseca , E. , Carro , R.M. , Martin , E. , Ortigosa , A. , Paredes , P. The impact of learning styles on student grouping for collaborative learning: a case study . User Modeling and UserAdapted Interaction , 16 ( 3 ), pp. 377 - 401 . ( 2006 ).

6. Andresen , M.A.. Asynchronous discussion forums: Success factors, outcomes, assessments, and limitations . J. Educ. Techn. & Society , 12 ( 1 ), pp. 249 - 257 .( 2009 ).

7. Bromme , R. , Hesse , F.W. , Spada , H. Barriers and biases in computer-mediated knowledge communication: and how they may be overcome . Springer. ( 2006 ).

8. Cohen , E.G. , Brody , C.M. , Sapon-Shevin , M. Teaching cooperative learning: The challenge for teacher education . Suny Press. ( 2004 ).

9. Cukurova , M. , Luckin , R. , Mavrikis , M. , Millan , E.. Machine and human observable differences in groups collaborative problem-solving behaviours . In European Conference on Technology Enhanced Learning , pp. 17 - 29 . Springer. ( 2017 ).

10. Curtis , D.D. , Lawson , M.J. Exploring collaborative online learning . Journal of Asynchronous learning networks , 5 ( 1 ), pp. 21 - 34 .( 2001 ).

11. De Wever , B. , Schellens , T. , Valcke , M. , Van Keer , H. Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review . Computers & education , 46 ( 1 ), pp. 6 - 28 ( 2006 ).

12. Dringus , L.P. , Ellis , T. Using data mining as a strategy for assessing asynchronous discussion forums . Computers & Education , 45 ( 1 ), pp. 141 - 160 . ( 2005 ).

13. Driscoll , A. , Jicha , K. , Hunt , A.N. , Tichavsky , L. , Thompson , G. Can online courses deliver in-class results? A comparison of student performance and satisfaction in an online versus a face-to-face introductory sociology course . Teaching Sociology , 40 ( 4 ), pp. 312 - 331 . ( 2012 ).

14. Hancock , J.T. , Landrigan , C. , Silver , C.. Expressing emotion in text-based communication . In SIGCHI conf. on Human factors in comp. Sys. ACM . ( 2007 ).

15. Ho , M.-c. Academic discourse socialization through small-group discussions . System , 39 ( 4 ), pp. 437 - 450 .( 2011 ).

16. Liu , S. , Joy , M. , Griffiths , N. Incorporating learning styles in a computer-supported collaborative learning model . ( 2008 ).

17. Loncar , M. , Barrett , N.E. , Liu , G.- Z. Towards the refinement of forum and asynchronous online discussion in educational contexts worldwide: Trends and investigative approaches within a dominant research paradigm . Computers & Education , 73, pp. 93 - 110 . ( 2014 ).

18. Macdonald , J. Assessing online collaborative learning: process and product . Computers & Education , 40 ( 4 ), pp. 377 - 391 . ( 2003 ).

19. Maldonado , R.M. , Yacef , K. , Kay , J. , Kharrufa , A. , Al-Qaraghuli , A. Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop . In Educational Data Mining 2011 , ( 2010 ).

20. Martinez , R. , Wallace , J. , Kay , J. , Yacef , K. Modelling and identifying collaborative situations in a collocated multi-display groupware setting . In Artificial Intelligence in Education , pp. 196 - 204 . Springer.( 2011 ).

21. Martinez-Maldonado , R. , Dimitriadis , Y. , Martinez-Mones , A. , Kay , J. , Yacef , K. Capturing and analyzing verbal and physical collaborative learning interactions at an enriched interactive tabletop . I.J of CSCL , 8 ( 4 ), pp. 455 - 485 ( 2013 ).

22. Meier , A. , Spada , H. , Rummel , N. A rating scheme for assessing the quality of computersupported collaboration processes . I.J. of CSCL , 2 ( 1 ), pp. 63 - 86 , ( 2007 ).

23. Meyer, K.A. Evaluating online discussions: Four different frames of analysis . Journal of Asynchronous Learning Networks , 8 ( 2 ), pp. 101 - 114 ( 2004 ).

24. Newman , D.R. , Webb , B. , Cochrane , C. A content analysis method to measure critical thinking in face-to-face and computer supported group learning . Interpersonal Computing and Technology , 3 ( 2 ), pp. 56 - 77 . ( 1995 ).

25. Pena-Shaff , J. , Nicholls , C. Analyzing student interactions and meaning construction in computer bulletin board discussions . Comp. & Educ ., 42 ( 3 ), pp. 243 - 265 .( 2004 ).

26. Savery , J.R. , Duffy , T.M.. Problem based learning: An instructional model and its constructivist framework . Educational technology , 35 ( 5 ), pp. 31 - 38 . ( 1995 ).

27. Schellens , T. , Valcke , M.. Collaborative learning in asynchronous discussion groups: What about the impact on cognitive processing? Computers in Human behavior , 21 ( 6 ), pp. 957 - 975 . ( 2005 ).

28. Schneider , S. , Kerwin , J. , Frechtling , J. , Vivari , B. Characteristics of the discussion in online and face-to-face focus groups . Soc.Sc.Comp.Rev., 20 , pp. 31 - 42 . ( 2002 ).

29. Schwarz , B.B. , Asterhan , C.S.. E-moderation of synchronous discussions in educational settings: A nascent practice . J. of the Learn. Sc. , 20 ( 3 ), pp. 395 - 442 .( 2011 ).

30. Soller , A. Supporting social interaction in an intelligent collaborative learning system . Int. JAIED , 12 , pp. 40 - 62 .( 2001 ).

31. Soller , A. , Martinez , A. , Jermann , P. , Muehlenbrock , M. From mirroring to guiding: A review of state of the art technology for supporting collaborative learning . Int. JAIED , 15 ( 4 ), pp. 261 - 290 .( 2005 ).

32. Suh , H. Lee , L.. Collaborative learning agent for promoting group interaction . ETRI journal , 28 ( 4 ), pp. 461 - 474 . ( 2006 ).

33. Webb , N.M. The teacher's role in promoting collaborative dialogue in the classroom . British Journal of Educational Psychology , 79 ( 1 ), pp. 1 - 28 ( 2009 ).