Computer-supported collaborative learning (CSCL) is a method in education where the students work together on a task while the teacher takes on the role of a coach who --- aided by information technology --- scaffolds their progress and allows them to discover a solution on their own. CSCL exercises are often run following a script, which breaks the activity in a set number of steps to facilitate productive collaboration. This makes it easier for the teacher to orchestrate the exercise --- controlling the flow of the activity and attending to the students' needs as they arise. Teacher-facing dashboards are often used to enable orchestration by providing information about and controls to manipulate the state of the activity. Our research is centered on analyzing whether teachers and students can benefit from visualizing epistemic information, i.e. learning analytics data derived from examining the content of students' input. We expect that giving teachers access to epistemic information will facilitate orchestration, reduce the cognitive load required to oversee a CSCL activity, and create the opportunity for teacher-led debriefing --- a technique used by educators to make students reflect on the activity they engaged in and thus help them get a deeper understanding of the content that was covered. We also expect that this will ultimately have a positive impact on students' learning gains. We will extend the dashboard of “PyramidApp” --- a software tool that implements the CSCL “Pyramid” script --- with epistemic information to test our hypothesis. Subsequently, we will analyze how our findings transfer to other CSCL scripts and tools. We thus hope to contribute to the existing knowledge of how learning analytics data can successfully be employed in a CSCL context. We will follow the design-based research method which emphasizes cooperation with teachers and aims to test and apply interventions in realistic scenarios.
The idea of using computers in education dates back to the 1960s [1]. What was initially a fringe approach has become more and more common and shows no signs of slowing down [2]. Using this technology for teaching and learning has great appeal for both educational institutions and researchers. Subsequently, the field of technology enhanced learning (TEL) emerged and with it a plethora of studies. This is particularly evident since the beginning of the Corona-crisis, as many institutions were forced to conduct at least part of their lessons online [3]. While the actual impact of using technology for education has been criticized, the endeavor is still viewed as promising [4]. Another frequent criticism is that the results from the lab don't translate to the reality of the classroom --- or that they never make it there in the first place [5]. However, with further development comes further progress: Many researchers place an emphasis on developing and testing their interventions in realistic scenarios and are adding to the growing amount of evidence that enhancing learning through technology is not only possible, but worthwhile.
Learning analytics is a fast-growing area of TEL and is defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” [6]. Typically, learning analytics data is automatically collected and processed by machines. One benefit of this approach is that large amounts of data can be handled and made use of --- potentially in real time.
Another relatively modern trend in education is collaborative learning [7]. This means that the students will work together on a task and try to find a solution, rather than being directly told how to get there. The role of the teacher becomes that of a coach, who scaffolds the students’ progress rather than giving them the correct answers / techniques outright. This is also referred to as “guided participation”. There are many forms of collaborative learning, but the most effective approaches seem to be those that put a focus on intrinsic incentives (e.g. the student’s natural search for knowledge, competence, and stimulating communication) and frame the task in a way that emphasizes collaboration rather than competition. The positive effects of this method are most notable when looking at conceptual insights that are acquired by the students --- something that is notoriously difficult to teach. However, collaborative learning is no more successful than direct instruction when teaching formulas, procedures, or the application of an existing model.
Computer supported collaborative learning
(CSCL) is the combination of collaborative
learning and technology enhanced learning [
While a CSCL script gives the task a clear
structure --- with all the upsides that such a
guide brings ---, technology can help make its
implementation more flexible to its specific
context. This is described by the notion of
orchestration: The teacher needs to respond to
the students' needs as they arise and adapt the
exercise to the current situation [
There have been several implementations of teacher-facing dashboards that visualize learning analytics data. Our focus will be on the visualization of epistemic information derived from analyzing the content of the students' inputs (answers, chat messages etc.). We expect that visualizing synthesized epistemic information can reduce teacher cognitive load as it drastically reduces the amount of text a teacher has to read to follow the students' progress. Additionally, we expect this to have a positive impact on orchestration by making it easier to identify when and where to intervene, as well as to facilitate teacher-led debriefing by highlighting the most relevant student contribution for further discussion.
In teacher-led debriefing lectures, students'
answers are put into perspective and addressed
in the light of new course content. Students are
required to justify their beliefs, receive
feedback on their performance and thus get to
structure their newly acquired knowledge
before integrating it into a theoretical
framework [
We are basing the assumptions on the impact of our intervention in part on a study similar to our own, in which content analysis data was added to a teacher-facing dashboard to support the CSCL activity EthicApp [18]. The data visualizations were derived using natural language processing (NLP) techniques on student data, rank ordering comments by relevance and comparing the work groups by how homogeneous their members opinions are. Results were promising: Experts judged about 80% of the selected comments as viable, which indicates that this approach could be useful in reducing the number of comments teachers have to consider when monitoring an activity and thus reducing cognitive load.
The approach to use NLP technology to analyze students' artefacts and utterances for learning analytics is not without precedence and there are several techniques that seem promising [19, 20, 21]. One such technique is the analysis of text to gain a measure on the level of confusion and precision in the students' answers [22, 23]. Other studies showed the potential to investigate semantic similarity, sentiment, and point-of-view --- going as far as being able to gauge the degree of collaboration within a group that is working on a CSCL task [21, 24, 25].
Ultimately, we expect that the effects of our intervention will extend from the teachers to the students and have a positive impact on their learning gains.
An example of a CSCL script is the “Pyramid” (sometimes referred to as “Snowball”), which is structured as follows [26]:
The teacher will initially give a task to the students, usually to answer an open question. In the first stage, the students will each individually think about and write down their answer. In the second stage, they are presented with a selection of answers from their peers and rate these answers by what they think are the most correct and complete. In the third stage, the collaboration truly begins, as the students are assigned to groups where they discuss the previously rated answers and synthesize an answer for the group. Finally, the group answers are rated by all students and thus the class agrees on one final answer. Depending on the size of the class, stages 2 and 3 will be repeated with larger and larger groups, until a final consensus is reached.
Another example of a CSCL script is the “Jigsaw”: First, students work on their own on one of several topics. Then, expert groups get formed by grouping the students by the topic that they worked on. In these groups, the students help each other understand their topic in depth and prepare to present it to nonexperts. In the last phase, groups are formed heterogeneous by mixing students in a way that each group has at least one expert of each topic. They then take turns explaining what they are now proficient in to the non-experts until the whole group understands the entire range of topics.
PyramidApp is a software that implements the “Pyramid” script, making it easy to integrate it into a classroom lesson or online course [27, 28]. Figure 1 shows the group stage of a “Pyramid” script in PyramidApp. PyramidApp also comes with a teacher-facing dashboard, which provides information about the state of the activity and gives the teacher controls for orchestration (see Figure 2) [14].
We will initially focus on the “Pyramid” script and PyramidApp, but we are hoping to extend the research by analyzing to what extent the interventions that will be designed and evaluated are transferable to other CSCL scripts such as “Jigsaw” or “ArgueGraph” [29, 30].
To sum up, the research questions that we want to answer are the following: 1. How can teacher-oriented dashboards with learning analytics (LA) indicators based on epistemic information facilitate teacher-led debriefing in CSCL scripts? 2. How can teacher-oriented dashboards with LA indicators based on epistemic information facilitate real-time orchestration in
CSCL scripts? 3. Do teacher interventions informed by LA indicators related to epistemic information improve learning gains?
Section 1 covers the background and motivation of our questions, section 2 introduces a concrete implementation of a CSCL script that we will build upon to test our questions, section 4 lays out the methodology we will use to attempt to answer our questions, and section 5 concludes with describing what we expect the impact answering our questions will have.
Design-based research is a paradigm that
aims to bring educational research back to
where it has the most impact [
We are going to explore several approaches to gather and present epistemic information in the PyramidApp dashboard and implement the interventions in practice. We will use existing data from previous experiments with PyramidApp to analyze the feasibility of the different presentation approaches and codesign prototypes in cooperation with the stakeholders.
Following the design-based research methodology, the project will go through several cycles. Figure 3 shows the typical phases of each cycle (taken from [32]). 4.1.
In the analysis stage, we conducted a literature review and identified that providing epistemic information to the teacher during a CSCL activity could lead to improved orchestration and debriefing. We then gathered several ideas for possible ways in which epistemic information could be gathered (see Table 1) and integrated (see Table 2) into the PyramidApp dashboard. It should be mentioned that they are not mutually exclusive and we hope to be able to implement several of them simultaneously.
Before moving to the second stage (development), we will need to identify which of these options are the most promising in terms of feasibility and impact. To achieve this, we will analyze existing PyramidApp data that we have access to. This data comes from previous applications of PyramidApp in real classroom scenarios. It consists of all inputs made in the application, both from teachers (e.g. interactions with the dashboard) and students (e.g. answers and chat messages), as well as metadata such as timestamps. Some of the students’ answers have also been rated by teachers, giving us additional information that could help to automatically identify the quality of a student-submitted text. In some cases, we might also develop low-fidelity prototypes to gauge the technical feasibility of our ideas. Finally, we will create mock-up visualizations and seek feedback from teachers. This preliminary work should allow us to identify the most promising approaches and might lead us to discard or add ideas. 4.2.
In the development stage, we will now be able to make an informed decision on which and how many of the visualizations we want to implement and will begin by creating a lowfidelity, “proof-of-concept” prototype. We will seek feedback from colleagues and teachers and improve it until we have a first version that is sophisticated enough for a realistic test. 4.3.
We will then enter the testing stage, where we intent to conduct multiple within-subjects experiments running a PyramidApp activity with and without epistemic information in a realistic classroom or Massive Open Online Course (MOOC) setting. This is the phase where we collect our data: we will use the PyramidApp software to automatically log all inputs of both students and teachers during the activity (the data we analyzed in stage one was collected in the same way in the past). We will also need to keep track of what was displayed in the dashboard at any time, ask experts to rate the students' answers, and have teachers and students answer questionnaires. We will consider using a dual-task method to directly measure teacher cognitive load [35]. If necessary, we will fix errors, improve the software and conduct additional tests until we have preliminary results. 4.4.
This data will then be analyzed in the reflection stage. We will attempt to integrate the findings into our understanding of the underlying theory and identify where things went well and where there were problems. We will reflect on the impact that our intervention had by comparing it to the activities where teachers did not have access to epistemic information. We expect to see a positive impact in the form of a measurable reduction in cognitive load, increase in the ease of orchestration, facilitation of teacher-led debriefing, and student learning gains.
When considering learning gains, it has to be kept in mind that giving a correct answer does not necessarily mean that one knows what they are doing, but measuring --- or even defining --- understanding is challenging [36]. We will focus on tangible expert scores for the time being, but might incorporate alternative measurements in the future.
We will then use all the insights that we've gained to begin the second design-based research cycle. We will ask ourselves whether the data we gather and analyzed was sufficient to confirm or deny our expectations and answer our research questions. We will consider what would be necessary to extend our results to other CSCL scripts. Our considerations will let us decide whether we need to run additional experiments, formulate new research questions, or further develop our epistemic data visualizations.
Figure 4 summarizes how the first designbased research cycle looks like for this project.
Following the design-based research philosophy, the ultimate goal of our research is the application of the findings in real teaching situations in a way that improves learning gains and / or reduces the workload of the people involved.
Our expected contribution is the development of visualizations of learning analytics data based on epistemic information to reduce cognitive load, support orchestration, and facilitate debriefing of CSCL scripts. We expect that this will improve learning gains and we will directly implement and validate it for the “Pyramid” script as well as critically examine and discuss its value for other types of CSCL scripts such as “Jigsaw” or “ArgueGraph”.
The indirect influence of the research would be through the insights gained. The theory of the science of learning could be extended by getting valuable information on the effects and effectiveness of debriefing and orchestration in a CSCL context. Proving -- or disproving -- its impact can inform the direction of further research and lead to the development of successful interventions in the future.
It should not be forgotten that even a “negative” result would be significant, as it could suggest that a specific type of intervention is inferior and the time of educators is better spent elsewhere.
In this way, we hope to make a contribution to the further improvement of educational practice.
Thanks to Ishari Amarasinghe for providing me with ideas for ways in which epistemic information could be gathered and integrated into the PyramidApp dashboard.
[1] G. Paquette, Technology-based instructional design: Evolution and major trends, Handbook of Research on Educational Communications and Technology: Fourth Edition (2014) 661– 671. doi:10.1007/978-1-4614-3185-5_53. [2] H. Jeong, C. E. Hmelo-Silver, K. Jo, Ten years of computer-supported collaborative