This in-progress research study investigates middle school students' use of a wiki-based e-learning platform as a coordinating representation in the context of their guided discovery-based game design work. The study aims to (a) consider/validate the quality of wiki trace data and Google Analytics page read data as a source of insight for research; (b) describe group activity patterns using wiki trace data and Google Analytics page read data; (c) investigate relationships between measured activity patterns and student learning outcomes; (d) develop appropriate algorithms for early detection of success trajectories, and to establish formative assessment diagnostic tools deriving from actual user behavior patterns in situ. This research holds implications for instructional design optimization of the e-learning system under investigation, curriculum and professional development support for educators involved, for quality of actual student learning outcomes, and, for the wider field of e-learning systems design and learning analytics.
“Guided discovery-based” learning experiences are those in which learners are given a particular task (e.g., a problem or a project) that must be supported by inquiry. That is, in order to successfully complete a task, the learner must develop core disciplinary knowledge as well as practices (e.g., the technical means of creating a multimedia project). Discovery denotes the need for student engagement in autonomous inquiry to support development of the core knowledge and expertise in the practices, to complete the given problem or project task. Often such complex activity is completed in teams. Learners who engage in such experiences are often provided affordances and scaffolds to “guide” and structure their discovery process, such as teacher facilitation, peer support in collaboration, digital/print graphic organizers to frame and sequence their activities, certain software and hardware technologies to support their creation, information resources in variety of multimedia formats to support inquiry, and other web tools and web services, e.g., social media.
One example of this type of intervention is the Globaloria program, which
embodies the principles of Constructionism and distributed cognition (Harel & Papert, 1991;
Salomon, 1997). The program is being implemented in middle and high schools in
several U.S. states. Participating students engage in collaborative game design activity
within a formal, in-school class. The primary goal from the students’ perspective is
successful completion and online publishing of a functioning web game, which they
also enter into an annual competition. To complete a game, students participate in
several integrated technology-supported activities to meet a range of instructional
objectives
One type of support that students use in guided discovery based and blended learning interventions is the e-learning platform, which can serve a range of purposes including affording access to information resources, and communication and collaborative tools. Examples of e-Learning platforms include corporate sites like Blackboard and eCollege, as well as free and inexpensive content management services and social media including GoogleDocs, Google spreadsheets, Moodle, and, Wikis.
Wikis are a form of participatory media that lend themselves to the
codevelopment of content in learning environments. They serve both as a point of
shared reference and shared production. The use of Wikipedia, the world’s most
extensive information reference wiki, is widely studied in educational contexts, with
researchers asking and answering a number of questions focused on how the
authenticity of editing a large, public reference tool influences student participation and
motivation
The current research underway aims to investigating the role of a wiki-based elearning platform to support students’ project-based work in game design. A coordinating representation is a type of scaffolding support that Salomon, Perkins & Globerson (1991) describe as “an intelligent technology that can undertake a significant part of the cognitive process that otherwise would have to be managed by the person.” Larussen & Alterman (2009) found that wikis can be used to support project-based work, making it easier for actors to work in parallel, multitask and make ‘common sense’ of the situation and how to proceed with the action (p. 375).
In Globaloria, student use of the wiki allows them to engage in development of an individual and team online identity, within-team and intra-team collaboration and project management of the game development process, including activities such as:
Game project file sharing (e.g., SWFs, FLAs, Actionscript code, JPGs, design documents)
Ongoing documentation of the product management process, Updating of a log file grid in which students outline their daily tasks completed Communication and feedback among team and class members
Viewing and using syllabus topic pages and on- and off-site game design tutorial resources as they proceed through the syllabus assignments
Our research investigates student uses of the wiki as a coordinating representation to guide their discovery-based game design activity in Flash. One data source available to us is the wiki trace log file data, which features the imprint of student individual and group edit activity, and upload activity on the wiki. Another data source is the Google Analytics page read data for each school, at the class level of analysis.
Our current work is outlined as follows. 1. Considering/validating the quality of the wiki trace data and Google Analytics page read data as a source of insight for research:
What do the team page edit and upload trace findings really represent? What do page read data represent?
What insights do our classroom observational and interview data lend to their credence?
What are the limitations of these sources? 2. Describing group activity patterns using wiki trace data and Google Analytics page read data:
Frequencies of edits/uploads (team means, SD) Sequential analysis of participation across time Cluster analysis of patterns (individuals within teams, teams within classes, etc.) Temporal social network analysis (measures of centrality for team wiki page building across time) 3. Investigating relationships between measured activity patterns and student learning outcomes:
Defining activity pattern clusters
Connecting activity to outcomes in which the dependent variable is quality of student final game artifacts as measured by content analysis using a reliable coding scheme
Considering other dependent variables that measure and indicate learning and performance
Identifying what levels of analysis and what variables in particular are predictive of outcomes. 4. If predictable patterns of behavior and relationships to outcomes emerge, then we aim to develop appropriate algorithms for early detection of success trajectories, and to establish formative assessment diagnostic tools deriving from actual user behavior patterns in situ.
Patterns we expect based on our research to date suggest three specific pattern categories. First, aggregated behavioral data will identify categorical differences for individual students and student groups that provide cues for teachers when deciding where additional facilitation is required. Second, sequential interaction patterns associated with particular lessons or pedagogical strategies recur across groups, and could help teachers develop a sense of what time segments of online collaboration to attend to. Finally, the intersection of individual and group roles with the time dimension holds potential for helping teachers label specific learning trajectories, and actively design curriculum to foster trajectories with more positive learning outcomes.
There are significant challenges associated with making sense of the rich traces of data available in technologically mediated learning environments, especially those related to discovery based learning. In the long view, establishing predictive models of individual student learning and group performance and outcomes drawing upon system trace data is a laudable goal. More pragmatically, our work to date suggests that indicators of performance at the group level hold greater weight than individual level activity, which makes sense because the group is the main locus of game design creation in this context. For example, imagine the simplest indicators of student participation over time, and group participation over time. A teacher presented with these basic measures in an xy coordinate graph could draw comparisons between groups and develop a heuristic sense of patterns that correspond with excellent group performance and groups in crisis (more so than they can for individuals in this context).
Discovery based learning poses unique challenges. In a discovery based learning environment students pursue less well defined objectives than in a traditional curriculum, and student game design work tasks vary substantially at any given time. This lack of definition that corresponds with an encouragement to discover also makes it more difficult for teachers to identify challenges being experienced by learning groups. This challenge is multiplied when a significant portion of group production occurs in technology mediated environments like wikis, involving varying types of software code for different functions like that used in the project we are investigating. Unlike a classroom environment where students are working on paper projects, an environment focused on technology is more difficult to observe. Even when students are in the same room they may be interacting through technology; and as parts of the curriculum involve routine work outside the classroom, teachers face a diminishing number of intermediate opportunities for feedback in between graded assignments. 2.1
Levels, Time and Perspective
The simple measures of participation described above represent a starting point for teaching analytics design for technologically mediated discovery learning environments like Globaloria. Through our individual work on Globaloria and the Virtual Math Teams environment we the authors have developed a shared understanding of three primary considerations for teaching analytics design. The first are levels of participation. These are individual, small group, classroom and school levels. Learning measures frequently account for these levels. Our conceptualization of teaching analytics does not expect measurement as robust as those expected from measurement of learning outcomes. We seek to design heuristic indicators at each level.
Some indicators of participation will be useful at a point in time. For example, as a milestone approaches in a learning module, individual and small group measures of work change, file management and simple login activity can set off alarms related to groups in trouble. Static measures across levels may not, however, prove as useful as measures of participation trajectory, which indicates the levels of participation over time. We expect comparative measures within learning groups and across learning groups over time will provide a stronger signal for teachers to develop heuristic indicators from than static measures; though this is in the hypothesis stage for our work.
Finally, discovery based blended e-learning in the Globaloria project involves both technologically mediated coordination activities, and face to face work. These two perspectives on the learning activity likely interact to some extent, and we think it is essential to build teaching analytics that enable teachers to view a synthetic construction of the total perspective in a single place. For example, imagine a teacher supervising ten learning groups of three in a discovery based learning program like Globaloria. Further, imagine that three of the groups engage deeply in face to face design activities, but use the wiki sparingly; three groups perform most design work almost exclusively through the wiki in an asynchronous manner, utilizing the class time as a more social period; and four groups shift in between these two polarized perspectives on how to perform the work. Indicators derived exclusively from the traces of wiki system use will not enable comparison across groups; and fine grained classroom observation systems are unlikely to be used by teachers. To account for this challenge, we hypothesize that period classification of group interactions in the classroom will enable teachers to see comparisons of groups with similar perspectives on how to pursue discovery based curriculum without placing too heavy a burden on teachers for record keeping. 3
To date we have performed network and participation analysis on traces from three
years of the Globaloria project. We are able to classify groups according to both
“point in time” participation levels, overall participation trajectories and changes in
the apparent “engagement roles” of members. Engagement roles are identified by
focusing on which members of the group perform work and interact with other
specific members both at a point in time and across time. We are presently in the process of
refining our classification of individuals within groups and between groups into
formal categories. This approach to our data analysis follows a qualitative, network
analytic approach that we are refining from Goggins Group Informatics
methodological approach and ontology
Our next phase of work is to complete our classification of learning groups within specific classrooms. We will then perform a correspondence analysis of the specific patterns with learning outcomes as measured by a detailed, rubric based assessment of student work products. This initial analysis will frame additional inquiry and iteration on the ways we analyze the trace data from the first step. We will continue this reflexive analysis on a single classroom group until we reach saturation, after which we will test the resulting model against groups from other schools.
At the same time as we are conducting the reflexive analysis of participation patterns and member role development we will begin to apply a model of contemporary learning abilities to the data we have. This will include content analysis of intermediate participation artifacts and final work projects. The result of this work will be an expanded understanding of how teaching analytics can be designed and developed for discovery based learning in partially technologically mediated contexts.
Through our design and analysis of participation and student work products we will begin to construct a teaching analytics dashboard for discovery based learning. This dashboard will incorporate probabilistic estimates of likely group performance based on electronic trace data from the Globaloria wiki and historical assessment of performance by groups with similar patterns of engagement. Our indicators will be presented as what we reasonably estimate they can be, and not as a confusing and ultimately untrustworthy predictor of outcomes. The essential feature of our objective is the probabilistic nature of the teaching analytics dashboard.
The Globaloria model -- utilizing a wiki e-learning platform as a coordinating representation and requiring students’ discovery-based inquiry to complete a complex task -- has important design parallels with other blended e-learning contexts that are becoming more pervasive. While critiques of such models exist, they are pragmatic in approach, given the state of technology today, and the level of technology expertise of today’s educators. Overall, our work aims to develop scholarly understanding of patterns and measures of engagement and their connection to outcomes in such environments, and ultimately, to use this understanding to develop teaching analytics tools that can improve upon the structure and guidance provided to students in guided discovery-based settings.