=Paper=
{{Paper
|id=None
|storemode=property
|title=Designing Socio-Technical Systems to Support Guided "Discovery-Based" Learning in Students: The Case of the Globaloria Game Design Initiative
|pdfUrl=https://ceur-ws.org/Vol-985/paper1.pdf
|volume=Vol-985
|dblpUrl=https://dblp.org/rec/conf/lak/ReynoldsG13
}}
==Designing Socio-Technical Systems to Support Guided "Discovery-Based" Learning in Students: The Case of the Globaloria Game Design Initiative==
https://ceur-ws.org/Vol-985/paper1.pdf
Designing Socio-Technical Systems to Support Guided
“Discovery-Based” Learning in Students: The Case of
the Globaloria Game Design Initiative
Rebecca Reynolds1, Sean P. Goggins2
1 Rutgers University School of Communication & Information,
Dept. of Library and Information Science.
rebecca.reynolds.at.rutgers.edu
2 Drexel University etc.
outdoors.at.acm.org
Abstract. This in-progress research study investigates middle school stu-
dents’ use of a wiki-based e-learning platform as a coordinating representation
in the context of their guided discovery-based game design work. The study
aims to (a) consider/validate the quality of wiki trace data and Google Analytics
page read data as a source of insight for research; (b) describe group activity
patterns using wiki trace data and Google Analytics page read data; (c) investi-
gate relationships between measured activity patterns and student learning out-
comes; (d) develop appropriate algorithms for early detection of success trajec-
tories, and to establish formative assessment diagnostic tools deriving from ac-
tual user behavior patterns in situ. This research holds implications for instruc-
tional design optimization of the e-learning system under investigation, curricu-
lum and professional development support for educators involved, for quality of
actual student learning outcomes, and, for the wider field of e-learning systems
design and learning analytics.
1 Introduction
“Guided discovery-based” learning experiences are those in which learners are
given a particular task (e.g., a problem or a project) that must be supported by inquiry.
That is, in order to successfully complete a task, the learner must develop core disci-
plinary knowledge as well as practices (e.g., the technical means of creating a multi-
media project). Discovery denotes the need for student engagement in autonomous
inquiry to support development of the core knowledge and expertise in the practices,
to complete the given problem or project task. Often such complex activity is com-
pleted in teams. Learners who engage in such experiences are often provided af-
fordances and scaffolds to “guide” and structure their discovery process, such as
teacher facilitation, peer support in collaboration, digital/print graphic organizers to
frame and sequence their activities, certain software and hardware technologies to
support their creation, information resources in variety of multimedia formats to sup-
port inquiry, and other web tools and web services, e.g., social media.
One example of this type of intervention is the Globaloria program, which embod-
ies the principles of Constructionism and distributed cognition (Harel & Papert, 1991;
Salomon, 1997). The program is being implemented in middle and high schools in
several U.S. states. Participating students engage in collaborative game design activity
within a formal, in-school class. The primary goal from the students’ perspective is
successful completion and online publishing of a functioning web game, which they
also enter into an annual competition. To complete a game, students participate in
several integrated technology-supported activities to meet a range of instructional
objectives (Reynolds & Harel, 2009).
One type of support that students use in guided discovery based and blended learn-
ing interventions is the e-learning platform, which can serve a range of purposes in-
cluding affording access to information resources, and communication and collabora-
tive tools. Examples of e-Learning platforms include corporate sites like Blackboard
and eCollege, as well as free and inexpensive content management services and social
media including GoogleDocs, Google spreadsheets, Moodle, and, Wikis.
Wikis are a form of participatory media that lend themselves to the co-
development of content in learning environments. They serve both as a point of
shared reference and shared production. The use of Wikipedia, the world’s most ex-
tensive information reference wiki, is widely studied in educational contexts, with
researchers asking and answering a number of questions focused on how the authen-
ticity of editing a large, public reference tool influences student participation and
motivation (Forte & Bruckman, 2006). Less explored is the use of wikis in the class-
room as a design and coordination tool for long running projects, and the correspond-
ing utilization of the trace data generated by these wikis for providing teachers with
insight related to student participation and performance.
The current research underway aims to investigating the role of a wiki-based e-
learning platform to support students’ project-based work in game design. A coordi-
nating representation is a type of scaffolding support that Salomon, Perkins & Glober-
son (1991) describe as “an intelligent technology that can undertake a significant part
of the cognitive process that otherwise would have to be managed by the person.”
Larussen & Alterman (2009) found that wikis can be used to support project-based
work, making it easier for actors to work in parallel, multitask and make ‘common
sense’ of the situation and how to proceed with the action (p. 375).
In Globaloria, student use of the wiki allows them to engage in development of an
individual and team online identity, within-team and intra-team collaboration and
project management of the game development process, including activities such as:
Game project file sharing (e.g., SWFs, FLAs, Actionscript code, JPGs, design doc-
uments)
Ongoing documentation of the product management process,
Updating of a log file grid in which students outline their daily tasks completed
Communication and feedback among team and class members
Viewing and using syllabus topic pages and on- and off-site game design tutorial
resources as they proceed through the syllabus assignments
Our research investigates student uses of the wiki as a coordinating representation
to guide their discovery-based game design activity in Flash. One data source availa-
ble to us is the wiki trace log file data, which features the imprint of student individual
and group edit activity, and upload activity on the wiki. Another data source is the
Google Analytics page read data for each school, at the class level of analysis.
Our current work is outlined as follows.
1. Considering/validating the quality of the wiki trace data and Google Analytics
page read data as a source of insight for research:
What do the team page edit and upload trace findings really represent?
What do page read data represent?
What insights do our classroom observational and interview data lend to their cre-
dence?
What are the limitations of these sources?
2. Describing group activity patterns using wiki trace data and Google Analytics page
read data:
Frequencies of edits/uploads (team means, SD)
Sequential analysis of participation across time
Cluster analysis of patterns (individuals within teams, teams within classes, etc.)
Temporal social network analysis (measures of centrality for team wiki page build-
ing across time)
3. Investigating relationships between measured activity patterns and student learn-
ing outcomes:
Defining activity pattern clusters
Connecting activity to outcomes in which the dependent variable is quality of stu-
dent final game artifacts as measured by content analysis using a reliable coding
scheme
Considering other dependent variables that measure and indicate learning and per-
formance
Identifying what levels of analysis and what variables in particular are predictive of
outcomes.
4. If predictable patterns of behavior and relationships to outcomes emerge, then we
aim to develop appropriate algorithms for early detection of success trajectories, and
to establish formative assessment diagnostic tools deriving from actual user behavior
patterns in situ.
Patterns we expect based on our research to date suggest three specific pattern catego-
ries. First, aggregated behavioral data will identify categorical differences for indi-
vidual students and student groups that provide cues for teachers when deciding
where additional facilitation is required. Second, sequential interaction patterns asso-
ciated with particular lessons or pedagogical strategies recur across groups, and could
help teachers develop a sense of what time segments of online collaboration to attend
to. Finally, the intersection of individual and group roles with the time dimension
holds potential for helping teachers label specific learning trajectories, and actively
design curriculum to foster trajectories with more positive learning outcomes.
2 Teaching Analytics for Discovery Based Learning
There are significant challenges associated with making sense of the rich traces of
data available in technologically mediated learning environments, especially those
related to discovery based learning. In the long view, establishing predictive models
of individual student learning and group performance and outcomes drawing upon
system trace data is a laudable goal. More pragmatically, our work to date suggests
that indicators of performance at the group level hold greater weight than individual
level activity, which makes sense because the group is the main locus of game design
creation in this context. For example, imagine the simplest indicators of student par-
ticipation over time, and group participation over time. A teacher presented with
these basic measures in an xy coordinate graph could draw comparisons between
groups and develop a heuristic sense of patterns that correspond with excellent group
performance and groups in crisis (more so than they can for individuals in this con-
text).
Discovery based learning poses unique challenges. In a discovery based learning
environment students pursue less well defined objectives than in a traditional curricu-
lum, and student game design work tasks vary substantially at any given time. This
lack of definition that corresponds with an encouragement to discover also makes it
more difficult for teachers to identify challenges being experienced by learning
groups. This challenge is multiplied when a significant portion of group production
occurs in technology mediated environments like wikis, involving varying types of
software code for different functions like that used in the project we are investigating.
Unlike a classroom environment where students are working on paper projects, an
environment focused on technology is more difficult to observe. Even when students
are in the same room they may be interacting through technology; and as parts of the
curriculum involve routine work outside the classroom, teachers face a diminishing
number of intermediate opportunities for feedback in between graded assignments.
2.1 Levels, Time and Perspective
The simple measures of participation described above represent a starting point for
teaching analytics design for technologically mediated discovery learning environ-
ments like Globaloria. Through our individual work on Globaloria and the Virtual
Math Teams environment we the authors have developed a shared understanding of
three primary considerations for teaching analytics design. The first are levels of
participation. These are individual, small group, classroom and school levels. Learn-
ing measures frequently account for these levels. Our conceptualization of teaching
analytics does not expect measurement as robust as those expected from measurement
of learning outcomes. We seek to design heuristic indicators at each level.
Some indicators of participation will be useful at a point in time. For example, as a
milestone approaches in a learning module, individual and small group measures of
work change, file management and simple login activity can set off alarms related to
groups in trouble. Static measures across levels may not, however, prove as useful as
measures of participation trajectory, which indicates the levels of participation over
time. We expect comparative measures within learning groups and across learning
groups over time will provide a stronger signal for teachers to develop heuristic indi-
cators from than static measures; though this is in the hypothesis stage for our work.
Finally, discovery based blended e-learning in the Globaloria project involves both
technologically mediated coordination activities, and face to face work. These two
perspectives on the learning activity likely interact to some extent, and we think it is
essential to build teaching analytics that enable teachers to view a synthetic construc-
tion of the total perspective in a single place. For example, imagine a teacher super-
vising ten learning groups of three in a discovery based learning program like Glob-
aloria. Further, imagine that three of the groups engage deeply in face to face design
activities, but use the wiki sparingly; three groups perform most design work almost
exclusively through the wiki in an asynchronous manner, utilizing the class time as a
more social period; and four groups shift in between these two polarized perspectives
on how to perform the work. Indicators derived exclusively from the traces of wiki
system use will not enable comparison across groups; and fine grained classroom
observation systems are unlikely to be used by teachers. To account for this chal-
lenge, we hypothesize that period classification of group interactions in the classroom
will enable teachers to see comparisons of groups with similar perspectives on how to
pursue discovery based curriculum without placing too heavy a burden on teachers for
record keeping.
3 Work to Date: Framing the Challenge
To date we have performed network and participation analysis on traces from three
years of the Globaloria project. We are able to classify groups according to both
“point in time” participation levels, overall participation trajectories and changes in
the apparent “engagement roles” of members. Engagement roles are identified by
focusing on which members of the group perform work and interact with other specif-
ic members both at a point in time and across time. We are presently in the process of
refining our classification of individuals within groups and between groups into for-
mal categories. This approach to our data analysis follows a qualitative, network
analytic approach that we are refining from Goggins Group Informatics methodologi-
cal approach and ontology (Goggins, Mascaro & Valetto, 2013), which systematically
integrates ethnographic and network analytic methods to ensure reliability, validity
and theoretical coherence in network analysis of electronic trace data. We are not
merely analyzing traces, but instead aggregating and weighting interactions so that the
network analysis is closely connected to the specific context of the Globaloria project.
Our next phase of work is to complete our classification of learning groups within
specific classrooms. We will then perform a correspondence analysis of the specific
patterns with learning outcomes as measured by a detailed, rubric based assessment of
student work products. This initial analysis will frame additional inquiry and iteration
on the ways we analyze the trace data from the first step. We will continue this re-
flexive analysis on a single classroom group until we reach saturation, after which we
will test the resulting model against groups from other schools.
At the same time as we are conducting the reflexive analysis of participation pat-
terns and member role development we will begin to apply a model of contemporary
learning abilities to the data we have. This will include content analysis of intermedi-
ate participation artifacts and final work projects. The result of this work will be an
expanded understanding of how teaching analytics can be designed and developed for
discovery based learning in partially technologically mediated contexts.
Through our design and analysis of participation and student work products we
will begin to construct a teaching analytics dashboard for discovery based learning.
This dashboard will incorporate probabilistic estimates of likely group performance
based on electronic trace data from the Globaloria wiki and historical assessment of
performance by groups with similar patterns of engagement. Our indicators will be
presented as what we reasonably estimate they can be, and not as a confusing and
ultimately untrustworthy predictor of outcomes. The essential feature of our objec-
tive is the probabilistic nature of the teaching analytics dashboard.
The Globaloria model -- utilizing a wiki e-learning platform as a coordinating rep-
resentation and requiring students’ discovery-based inquiry to complete a complex
task -- has important design parallels with other blended e-learning contexts that are
becoming more pervasive. While critiques of such models exist, they are pragmatic in
approach, given the state of technology today, and the level of technology expertise of
today’s educators. Overall, our work aims to develop scholarly understanding of pat-
terns and measures of engagement and their connection to outcomes in such environ-
ments, and ultimately, to use this understanding to develop teaching analytics tools
that can improve upon the structure and guidance provided to students in guided dis-
covery-based settings.
References
1. Forte, A., & Bruckman, A. (2006, June). From Wikipedia to the classroom: exploring
online publication and learning. In Proceedings of the 7th international conference on
Learning sciences (pp. 182-188). International Society of the Learning Sciences.
2. Goggins, S. P., Mascaro, C., & Valetto, G. (2013). Group informatics: a methodological
approach and ontology for understanding socio-technical groups. Journal of the American
Society for Information Science and Technology. In Press.
3. Kirschner, P.A., Sweller, J. & Clark, R.E. (2006). Why minimal guidance during instruc-
tion does not work: An analysis of the failure of constructivist, discovery, problem-based,
experiential, and inquiry based teaching. Educational Psychologist, 41, 75-86.
4. Larusson, J. A. & Alterman, R. (2009). Wikis to support the “collaborative” part of collab-
orative learning. Computer-Supported Collaborative Learning, 4:371-402
5. Reynolds, R., Harel Caperton, I. (2009). The emergence of 6 contemporary learning abili-
ties in high school students as they develop and design interactive games. Paper presented
at the American Educational Research Association annual conference, April 2009.
6. Salomon, G., Perkins, D. & Globerson, T. (1991). Partners in Cognition: Extending Hu-
man Intelligence with Intelligent Technologies. Educational Researcher, April 1991, pgs
2-9