This paper analyses a gamified collaborative formal-informal outdoor activity and uses the findings to inform the design of future mobile collaboration tools. We present a case study framed in two editions (in years 2018 and 2019) of a (math) gymkhana1 for 15- and 16-year-old students, during which small groups worked collaboratively outside school, with no teacher supervision. From the case study, we present the analysis of the observations of three groups that participated in the activity and the post-activity questionnaire answered by 80 students. The analysis of the questionnaire reveals factors that students appreciated as promoting productive work (i.e. working together, a sense of agency because they were completely on their own and a change of perception of the nature of the subject matter). The analysis of the observations points to other elements that promote successful collaborative work, most of them confirming findings from previous research, such as engaged feedback, joint attention and alignment of goals. But other elements require more research, such as the importance of role changes over time and a broader view of the subject matter, as well as those specific to collaborating outdoors, such as finding/locating the task and matching the description of the task with the real object/place. From those elements, we derive design implications that should inform the design of future mobile educational location-based apps (ELbAs) for collaboration.
Collaboration is one of the key components of the digital literacy framework of the EU
Copyright © 2020 for this paper by its authors. 2
This section identifies the gaps in the literature and summarizes the main theories and principles that guided our study.
There are numerous studies that aimed to understand which collaborative learning (CL) conditions lead to good learning outcomes. There is evidence of multiple factors. We summarize here the findings from reviews of group dynamics and computersupported collaborative learning (CSCL) conducted by Barron (2003), Dillenbourg (2009), Greeno (2000), Stahl (2005), Stahl et al. (2014), and Tchounikine (2019). Their findings were used to categorize the observations in the case study presented in this paper (Analysis of the Observations): ● The amount and/or frequency of discussions. Opportunities to explain one’s thinking, negotiate and share knowledge and ideas (see design implication 6 in Table 2). ● Making many proposals and actively listening to proposals with constructive feedback (see design implication 4 in Table 2). ● Alignment of goals (see design implications 2 and 8 in Table 2). ● Body language for keeping/managing joint attention: silences, intonation, facial expressions, pointing, tapping, turn taking, coherency, gestures, laughter, jokes and eye gaze, for example (see design implication 4 in Table 2).
While the primary focus of our study was location-based learning apps, we proceed now to give an overview of the existing research into mobile collaboration because numerous studies have been done on mobile learning but only a handful have examined mobile and collaborative learning. Fu & Hwang’s (2018) literature review is one of the few papers especially focused on mobile and collaborative learning. The authors conclude with the importance of advancing the research in this field: “How can researchers or teachers design activities to engage students in more meaningful and authentic collaborative learning contexts to provide them with better chances to connect the learning content with real-life experiences, and hence construct knowledge and develop higher order thinking competences.” (p. 21). Our approach to advancing the research into collaborative location-based apps was to observe and analyze face-to-face mobile collaboration not mediated through technology and not constrained by the supervision of teachers. With this evidence, we used the findings to propose a set of design implications that could potentially improve future collaborative ELbAs.
Our main research questions (RQs) are: ● What factors shape collaboration in outdoor location-based activities without teacher supervision? ● What are students' experiences with outdoor location-based activities? Table 1 provides details about the structure of this research.
In order to answer our RQs, a case study design was selected due to the opportunities it provides to holistically understand phenomena (Baškarada, 2014). A case study is a good choice for testing and expanding upon existing research with new contexts (Baškarada, 2014), which is the purpose of this paper.
Description of the Gymkhana Activity. This particular gymkhana has been taking place for 20 years, with around 20 high schools participating each year. It gathers around a thousand fourth-grade secondary school students(15-year-olds) in groups of four. The groups disperse around base points that have math word problems located in situ. Students must first find these base points and locate the object of the problem in order to solve them and move forward. Students get points for the problems they solve. During the gymkhana, the groups of students work on their own, with no teacher supervision, and are free to use any resource they have in solving the problems. 3.2
The research instruments used for this case study were ● Observation during the activity of three groups of students (11 students per group) as they solved math problems in real outdoor settings during two editions of the gymkhana: 2018 and 2019. The observations included field notes and still pictures.
● Post-activity evaluation questionnaire completed by 80 participants, The main purpose of the observations is to give us a view of the group collaboration while the data collected from the questionnaire gave us insight into individual experiences of the activity. By combining the two, we can obtain a more holistic view of the activity.
We conducted a semi-structured observation following the details suggested by
Cohen et al. (2007). The main criteria for the observation was to document any
interactions that were new and different from the existing research. Field notes were
used to describe conversations, attitudes and processes and still pictures to capture body
language and personal interactions. The role of the researcher was
observer-asparticipant
The purpose of the questionnaire was to obtain information about the individual experience of the gymkhana. The purpose was to find out what was the best of the activity and why, what was their opinion about collaborating and what they thought they had learned. Questions were generated around RQ2 (see Table 1). Those categories were chosen in order to (a) test if they think collaboration is useful in terms of learning outcomes and fun and (b) detect if their personal attitude towards math affected somehow the experience of the gymkhana. A mix of quantitative and qualitative questions was used (See https://tinyurl.com/design-collaborative-ELbAs for more details) with the goals of the qualitative questions aimed at obtaining details about (a) what they thought was the best part of the gymkhana, (b) their perceived learning outcomes and (c) their opinion on the best aspects of the activity.
In order to analyze the data, a thematic analysis approach (Braun & Clarke, 2006) was used to identify, analyze and report on patterns (themes) in the collected data. 4 4.1
Fortunately, the groups making up the gymkhana teams followed Stahl’s recommendation (2014) of four members as the most fruitful unit of analysis when studying collaborative meaning-making (one of our groups had three members because one was sick). Our target was secondary school students, a group that was identified by Fu & Hwang (2018) as needing further research in collaborative location-based activities. Data collection took place during two editions of the activity (the years 2018 and 2019). In 2018, one researcher shadowed two groups (A and B, seven students in total) for 2h 30min each. In order to get a deeper observation of the interactions in groups, in 2019, a third group (C) was observed for 5 hours. Group A was composed of three girls and a boy, group B was composed of three girls and group C was composed of four girls. In all cases the groups were formed on the basis of previous friendships and independently of the teacher’s opinion. 4.2
The 14-item questionnaire was completed by a total of 80 students from five different high schools. They completed the questionnaire during their regular math class with their regular math teacher in their regular high school within two weeks after the gymkhana. 5 5.1
The observation notes were reviewed and clustered using ATLAS.ti software (ATLAS.ti Scientific Software Development, Berlin) in two categories derived from our RQ1: Personal Interactions Among Members (testing findings from previous research): experience, feelings (positive or negative), roles, conversations, proposals, alignment of goals, body language, problems/frictions
problems outdoors (connecting the formal and the informal), tools, problems/frictions
Second, we coded the pictures taken during the observations. This process revealed the importance of perception, meaning not only looking and searching with the eyes but also touching and pointing. Here, we use perception in terms of sensory processes like seeing, touching or hearing. Perception was the cause of a wide variety of interactions and became a new category that encompassed many of the notes categorized in new phases and factors/opportunities detected for collaborating outside school.
We reviewed the notes and pictures clustered in each category, and from those categories (see Table 2 at https://tinyurl.com/design-collaborative-ELbAs), we defined themes, following the methodology proposed by Braun and Clark (2006). 5.2
The analysis of the qualitative answers was created by clustering similar answers and detecting and highlighting atypical cases.
Combining our observations with their opinions and feelings provided us with a more holistic picture of the activity and a student centred design approach. 6
In this section we present the results of the questionnaire and a holistic view of the observations. 6.1
Identity and Individual Experience during Collaboration. Fig. 1 (see https://tinyurl.com/design-collaborative-ELbAs) shows that students had a very positive experience of the gymkhana, even if they had a low self-perception of their math performance. This indicates that the students generally think math is useful in their everyday lives, even if they consider their performance low. The responses also reflected that students valued working in groups much more than their feelings about their math performance.
Collaboration. Responses to the open question “What did you enjoy most about this gymkhana?” all fell into one of the following: “collaboration” (28.4%), “having fun with math” (25.7%) and “being outside/on our own” (20.3%). This final answer raised the new topic of agency (i.e. being on their own, without teacher intervention). Roles. Most groups did not have any kind of organization of tasks, but that also highlights that an important percentage of them did (38.8%). More insights about this topic are presented with the results of the roles observations. In conclusion, the results of the questionnaire show that collaborating and solving problems outside school is something that should be promoted because students find it to be a positive experience. 6.2
We followed the recommendations of Cohen et al. (2007) and Braun and Clarke (2006) for making meaning from data and now present the results of the holistic observations of the activity.
gymkhana: getting points by solving problems according to difficulty, having to move around the city to find the problems and having to find the objects of the problems in a given amount of time. These were the origins of authentic and situated math problems; for example, optimizing time and distances resulted in the unfolding of a number of interesting math problems that students were probably unaware of.
Strategies refers to the decisions taken in order to maximize the possibilities of collecting and solving problems. Basically, this meant saving time walking and choosing the best problems to solve and the best moments to do it. For example, if a group chose to collect as many problems from base points as possible in order to try to solve them later, the interactions between the members were focused on getting to the base points. Alternatively, if a group chose to solve problems as they were collected, the dynamics of that group were more collaborative in all aspects. As already noted, the game dynamics embedded in the activity add an extra layer of math thinking that was enjoyed by students (see design implication 7 in Table 2).
Roles. Roles/tasks refers to the functions assumed by each member of the group,
including whether these functions changed over time. Because these students had only
minimal experience with collaborating, their emergent roles were either tasks or roles
that existed before the gymkhana and were dependent on each member’s grades. If roles
changed among members of the group (as they did in group A), the interactions were
more productive. If the roles were clearly established and fixed, the members tended to
accept the opinions of the leaders and/or students who usually obtain better grades.
These groups also had more difficulty making decisions and had a tendency to remain
blocked if the leader did not know what to do. This effect is not entirely new
Among the positive outcomes noted in the questionnaires, 35% of students referred to enjoying working in a group. Most of these kinds of answers were about being useful to the group, even when the respondents did not consider themselves “good at math”. They graded themselves, on average, 3.20/5 points (1.10 SD) on this aspect. We observed that they seemed more confident and freer to voice their opinions when working without the teacher’s presence. One student described the best part of the experience: “Even being bad at math, I could still propose things that were useful for the others” (all responses translated from Spanish by the author). These students did not have much experience collaborating, and the fact that they were among friends without any teachers involved and far from the classroom culture of right/wrong opened up far-reaching opportunities for sharing and learning through their conversations.
For the students who had little prior experience (and even less with math), being in this situation made the entire situation easier and more enjoyable.
There were also opportunities for learning from experiences that were not altogether positive. Some students complained that other members of the group did not work as hard as they did or that not all members were engaged equally with the goals of the group. Others complained that even if they gave their opinions and made proposals, they felt it was futile because the group always did “what the members with better grades said”.
We find that tools for trying to promote collaborative learning should afford anonymous sharing of opinions in order to avoid the fear of proposing wrong answers and the kind of negative situations mentioned (see design implication 3 in Table 3).
questionnaire, 17.5% of the students referred to some degree of surprise about math.
This was reflected in responses that included “taking a closer look at things” or “math
is more important than we thought” as one of their learning outcomes (this was also an
open question). This is as though being in a group outside the classroom and looking at
things in a new, closer way was an “eye-opening” experience that led to a new or at
least broader perception of math; some students remarked that they “didn’t know that
was math”. We interpreted these answers as confirmation that the perception and
relationship students have with math can be significantly improved. On the
questionnaire, these students ranked the usefulness of math in their everyday lives at
3.56/5 avg. (1.01 SD). Their perception of math was as something they do at school
From the results just discussed, we distilled design implications that should promote more active participation and could inform the design of future asynchronous mobile collaborative tools. Some of these design implications come from existing literature and some are new proposals that need to be tested in the future.
Type Collaboration: group awareness Collaboration: visualization of group activity Roles/tasks Tools Design Implication 1. Always show all the members of the group (and of course, who is sending any message). This should help to have an active image of all the people as a working group. 2. Allow evaluations to be made of the general performance of the group, and let that information be visible to the group: For example, a simplified version of the Radar and OurEvaluator tool (Järvelä et al., 2015) . 3. Allow sending of anonymous/private messages to the group to provide
an error-safe space.
4. Provide affordances for showing that members are paying attention
through quick answer/feedback icons. (“understood”, “I need more
information”, “I didn’t understand”, etc.)
5. Visually show different kinds of messages: proposals
would facilitate taking decisions and moving forward to next steps. 9. Awareness of roles and tasks and let them evaluate each other in the
roles. However, do not force students into assigned or scripted roles. 10. Provide affordances for a shared creative/modelling canvas to draw
together
Instagram). Quick access to calculator and formulas cheat-sheet.
This study was built on top-down and bottom-up methodologies, specifically, a literature review and a case study, to identify factors that shape collaborative problemsolving in hybrid contexts. The observed case study is a gamified activity that combines formal education with an informal setting. While the gymkhana was enjoyed by students, some questions arose. First, if we implement these design implications in a mobile collaborative ELbA, we wonder if students would engage as much as they did with this face-to-face activity or if they would just try to finish the activity as quickly as possible, without truly collaborating. Second, how much of the positive experience of collaboration was about being physically together out of school for one day and how much was collaboration and learning. Finally, data needs to be obtained from other types of groups, especially those who do not work together as well or who are minimally motivated.
From the questionnaires, we learned that students not only enjoyed collaborating and the sense of agency they got from working on their own but that this experience also helped them change their perception of math. At the same time, the problems were very similar to those you can find in a regular textbook except that they were located in a real place. We wonder if the experience of the activity would improve if the problems were more connected to the students’ interests. The problems were also of the “wellstructured”, one-solution type. Problems that are less structured and more open-ended would require more personal interaction and collaboration. We hypothesize that those kinds of tasks would offer better opportunities for learning. We also cannot be certain the students actually learned any new mathematics with the activity, and if they did not, a thoughtful determination of what can be done will be required.
These design implications should inform the design of future mobile collaborative ELbA’s, but we are also curious how these design implications can be organically integrated into a collaboration tool. 9.
Most of the groups that are created without teacher intervention are not gender balanced, and this was true in our study. Another limitation is our lack of video recorded material. The decision to not take video recordings was taken because the students being observed preferred to not be recorded. The chosen method of data collection was aimed at keeping the researcher in a more invisible and unobtrusive position while taking field notes. Ethical issues and the potential emotional implications for the students led to the decision that video recording could produce awkward behaviour from the students and affected their personal interactions. A final limitation of the study is the small number of groups observed with only three. Future research must thoroughly test and iterate the prototypes based on the design implications proposed here. 10.
The observation notes and questionnaire results are published online at
https://tinyurl.com/design-collaborative-ELbAs. The ethics procedure for this study
followed the principles, tools and procedures for the quality of research