Computational Thinking (CT) is one of the key skills within adequate digital literacy for the 21st century. Over the last few decades, several initiatives fostered the development of CT among primary and secondary school students (e.g., The Hour of Code, CodeWeek EU, Scratch Day) using online learning platforms (e.g., Code.org). Many of the activities on these platforms are based on 2D maze-based challenges that students solve using visual code blocks. Our findings are based on the analysis of the platform log data gathered from 326 learners during three studies in which students were asked to solve maze-based programming challenges in our online platform, Kodetu. According to our results, a Kodetu challenge is difficult when: a) the maze has turns and the total number of steps needed to go from the initial position to the endpoint is high, and b) when not only movement blocks, but also loop and conditionals blocks are needed to solve the challenge. The results of this research should be considered when designing learning activities to develop and enhance CT skills through maze-based programming challenges.
The increasing digitization of all aspects of our daily lives demands the development
of adequate digital literacy [
The most specific of all the skills that make up CT is probably the algorithmic
skill, i.e. the ability to define a set of steps in the form of sequences, conditionals, and
loops to solve a problem. For this reason, many of the online platforms, mobile
applications, and even "unplugged" learning materials (e.g., board games, activity books,
toys) focus on the design of algorithms by learners. A common approach in these
This work was supported by the European Union’s Horizon 2020 Research and Innovation Program
through the Marie Sklodowska-Curie Grant under Agreement 665959.
cases is to use games where players have to guide a character through a 2D grid that
may contain obstacles, hazards, or even moving enemies [
Our goal with this study is to know which features of the Kodetu programming challenges influence learners' performance. Therefore, we designed a wide set of maze-based programming challenges and tested them with a large sample of primary and secondary school students. Based on the performance of these students, we have estimated which factors have more or less influence on their performance and thus on the difficulty associated with each challenge.
Considering this, our research questions are focused on the two aspects that define a challenge in Kodetu: 1. The features of the maze: How do maze characteristics (width, height, total number of steps in the maze, optimal path, maze loops, turns, and numbers of x-crosses and t-crosses) affect the performance in a maze-based programming challenge? 2. Programming constraints: How do coding limitations (blocks provided and block limit) affect the performance in a maze-based programming challenge? Knowing the answers to these research questions will allow us to design challenge sequences adapted to the proficiency level of each learner and maximize their learning. 2
Difficulty is generally defined as the commitment taken to effectively perform an
operation [
Difficulty is considered a key factor in promoting the motivation of learners in
educational games and resulting in better learning outcomes [
Some authors have studied the difficulty associated with educational games that
employ mazes [
However, existing research does not investigate the specific characteristics of
block-based maze games that affect the performance of the learners. Given the wide
usage of this type of games to promote CT, it is important to define how these
characteristics affect the performance and use them to design effective learning paths of
adaptive difficulty. To achieve this, it is necessary to record learners’ interactions in
as much detail as possible. Numerous authors have followed this approach [
Kodetu is an online platform where participants must solve challenges using a blockbased programming interface. It is an educational game that allows researchers and teachers to easily create new challenges and challenge sequences.
The interface of Kodetu is divided into three parts (see Fig. 1). The left panel shows the maze-like spaceship, the initial position of the astronaut, and the endpoint marker. Players must lead the astronaut from the beginning of the maze to the endpoint using the programming blocks provided in the center panel of the interface. There are movement blocks (go forward, turn left, and turn right), loop blocks, and one-branch or two-branch conditional blocks. Players drag and drop these blocks to the right panel, i.e. the workspace, where they define the visual program that leads the astronaut to the endpoint. When players click the “play” button, the program defined in the workspace is executed, and the astronaut moves according to the programmed instructions. A Kodetu challenge is solved successfully when the player’s program leads successfully the astronaut to the exit of the maze. We use the term sequence to define a group of consecutive Kodetu challenges.
Throughout the game, Kodetu logs all player interactions with the interface. All players’ data are gathered anonymously and stored in the database under a unique identifier that is automatically generated when the player accesses the platform. By analyzing this usage data, we can estimate the performance of each player.
The aim of this study is to determine which maze features and which programming constraints affect the difficulty of each challenge in Kodetu. Therefore, we conducted 3 studies with a very similar design. After a preliminary analysis of data from over 19.000 participants in Kodetu sessions gathered during the last 5 years, we concluded that these factors could affect the difficulty of each challenge: the width and height of the maze, the total number of steps in the maze, the length of the optimal path (from the starting point to the exit), no turns on the optimal path/one-direction turns on the optimal path (that is, only right- or only left-direction turns)/two-direction turns on the optimal path, X-crosses (the possibility to move north, south, west and east from a certain point of the maze), T-crosses (the possibility to move south, west and east from a certain point of the maze), maze loops (a path that allows users to go from one position in the maze to the same position used in the maze without passing through any previous position), blocks available (movement only blocks, loops + movement blocks, and conditionals + loops + movement blocks), and block limits (the number of blocks allowed to be used in a maze challenge). To assess the influence of each factor, we designed sequences of Kodetu challenges and conducted several workshops with primary and secondary school students.
In Study 1, we wanted to know how much maze loops affect the performance in a maze-based programming challenge. We designed 34 Kodetu challenges, separated into pairs that differed only in the number of maze loops(e.g., one challenge was defined as {width: 7, height: 7, optimal path: 24, total steps: 24, maze loops: 0, xcrosses: 0, t-crosses: 0, turns: 2, no block limit, blocks: all available} and another challenge was the same but instead of 0 maze loops, it has 2 maze loops). With these 34 challenges, we prepared 7 sequences of 5 challenges each. A total of 70 participants aged between 11 and 15 years old (44% female, 56% male) had 30 min to solve ten challenges in Kodetu. The first 5 challenges were training challenges and were not considered in the analysis. The last 5 challenges corresponded to one of the 7 challenge sequences mentioned before, randomly assigned to each participant.
The design of the sequences (Fig. 2) aimed to achieve increasing difficulty and a smooth transition from one challenge to the next based on an initial estimation of the difficulty of the challenges.
To determine whether the number of maze loops in a challenge affects success, we performed one-way ANOVA in which the dependent variable was the percentage of success in a challenge and the independent variable was the number of maze loops (four groups: 0, 1, 2, or 3 maze loops in each challenge). The results showed that no statistically significant difference existed between groups [F(3,31) = 0.705, p = 0.556]. We ran another one-way ANOVA in which the groups of the independent variables were challenges with no maze loops and challenges with maze loops. The results showed that no statistically significant difference existed between the groups [F(1,33) = 1.604, p = 0.214]. Therefore, these results suggest that maze loops in Kodetu challenges do not cause additional difficulty for the players.
In Study 2, we wanted to know how maze characteristics (turns, and numbers of xcrosses and t-crosses) and coding limitations (blocks provided and block limit) affect the performance of Kodetu players. We designed forty challenges following the same principles of Study 1 (i.e., 20 pairs of challenges where all the variables were the same except for one), and we created 6 sequences of 7 challenges each. A total of 197 participants aged 9 to 11 years old (49% female, 49% male, and 2% other) had 60 min to solve 12 challenges in Kodetu. The first 5 challenges were training challenges with no block limit, and they were not considered in the analysis. The next 7 challenges corresponded to one of the six challenge sequences randomly assigned to each participant.
We conducted a one-way ANOVA test to compare the effect of the maze turns on the percentage of success on challenges with no turns, one-direction turns, and twodirection turns. There was a significant effect of the type of turn on the success rate at the p<0.05 level for the three conditions [F(2,39) = 3.722, p = 0.033]. However, we did not find a significant effect using the Tukey HSD and Duncan post hoc tests in terms of pairwise comparisons.
Regarding programming constraints, an analysis of variance showed that the effect of the blocks available was significant [F(2,39) = 20.032, p = 0.000]. Post hoc analyses using the Tukey HSD post hoc criterion for significance indicated that the success percentage was significantly higher in the conditions in which movement only blocks (go forward-turn left-turn right) (M = 0.934, SD = 0.0755) and loops + movement blocks (M = 0.945, SD = 0.0544) were provided than in the other condition (conditionals + loops + movement blocks) in which M = 0.55 and SD = 0.2899. Similarly, statistically significant differences in the means of the challenges with and without a block limit were observed with F(1,40) = 17.902 with p = 0.000.
One-way ANOVA showed that the analysis was not significant for the effect of the numbers of x-crosses [F(3,38) = 0.978, p = 0.413] and t-crosses [F(3,38) = 2.034, p = 0.125] on success.
Study 3 is a replica of Study 2 with older participants. A total of 59 participants aged 15-16 (37% female, 59% male, and 3% other) had 60 min to solve 12 challenges in Kodetu. The first 5 challenges were training challenges with no block limit and were not considered for analysis. The last 7 challenges corresponded to one of the six challenge sequences prepared before and randomly assigned to each participant.
With the data of Study 3, we wanted to compare the effects of the maze characteristics and the coding constraints on the success percentage of the participants. A oneway ANOVA test was conducted to compare the effect of the maze turns on the percentage of success on challenges with no turns, one-direction turns, and two-direction turns. There was a significant effect of the type of turn on the success rate at the p<0.05 level for the three conditions [F(2,39) = 3.997, p = 0.026]. An analysis of variance showed that the effect of the blocks available was significant [F(2,39) = 7.768, p < 0.001] for the three conditions (movement only blocks, loops + movement blocks and conditionals + loops + movement blocks). However, we did not find a significant effect using the Tukey HSD and Duncan post hoc tests in terms of pairwise comparisons. Furthermore, one-way ANOVA showed that the analysis was not significant for the effect of the numbers of x-crosses [F(3,38) = 0.113, p = 0.952] and t-crosses [F(3,38) = 2.054, p = 0.123] on success.
In order to analyze the differences in the results between Studies 2 and 3, we conducted a one-way ANOVA to compare the effect of age on success in the group of 9to 11-year-olds and in the group of 15- to 16-year-olds. We found that age had a significant effect on success at the p<0.05 level for the two groups [F(1,82) = 8.437, p = 0.005]. 5
The present research provides a quantitative analysis of data obtained from the Kodetu platform, which advances our understanding of the maze characteristics and coding limitations that affect participants’ performance in maze-based programming challenges. Our findings are based on the analysis of the platform log data gathered from 326 learners during three studies.
After analyzing the data obtained from Study 1, we found that the existence of maze loops in the challenges did not affect learners’ success rate. Thus, the high failure rate, especially in the last challenges of the sequences, cannot be explained by the maze loops. One of the reasons for this finding may be that as long as learners can cognitively solve the challenge, the maze loops do not affect their performance (i.e., they are not challenged by the maze, but by the code needed to make the astronaut get to the endpoint).
The results from Study 2 indicated that challenges that provide conditionals and
loop blocks (in addition to movement blocks), as well as challenges with block limits,
need more time/interactions/attempts to be solved. This is in line with the results from
previous research [
The results from Study 3 were consistent with the results from Study 2. The statistical analysis showed that the age of the participants turned out to be an important factor, which indicates that age plays a key role in the success rate of this kind of coding challenges. However the factors that affected the success rates were the same in both studies.
Consequently, a Kodetu challenge is difficult when: a) the maze has turns and the total number of steps needed to go from the initial position to the endpoint is high, and b) when not only movement blocks, but also loop and conditionals blocks are needed to solve the challenge. The results of this research should be considered when designing learning activities to develop and enhance CT skills through maze-based programming challenges.