Computational Thinking is a key set of skills, which actually represents an obstacle to the clear de nition of an e ective assessment strategy. In this work, rst we explain why designing an assessment framework is even more challenging for rst year elementary school. Based on these premises, we propose to combine the Computational Thinking educational contribution and the problem-solving skills connected to it with the speci c needs of the educational context. Taking into account age and expected educational outcomes, we propose to evaluate algorithmic thinking, problem-solving, and creativity. Finally, we discuss possible challenges related to this approach, and report a set of lessons learned that could contribute to solving these challenges.
For some years, many of the existing assessment strategies have been based on code analysis [Cro14, Net13], which can result in misunderstanding the development of CT skills [RGMLR17] by ignoring a set of skills that cannot be measured by looking just at one's code. Therefore, it is suggested to consider multiple measures that are complementary, encourage and re ect deeper learning, and contribute to a comprehensive picture of students' learning [Gro15, BFCP18].
In general, the advantage of de ning an algorithmic solution of a problem lies in the re-usability of the dened solution, which can be useful at some point both to who solved the problem and to those who may face the same problem in the future. From this perspective, nding a solution can be considered as a social advantage, and this opens the possibility to motivate people to work together towards a solution, which requires at the same time the skill of e ective communication.
Based on these premises, we are working on a research project, called COmbining COmputational thiNking didAcTics and Software engineering in K1 2 (COCONATS1). The project aims at designing activities for K-12, which have as a principal output the acquisition of a reasoning approach that leads to clear programming, display, and implementation of the product. This not only allows pupils to address any discipline systematically and e ectively but also promotes a dimension of meta-cognitive reasoning, which in turn allows them to address and connect further complexities.
In this work, we propose to combine the CT educational contribution and the problem-solving skills connected to it with the speci c needs of the educational context. For the speci c case of primary school rst year, taking into account age and expected educational outcomes, we propose to evaluate algorithmic thinking, problem-solving, and creativity. Moreover, we discuss possible challenges related to this approach and report a set of lessons learned that could contribute to solving these challenges.
Section 2 reports the state of the art of existing CT systems of assessments. Section 3 brie y describes the objectives of the COCONATS project. Section 4 details the approach proposed in this paper, while Section 5 discusses possible problems that might emerge while developing this solution. Section 6 draws conclusions from this work, also proposing possible directions for future work. 2
As above mentioned, one of the main barriers to the implementation of Computational Thinking in the school context is the absence of an agreed assessment strategy. Indeed, assessment determines whether or not educational goals are being met and, at the same time, it drives the design of the curriculum itself [HL15].
In 2017 Roman-Gonzalez et al. [RGMLR17] provided an overview of the existing research works on this topic, and classi ed them based on their perspective (e.g., summative assessment, perceptionsattitudes scales, etc.). Some relevant examples are the Computational Thinking Test [Gon15, RGPGJF17], the Test for Measuring Basic Programming Abilities [MRH15], and the Commutative Assessment Test [WW15]. All the tests mentioned above have been designed for middle- and/or high-school students.
A large number of proposals and perspectives for Computational Thinking assessment re ects the extreme di culty of measuring it, due to a large number 1coconats.inf.unibz.it of variables both at a personal and social level, especially in lower school levels. Indeed, Roman-Gonzalez and his co-authors [RGPGMLR18] asserted that CT is currently still a confused psychological construct and that its evaluation remains a thorny and unresolved issue.
In recent years several evaluation tools have been developed from di erent approaches and various operational tools have been proposed [FP18]; nevertheless, very little research has been conducted to study whether these tools provide convergent measures and how to combine them in an educational environment [RGPGMLR18, BFCP18, FIC17].
Moreover, recent literature for assessment is present for universities or high schools. Nevertheless, there are just a few examples designed for lower school grades. The reason is clear: until the end of the primary school, pupils do not yet have the concept of abstraction, which is a founding concept of Computational Thinking [KA19]. The skills that are being built at this age are so many that it is very problematic to understand whether a positive result in the eld of CT is to refer to the activities prepared for that purpose or if a series of external factors a ect the assessment, such as the richness of the vocabulary, the familiar context, the extra-school opportunities to use technologies, and so on.
To this consideration, it should be added the peculiarity of the rst class of primary school (age: 6) when compared to the other classes of primary school. This class represents, in fact, a delicate phase (sometimes with many obstacles) in which children need to develop many skills and knowledge at the same time, such as learning to write, read, count, and be with others. This learning phase requires great commitment and psycho-physical energy. Moreover, the learning process does not proceed at the same pace for each individual; therefore, any assessment should take into account the di erences in individual learning styles, which represent an obstacle to obtaining comparable levels in any result. It is therefore tough to think of an evaluation of the process because there are too many individual variables that a ect the process in this age group.
One alternative possibility would then be to evaluate the outcome of the performed activities. However, Israel et al.[ICD+95] argued that in the outcome evaluation a very large sample of subjects would be required to obtain statistically signi cant results and, above all, the objectives that guide an outcome search often mature too long before being evaluated.
Moreover, when the outcome consists of a piece of software, as above mentioned evaluating the outcome by focusing only on the analysis of the code is not the optimal solution when the goal is evaluating the impact that CT has on ordinary activities, being CT by de nition a set of competences [BFCP18], also described by Corradini et al. [CLN17] and classi able in four general categories: mental processes, methods, practices, transversal objectives (such as creative, communicative and collaborative skills). 3
The COCONATS project involves both the Faculty of Computer Science and the Faculty of Education of the Free University of Bolzano, Italy.
Part of the project objectives is designing a set of educational activities to promote computational thinking in primary and secondary schools. The plan is to design a progression of activities during the curriculum, which starts with unplugged activities and endsup with hands-on exercises [CFPB18].
Computational Thinking has several concepts in common with the promotion of cognitive and relational Life Skills .Thus, COCONATS aims at promoting the acquisition of this second set of skills, which a vast literature indicates as essential not only for the structuring of ordinary life but also for the future workers.
Moreover, in line with the recent research interest in bringing Software Engineering to K-12 [PM19] to foster design skills and ability to manage the process towards the solution, the COCONATS project aims at understanding how Software Engineering can be fostered at di erent ages, at individual and collaborative level. 4
The research work that inspired us in formulating our approach is the one by Siu-Cheung Kong [Kon19]; in this work, Kong asked elementary school pupils to write simple re ections on what types of Computational Thinking concepts they used in carrying out their tasks and projects. The author considers the following components of CT among those that could be possibly assessed at the end of the primary school curriculum: 1. problem formulating,
Among the existing ones, this approach is probably the most suitable for the age group 6-11, but it is still too complicated for rst-grade children. For this reason, we propose to match the educational contribution of the CT model and the problem-solving skills connected to it with the curriculum, and therefore also with the educational needs of the speci c school. The strength of this approach is not fragmenting these educational objectives.
Taking into account these considerations, the age of the pupils and the expected educational outcomes, we propose to evaluate the activities speci cally prepared explicitly concerning these concepts: algorithm, generalization, problem-solving, creativity.
As shown in Figure 1 we, therefore, break up the problem of abstraction by evaluating: algorithmic thinking, problem-solving, and creativity. These aspects are further detailed in the remaining part of this section. In 2009, P. J. Denning [Den09] stated that algorithmic thinking is the basic idea behind computational thinking. In the computer science eld, an algorithm is de ned as any well-de ned sequence of actions that take a set of values as input and procedures some set of values as output [RH14]. An algorithmic view of the problem-solving process is valuable because it facilitates many activities essential to daily life. Problemsolving has been associated to CT in recent literature, for example in [KCO 17, RGMLR17].
For the evaluation of the algorithm, it has been shown that the researchers' observation plays a crucial role. There are references to the validity, in this eld, of the researchers' observation [Bur12, FGM13] and to the progress of the children's work; however, the object of assessment is only their nished product.
One possible criterion, in this case, could be represented by the information collected by using a thinkaloud protocol [EA17]: when the researcher thinks that a child has made with this activity a relevant experience of progressive sequences to achieve a goal (for example, an artifact), she/he could ask the child to verbalize that sequence of actions. 4.2
In recent literature, Computational Thinking has been associated with problem-solving [KCO 17, RGPGJF17]; nevertheless, the acquisition of problem-solving skills is still under discussion. A process frequently described is the 7-step process of Pretz et al. [PNS03], which consists of the following seven steps: 1. recognition or identi cation of a problem, 2. de nition and mental representation of the problem, 3. development of a strategy to solve the problem, 4. organization of knowledge concerning the problem, 5. allocation of mental and physical resources to solving the problem, 6. monitoring of progress toward the goal,
4.3
This concept shares with Computational Thinking the ability to add original solutions or improvements to a simple work or artifact. However, we are persuaded that creativity also helps to overcome problems creatively or, in other words, it can be considered a part of the problem-solving skill. The creative ability is a powerful resource to face personal and social situations, converting them in growth and learning opportunities. In this respect, WHO de nes creative thinking as a fundamental life skill that \contributes to both decision making and problem-solving by enabling us to explore the available alternatives and various consequences of our actions or non-action"; furthermore, it \can help to respond adaptively and with exibility to the situations of our daily lives" [O+94]. 5
The main question emerging is: which methods are appropriate for evaluating the CT components in each dimension for rst-year elementary school?
To answer this research question, we believe that we rst need to address some sub-questions: 1. Which didactic activities are functional for the purpose? 2. Which activities are the most appreciated and effective? 3. Which components of the model could present difculties? 4. What indications emerge for the creation of a pattern for assessment in this age group?
In the remaining part of this section, we describe some lessons learned from the rst part of the COCONATS project, together with possible problems (and solutions) that might emerge while working to provide an answer to this research question. 5.1
The rst activities that we have designed in the context of the COCONATS project for rst-year elementary school are manipulative activities using cubes (see Figure 2).
Speci cally, we have adopted cubes with square holes on each face, and a single, connecting stud located o -center so that children can connect the faces in any way they choose. The cubes come in two basic shapes. The rst is a cube that measures two centimeters on each side; the second a right prism where the base is a right-angled isosceles triangle with two equal rectangular faces of the same measure as those of the cube. A quick note on the technical side refers to the fact that cubes used in pre-school environments are exactly four times bigger than those we use with higher school level and for educational robotics experiments.
We believe that this is a particularly suitable tool, because children have to solve various problems to carry out the construction, as the cubes are equipped with a single protuberance for the attachment between one and the other, so it is necessary not only design the object to be made but also found a solution to hook the cubes. An algorithm of this type, the phases of which can be observed and a nished product can be obtained, can subsequently be obtained also through other activities, linked to other disciplinary elds, such as, the natural sciences.
Our opinion is that children should carry out these activities in groups of 2, to be observed and assisted by researchers almost individually.
The activities should be carried out not in a class, but in a specially prepared setting, bright and welcoming, to make children feel at ease and to present the activity as a moment of creative play. Children can build on the ground or a bench. The cubes that we have chosen are usually welcome by children because they look like Lego cubes, which are generally already known and accepted as suitable play. 5.3
As above mentioned, the skills that the children built in this age are already many. Thus, it is very problematic to understand if a positive result in the eld of CT is to refer to the activities prepared for the purpose or if a series of extra-school factors a ect the correct assessment. Moreover, this age represents a delicate phase in which children learn at the same time how to write, read, count, and be with others. In this process, each child has a di erent pace; therefore, individual di erences should be taken into account by an assessment strategy.
As previously noted, it is tough to think of an evaluation of the process because too many individual variables a ect the process in this age group. As to the process, the only description, but not the evaluation, is grounded on the observation of the researchers.
We believe that the alternative of the outcome's assessment remains the only possible for this age-level. In our context, for the realization of constructions with cubes, outcome assessment means:
Building the object according to the project, respecting its operational sequences;
5.4
Based on these considerations, for the three aspects that we proposed to evaluate (i.e., algorithm, problemsolving, and creativity), we recommend proceeding as follows.
The Algorithm that we believe to be correct to evaluate is how the product ts the original project. This allows us to observe the following phases: construct by following instructions and topological concepts; in the game with the cubes, establish a sequence of actions to complete the chosen construction; build freely with the cubes explaining the project before starting, with the possibility of using the instruction booklet; reproduce a gure according to a two-dimensional model; debugging: knowing how to correct.
As to Problem-Solving :
When there is a problem, pupils stop and think about how to solve it;
They elaborate a strategy (verbalizing it);
As to Creativity : children nd original solutions and add personal and relevant elements to the construction; they try to combine similar artifacts by categories (for example, a tree and a house).
The following section draws conclusions from this work, also proposing possible directions for future work. 6
This discussion aims to highlight the open question of Computational Thinking assessment at lower school levels, a context in which the variables are so many that it is challenging to allow assessments other than outcome assessments.
We believe that the evaluation of the nished product is the most reliable; however, it is essential that the researchers, together with the teacher, observe and compare the results. Indeed, school teachers can assess whether the concepts expressed in the activities are in line with the plan for the speci c class and whether the objectives correspond to them, to avoid overwhelming children with requests for performance that go beyond their real capabilities.
Future perspectives encourage us to continue to think about possible evaluation of the implementation of CT even at lower levels of education, and in those ages where cognitive development such as abstraction has not yet unfolded.
We, therefore, intend to propose similar activities in other schools in order to broaden our sample, and possibly create and test an evaluation framework that can be disseminated to other interested parties. [A+16] [BFCP18]
Meeting of the American Educational [Bur12] [CFPB18] [CLN17] [Cro14] [Den09] [EA17] [FGM13] [FIC17]
Research Association (AERA'12), Vancouver, Canada, pages 1{25, Vancouver, Canada, 2012. AERA.
pencil: Introducing programming-aswriting in the middle school classroom. Journal of Media Literacy Education, 4(2):121{135, 2012.
Claus Pahl, and Demis Basso. Coconats: Combining computational thinking didactics and software engineering in k-12. In Proceedings of the 19th Annual SIG Conference on Information Technology Education, pages 162{162. International World Wide Web Conferences Steering Committee, 2018.
Isabella Corradini, Michael Lodi, and Enrico Nardelli. Conceptions and misconceptions about computational thinking among italian primary school teachers. In Proceedings of the 2017 ACM Conference on International Computing Education Research, pages 136{144. ACM, 2017.
tional thinking. Communications of the ACM, 52(6):28{30, 2009.
think aloud method: what is it and how do i use it? Qualitative Research in Sport, Exercise and Health, 9(4):514{531, 2017.
and E Mavroudi. Problem solving by 5{6 years old kindergarten children in a computer programming environment: A case study. Computers & Education, 63:87{97, 2013.
Luis Corral. Teaching computational thinking using agile software engineering methods: A framework for middle schools. ACM Transactions on Computing Education (TOCE), 17(4):19, 2017.
Siu-Cheung Kong. Components and methods of evaluating computational thinking for fostering creative problem-solvers in senior primary school education. In Computational Thinking Education, pages 119{141. Springer, 2019. [RGPGMLR18] Marcos Roman-Gonzalez, JuanCarlos Perez-Gonzalez, Jesus Moreno-Leon, and Gregorio Robles.
Extending the nomological network of computational thinking with [RH14] [Win06] [Win14]