=Paper=
{{Paper
|id=Vol-2434/invited1
|storemode=property
|title=About Computational Thinking Assessment: a Proposal for Primary School First Year from a Pedagogical Perspective (invited paper)
|pdfUrl=https://ceur-ws.org/Vol-2434/invited1.pdf
|volume=Vol-2434
|authors=Marina Sartor Hoffer,Sara Baroni,Ilenia Fronza,Claus Pahl
|dblpUrl=https://dblp.org/rec/conf/ectel/HofferBFP19
}}
==About Computational Thinking Assessment: a Proposal for Primary School First Year from a Pedagogical Perspective (invited paper)==
https://ceur-ws.org/Vol-2434/invited1.pdf
About Computational Thinking Assessment: a Proposal
for Primary School First Year from a Pedagogical
Perspective
Marina Sartor Hoffer Sara Baroni
Free University of Bozen/Bolzano Free University of Bozen/Bolzano
Piazza Domenicani 3, 39100 Bolzano, Italy Piazza Domenicani 3, 39100 Bolzano, Italy
marina.sartor@unibz.it sara.baroni@unibz.it
Ilenia Fronza Claus Pahl
Free University of Bozen/Bolzano Free University of Bozen/Bolzano
Piazza Domenicani 3, 39100 Bolzano, Italy Piazza Domenicani 3, 39100 Bolzano, Italy
ilenia.fronza@unibz.it claus.pahl@unibz.it
sumed a broad meaning, and it took some years of
discussion in the research community to obtain an
Abstract agreed operational definition [BR12, Win14]. In 2016,
the International Society for Technology in Education
Computational Thinking is a key set of skills, (ISTE) defined CT as the ability to develop creative
which actually represents an obstacle to the solutions to a problem, which can be described through
clear definition of an effective assessment an algorithmic strategy [A+ 16].
strategy. In this work, first we explain why
Computational Thinking is considered as a key set
designing an assessment framework is even
of skills that should be learned by everybody [Gro17]
more challenging for first year elementary
regardless of the chosen career path. However, being a
school. Based on these premises, we propose
set of skills represents an obstacle to the precise defini-
to combine the Computational Thinking edu-
tion of an effective strategy for CT learning assessment
cational contribution and the problem-solving
and evaluation, which is in turn of paramount impor-
skills connected to it with the specific needs
tance to incorporate CT in a curriculum [HL15].
of the educational context. Taking into ac-
count age and expected educational outcomes, For some years, many of the existing assessment
we propose to evaluate algorithmic thinking, strategies have been based on code analysis [Cro14,
problem-solving, and creativity. Finally, we Net13], which can result in misunderstanding the de-
discuss possible challenges related to this ap- velopment of CT skills [RGMLR17] by ignoring a set
proach, and report a set of lessons learned that of skills that cannot be measured by looking just at
could contribute to solving these challenges. one’s code. Therefore, it is suggested to consider mul-
tiple measures that are complementary, encourage and
1 Introduction reflect deeper learning, and contribute to a comprehen-
sive picture of students’ learning [Gro15, BFCP18].
After Jeannette Wing’s seminal article in 2006
[Win06], the term Computational Thinking (CT) as- In general, the advantage of defining an algorithmic
solution of a problem lies in the re-usability of the de-
Copyright c 2019 for this paper by its authors. Use permitted fined solution, which can be useful at some point both
under Creative Commons License Attribution 4.0 International to who solved the problem and to those who may face
(CC BY 4.0). the same problem in the future. From this perspective,
In: I. Fronza, C. Pahl (eds.): Proceedings of the 2nd Systems of finding a solution can be considered as a social advan-
Assessments for Computational Thinking Learning Workshop
(TACKLE 2019), co-located with 14th European Conference
tage, and this opens the possibility to motivate people
on Technology Enhanced Learning (EC-TEL 2019), 17-09-2019, to work together towards a solution, which requires at
published at http://ceur-ws.org. the same time the skill of effective communication.
Based on these premises, we are working on a of variables both at a personal and social level, espe-
research project, called COmbining COmputational cially in lower school levels. Indeed, Román-González
thiNking didAcTics and Software engineering in K- and his co-authors [RGPGMLR18] asserted that CT is
1 2 (COCONATS1 ). The project aims at designing ac- currently still a confused psychological construct and
tivities for K-12, which have as a principal output that its evaluation remains a thorny and unresolved
the acquisition of a reasoning approach that leads to issue.
clear programming, display, and implementation of the In recent years several evaluation tools have been
product. This not only allows pupils to address any developed from different approaches and various op-
discipline systematically and effectively but also pro- erational tools have been proposed [FP18]; neverthe-
motes a dimension of meta-cognitive reasoning, which less, very little research has been conducted to study
in turn allows them to address and connect further whether these tools provide convergent measures and
complexities. how to combine them in an educational environment
In this work, we propose to combine the CT ed- [RGPGMLR18, BFCP18, FIC17].
ucational contribution and the problem-solving skills Moreover, recent literature for assessment is present
connected to it with the specific needs of the educa- for universities or high schools. Nevertheless, there are
tional context. For the specific case of primary school just a few examples designed for lower school grades.
first year, taking into account age and expected ed- The reason is clear: until the end of the primary
ucational outcomes, we propose to evaluate algorith- school, pupils do not yet have the concept of abstrac-
mic thinking, problem-solving, and creativity. More- tion, which is a founding concept of Computational
over, we discuss possible challenges related to this ap- Thinking [KA19]. The skills that are being built at
proach and report a set of lessons learned that could this age are so many that it is very problematic to un-
contribute to solving these challenges. derstand whether a positive result in the field of CT is
Section 2 reports the state of the art of existing to refer to the activities prepared for that purpose or if
CT systems of assessments. Section 3 briefly describes a series of external factors affect the assessment, such
the objectives of the COCONATS project. Section 4 as the richness of the vocabulary, the familiar context,
details the approach proposed in this paper, while Sec- the extra-school opportunities to use technologies, and
tion 5 discusses possible problems that might emerge so on.
while developing this solution. Section 6 draws conclu- To this consideration, it should be added the pe-
sions from this work, also proposing possible directions culiarity of the first class of primary school (age: 6)
for future work. when compared to the other classes of primary school.
This class represents, in fact, a delicate phase (some-
times with many obstacles) in which children need to
2 State of The Art develop many skills and knowledge at the same time,
As above mentioned, one of the main barriers to such as learning to write, read, count, and be with oth-
the implementation of Computational Thinking in the ers. This learning phase requires great commitment
school context is the absence of an agreed assess- and psycho-physical energy. Moreover, the learning
ment strategy. Indeed, assessment determines whether process does not proceed at the same pace for each
or not educational goals are being met and, at the individual; therefore, any assessment should take into
same time, it drives the design of the curriculum itself account the differences in individual learning styles,
[HL15]. which represent an obstacle to obtaining comparable
In 2017 Román-González et al. [RGMLR17] pro- levels in any result. It is therefore tough to think of an
vided an overview of the existing research works on evaluation of the process because there are too many
this topic, and classified them based on their per- individual variables that affect the process in this age
spective (e.g., summative assessment, perceptions- group.
attitudes scales, etc.). Some relevant examples are the One alternative possibility would then be to eval-
Computational Thinking Test [Gon15, RGPGJF17], uate the outcome of the performed activities. How-
the Test for Measuring Basic Programming Abili- ever, Israel et al.[ICD+ 95] argued that in the outcome
ties [MRH15], and the Commutative Assessment Test evaluation a very large sample of subjects would be
[WW15]. All the tests mentioned above have been de- required to obtain statistically significant results and,
signed for middle- and/or high-school students. above all, the objectives that guide an outcome search
A large number of proposals and perspectives for often mature too long before being evaluated.
Computational Thinking assessment reflects the ex- Moreover, when the outcome consists of a piece of
treme difficulty of measuring it, due to a large number software, as above mentioned evaluating the outcome
by focusing only on the analysis of the code is not
1 coconats.inf.unibz.it the optimal solution when the goal is evaluating the
impact that CT has on ordinary activities, being CT 5. reusing and remixing,
by definition a set of competences [BFCP18], also de-
scribed by Corradini et al. [CLN17] and classifiable in 6. being iterative and incremental,
four general categories:
7. testing and debugging.
• mental processes,
Among the existing ones, this approach is probably
• methods, the most suitable for the age group 6-11, but it is still
too complicated for first-grade children. For this rea-
• practices,
son, we propose to match the educational contribution
• transversal objectives (such as creative, commu- of the CT model and the problem-solving skills con-
nicative and collaborative skills). nected to it with the curriculum, and therefore also
with the educational needs of the specific school. The
3 The COCONATS project strength of this approach is not fragmenting these ed-
ucational objectives.
The COCONATS project involves both the Faculty of Taking into account these considerations, the age
Computer Science and the Faculty of Education of the of the pupils and the expected educational outcomes,
Free University of Bolzano, Italy. we propose to evaluate the activities specifically pre-
Part of the project objectives is designing a set of pared explicitly concerning these concepts: algorithm,
educational activities to promote computational think- generalization, problem-solving, creativity.
ing in primary and secondary schools. The plan is to As shown in Figure 1 we, therefore, break up the
design a progression of activities during the curricu- problem of abstraction by evaluating: algorithmic
lum, which starts with unplugged activities and ends- thinking, problem-solving, and creativity. These as-
up with hands-on exercises [CFPB18]. pects are further detailed in the remaining part of this
Computational Thinking has several concepts in section.
common with the promotion of cognitive and relational
Life Skills .Thus, COCONATS aims at promoting the
acquisition of this second set of skills, which a vast
literature indicates as essential not only for the struc-
turing of ordinary life but also for the future workers.
Moreover, in line with the recent research interest
in bringing Software Engineering to K-12 [PM19] to
foster design skills and ability to manage the process
towards the solution, the COCONATS project aims at
understanding how Software Engineering can be fos-
tered at different ages, at individual and collaborative
level.
4 Proposed approach
The research work that inspired us in formulating our
approach is the one by Siu-Cheung Kong [Kon19]; in
this work, Kong asked elementary school pupils to
write simple reflections on what types of Computa-
tional Thinking concepts they used in carrying out Figure 1: The three aspects considered in the proposed
their tasks and projects. The author considers the CT assessment model.
following components of CT among those that could
be possibly assessed at the end of the primary school
curriculum: 4.1 Algorithmic Thinking
1. problem formulating, In 2009, P. J. Denning [Den09] stated that algorith-
mic thinking is the basic idea behind computational
2. problem decomposition,
thinking. In the computer science field, an algorithm
3. abstracting and modularizing, is defined as any well-defined sequence of actions that
take a set of values as input and procedures some set of
4. algorithmic thinking, values as output [RH14]. An algorithmic view of the
problem-solving process is valuable because it facili- to explore the available alternatives and various con-
tates many activities essential to daily life. Problem- sequences of our actions or non-action”; furthermore,
solving has been associated to CT in recent literature, it “can help to respond adaptively and with flexibility
for example in [KCÖ17, RGMLR17]. to the situations of our daily lives” [O+ 94].
For the evaluation of the algorithm, it has been
shown that the researchers’ observation plays a cru- 5 Emerging problems and possible so-
cial role. There are references to the validity, in this lutions
field, of the researchers’ observation [Bur12, FGM13]
and to the progress of the children’s work; however, The main question emerging is: which methods are
the object of assessment is only their finished product. appropriate for evaluating the CT components in each
One possible criterion, in this case, could be repre- dimension for first-year elementary school?
sented by the information collected by using a think- To answer this research question, we believe that
aloud protocol [EA17]: when the researcher thinks we first need to address some sub-questions:
that a child has made with this activity a relevant 1. Which didactic activities are functional for the
experience of progressive sequences to achieve a goal purpose?
(for example, an artifact), she/he could ask the child
to verbalize that sequence of actions. 2. Which activities are the most appreciated and ef-
fective?
4.2 Problem-solving
3. Which components of the model could present dif-
In recent literature, Computational Think- ficulties?
ing has been associated with problem-solving
[KCÖ17, RGPGJF17]; nevertheless, the acquisition 4. What indications emerge for the creation of a pat-
of problem-solving skills is still under discussion. A tern for assessment in this age group?
process frequently described is the 7-step process of In the remaining part of this section, we describe
Pretz et al. [PNS03], which consists of the following some lessons learned from the first part of the CO-
seven steps: CONATS project, together with possible problems
(and solutions) that might emerge while working to
1. recognition or identification of a problem,
provide an answer to this research question.
2. definition and mental representation of the prob-
lem, 5.1 Activities.
The first activities that we have designed in the con-
3. development of a strategy to solve the problem,
text of the COCONATS project for first-year elemen-
4. organization of knowledge concerning the prob- tary school are manipulative activities using cubes (see
lem, Figure 2).
Specifically, we have adopted cubes with square
5. allocation of mental and physical resources to holes on each face, and a single, connecting stud lo-
solving the problem, cated off-center so that children can connect the faces
in any way they choose. The cubes come in two basic
6. monitoring of progress toward the goal, shapes. The first is a cube that measures two centime-
ters on each side; the second a right prism where the
7. evaluation of the solution for accuracy.
base is a right-angled isosceles triangle with two equal
rectangular faces of the same measure as those of the
4.3 Creativity
cube. A quick note on the technical side refers to the
This concept shares with Computational Thinking the fact that cubes used in pre-school environments are ex-
ability to add original solutions or improvements to a actly four times bigger than those we use with higher
simple work or artifact. However, we are persuaded school level and for educational robotics experiments.
that creativity also helps to overcome problems cre- We believe that this is a particularly suitable tool,
atively or, in other words, it can be considered a part because children have to solve various problems to
of the problem-solving skill. The creative ability is carry out the construction, as the cubes are equipped
a powerful resource to face personal and social situa- with a single protuberance for the attachment between
tions, converting them in growth and learning oppor- one and the other, so it is necessary not only design the
tunities. In this respect, WHO defines creative think- object to be made but also found a solution to hook
ing as a fundamental life skill that “contributes to both the cubes. An algorithm of this type, the phases of
decision making and problem-solving by enabling us which can be observed and a finished product can be
obtained, can subsequently be obtained also through phase in which children learn at the same time how
other activities, linked to other disciplinary fields, such to write, read, count, and be with others. In this pro-
as, the natural sciences. cess, each child has a different pace; therefore, indi-
vidual differences should be taken into account by an
assessment strategy.
As previously noted, it is tough to think of an evalu-
ation of the process because too many individual vari-
ables affect the process in this age group. As to the
process, the only description, but not the evaluation,
is grounded on the observation of the researchers.
We believe that the alternative of the outcome’s
assessment remains the only possible for this age-level.
In our context, for the realization of constructions with
cubes, outcome assessment means:
• Building the object according to the project, re-
specting its operational sequences;
• Finishing the work and present it.
5.4 Strategy.
Based on these considerations, for the three aspects
that we proposed to evaluate (i.e., algorithm, problem-
solving, and creativity), we recommend proceeding as
follows.
The Algorithm that we believe to be correct to eval-
uate is how the product fits the original project. This
allows us to observe the following phases:
• construct by following instructions and topologi-
cal concepts;
Figure 2: An example of manipulative activities using • in the game with the cubes, establish a sequence
cubes. of actions to complete the chosen construction;
• build freely with the cubes explaining the project
5.2 Setting.
before starting, with the possibility of using the
Our opinion is that children should carry out these instruction booklet;
activities in groups of 2, to be observed and assisted
by researchers almost individually. • reproduce a figure according to a two-dimensional
The activities should be carried out not in a class, model;
but in a specially prepared setting, bright and welcom- • debugging: knowing how to correct.
ing, to make children feel at ease and to present the
activity as a moment of creative play. Children can As to Problem-Solving:
build on the ground or a bench. The cubes that we
have chosen are usually welcome by children because • When there is a problem, pupils stop and think
they look like Lego cubes, which are generally already about how to solve it;
known and accepted as suitable play. • They produce various options to solve it;
5.3 Outcome assessment. • They correct wrong solutions;
As above mentioned, the skills that the children built • They detect a problem;
in this age are already many. Thus, it is very prob-
lematic to understand if a positive result in the field of • They elaborate a strategy (verbalizing it);
CT is to refer to the activities prepared for the purpose • They apply a strategy;
or if a series of extra-school factors affect the correct
assessment. Moreover, this age represents a delicate • They check if it works.
As to Creativity: Research Association (AERA’12),
Vancouver, Canada, pages 1–25, Van-
• children find original solutions and add personal couver, Canada, 2012. AERA.
and relevant elements to the construction;
[Bur12] Quinn Burke. The markings of a new
• they try to combine similar artifacts by categories
pencil: Introducing programming-as-
(for example, a tree and a house).
writing in the middle school class-
The following section draws conclusions from this room. Journal of Media Literacy Ed-
work, also proposing possible directions for future ucation, 4(2):121–135, 2012.
work.
[CFPB18] Alessandro Colombi, Ilenia Fronza,
6 Conclusion and Future Work Claus Pahl, and Demis Basso. Co-
conats: Combining computational
This discussion aims to highlight the open question of thinking didactics and software en-
Computational Thinking assessment at lower school gineering in k-12. In Proceedings
levels, a context in which the variables are so many of the 19th Annual SIG Conference
that it is challenging to allow assessments other than on Information Technology Educa-
outcome assessments. tion, pages 162–162. International
We believe that the evaluation of the finished prod- World Wide Web Conferences Steer-
uct is the most reliable; however, it is essential that ing Committee, 2018.
the researchers, together with the teacher, observe and
compare the results. Indeed, school teachers can assess [CLN17] Isabella Corradini, Michael Lodi,
whether the concepts expressed in the activities are in and Enrico Nardelli. Conceptions
line with the plan for the specific class and whether the and misconceptions about computa-
objectives correspond to them, to avoid overwhelming tional thinking among italian primary
children with requests for performance that go beyond school teachers. In Proceedings of
their real capabilities. the 2017 ACM Conference on In-
Future perspectives encourage us to continue to ternational Computing Education Re-
think about possible evaluation of the implementation search, pages 136–144. ACM, 2017.
of CT even at lower levels of education, and in those
ages where cognitive development such as abstraction [Cro14] Dan Crow. Why every child should
has not yet unfolded. learn to code, feb 2014.
We, therefore, intend to propose similar activities
[Den09] Peter J Denning. Beyond computa-
in other schools in order to broaden our sample, and
tional thinking. Communications of
possibly create and test an evaluation framework that
the ACM, 52(6):28–30, 2009.
can be disseminated to other interested parties.
[EA17] David W Eccles and Güler Arsal. The
References think aloud method: what is it and
[A+ 16] Computer Science Teachers Associa- how do i use it? Qualitative Re-
tion et al. Operational definition of search in Sport, Exercise and Health,
computational thinking, 2016. 9(4):514–531, 2017.
[BFCP18] Demis Basso, Ilenia Fronza, Alessan- [FGM13] Georgios Fessakis, Evangelia Gouli,
dro Colombi, and Claus Pahl. Im- and E Mavroudi. Problem solving by
proving assessment of computational 5–6 years old kindergarten children
thinking through a comprehensive in a computer programming environ-
framework. In Proceedings of the ment: A case study. Computers &
18th Koli Calling International Con- Education, 63:87–97, 2013.
ference on Computing Education Re-
search, page 15. ACM, 2018. [FIC17] Ilenia Fronza, Nabil El Ioini, and
Luis Corral. Teaching computational
[BR12] Karen Brennan and Mitchel Resnick. thinking using agile software engi-
New frameworks for studying and as- neering methods: A framework for
sessing the development of compu- middle schools. ACM Transactions
tational thinking. In 2012 Annual on Computing Education (TOCE),
Meeting of the American Educational 17(4):19, 2017.
[FP18] Ilenia Fronza and Claus Pahl. En- [MRH15] Andreas Mühling, Alexander Ruf,
visioning a computational think- and Peter Hubwieser. Design and
ing assessment tool. In CC- first results of a psychometric test for
TEL/TACKLE@ EC-TEL, CEUR measuring basic programming abili-
Workshop Proceeding 2190, 2018. ties. In Proceedings of the workshop
in primary and secondary computing
[Gon15] Marcos Román González. Compu-
education, pages 2–10. ACM, 2015.
tational thinking test: Design guide-
lines and content validation. In Pro- [Net13] Warren Nettleford. Primary school
ceedings of EDULEARN15 confer- children learn to write computer
ence, pages 2436–2444, 2015. code, jul 2013.
[Gro15] Shuchi Grover. “systems of assess-
[O+ 94] World Health Organization et al.
ments” for deeper learning of com-
Life skills education for children and
putational thinking in k-12. In An-
adolescents in schools. pt. 3, train-
nual Meeting of the American Edu-
ing workshops for the development
cational Research Association, pages
and implementation of life skills pro-
1–9, Washington, DC, 2015. AERA.
grammes. Technical report, Geneva:
[Gro17] Shuchi Grover. Assessing algorithmic World Health Organization, 1994.
and computational thinking in k-12:
Lessons from a middle school class- [PM19] David Parsons and Kathryn Mac-
room. In Emerging research, practice, Callum. Agile and lean concepts
and policy on computational thinking, for teaching and learning: Bringing
pages 269–288. Springer, 2017. methodologies from industry to the
classroom. Springer, 2019.
[HL15] Roxana Hadad and Kimberly A Law-
less. Assessing computational think- [PNS03] Jean E Pretz, Adam J Naples, and
ing. In Encyclopedia of Information Robert J Sternberg. Recognizing,
Science and Technology, Third Edi- defining, and representing problems.
tion, pages 1568–1578. IGI Global, The psychology of problem solving,
2015. 30(3), 2003.
[ICD+ 95] Barbara A Israel, K Michael Cum- [RGMLR17] Marcos Román-González, Jesús
mings, Mark B Dignan, Catherine A Moreno-León, and Gregorio Robles.
Heaney, Daniel P Perales, Bruce G Complementary tools for compu-
Simons-Morton, and Marc A Zim- tational thinking assessment. In
merman. Evaluation of health edu- Proceedings of International Con-
cation programs: current assessment ference on Computational Thinking
and future directions. Health educa- Education (CTE 2017), S. C Kong,
tion quarterly, 22(3):364–389, 1995. J Sheldon, and K. Y Li (Eds.). The
Education University of Hong Kong,
[KA19] Siu-Cheung Kong and Harold Abel-
pages 154–159, 2017.
son. Computational Thinking Educa-
tion. Springer, 2019. [RGPGJF17] Marcos Román-González, Juan-
[KCÖ17] Özgen Korkmaz, Recep Cakir, and Carlos Pérez-González, and Carmen
M. Yasar Özden. A validity and re- Jiménez-Fernández. Which cognitive
liability study of the computational abilities underlie computational
thinking scales (cts). Computers in thinking? criterion validity of
Human Behavior, 72:558 – 569, 2017. the computational thinking test.
Computers in Human Behavior,
[Kon19] Siu-Cheung Kong. Components 72:678–691, 2017.
and methods of evaluating compu-
tational thinking for fostering cre- [RGPGMLR18] Marcos Román-González, Juan-
ative problem-solvers in senior pri- Carlos Pérez-González, Jesús
mary school education. In Compu- Moreno-León, and Gregorio Robles.
tational Thinking Education, pages Extending the nomological network
119–141. Springer, 2019. of computational thinking with
non-cognitive factors. Computers in 2014.
Human Behavior, 80:441–459, 2018.
[RH14] David Riley and Kenny A Hunt.
Computational thinking for the mod-
ern problem solver. Chapman and
Hall/CRC, 2014. [WW15] David Weintrop and Uri Wilensky.
[Win06] Jeannette M. Wing. Computational Using commutative assessments to
thinking. Comm. ACM, 49(3), 2006. compare conceptual understanding
in blocks-based and text-based pro-
[Win14] Jeannette M. Wing. Computational grams. In ICER, volume 15, pages
thinking benefits society, January 101–110, 2015.