=Paper=
{{Paper
|id=Vol-2190/TACKLE_2018_paper_2
|storemode=property
|title=Envisioning a Computational Thinking Assessment Tool
|pdfUrl=https://ceur-ws.org/Vol-2190/TACKLE_2018_paper_2.pdf
|volume=Vol-2190
|authors=Ilenia Fronza,Claus Pahl
|dblpUrl=https://dblp.org/rec/conf/ectel/FronzaP18
}}
==Envisioning a Computational Thinking Assessment Tool==
https://ceur-ws.org/Vol-2190/TACKLE_2018_paper_2.pdf
Envisioning a Computational Thinking Assessment Tool
Ilenia Fronza Claus Pahl
Free University of Bolzano Free University of Bolzano
Piazza Domenicani 3, 39100 Bolzano, Italy Piazza Domenicani 3, 39100 Bolzano, Italy
ilenia.fronza@unibz.it claus.pahl@unibz.it
create an agreed definition, as CT was rather a broad
term [BR12, Win14]. Then, researchers and educa-
Abstract tors have focused on designing new activities to foster
CT across the curriculum. Unfortunately, in many
Recent work on Computational Thinking
cases more weight has been given to the development
(CT) has focused on proposing new curricula
of methods to teach CT than on proposing how learn-
but in many cases the assessment phase has
ing will be assessed and evaluated. This is a relevant
been overlooked. The issue is critical because
issue, because appropriate assessment instruments are
appropriate assessment is needed to facilitate
needed to incorporate CT in the curriculum [HL15].
the incorporation of CT in the curriculum.
What is now clear from the existing literature Most of the existing assessment procedures focus
is that there is a need to build on top of the on code analysis. This approach might be convenient
existing multiple forms of assessments, in or- in a professional environment [CF15], but it neglects
der to integrate multiple approaches and reach CT broader aims [Cro14, Net13]. Other procedures,
a comprehensive assessment of CT learning. while valuable for research and for providing a view
In this paper, we envision a system that in- of student’s learning, are subjective, time-consuming,
tegrates different types of assessments while and not easily usable regularly in classrooms [Gro17].
providing an intuitive interface in order to al- Nowadays, many researchers agree that using only
low teachers to see and supervise the overview one type of assessment can lead to misunderstand
of the learning process, with the possibility to the development of CT skills [RGMLR17]. There-
assess individually the student’s learning. To fore, in order to reach a comprehensive assessment of
assess the suitability of our idea, we describe CT learning, a “system of assessments” needs to be
the Proof of Concept of a mobile application adopted, i.e., a combination of omplementary assess-
to assist CT assessment, and we discuss the ments tools. To this end, S. Grover [Gro15] suggests
challenges that need to be solved to create to consider multiple measures that are complementary,
such an application. encourage and reflect deeper learning, and contribute
to a comprehensive picture of students’ learning.
1 Introduction At this point it should be considered that the use of
a system of assessments could make the teacher’s work
Computational Thinking (CT) is considered as a key even more complicated, which could result in limiting
set of skills that must be learned by today’s gener- considerably the adoption of this type of systems in the
ation of learners, both in the context of STEM and classroom. In fact, the teacher would need to regularly
other subjects [Gro17]. Therefore, CT has caught the apply different types of assessment (e.g., code analysis,
attention of a broad academic community. Many stud- interview) and then integrate the results to obtain an
ies have first tried to capture the essence of CT and to overall assessment.
Copyright c by the paper’s authors. Copying permitted for To address this issue, we envision a system that in-
private and academic purposes. tegrates different types of assessments while providing
In: A. Piotrkowicz, R. Dent-Spargo, S. Dennerlein, I. Koren, an intuitive interface in order to allow teachers to see
P. Antoniou, P. Bailey, T. Treasure-Jones, I. Fronza, C. Pahl and supervise the overview of the students’ learning
(eds.): Joint Proceedings of the CC-TEL 2018 and TACKLE
2018 Workshops, co-located with 13th European Conference
process, with the possibility to assess individual learn-
on Technology Enhanced Learning (EC-TEL 2018), 03-09-2018, ing. To assess the suitability of our idea, we describe
published at http://ceur-ws.org the Proof of Concept (PoC) of a mobile application,
and then we discuss possible challenges that need to to the classroom, in addition to building these systems,
be solved in order to create such application. another goal must be to reflect on how to facilitate
Section 2 reports the state of the art of existing their adoption in the classroom, without leaving on
CT systems of assessments. Section 3 describes the the teacher’s shoulders the task of manually integrat-
proposed Proof of Concept of a mobile application to ing different types of assessments to achieve an overall
assist CT assessment, and Section 4 discusses the chal- assessment.
lenges that need to be faced in order to create such ap- Despite this need, few if any tools exist that enable
plication. Section 5 draws conclusions from this work, real-time, overall, formative assessment of CT. As we
also proposing possible directions for future work. describe in the next section, we aim to fill this research
gap envisioning an assessment tool that assists CT as-
2 State of The Art sessment, by integrating different types of assessments
and providing an intuitive interface.
Recent work on Computational Thinking (CT) has
focused on environments and tools that foster CT
[GP13, RWI10], new curricula for CT (e.g., [SFH+ 12,
3 CT Assessment Tool: A Proof Of
FEIJ+ 14]), also using CT as a medium for teaching Concept
other subjects (e.g., [FEIC15, FZ15, Edw11]). In this We envision a mobile application that supports CT
scenario, a major gap has emerged in research on CT assessment as follows. It integrates different types of
assessment. Indeed, in many cases more weight is assessments, also providing an intuitive interface in or-
given to the development of methods to teach and fos- der to allow teachers to see and supervise the overview
ter CT without thinking about how those methods will of the students’ learning process, with the possibility
be assessed and evaluated. This issue is critical, be- to assess individually the students’ learning. In this
cause assessment not only determines whether or not section, we describe a Proof of Concept of this appli-
educational goals are being met, it also drives the de- cation. First, we describe a possible approach to define
sign of the curriculum [HL15]. Grover and Pea make the assessment framework that the application should
the gravity of the lack of CT assessment clear: “With- implement. Then, we detail possible design choices
out attention to assessment, CT can have little hope that would ease the assessment process. Finally, we
for making its way successfully into any K-12 curricu- report on architectural considerations.
lum” [GP13].
Given the absolute need for an assessment method-
3.1 Assessment Framework
ology, many efforts in the last years aimed specifically
at tackling the issue of CT assessment. An overview Following the guidelines of the existing literature, our
of the proposed approaches, divided according to their assessment framework, being a systems of assessments,
perspective (e.g., summative assessment, perceptions- should assess different skills, such as: CT concepts,
attitudes scales, etc.), can be found in the recent work practices, and perspectives [BR12], cognitive skills,
of Román-González et al. [RGMLR17]. and also social and relational skills [Wor94]. The
What is clear from the existing literature, is the “Goal-Question-Metrics” (GQM) measurement model
need to build on top of the multiple forms of assess- [BCR94] could be adopted in order to provide an ef-
ments that have been proposed so far, in order to reach fective view of student’s learning. In fact, the GQM
a comprehensive assessment of CT learning. Research approach helps to clearly specify the object of study,
is now moving in this direction, and some examples of the aspect of study, the purpose of the assessment and
“systems of assessments” have been proposed [Gro15]. the environment in which the data is collected.
For example, Brennan and Resnick [BR12] de- The GQM approach foresees the definition of mea-
scribed three approaches for assessing the development surement goals, questions, and metrics. In our case,
of CT concepts, practices, and perspectives. Fronza et goals specify the assessment needs (i.e., CT skills) in
al. [FIC17] developed a framework to assess the de- a formal way; questions define information gaps that
velopment of computational concepts and practices. need to be filled to understand whether a measurement
Román-González et al. [RGMLR17] aimed at study- goal can be achieved or not; and measurements help
ing the convergent validity of their CT summative- to answer the measurement questions.
aptitudinal assessment test with respect to other as- An example of such a model is depicted in Figure 1:
sessment tools. S. Grover [Gro17] described the mul- the problem of assessing the learning of computational
tiple forms of assessments designed and empirically concepts is modeled as a GQM goal. The goal is then
studied in a middle school introductory computing assessed using measurement questions: in the example
curriculum. of Figure 1, these questions are how well the student
If we want these systems of assessments to make it understands conditionals, sequences, and loops. Each
question is then answered using one or more measure- feedback on those aspects that have not been learned
ments, for example, one measurement to understand yet or require a more detailed explanation. In the ex-
whether the student understands the correct usage of ample shown in Figure 3, it would be immediately clear
a loop is the presence of a “for” construct in the source to the teacher that “CT skill 2” has been achieved.
code provided by the student, when this is required by The rules defining whether a tile becomes green,
the given problem. yellow, or red embed the knowledge extracted during
the GQM definition, i.e., the conditions under which
3.2 Design of the Visual Support teachers can assess whether a skill has been acquired
or not.
A dashboard would be an effective means to visually
display the outcome of the assessment, which is mod- 3.3 Architectural Considerations for the As-
elled in form of GQM models. In dashboards, in fact, sessment Tool
pre-attentive properties (e.g., color, shape, location)
The architectural considerations for the CT measure-
are used to maximize the understanding of the dis-
ment tool we consider are the following:
played information and to guide attention [Few13].
Each tile in the dashboard shall represent a spe- • Functionality. The assessment tool shall have
cific measurement. Suppose that, for example, the tile the necessary features to receive the input of key
in Figure 2 shows the number of “for” statements in concept or areas from each student, as well as
the code. In this case, there are 12 loops in the code the comments and preliminary assessment of the
and this number is higher respect to the previous mea- teacher. It shall allow the track, follow up, com-
surement (as shown by the upward arrow). The tile parison, evaluation and projection of each con-
is colored in green to show that a sufficient level has sidered skill. Due to the scope of the tool, the
been reached, therefore the teacher does not have to modules and features pertaining data reports will
focus on this tile anymore. If, instead, no loops in be of particular importance through the develop-
the code are found, then the tile shall be red. The ment and acceptance of this tool.
teacher, in this case, needs to find out if there was a
• Maintainability. The CT model, formulated as
need of a loop in the project and the students did not
a GQM measurement model, is of crucial impor-
use the appropriate block but, for example, just re-
tance for the interpretation of the data. There-
peated the same command many times. In this case,
fore, it has to be implemented in a modular, con-
the computational concept “loops” would not be as-
figurable way for two reasons: it has to possible to
sessed positively [FIC17].
configure an existing model to accomplish small to
Figure 3 shows an example in which three skills (i.e.,
medium changes but we want also that it is possi-
goals) are depicted with associated questions and met-
ble to replace the entire model with a new one,
rics. For example, metric B is colored in red and there-
if new considerations arise. Moreover, changes
fore requires the attention of the teacher. Metric E
could be necessary to allow using the assessment
is colored in yellow, therefore depicts a warning level
tool at different education levels (such as K-12
which should be kept under observation. Tiles that
or university). In consequence, the assessment
are colored in green depict a satisfactory fulfillment of
framework has to be implemented as a pluggable,
the metric.
configurable component of the final tool, which in-
The same rule applies to the each skill tile. When
teracts with the designed dashboard through clear
all the corresponding metrics are colored in green, also
interfaces.
the skill tile is colored with the same color in order to
show immediately to the teacher that there is no need • Usability. A desirable characteristic for the as-
of attention for that skill (for example, “CT skill 2” in sessment tool is to be designed to be used on a
Figure 3). The skill tile is red, instead, when immedi- tablet because it would allow teachers to move
ate intervention is needed because all the metrics are easily in the room (from one student’s worksta-
colored in red (for example, “CT skill 3”). A yellow tion to another one).
skill tile (for example, “CT skill 1”) indicates that the
The next section discusses possible challenges that
corresponding skill should be kept under observation.
need to be faced in order to create our envisioned CT
The existing literature recommends to use an as-
assessment tool.
sessment framework in several points of the learning
process [BR12]. The proposed dashboard would help
the teacher to focus on critical aspects: she/he will
4 Discussion
ignore the green tiles and focus on the orange and red Before starting the development of a real application
ones, which means she/he would assess and provide from the PoC that we have envisioned in Section 3,
Figure 1: Illustration of CT assessment, modeled in form of a GQM.
Social skills. One of the most critical aspects will
probably be assessing social and relational skills. In-
deed, a comprehensive method has not been proposed
so far, and the current approach is to evaluate life-
skills trainings through observations, questionnaires
Figure 2: Structure of a tile in the assessment dash-
[AVM15], or self-report [AM15]. The tool shall pro-
board.
vide an opportunity to collect and analyse data from
these sources, at different points in the learning pro-
a number of challenges need to be solved. In the fol- cess.
lowing, we discuss the main challenges that we have
already identified. Iterative development and empirical re-
Definition of measurements and thresholds. search. The tool shall be developed in an itera-
The definition of the CT assessment model (Section tive fashion, performing empirical research with stu-
3.1) is a crucial step towards the implementation of dents and teachers, and adapting it to the feedback
our PoC, and requires the definiton of measurements obtained. A research method that supports such ap-
and thresholds. This requires extensive effort and em- proach is “design research”. By its nature, design re-
pirical research, by iterations, also involving teachers search is relevant for educational practice as it aims
to collect their feedback. to develop research-based solutions for complex prob-
lems in educational practice or to develop or vali-
Customization. To facilitate the adoption of this
date theories about processes of learning and teaching
tool in the classroom, attention needs to be taken to
[VdAGMN06]. Design research incorporates system-
make it customizable. For example, a teacher may
atic educational design processes and, like all system-
decide to carry out activities to improve only a specific
atic educational and instructional design processes, it
subset of skills; in this case, he/she should be able
is cyclical in character: analysis, design, evaluation
to select (and integrate) only a part of the available
and revision activities are iterated until an appropri-
measurements in the tool.
ate balance between ideals (“the intended”) and real-
Code analysis. The tool shall be adaptable to ization has been achieved [VdAGMN06].
each level of education. From the code analysis per-
spective, this means for example foreseeing the possi- Clearly, this type of method requires numerous
bility of analyzing code written in different program- classroom experiments to be carried out, and repli-
ming languages (e.g., block- or text-based). cated, to draw relevant information from them.
Figure 3: Dashboard structured using pre-attentive visualization techniques (adapted from [JSS13]).
5 Conclusion and Future Work dren with hearing impairments in in-
clusive schools. Childhood Education,
In this paper, we described our vision of a mobile appli-
91(6):469–476, 2015.
cation to assist CT assessment, and then we discussed
possible challenges that need to be solved in order to [BCR94] Victor R. Basili, Gianluigi Caldiera,
create such application. The results of this analysis al- and H. Dieter Rombach. The goal ques-
low us to outline a research agenda that we believe will tion metric approach. In Encyclopedia
help to implement the idea we have described. An in- of Software Engineering. Wiley, 1994.
terdisciplinary approach is needed, which draws from
[BR12] Karen Brennan and Mitchel Resnick.
software engineering, didactics, and cognitive psychol-
New frameworks for studying and as-
ogy. Research efforts should focus primarily on: i)
sessing the development of computa-
definition of the assessment framework (using a three-
tional thinking. In 2012 Annual Meet-
step approach: what? so what? now what?); ii) im-
ing of the American Educational Re-
plementation of a software product that will sustain
search Association (AERA’12), Van-
operationally the assessment framework; iii) data min-
couver, Canada, pages 1–25, Vancou-
ing and effective visualization; iv) empirical research
ver, Canada, 2012. AERA.
in classroom.
[CF15] L. Corral and I. Fronza. Better code
References for better apps: A study on source code
quality and market success of android
[AM15] Hayley Anthony and Louise A McLean. applications. pages 22–32, 2015.
Promoting mental health at school:
short-term effectiveness of a popu- [Cro14] Dan Crow. Why every child should
lar school-based resiliency programme. learn to code, feb 2014.
Advances in School Mental Health Pro- [Edw11] Michael Edwards. Algorithmic com-
motion, 8(4):199–215, 2015. position: computational thinking in
music. Communications of the ACM,
[AVM15] Narges Adibsereshki, Abas Mahvashe
54(7):58–67, 2011.
Vernosfaderani, and Guita Movallali.
The effectiveness of life skills training [FEIC15] Ilenia Fronza, Nabil El Ioini, and Luis
on enhancing the social skills of chil- Corral. Students want to create apps:
Leveraging computational thinking to and Technology, Third Edition, pages
teach mobile software development. In 1568–1578. IGI Global, 2015.
Proceedings of the 16th Annual Confer-
ence on Information Technology Edu- [JSS13] Andrea Janes, Alberto Sillitti, and Gi-
cation, SIGITE ’15, pages 21–26, New ancarlo Succi. Effective dashboard de-
York, NY, USA, 2015. ACM. sign. Cutter IT Journal, 26(1), 2013.
[FEIJ+ 14] Ilenia Fronza, Nabil El Ioini, Andrea [Net13] Warren Nettleford. Primary school
Janes, Alberto Sillitti, Giancarlo Succi, children learn to write computer code,
and Luis Corral. If i had to vote on jul 2013.
this laboratory, i would give nine: In- [RGMLR17] Marcos Román-González, Jesús
troduction on computational thinking Moreno-León, and Gregorio Robles.
in the lower secondary school: Results Complementary tools for compu-
of the experience. Mondo Digitale, tational thinking assessment. In
13(51):757–765, 2014. Proceedings of International Con-
[Few13] Stephen Few. Information Dashboard ference on Computational Thinking
Design: Displaying data for at-a-glance Education (CTE 2017), S. C Kong,
monitoring, volume 5. Analytics Press J Sheldon, and K. Y Li (Eds.). The
Burlingame, CA, 2013. Education University of Hong Kong,
pages 154–159, 2017.
[FIC17] Ilenia Fronza, Nabil El Ioini, and Luis
Corral. Teaching computational think- [RWI10] Alexander Repenning, David Webb,
ing using agile software engineering and Andri Ioannidou. Scalable game
methods: A framework for middle design and the development of a check-
schools. ACM Transactions on Com- list for getting computational thinking
puting Education (TOCE), 17(4):19, into public schools. In Proceedings of
2017. the 41st ACM Technical Symposium on
Computer Science Education, SIGCSE
[FZ15] I. Fronza and P. Zanon. Introduction of ’10, pages 265–269, New York, NY,
computational thinking in a hotel man- USA, 2010. ACM.
agement school [introduzione del com-
putational thinking in un istituto al- [SFH+ 12] Amber Settle, Baker Franke, Ruth
berghiero]. Mondo Digitale, 14(58):28– Hansen, Frances Spaltro, Cynthia Ju-
34, 2015. risson, Colin Rennert-May, and Brian
Wildeman. Infusing computational
[GP13] Shuchi Grover and Roy Pea. Compu- thinking into the middle- and high-
tational thinking in k–12. a review of school curriculum. In Proceedings of the
the state of the field. Educational re- 17th ACM Annual Conference on In-
searcher, 42(1):38–43, Jan/Feb 2013. novation and Technology in Computer
Science Education, ITiCSE ’12, pages
[Gro15] Shuchi Grover. “systems of assess-
22–27, New York, USA, 2012. ACM.
ments” for deeper learning of com-
putational thinking in k-12. In An- [VdAGMN06] Jan Van den Akker, Koeno Grave-
nual Meeting of the American Educa- meijer, Susan McKenney, and Nienke
tional Research Association, pages 1–9, Nieveen. Educational design research.
Washington, DC, 2015. AERA. Routledge, 2006.
[Gro17] Shuchi Grover. Assessing algorithmic [Win14] Jeannette M. Wing. Computational
and computational thinking in k-12: thinking benefits society, January 2014.
Lessons from a middle school class-
room. In Emerging research, practice, [Wor94] World Health Organization and others.
and policy on computational thinking, Life skills education for children and
pages 269–288. Springer, 2017. adolescents in schools. In Life skills ed-
ucation for children and adolescents in
[HL15] Roxana Hadad and Kimberly A Law- schools, pages 2pts–in. 1994.
less. Assessing computational thinking.
In Encyclopedia of Information Science