Computational thinking (CT) is widely accepted as a fundamental practice for equipping students to formulate and solve problems in the digital era. Many countries are adopting mandatory curricula for computer science (CS) lessons and CT education at high school. However, assessing how well students perform in CT activities is hard. Teachers face many challenges in the assessment process, because there is a limited number of resources for assessment and a lack of online access to resources. Therefore, the goal of this paper is to support teachers by developing an e ective automated Computational Thinking Assessment System (CoTAS) for instructing and evaluating CT skills of high school students in Python course. CoTAS facilitates the assessment of students' CT skills. Supported by CoTAS, teachers will be able to determine students' CT skill levels and shape their learning by continuously observing students' individual levels of development during the learning process. Teachers can access di erent resources to evaluate CT concepts, practices and perspectives. CoTAS can provide automatic feedback, so teachers can guide students directly when misconceptions arise. Moreover, Co-
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
TAS o ers multi-disciplinary assessment tools which can be used not only in the programming lessons, but also in other disciplines such as science, mathematics and social sciences in which CT skills are integrated. 1
Assessment plays a key role in the development of many educational reform movements. In the Horizon 2017 report, the key trends accelerating adoption of higher education technology are de ned [1]. According to this report, there is a growing focus on measuring learning. This trend describes an interest in assessment and a wide variety of methods and tools that educators use to evaluate, measure, and document academic readiness, learning progress, skill acquisition and other educational needs of students. On the other hand, in a study supported by European Commission called \Developing CT in compulsory education", the authors emphasize that the evaluation of CT skills is at an early stage and there is a need for further study, and that current assessment methods are insu cient for evaluating all aspects of CT skills [2]. Research on assessment literacy indicates that particularly teachers new to a content area and teaching practice often face many challenges when it comes to engaging in robust assessment practices in their instructions [3], [4], also many teachers teaching CT lack a strong background in programming [5]. In addition, constructionist approaches are actively used in CS education. The process of evaluating students in the constructionist learning environment has several di culties, because it is an open-ended and unde ned situation. Designoriented learning environments based on constructionist approaches require frequent evaluation in various forms.
For assessing CT, various evaluation tools are used: tests, observations, open-ended questions, computerbased coding exams to determine levels of comprehension of programming-related terms and to shape teaching processes [6], [7], [8], [9]; artifact based interviews, portfolios, thinking aloud method, projects, rubrics to assess students' e ort and development in the process [10], [11], [12], [13]; performance-based assessment, problem-based assessment, design scenarios to evaluate problem solving, algorithmic thinking and abstraction skills [14], [15], [16]; automated programming assessment tools to provide quick feedback [17], [18], [19], [20] and automated CT assessment tools to determine the CT skill levels of students [21], [22]. However, nding and validating CT measures that assesses CT with the holistic approach remains challenging. Brennan and Resnick [23] have six suggestions for assessing CT: supporting further learning, incorporating artifacts (portfolio analysis), illuminating artifact development processes, checking in at multiple waypoints, valuing multiple ways of knowing and including multiple viewpoints. Therefore, CoTAS aims to evaluate the CT concepts, practices and perspectives of high school students during education in a text-based programming language (Python). For evaluating students' comprehension of CT concepts, the automated quality assessment tool and test tool will be used [21], [24], [25], [26], [27]. CoTAS will o er problem-based assignments and problem-based tests to evaluate CT practices. Finally, CoTAS will present survey and interviews for evaluating students' CT perspectives. 2
The goal of CoTAS is to improve the e ectiveness of CT education and support teachers during the evaluation of CT skills of high school students. CoTAS provides both summative and formative evaluation tools for evaluating students' CT concepts, practices and perspectives [23], [28]. 2.1
CoTAS Tools for CT Concepts In order to follow students' comprehension of CT concepts (such as data structures, operators, conditionals, sequences, loops and functions), the automated quality assessment tool and the test tool of CoTAS will be used (Table 1). (1.1)The automated quality assessment tool will measure the pro ciency level of students' codes without the need of human supervision. CT metrics are identi ed to measure the pro ciencycomplexity level of code [21], [22], [24], [25], [26]. This tool of CoTAS will alleviate teachers' struggle with manual assessment of students' CT skills, as well as providing real-time feedback on how students develop CT competency over time. (1.2) Students' knowledge level about CT concepts will be evaluated at the end of each unit with the test tool consisting of multiple choice questions in order to follow the development during the training, to detect misconceptions and to identify issues that are not understood. The contents of the test will be programming related and Bloom's lower order thinking (knowledge, comprehension and application level) questions.
The evaluation results for CT concepts: In \My Progress" page of CoTAS, four di erent types of assessment scores are shown in Figure 1 (1 to 4). The rst part (Fig1.1) shows the pro ciency level of students' projects. The percentage of projects' pro ciency level will be presented in three levels (basic, developing and pro cient). The second part (Fig1.2) shows the usage frequency for CT concepts according to pro ciency levels. In order to evaluate students' pro ciency level of CT practices (such as formulation, abstraction, algorithmic thinking, reusing and remixing, being iterative and incremental, debugging) problem-based assignment tool and problem-based test tool of CoTAS will be used (Table 1). (2.1) The problem-based assignment tool will include authentic assessment questions [13] and problem solving scenarios to evaluate students' CT practices. As the automatic feedback, the information comprising of the number of attempts performed while solving a problem, the duration of code writing and the similarity of the nal output to the desired output will be provided to analyze the problem solving process of students. Teachers can also score assignments manually according to predetermined rubrics. The rubrics will o er descriptors of performance levels for each CT practice. (2.2) Problembased test tool will consist Bloom's higher order thinking (evaluation, synthesis and analysis level) multiple choice questions in order to detect development in the CT practices. This tool can be used not only in the programming lessons, but also in other disciplines such as science, mathematics and social sciences in which CT skills are integrated.
The evaluation results for CT practices: The third part of \My Progress" page (Fig1.3) presents problembased assignment and test scores for CT practices. 2.3
CoTAS Tools for CT Perspectives In order to evaluate students' CT perspectives (such as computational identity, programming empowerment, perspectives of expressing, connecting and questioning), CoTAS will o er the survey and interview questions (Table 1). (3.1) Students will be required to rate their agreement or disagreement with the statements in the survey. Surveys will be conducted at di erent time points to capture the development of students' CT perspectives. (3.2) Interviews will be used to obtain more details on students' CT perspectives; however the interview results will be manually evaluated according to predetermined rubrics so it will require time and e ort.
The evaluation results for CT perspectives: The fourth part of \My Progress" page (Fig1.4) shows the survey scores for CT perspectives of students at different time points. Finally, Fig1.5 shows all actions a student can perform in CoTAS. 3
CoTAS will provide di erent facilities for teachers, students and researchers during the assessment of CT skills of high school students. With the help of CoTAS, teachers will be able to access CT evaluation resources. The time spent for evaluation will be reduced through automatic feedback and provided resources. Teachers will be able to follow the progress of students during learning and they can manage the evaluation content easily. Teachers will see students' mistakes and misconceptions those are frequently made.
Students will be able to follow their own progress with the help of CoTAS. Students will receive instant and guiding feedback related during learning and evaluation. CoTAS will provide opportunity for accessing to resources anytime and anywhere for students. Through di erent assessment tools, students will be able to realize their own inadequacies and receive guidance to support their individual development.
Researchers will be able to identify the factors those are e ective in improving students' CT skills by making predictive assessments. They will be able to examine whether there is a relationship between the data obtained from CoTAS (such as access frequency to learning resources or number of shared projects etc.) and students' level of CT skills. Researchers will have the opportunity to examine the e ectiveness of different assessment tools in predicting students' nal achievements. 4
Many countries have taken steps to bring the CT concepts into their respective curriculum. Although there is a high level of consensus regarding the inclusion of the concepts related to CT into the curriculum, it is known that there is resource shortage and insecurity about how this high-level thinking skill should be evaluated. In this context, CoTAS will contribute to the (inter)national CS education eld. Moreover, CoTAS will provide di erent advantages for improving the CT skills of students. Coding is generally perceived as difcult by students, and it is one of the most di cult learning outcomes to evaluate for teachers. CoTAS will provide guided and instant feedback to students for improving the CT learning processes. CoTAS proposes to carry out formative and summative evaluation tools together in a holistic approach. Thus, it can be used not only in the programming lessons, but also in other disciplines such as science, mathematics and social sciences in which CT skills are integrated. The continuation of learning with the help of CoTAS (using automatic feedback) outside of regular lessons will provide the maximum bene t for students to reach their learning goals about CT skills. u
u A
A M A
A M s T l
o C o r T f S h e o m l t tou too tse i a t t n s o n s i n t e n e e s e en e i h h e h
m ) le l
b o o ro to r ,
p t A T s T ) s ) 1 e 2
e u m T r q
h t s T m T a i
n u s p l m ev .
) g ) l 1 i
y 2 a l
v a , s l a n o i t i d n o C , ) s s n o r o t i a t r c e n p u O F , , s s e p r o u o t c L s t n e n o
P
P , g n i k n i h d T an ic e
v m i h t t a i r r e og It l
g A n ,n ie o B ) it , g c ra i ts i b gn ign x g
u m b A e e
R D , n o i
, d l an ta
n g e t s a e l c i u n
m m is e t c r ra o e
F
n P ( R I u rc
) , g t in n n e
o m it r s e e w u o Q p m dn E a g g n n i i
t m c am en r n g o o C r P ,g , n y i t i t s s e n r e p Id x l E a f n o s io s e t e v v
ta i i t
t c u c e p e p
m sp s r o r e [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]
Kotini, I., Tzelepi, S. (
Bers, M. U. (2010). The TangibleK Robotics program: Applied computational thinking for young children. Early Childhood Research and Practice, 12(2).
Fronza, I., Ioini, N. E., Corral, L. (
Werner, L., Denner, J., Campe, S., Kawamoto, D. C. (2012). The fairy performance assessment: measuring computational thinking in middle school. In Proceedings of the 43rd ACM technical symposium on Computer Science Education , 215-220, ACM.
Webb, D. C. (2010). Troubleshooting assessment: an authentic problem solving activity for it education. Procedia-Social and Behavioral Sciences , 9, 903-907.
Djambong, T., Freiman, V. (2016). TaskBased Assessment of Students' Computational Thinking Skills Developed through Visual Programming or Tangible Coding Environments. International Association for Development of the Information Society.
Ala-Mutka, K. (2005). A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2), 83{102.
Edwards, S. H., Perez-Quinones, M. A. (2008). Web-CAT: automatically grading programming assignments. In ACM SIGCSE Bulletin, 40(3), 328-328. ACM.
Joy, M., Gri ths, N., Boyatt, R. (2005). The boss online submission and assessment system. Journal on Educational Resources in Computing (JERIC), 5(3), 2.
Korhonen, A., Malmi, L., Silvasti, P. (2003). TRAKLA2: a framework for automatically assessed visual algorithm simulation exercises. In Proceedings of Kolin Kolistelut/Koli Calling{Third Annual Baltic Conference on Computer Science Education, 48-56.