Learning Process Assessment and Improvement Oleg Mirzianov, Antanas Mitasiunas, Saulius Ragaisis Vilnius University, Vilnius, Lithuania {oleg.mirzianov, antanas.mitasiunas, saulius.ragaisis}@mif.vu.lt Abstract. The necessity of lifelong learning is more and more recognized. Therefore, improvement of learning is very important. The process capability maturity modeling elaborated by the Software Engineering community could be employed for this purpose. This paper contributes to the solution of learning improvement problem based on process quality attributes modeling approach. The consciousness as a learning process quality characteristic is introduced. Learning process assessment model based on R. M arzano taxonomy of learning objectives has been developed and validated. Keywords: Learning process assessment model, ISO/IEC 33000 compatible, learning improvement. 1 Introduction Euro-Inf framework standards and accreditation criteria for informatics degree programs [1] adopted by the Executive board of EQANIE (European Quality Assurance Network for Informatics Education) stress that graduates of the degree should be able to plan self-learning and improve personal performance as a foundation for lifelong learning and ongoing professional development. Such ability is particularly important for IT professional carrier because of the biggest ratio of knowledge to be acquired after university’s studies but it is undoubtedly essential for any studies . Lifelong learning is at some extent unavoidable necessity for successful professional carrier of everybody. Lifelong learning is not regular attendance at some formal courses. Lifelong learning is a part of regular daily work. Right attitude to lifelong learning is very important but it should be enforced by the conscious approach to learning activity, understanding of learning process , and an ability to learn, i.e. by learning improvement. The main question is: how to improve learning or in more generic terms how to improve the results of learning activity? Traditional industry gave answer to this question – a systematic method to improve the results is improvement of processes that produce these results. Software engineering community confirmed the validity of such thesis for software industry [2]. The focus of the authors of this paper is targeted to the research of applicability of process capability modeling, assessment and improvement methodology elaborated by software engineering community to improvement of other process oriented activities traditionally considered as creative activities like innovation and technology transfer [3], and, particularly, learning as a mental activity. The goal of this research is to create learning process assessment model, including base practices and generic practices, based on the learning process reference model [4, 5] and to validate it by experiments performed with participation of students of Vilnius University. The state of the art of learning process capability maturity modeling is provided in Section 2. Section 3 contains authors’ contribution - two-dimensional learning process assessment model. Validation of the model is presented in Section 4. The last Section concludes the results achieved and shares the future work. 2 Learning process modeling approaches Authors’ initial idea on learning as process oriented activity found confirmation in the work of R. Marzano on New Taxonomy of learning objectives [6] as an evolution of the well-known Bloom’s Taxonomy [7], based on knowledge complexity and knowledge structure based SOLO Taxonomy [8] in the area of education and psychology. Marzano explicitly treats the learning as a process and introduces well- structured approach to understanding of learning activity. Marzano’s New Taxonomy is based on consciousness of learning activities from automatically performed to conscious actions. Thomson’s approach [9] to learning process capability maturity modeling is based on parallels between software development and learning. Extended research on e-learning [10] is rather a special case of education than learning. The transition from process assessment standard ISO/IEC 15504 to process assessment standard series ISO/IEC 330xx ensures the possibility to address process quality characteristics other that process capability. Such need arises when building the learning process assessment model grounded by Marzano’s consciousness based New Taxonomy. The consciousness of process performance is considered as the essential measurable learning processes characteristic determining learning success . The pioneering idea of modeling process characteristics other than process capability is provided in [11]. ISO/IEC 33003 [12] allows to define own process quality characteristics. 3 Continuous Learning Process Assessment Model Continuous Learning process assessment model (PAM) has been developed according to ISO/IEC 33000 series requirements. The consciousness has been selected as learning process quality characteristic. So, the model has two dimensions: consciousness dimension and process dimension. 3.1 Consciousness Dimension Consciousness dimension establishes the measurement framework of consciousness of learning process. The requirements for process measurement frameworks are defined in ISO/IEC 33003 [12] and the reference implementation of such framework is the process capability measurement framework [13]. The process consciousness measurement framework presented in [5] specifies: the process consciousness levels from 0 (Incomplete) to 3 (Conscious), the process attributes (PA), the required process achievements , and the process of measurement. PAM expands each of the 4 process attributes through the inclusion of a set of generic practices (Table 1). The Generic Practices (GP) are activities of a generic type and provide guidance on the implementation of the attribute's characteristics. During the evaluation of process consciousness, the primary focus is on the performance of the generic practices. The performance of all generic practices ensures the full achievement of the process attribute. Table 1. Generic practices by process atributes and consciousness levels Level 0: Incomplete process Process attributes Generic practices - - Level 1: Performed process PA1.1.Process performance GP 1.1.1. Achieve the defined process outcomes. attribute. Level 2: Motivated process PA 2.1. M otivated process GP 2.1.1. Self-assess the importance of process performance. performance attribute. GP 2.1.2. Self-assess the ability to perform the process effectively. GP 2.1.3. Self-assess the positive emotions concerning the process performed. GP 2.1.4. Self-assess the motivation to perform the process. GP 2.1.5. M ake the decision to perform process. Level 3: Conscious process PA 3.1. Planned process GP 3.1.1. Define the clear goal of the process performed and performance attribute. the target knowledge state. GP 3.1.2. Create the strategy to achieve process goal and the target knowledge state. GP 3.1.3. Develop the plan to achieve the process goal and the target knowledge state. GP 3.1.4. Determine the resources, milestones and schedule for the achievement of process goal and the target knowledge state. PA 3.2. Tracked process GP 3.2.1. Track the process performance against the plan. performance attribute GP 3.2.2. Assess the clarity and unambiguity of the knowledge learned. GP 3.2.3. Assess the precision and trustworthiness of the knowledge learned. The rating scale of a process quality characteristic or process attribute should represent the extent of its achievement [12]. Standard rating scale has been inherited from the process capability measurement framework [13]. This scale is used for measuring achievements of process attributes and performance of practices. The process consciousness level model has been also inherited from the reference implementation of process measurement framework [13]. 3.2 Process Dimension Process dimension is based on the Learning Process Reference model (PRM). According to ISO/IEC 33004 requirements for Process Reference Model [14] the processes are defined specifying a statement of the purpose of the process and a set of outcomes which demonstrate successful achievement of the process purpose. This PRM consists of 7 processes to be performed by a learner and it has been presented in [5]. PAM should expand the PRM process definitions by including a set of process performance indicators called base practices for each process [13]. A base practice (BP) is an activity that addresses the purpose of a particular process. Consistently performing the base practices associated with a process help the consistent achievement of its purpose. The performance of all base practices associated with the process ensures the full achievement of the process outcomes. The following sets of base practices are defined for each Learning process in the process dimension (LEAR.1 – LEAR.7 are the identifiers of the processes ). LEAR.1. Knowledge Retrieve Ability Development BP 1.1. Recognize knowledge items. Identify the first occurrence of knowledge items; recognize them when faced with them again. BP 1.2. Reproduce knowledge and perform the procedures. Remember the common features and purpose of subject area procedures. Learn to perform them without major errors, but do not necessarily understand how and why the procedure is performed. LEAR.2. Knowledge Synthesis Ability Development BP 2.1. Recognize the essential and non-essential features of the knowledge items. Identify the essential and non-essential features of the knowledge items and distinguish which knowledge items and features of knowledge items are related to the subject area, and which are not related, or related to, but are not significant. BP 2.2. Generalize a set of knowledge items by single essential feature. Search for the essential linking feature of similar knowledge items. BP 2.3. Represent the abstract information of the subject area in yourself comprehensible form. Represent the abstract information acquired in the form suitable for easy understanding and further operation (e.g., symbols, images, self-acceptable explanations). BP 2.4. Aggregate knowledge items and structures. Integrate the knowledge items acquired and their structures into a whole. LEAR.3. Knowledge Analysis Ability Development BP 3.1. Compare the subject area knowledge items and procedures among. Identify the similarities and differences of the comparable subject area knowledge items and procedures. BP 3.2. Classify the knowledge acquired. Reasonably distribute the knowledge into meaningful classes according to the similarities and differences of the subject area knowledge items and procedures. BP 3.3. Analyze mistakes in subject area knowledge. Evaluate the correctness of the subject area information, when finding a fault repair and identify the cause. BP 3.4. Identify special cases of subject area knowledge. Analyze the specific features of the exclusive knowledge items comparing them with learned knowledge. BP 3.5. Search for the functioning of known principles in practical situations. Search for the functioning of known principles in specific situations, check whether the conditions necessary for the principles are satisfied. LEAR.4. Knowledge Application Ability Development BP 4.1. Solve the problems based on possessed knowledge aggregate. Derive the problem solution based on possessed knowledge aggregate. In cases of knowledge deficiency, determine what knowledge is missing, acquire the knowledge need and solve the problem. BP 4.2. Assess the alternative solutions. Identify for the possible solutions of the subject area problem, establish criteria for the comparison of the solutions, and select the most suitable solution. LEAR.5. Motivation Assessment BP 5.1. Assess the importance for learner of knowledge and skills to be acquired. Assess how much the subject-related knowledge and skills are important for the learner. Argue the assessment and verify the correctness of the arguments . BP 5.2. Evaluate the own abilities to acquire the knowledge and skills. Evaluate how the learner himself aware of the chance to learn the subject. Argue the evaluation and verify the correctness of the arguments . BP 5.3. List the positive emotions caused by learning. Identify the positive emotions caused by learning of the subject, collect them, determine the reasons of these emotions, and verify the correctness of the arguments . BP 5.4. Identify the reasons for motivation to learn. Identify which factors determine the most current disposition to learn, argue this assessment, and verify the correctness of the arguments . LEAR.6. Learning Goals Definition BP 6.1. Identify the target the knowledge level. Identify the target the knowledge level: knowledge retrieve, synthesis, analysis, or application ability. BP 6.2. Define the learning goals. Define the learning goals corresponding the target knowledge level. BP 6.3. Select the learning strategy. Consider alternative learning strategies (taking into account factors such as environmental conditions, human inclinations, the specifics of the subject, etc.), evaluate them according to criteria defined in advance, and select the strategy most suitable for achieving the learning goals. BP 6.4. Develop the learning plan. Depending on the chosen strategy, create the learning plan and foresee the necessary resources to carry out the plan. BP 6.5. Select the suitable sources for learning. Based on learning plan and strategy define the criteria for selection of learning sources. Select the sources most suitable for the learning goals. LEAR.7. Learning Results Tracking BP 7.1. Assess the learning achievements correspondence to learning goals. Assess how much learning achievements fit for the learning objectives . In case of non-compliance, identify the causes and corrective actions. BP 7.2. Assess the consistency, clarity, and unambiguity of knowledge learned. Assess how much the knowledge learned is clear for the learner, which knowledge items are clear and understandable, and for which the learner is not entirely true. Upon detecting any inconsistencies, determine the causes. BP 7.3. Assess the trustworthiness of knowledge being learned. Reasonably assess the correctness and accuracy of the acquired knowledge in the subject area. Determine the causes of incorrect or inaccurate knowledge. 4 Model Validation Validation of an adequacy of the model was performed by assessing learning process before and after learning session at Vilnius University with the same students for the subject “Computer Architecture”. Learning process assessment was performed in guided self- assessment style with each student individually 2-3 hours long. The outcome of assessment process is the documented student’s learning process consciousness profiles. The profiles are acquired via guiding students through Learning process model processes and establishing to what extent students are performing their learning consciously. The learning process was assessed for level 1 only because absolute majority of students wouldn’t reach the level 2. In total 22 participants agreed on being assessed. These are students to whom assessor was giving practical lectures of Computer Architecture. The selection of the subject was motivated by the fact that its examination tasks are meant to assess students’ ability to apply the knowledge gained during semester. Awareness of both students and subject has allowed the assessor to be more accurate in explaining and giving examples along with comparing how students described their learning and the learning abilities they have demonstrated during the semester. Two assessments have been performed. The assessment before university session was held to receive and analyze model adequacy towards terminology and ability to cover learning activity. Based on the received feedback some minor adjustments have been made in the model. The assessment after university session was done with already slightly improved Learning process model and assessment process. This assessment was more accurate because the students already knew the model terminology and what is expected during the assessment. Also students already knew the whole scope of the subject needed for the exam and to what extent they covered it. 15 out of 22 students agreed to be assessed. Figure 1 displays the assessment results (after session) in percent of performance of all processes by 2 groups of students respectively. Fig. 1. Validation of the Learning process assessment model Group A consists of students who got during the exam marks 1-3 out of 6 possible, when group B consists of students with marks 4-5. The process performance for each group is calculated as the average of individual assessments of all students of the group. It should be noted that group B students performed learning more consciously. The 4th process of the model is meant to assess the ability to apply aggregated knowledge in solving new tasks , i.e. similar to the requirements of the exam. Therefore, the exam marks and the results of 4th process assessment of each students have been compared. Table 2 provides the results that shows the relation between them. Table 2. Assessed students with examination mark and Learning process model 4th process Exam 1 2 2 3 3 3 3 3 4 4 4 4 5 5 5 4th proc. 36 34 63 55 62 63 65 72 73 73 73 73 74 74 85 Student 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 5 Conclusions and Future Work The consciousness is selected as learning process quality characteristic. ISO/IEC 33000 compatible Learning process assessment model based on R. Marzano taxonomy of learning objectives is developed. The first levels of the model has been validated with the students of Vilnius University. Further validation could be performed for more accurate adequacy by increasing both amount and variety of learners. It is planned to include the final year students that hopefully will allow to assess the higher levels of consciousness. Assessments at the beginning and at the end of the first semester will allow to observe changes in students’ learning at the first year of adaptation to university. References 1. EQANIE. Euro-Inf framework standards and accreditation criteria for informatics degree programmes, Version: 2015-10-12: http://www.cski.cz/files/pdf/equanie.pdf 2. ISO/IEC 15504-5. Information Technology – Process Assessment – Part 5: An exemplar software life cycle process assessment model (2012). 3. Besson, J., Woronowicz, T., M itasiunas, A., Boronowsky, M .: Innovation, Knowledge- and Technology Transfer Process Capability M odel – innoSPICETM . In: Software Process Improvement and Capability Determination, Communications in Computer and Information Science, Vol. 290, pp. 75--84 (2012). 4. M arcinka, J., M irzianov, O., M itasiunas, A.: Learning Process M aturity M odel. In: Software Process Improvement and Capability Determination, Communications in Computer and Information Science, Vol. 477, pp. 261--267 (2014). 5. M irzianov, O., M itasiunas, A.: Continuous Learning Process Assessment M odel. In: CEUR Workshop Proceedings, Vol. 1368, pp. 55--62 (2015). 6. M arzano, R.J.: Designing a New Taxonomy of Educational Objectives. Corwin Press (2001). 7. Bloom, B. S.: Taxonomy of educational objectives: The Classification of Educational Goals. New York, Longmans, Green (1956). 8. Biggs, J. B., Collis, K. F.: Evaluating the quality of learning: the SOLO taxonomy (structure of the observed learning outcome). Academic Press, New York (1982). 9. Thompson, E.: Using a Subject Area M odel as a Learning Improvement M odel. In: The Eighth Australasian Computing Education Conference (ACE2006), pp. 197--203, January (2006). 10.Novickis, L., M itasiunas, A., Rikure, T., Jurenoks, A.: Promotion of e-learning solutions via information technology transfer concept and Baltic regional competence network. In: VARE 2011: Virtual and augmented reality in education: proceedings of annual international conference, pp. 71--80 (2011). 11.Wells, C., Ibrahim, L., LaBruyere, L.: A New Approach to Generic Attributes. Systems Engineering, Vol. 6, No.4, 301--308 (2003). 12.ISO/IEC TR 33003 Information technology –Process Assessment – Requirements for process measurement frameworks (2013). 13.ISO/IEC TR 33004 Information technology –Process Assessment – Requirements for process reference, process assessment and maturity models (2013). 14.ISO/IEC TR 33020 Information technology – Process Assessment – Process measurement framework for assessment of process capability (2013).