The article raises the problem of researching the metacognitive skills of software testers in modern IT companies. It figures out what are the specific features of software testing experts' activity and what soft skills (non-technical skills) are the leading ones for its implementation, as well as what is the place among them for metacognitive skills. Using a survey among professionals in the field of software testing, the article establishes a connection between the professional title of software testing professionals achieved and the level of their metacognitive skills.
The mass introduction of digital technologies into the life of modern man has greatly shifted the accents and vector of IT development. The main vector in software development has shifted from manufacturability and innovativeness to its convenience, stability, reliability, and most importantly, quality: the Internet banking user would prefer the absence of calculation errors in finances to the color of the interface, and the user of medical software the accuracy of medical evidence to the speed they get it.
Emphasis on software quality led to the emergence of a separate specialization of IT
professionals, whose main purpose is to directly control the compliance of the
developed product with the expectations of users and customers. Software testing is
the process of analyzing and operating software to identify differences in existing and
required operating conditions (defects) and to evaluate the features of this software
To date, testing has evolved into an independent industry of information technology with its unique techniques and theoretical framework. Demand for relevant professionals has grown significantly over the past 10 years. In many companies most of the time developers used to test their product for efficiency themselves, whereas in today's realities special departments and test teams are created, and they have their internal hierarchy and subordination.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Therefore, taking into account the growth of the IT industry in general and a particular focus on quality issues, training software testers is an urgent task for the modern IT sphere.
The analysis of domestic and foreign experience in preparing students of ITspecialties shows that theoretical and practical content of education was created based on the general concept of "information technology worker" or "software engineer", without paying attention to specific specializations (Classification of professions DK 003:2010). To date, Ukrainian higher education institutions do not single out testing as an independent specialty, and that requires a specific approach to training future professionals.
The analysis of specialized publications (Raluca Florea, Viktoria Stray, 2018) and current vacancies for the position of testing specialist shows that any tester must have basic knowledge of architecture, development and application administration and popular operating systems. In addition, the specialist must have a basic understanding of programming languages, be able to read code intuitively, as well as quickly adapt to new technological tools and environments. Software testing also has its theoretical framework, which includes a set of basic techniques and approaches to writing tests, passing them, analyzing results and potential vulnerabilities of the product. Moreover, the tester should apply in practice popular approaches to test documentation and reporting.
However, rather moderate requirements from employers for the quality of technical training of software testing specialists are offset by the extremely high requirements for their non-technical skills. Unimpeded communication with any team member, direct communication with the customer, active participation in planning and direct influence on the development process - all these tasks of professional activity require nontechnical skills of the specialist, and they should be formed at the stage of his professional training in the institution of higher education.
In basic documents of EU member countries (The European Qualifications Framework, European Skills, Competences, Qualifications and Occupations, 2019;), as well as the United States (IFTF, Al Forces Shaping Work and Learning in 2030, 2018), such non-technical non-professional skills are called 'essential skills' or 'soft skills'.
These documents make it possible to introduce a common understanding and a unified classification of soft skills for the global economic space.
To date, soft skills are defined by a set of non-specialized, super-professional characteristics, which are responsible for professional success, high productivity and, unlike specialized (technical) skills, are not related to a specific field of application. To the category of 'soft skills' researchers include: • individual qualities and attributes of a person (Cobb, 2015; Goleman, 2000;
Yarkova, Cherkasova, 2016), such as self-organization, the ability to speak in public;
• interpersonal communication skills (Robles, 2012) or teamwork, collaboration and
joint effort in a project office (Grugulis, Vincent, 2009; Yarkova, Cherkasova,
2016);
• skills focused on human relationships, as well as characteristics that form emotional
intelligence, that is the ability of a person to solve behavioral and cognitive tasks
(Peterson, Van Fleet, 2004);
• cognitive and methodological skills
The latter group of soft skills includes metacognitive skills or metacompetences, that is
the capacity to work on competencies, to reframe and transfer them from one field to
another, even from informal to formal learning
Metacognitive skills of a specialist are workaround techniques of metacognitive regulation of his professional activity and, based on activity and metacognitive approach (Karpov, Cholodna), they contain: • goal-setting ability (formulation of objectives, conversion of objectives into tasks, result objectification, defining criteria and indicators of the result, to divide the problematic situation into known and unknown); • the ability of anticipation (predicting scenarios of problematic situations, ways of solving a problem, as well as ways of conditions development, identifying probable risks, creating thinking models); • decision-making skills (formulation of hypotheses, alternatives, choosing a method of decision-making, analyzing the type and kinds of decision-making, working with errors and outcomes of decision-making); • planning ability (developing a work plan, correlating the work plan with the purpose
and the model); • reflexive or metacognitive self-control skills (analyzing the result of the activity, applying the mechanisms of reflection, determining the ways of correcting the activity, slowing down and termination of the activity, conducting a comprehensive reflective analysis of individual activity).
Among the competences of IT professionals related to the metacognitive group of soft
skills the most important are: the ability to work with complex problems in conditions
of uncertainty and ambiguity, the ability to plan their activities and adjust their course
(Software Engineering Body of Knowledge); ability to see the task simultaneously at
different levels of detail, the ability to formulate requirements and evaluate
opportunities, the ability to make decisions in the limited time, strategic thinking, the
ability to analyze their own mistakes, etc.
It is metacognitive skills that enable the monitoring and management of the
specialist's cognitive, emotional and regulatory processes that ensure the effectiveness
of his work
And whereas the problem of the development of software engineers' communicative
skills in the process of professional training and software testing professionals' in real
professional activity in recent years has been actively addressed in modern science and
practice by adjusting the content and teaching methods in higher education
establishments (general training disciplines)
Moreover, it should be noted that according to the Education Law (p. 12), stated by the Verkhovna Rada of Ukraine from 05.09.2017 No. 2145-VIII (# 11), the mandatory components in the content of teaching students in general secondary education institutions include the formation of such (metacognitive) skills, as the ability to express an opinion verbally and in writing, critical and systemic thinking, the ability to logically justify position, creativity, initiative, ability to constructively manage risks, evaluate and make decisions, solve problems, the ability to cooperate with others.
And so far, these requirements for learning outcomes of grades 5-9 students have been reflected in the Draft State Standard for Basic Secondary Education (Source # 12) in all educational fields, including Informatics, which is essentially a propaedeutic of professional IT education. For example, by the end of grade 9, after studying the subject "Informatics" according to the Concept of "New Ukrainian School" the student should have the following skills: • to argue and defend one's position, using various resources, comparing alternative
views from several information sources; • experiment with ideas and resources, solutions and technologies when creating information products, refining them for expression, solving learning and life problems, creating values or influencing the community; • to show persistence, adaptability, initiative, openness to creative experimentation
during the development of software projects; • offer solutions for real-world and virtual-based computer simulation; • handle challenges, eliminate mistakes and use them as an opportunity to improve
the project or develop it; • composes messages based on the visual representation of data; • explain the impact of emotions on teamwork, know and use ways to manage
emotions.
Therefore, it is important that the development of metacognitive skills of IT specialists continues at the next levels of education (in higher education) and that there will be continuity in education.
The objective of the study is to determine the level of metacognitive skills of software testing specialists of Ukrainian, Russian and Belorussian IT companies and to define the problem of the need for their purposeful development.
The study was conducted among Ukrainian software testing experts during DecemberJanuary 2020. The study involved 27 software testing experts working for leading IT companies (EPAM Systems, Plarium, Global Logic), ages 21 to 45 (44.4% women and 55.6% men), with the working experience in IT from 4 months to 12 years, with different professional levels (Junior -18,5%, Middle - 40,7%, Senior - 22,2%, Team Lead - 18,5%).
All of the participants in the study have higher education, 29.6% of them received education in the humanities in various specialties (journalism, variety singing, translation, management, finance), 22.2% have a technical education not related to IT (railway engineering. transportation, aeronautic engineering, marine engineering, etc.), 14.8% - mathematical education (applied mathematics) and 40.8% - IT education (software engineering, systems engineering, systems analytics, computer science, computer systems, software of automated control systems, applied IT). 100% of the interviewed specialists were trained in the area of IT companies' activities.
The methodology used in this piece of research was a mixed one combining qualitative and quantitative methods, the method of filling in the questionnaire is electronic.
For the statistical analysis of the collected data, the «IBM SPSS Statistics 23» was used. 2.1
The purpose of it is to determine: the level of specialists' awareness of the activity features in the field of software testing; requirements for professional qualifications, which primarily contribute to their successful professional careers, as well as requirements for the professional training of testers in higher education institutions. The questionnaire contained open-ended questions (In your opinion, what distinguishes the activity of a software testing specialist in an IT company from other IT specializations? What are the soft skills that first and foremost contribute to a successful professional career as a software tester? Do you think that you have sufficient professional qualifications at a higher education institution to continue to work as a software testing specialist (specify why)? In your opinion, how can the professional training of future software testing professionals in higher education institutions be improved? etc.); 2.2
Constructed by Rayne Sperling and Gregory Schraw (1994), the Metacognitive
Awareness Inventory (MAI) is a well-established and useful assessment of
metacognition. The MAI has been used in hundreds of studies, ranging from basic to
applied research. It is a 52-item inventory with two broad categories (knowledge of
cognition and regulation of cognition), with several sub-categories:
• knowledge about cognition - declarative knowledge, procedural knowledge and
conditional knowledge;
• regulation of cognition – planning, information management strategies,
comprehension monitoring, debugging strategies and evaluation
Declarative knowledge is the factual knowledge the specialist needs before being able to process or use critical thinking related to the topic, knowledge of one's skills, intellectual resources, and abilities (knowing about, what, or that). Procedural knowledge is the application of knowledge to complete a procedure or process, knowledge about how to implement cognitive procedures (e.g., strategies), know the process as well as when to apply the process in various situations. Conditional knowledge - the determination under what circumstances specific processes or skills should transfer, knowledge about when and why to use learning procedures, application of declarative and procedural knowledge with certain conditions presented. Regulation of cognition is the ability to regulate cognitive activity at different stages: • planning - planning, goal setting, and allocating resources before learning; • information management strategies - skills and strategy sequences used to process information more efficiently (e.g., organizing, elaborating, summarizing, selective focusing); • comprehension monitoring - assessment of one's learning or strategy use; • debugging strategies - strategies to correct comprehension and performance errors; • evaluation - analysis of performance and strategy effectiveness after a learning episode. 2.3
This technique was developed in 1998 and is a short questionnaire, fairly easy to use and interpret. Subjects are required to assess the cognitive strategies they use. The technique is recommended for use as an indicator of the level of metacognitive strategies development in the situation of purposeful learning and for the diagnosis of the metacognitive strategies of the professionals who use them in their work. The metacognitive strategies indicated in the methodology reflect the basic mechanisms of metacognitive skills actualization (objectification, schematizing, normalization, verbalization, reflection). It is also important that this technique is didactic in nature and stimulates thinking and reflexive activity of an interviewee, leads to a problematic situation, gives cause for reflection on existing metacognitive mechanisms/strategies. The scale consists of 12 statements that determine the degree of metacognitive skills generalization - the frequency of their use.
According to the results of the study, the following data was obtained. Among the specific features of the software testing specialist's work in IT companies, the respondents named (by frequency of statements): 1. responsibility for the quality of the product at all stages of its development, a specialist is a "quality engineer", which in turn requires analysis and the maximum of software inspections to ensure that the product is working properly; 2. metasystemic features, since it requires a professional programming competency of a tester (having programming experience, knowing programming code), and as well as the competencies of a business analyst and a project manager (knowing its organizational structure, having organizational and communication skills); 3. the need to analyze the product from several sides: from the customer's point of view, in addition to that - the software developer, and, what is more important - from the perspective of the user of the product; 4. creativity, since it requires a constant search for ways to test and optimize the test coverage.
The main non-technical professional qualities of the software testing specialist, according to the respondents, were: • communicative: communication, sociability, the ability to negotiate, the ability to put oneself in the position of others, the ability to seek and find compromises; • responsibility; • self-organization, the ability to overcome stress, the ability to perform repetitive mundane tasks; • the ability to solve problems, the ability to make decisions; • attentiveness, the ability to see the little things, thoroughness; • the ability to work in a team; • logic, analytical thinking, critical thinking, creativity, ability to formulate thoughts; • goal setting and the ability to plan activities; • organizational skills; • the ability to present the results of work; • desire for self-development; • foreign language proficiency.
Among the non-technical professional qualities related to metacognitive the interviewees named: the ability to solve problems (26%), decision-making ability (18.5%), goal setting (11%) and the ability to plan activities (9%), as well as generating hypotheses and risks assessment (7.4%).
These results are confirmed by the main employers' non-technical requirements to
a software testing specialist (based on the analysis of the positions)
Given the fact that metacognitive skills ensure successful completion of professional tasks, it was logical to assume that there is a connection between the level of development of metacognitive skills (metacognitive awareness) and the professional level of software testing professionals in IT companies. However, against all expectations, the obtained rate of Spearman correlation is very insignificant (r = 0.045) (Table 1). When dividing the sample into two polar groups (the lower professional level - junior - middle and the higher professional level - senior - team lead), metacognitive awareness indicators were also statistically insignificant. Therefore, it can be concluded that both software testing professionals who have a professional juniormiddle level and those who have a senior-team lead level have both high and low levels of metacognitive awareness. Thus, as the results of the study showed, as to the achieved professional level in IT companies experts in general and groups with lower and higher professional levels are not characterized by quantitative differences in metacognitive awareness. However, as the analysis showed, with minor general differences, metacognitive control of professionals with lower and higher professional levels is implemented by different strategies. Thus, calculating the correlation across the different subscales of Metacognitive awareness inventory showed that for senior-team lead professionals, the leading metacognitive strategies that ensure their effectiveness are the reflective assessment of their activities, their performance, and the development of alternative decision options, and for junior - middle - planning and reliance on available acquired knowledge.
Therefore, it can be argued that significant metacognitive soft skills of software testing professionals are those that provide metacognitive control at all stages of their professional activity - from goal setting and to reflectively evaluating its results and determining ways to adjust its progress. It is the metasystem, the relevance, the presence at all stages of solving professional problems that ensures the successful professional development of these specialists, their career growth. To identify the main metacognitive strategies most commonly used by software testing professionals in their professional activity, a 12-factor analysis of metacognitive skills was conducted using the method of Metacognitive Behavior Self-Assessment Scale provided by LaCosta.
To determine the number of contributing factors Cattell criterion was used. KaiserMeyer-Olkin Measure of Sampling Adequacy amounted to 0.618, which indicates that the factor model of the correlation matrix of this set of variables is satisfactory. Bartlett's test of sphericity gave the result that equals 0.000, which indicates the existence of a correlation between the variables of the source array and the possibility of grouping them by the tightness of the correlation. Four relatively independent factors were identified to explain 71.3% of the variations in the measures on the scales (Table 2). This indicator is considered sufficient for psychological research.
The skills that are included in the first factor, explaining 23.83% of the total variance and has a factor weight of 2.860, are the most influential on the metacognitive behavior of software testing professionals. It comprised: 1) overcoming subjective constraints (awareness of the ability to solve complex problems and persistent conscious search for solutions) (0.904); comprehension of achievements (correlation of subjective achievements with objective feedback) (0,893); strategic planning (purposeful planning, monitoring, and evaluation of activities) (0,699); formulation of questions (conscious formulation of questions that address gaps in certain areas of knowledge) (0.644). These skills are the most requested for a group of high-level professionals (senior-team lead).
The second factor (19.26% of the total variance, factor weight of 2,311) comprised: • modeling (construction of mental representations of experience) and schematizing (construction of schemes of phenomena and processes) (0.802); • definition of terminology (formulation of precise definitions for initially blurred, ambiguous or poorly understood terms) (0,690); paraphrasing and summarizing the information received (reframing ideas that come up) (0.639); conscious decision making (foreseeing the effect and consequences of each choice) (0.580). The third factor explains 15.75% of the total variance and has the factor weight of 1,890. It includes the following indicators: detection of cognitive behavior (definition of used cognitive strategies and their importance for solving the problem) (0,775), differential evaluation (reflective evaluation of their actions according to various criteria (0,760). The abilities included to the fourth factor (explains of 12.43% of the total variance and has factor weight of 1,492) - keeping a diary (writing down your personal thoughts) (0,845) and role-play (playback of partner/customer position of the communication, imaginative dialogue with him) have the least impact on metacognitive behavior of the testers. Regarding the consistency of the results obtained by the application of the Metacognitive Awareness Inventory and the Metacognitive Behavior Self-Assessment Scale provided by LaCosta, a statistically significant stable positive correlation of metacognitive awareness indicators and metacognitive behavior of professionals was found (r=0,709). Therefore, the higher the level of metacognitive awareness of respondents on the counts of metacognitive knowledge and metacognitive regulation, the more often they use metacognitive skills of the first and second factors.
Regarding the determination of changes that have to be made in the professional training of software testers for their successful professional activity, according to the survey results, 96.3% of software testing experts believe that there is a significant shortage of professional training for successful performance in IT companies in higher education institutions. Moreover, 59.2% of them stated that training programs in higher education institutions are inconsistent with the real professional activity of the software tester, that higher education represents a potential start, which should be compensated by long independent preparation for such activity, as well as training directly in the IT company.
89% of respondents said that training in higher education institutions does not ensure the formation of soft skills for future professionals, which causes the need for additional learning, doing special training, courses both within the IT company and externally (training centers, distance courses, etc.).
Among the measures to improve the training in higher education institutions, experts offered the following: internships for future testers in IT companies, on real projects (89%), enrollment in software testing courses for software engineers (59.2%), teaching professional courses exclusively by expert-practitioners in IT companies (40.7%), decision-making in real laboratory situations and tasks (introduction of contextual approach) (40.7%), rejection of outdated methodologies (18.5%). 4
The conducted research allows us to formulate the following conclusions.
Firstly, among employers' requirements for software testing professionals, apart from technical training, so-called non-technical, non-professional requirements or soft skills play an important role, which, in addition to communicative and teamwork skills, responsibility, stress management, include such metacognitive abilities as goal-setting, foreseeing, decision-making, planning and programming of personal activity, as well as carrying out its reflective analysis. It is metacognitive skills that monitor and manage the cognitive, emotional, and regulatory processes of the professional activities of software testing professionals, ensuring their performance and professional growth.
Secondly, the specific features of the professional activity of software testers in IT companies, as the experts surveyed stated, are responsibility for the quality of the product at all stages of its development, meta-systemic nature, multi-positioning, creativity. Their successful professional activity depends not only on technical, specialized knowledge and skills but also on non-technical ones, which include such metacognitive skills as problem-solving, decision-making skills, goal setting and planning skills, as well as generating hypotheses and risks assessment skills. This is confirmed by the lack of higher education of the surveyed testers in not only the field of IT but also by the lack of technical higher education in general.
Thirdly, empirical research shows that, in terms of professional level in IT companies, professionals as a whole and groups with lower and higher professional levels are not characterized by quantitative differences in metacognitive awareness. However, metacognitive control of professionals with lower and higher levels of professionalism is implemented by different strategies: for senior-team lead specialists, leading metacognitive strategies that ensure their performance are reflective assessment of their personal activity, its effectiveness and development of alternative solutions, and for junior - middle - level professionals, planning and relying on acquired knowledge. That reflects the importance of the professional growth of specialists in testing software metasystem, relevance, and representation of their metacognitive skills at all stages of solving professional problems. Factor analysis shows that the greatest influence on the metacognitive behavior of software testing professionals and their successful professional development is made by the ability to overcome subjective limitations, reflect on achievements, strategic planning, formulate questions. Metacognitive skills such as keeping a diary (writing down one's thoughts) and role-playing (putting oneself on a partner/client's position, imaginary dialogue) are of the least influence.
Fourthly, the training of IT professionals in Ukrainian higher education institutions serves mainly as a potential start and requires a time-consuming self-study, as well as training directly in an IT company. Given the lack of a separate direction of training for software testing specialists in Ukrainian universities, the IT sector needs to develop and implement appropriate higher education standards, which in turn will ensure the development of metacognitive skills for future professionals.