Automation of control over the formation of skills in the development of software documentation using a group expert assessment *

Abstract

The article is devoted to the development of e-learning tools and distance learning technologies in the training of IT specialists. A methodology for group expert assessment of the quality of software documentation has been developed. The use of the methodology allows, on the basis of mathematical methods, to control the formation of skills among IT specialists. On the basis of the methodology, a prototype of a subsystem for a group expert assessment of the quality of software documentation for an automated system for processing information on the development of competencies in the training of IT specialists has been developed. The use of the subsystem will reduce the complexity of the work of experts. The developed methodology and subsystem can be used: to control the formation of professional skills among trainees during training in educational organizations, during in-house training in IT companies and IT departments of enterprises; to assess the quality of real-life tasks for the preparation of software documentation by novice IT specialists.

Keywords

• IT specialist
• Software documentation
• Knowledge and skills control
• Group peer review
• Automated training systems

-

Documentation creation is an integral part of the life cycle of the development of complex software systems [ 1-2 ]. The quality of the created documentation at each stage of the life cycle affects the implementation of subsequent stages and the result of the development of a software product in general.

With regard to software documentation (examples of which are requirements for software systems, test cases, defect reports, etc. [ 1 ]), various criteria are used to assess the quality of its preparation (with the aim of subsequent elimination of deficiencies).

An IT specialist who is responsible for developing software documentation should know these criteria and have the skills to apply them to the execution of software projects.

Accordingly, in the course of professional training or retraining of IT specialists, the processes of formation and control of these knowledge and skills should be provided. In particular, according to the results of previous studies [ 3 ], a methodology for assessing the quality of exercises for the development of test documentation by one expert was proposed.

An urgent task is the development of this methodology in relation to the assessment by several experts of the implementation of exercises for the development of program documentation. For example, several experts can be involved: in an educational institution in the assessment of Olympiad tasks for the development of program documentation; when assessing the preparation of documentation by a novice IT specialist at the enterprise (experts can be the most qualified specialists). Involvement of several experts will allow taking into account the opinions of different specialists regarding the quality of the document created by the student.

The results of research in the field of solving this urgent problem are presented below. 2

Analysis of methods for assessing the quality of software documentation The works of some authors present the desired properties of software documentation. These properties can be used as criteria for assessing the quality of its compilation. For example, Kulikov S. highlights the following properties of test documentation [ 1 ]: ─ for software requirements: completeness, atomicity, consistency, continuity, unambiguity, feasibility, relevance, traceability, modifiability, ranking, verifiability; ─ for test cases: correct technical language; balance between specificity and generality; balance between simplicity and complexity; ensuring a high probability of error detection; sequence of actions to achieve a single goal; lack of unnecessary actions; non-redundancy in relation to other test cases; the ability to most clearly demonstrate the identified error; traceability; possibility of reuse; compliance with accepted design templates and company traditions; ─ for defect reports: filling of all fields with accurate and correct information; correct technical language; the specificity of the description of the steps; no unnecessary actions, long descriptions of actions; no duplicates; obviousness and clarity; traceability; a separate report for each new defect; compliance with accepted design templates and company traditions.

Orlov S. in [ 2 ] identifies the following properties of detailed requirements for software systems: tracking, testability, unambiguity, priority, completeness, consistency.

Based on this review, it can be seen that for different types of software documentation the criteria are largely similar. Each specific organization can use its own assessment criteria.

The use of computer technologies in the training of IT specialists, and, in particular, automated systems for assessing knowledge and skills, allows to improve the process of forming professional competencies [ 4-10 ]. Based on the results of previous studies [ 3 ], a methodology for expert assessment (by one expert) of the quality of exercises for the development of software (in particular, test) documentation was developed. This methodology is the basis for the development of software modules for an automated control system for knowledge and skills in the training of IT specialists.

Based on the analysis of criteria for assessing the quality of software documentation, existing approaches to automating the control of professional knowledge and skills and the use of modern mathematical methods [3; 11], it is proposed to develop the previously created and described methodology [ 3 ] in relation to the problem of document assessment by a group of experts (specialists). 3

Methodology for group expert assessment of software documentation in the training of IT specialists Let us consider the created methodology for group expert assessment of software documentation in the training of IT specialists (using the example of evaluating a practical task for the development of test documentation when training beginner IT specialists at an enterprise). The proposed methodology includes steps: Step No. 1. A group of experts prepares a practical task (exercise) for subsequent implementation by trainees. Consider an example in which such an exercise is to develop a defect report for a program.

During the preparation of the task, it is necessary to determine a set of quality indicators to assess its implementation: A  {ai | i  1, N ind. } , where ai is a separate quality indicator, Nind. is the total number of indicators.

For the example under consideration, we will choose 5 quality indicators based on the recommendations presented in [ 1 ] when creating reports on defects: a1 – filling of all fields with accurate and correct information; a2 – correct technical language; a3 – the specificity of the description of the steps; a4 – absence of unnecessary actions, too long descriptions of actions; a5 – traceability.

Step No. 2. The weights of quality indicators wi (showing the significance of each indicator ai in assessing the performance of the task) are determined by a group of Nexp. experts based on the following algorithm (using the method of direct assessment

2.1. Each j -th expert ( j  1, Nexp. ) compares with the i -th quality indicator the assessment of its significance b ji , measured on a certain scale (for example, 10-point). The result is a matrix B  b ji  .

For example, as a result of evaluating the Nind.  5 indicators (described above) by 10  Nexp.  3 experts, a matrix was obtained: B   3  2 task in the allotted time.

Step No. 4. Each expert checks the completed practical task. As a result, we obtain a set of matrices D  {Dj | j  1, Nexp.} , where D j  d jqi , d jqi [0;1] is the assessment by the j -th expert of the task performed by the q -th student ( q  1, Nstud. ) according to the i -th quality indicator. An example of a matrix with the assessments of the first  0,90 0,75 0,75 0,50 0,80   expert: D1   0,50 0,30 0,65 0,40 0,50 .

 0,75 0,60 0,40 0,60 0,75  0,25 0,25 0,50 0,40 0,50 Step No. 5. Based on the difference in the assessments for each quality indicator, set by experts at step 4, the generalized weights of the quality indicators wji are calculated. The sequence of actions for calculating these weights:

5.1. Calculation of average grades for each i -th quality indicator. We get the matrix Davg.  d ji  (here j  1, Nexp. , i  1, N ind. ). Here d ji  Nstud.d jqi Nstud. . q1 5.2. Calculation of the scatter values for each i -th quality indicator. We get the Nstud.

 d jqi  d ji matrix R  R ji , where R ji  q1 .

Nstud.  d ji 5.3. Calculating the sum of the scatter values. We get the matrix Rsum.  R j ,

Nind. where R j   R ji .

i1 5.4. Calculation of weights wji based on the scatter of assessments obtained in step the matrix W   wji , where wji  αwi  βwji . Here α and β are the coefficients of significance of the weights wi and wji , respectively. For example, α  β  0,5 [ 3, 11 ].

Step No. 6. Complex assessments of the task performed by each q -th student are calculated (for each j -th expert). We get the matrix Dcmp.  L jq , where

Nind.

L jq   wji d jqi .

i1 Step No. 7. The group assessments of the tasks performed by each q -th student are calculated (based on the complex assessments of each j -th expert):

7.1. Calculation of the same initial values of the competence coefficients for each j -th expert (at the iteration h  0 ): k 0j  1 N exp .  0,3333 .

7.2. Go to the next iteration (increase h by 1). 7.3. Calculation of the group assessment of each q -th student at the h -th iteration:

Nexp.

Lhq   L jq k hj1 .

j1 Nstud. Nexp. 7.4. Calculation of the normalization factor: h    (Lhq  Ljq ) . q1 j1 7.5. Calculation of coefficients of expert competence:

1 Nstud. k hj  h  (Lhq  Ljq ) at j  1, Nexp. 1;  q1

Nexp. 1 k hj  1   kvh at j  Nexp. ( according to the normalization condition Nexp. k hj  1).

v1 j1 7.6. Checking the condition max Lhq  Lh1  ε , where ε is some calculation accuq racy (for example, ε  0,001 [ 11 ]). If the condition is true, then the process of finding the group assessments of students ends. If false, then it returns to step 7.2.

The higher the group assessment, the better the quality of the document.

The functional requirements for the subsystem of the automated training system (ATS), through which the assessment of exercises according to the proposed methodology is implemented, are presented by the Use Case UML diagram in Figure 1 (based on improvements to a similar diagram from [ 3 ]).

As shown in Fig. 1, the expert team leader has access to all the functions of a regular expert. But in addition, there are functions for determining group assessments of weights, document quality, compiling a final list of comments and recommendations for the trainee (taking into account the comments and recommendations of other experts). The subsystem allows you to simplify many time-consuming calculations (use cases 4.1, 5.1, 8.1, 10.1). In this case, the person makes the final decision on the grades. You can convert the grouped document quality assessments (use case 10) to a different scale.

This methodology and the ATS subsystem can also be used, for example, when defending coursework and final qualification works by university students (future IT specialists). Moreover, in addition to the criteria for assessing the quality of documentation directly, it is possible to take into account other criteria, for example, the level of preparation of the report and presentation. ─ A methodology for group expert assessment of the quality of software documentation has been developed. The use of the methodology allows, on the basis of mathematical methods, to control the formation of skills among IT specialists; ─ On the basis of the methodology, a prototype of a subsystem for a group expert assessment of the quality of software documentation for an automated system for processing information on the development of competencies in the training of IT specialists has been developed. The use of the subsystem will reduce the complexity of the work of experts; ─ The developed methodology and subsystem can be used:  to control the formation of professional skills among trainees during training in educational organizations, during in-house training in IT companies and IT departments of enterprises;  to assess the quality of real-life tasks for the preparation of software documentation by novice IT specialists. 5

Acknowledgments The research is supported by a stipend of the President of the Russian Federation to young scientists and post-graduate students (No. SP-100.2018.5), which was assigned by the grants Council of the President of the Russian Federation.

References

1. Kulikov , S.S.: Software Testing . Basic course . Minsk: Four quarters ( 2017 ).

2. Orlov , S.A.: Software Engineering . Textbook for universities. 5th edition updated and expanded. Third generation standard . SPb .: Peter ( 2016 ).

3. Fayzrakhmanov , R.A. , Polevshchikov , I.S. , Bobrova , I.A. : Improving the Process of Training Specialists in the Development of Program Documentation Based on Automated Assessment of the Quality of Skills Formation, XVIII All-Russian Scientific and Practical Conference "Planning and Provision of Personnel Training for the Industrial and Economic Complex of the Region" (November 20-21 , 2019 ), Sat. reports, 21 - 25 , SPb .: Publishing house of ETU "LETI" ( 2019 ).

4. Bouhnik , D. , Carmi , G.: E-learning Environments in Academy: Technology, Pedagogy and Thinking Dispositions . Journal of Information Technology Education: Research , 11 , 201 - 219 ( 2012 ).

5. Kovacic , Z. , Green , J. : Automatic Grading of Spreadsheet and Database Skills . Journal of Information Technology Education: Innovations in Practice , 11 , 53 - 70 ( 2012 ).

6. Lisitsyna , L.S. , Smetyuh , N.P. , Golikov , S.P. : Models and Methods for Adaptive Management of Individual and Team-Based Training Using a Simulator . IOP Conference Series: Earth and Environmental Science , 66 ( 1 ), 012010 ( 2017 ).

7. Alshammari , M.T. , Qtaish , A. : Effective Adaptive E-Learning Systems According to Learning Style and Knowledge Level . Journal of Information Technology Education: Research , 18 , 529 - 547 ( 2019 ).

8. Candel , C. , Vidal-Abarca , E. , Cerdán , R. , Lippmann, M. , Narciss , S.: Effects of timing of formative feedback in computer‐assisted learning environments . J Comput Assist Learn , 36 ( 5 ), 718 - 728 ( 2020 ).

9. Chatwattana , P. , Phadungthin , R.: Web-based virtual laboratory for the promotion of selfdirected learning . Global Journal of Engineering Education , 21 ( 2 ), 157 - 164 ( 2019 ).

10. Gero , A. , Stav , Y. , Wertheim , I. , Epstein , A. : Two-tier multiple-choice questions as a means of increasing discrimination: case-study of a basic electric circuits course . Global Journal of Engineering Education , 21 ( 2 ), 139 - 144 ( 2019 ).

11. Gudkov , P.A. : Methods of Comparative Analysis . Tutorial. Penza: Publishing house of the Penza State University ( 2008 ).