=Paper=
{{Paper
|id=Vol-2864/paper20
|storemode=property
|title=Towards Designing of Recommendation System for Recruiting of Software Development Teams
|pdfUrl=https://ceur-ws.org/Vol-2864/paper20.pdf
|volume=Vol-2864
|authors=Karina Melnyk,Natalia Borysova,Zoia Kochuieva,Dina Huliieva
|dblpUrl=https://dblp.org/rec/conf/cmis/MelnykBKH21
}}
==Towards Designing of Recommendation System for Recruiting of Software Development Teams==
https://ceur-ws.org/Vol-2864/paper20.pdf
Towards Designing of Recommendation System for Recruiting
of Software Development Teams
Karina Melnyk, Natalia Borysova, Zoia Kochuieva and Dina Huliieva
National Technical University “Kharkiv Polytechnic Institute”, Kirpichova, 2, Kharkiv, 61002, Ukraine
Abstract
The problem of recruiting software development teams based on the choice of potential
candidates is considered. An analytical review of methods for creating recommendation
systems has been conducted. A suitable similarity measure based on the type of input data of
the domain area has been chosen. The recruiting task of software development teams in an
ІТ-company is formalized. The functional model of business-process of recruiting of
software development teams has been developed. The method of resolving the recruiting task
has been proposed. The algorithm for solving the issue is given. The proposed microservice
architecture of the information system, which has been used for solving the problem of
automation of the personnel selection process, has been improved at the expense of creating
and adding a new microservice to solve the recruiting task of software development teams.
The architecture of recommendation system in the form of deployment diagram and database
model are developed. The team building process according to technical appraisal is
demonstrated. The evaluation process of the quality of the team building is described. The set
of quality metrics and the convolution criterion are proposed.
Keywords 1
Recommendation system, team building, software development teams, content-based
filtering approach, similarity measure, quality assessment
1. Introduction
Recommender systems (RS) long ago have taken their rightful place in various spheres of human
life and activity. They are used to provide the user with goods, services, information, etc. The
conceptual framework of such actions are most relevant search requests, the analysis of user
preferences, the analysis of the goods, services, information, etc. However, the potential of
recommendation systems in the field of project management has not yet been fully disclosed. There is
no doubt that now there are many software products, which allow automating some of the routine
actions of a project-manager (PM) and head resource manager (HR). Some of them, such as Applicant
Tracking Systems, are used to search for candidates for vacant company positions [1, 2], others are
aimed directly at the project management process when a software development team has already
been formed [3, 4]. Nevertheless, the recruiting task of teams remains without support of RS.
Today, team building problems are mostly solved by HR-managers manually. The research [1]
proposes a solution to the task of personnel selecting for team building according to the technical
appraisal. As a result, a list of candidates with suitable technical skills for each vacant position is
formed without taking into account psychological characteristics. Further, HR and PM conduct face-
to-face interviews with each potential applicant to assess the candidate’s communication, leadership
and personal qualities. However, it is a time-consuming process. Therefore, some HR-managers use
CMIS-2021: The 4th International Workshop on Computer Modeling and Intelligent Systems, April 27, 2021, Zaporizhzia, Ukraine
EMAIL: karina.v.melnyk@gmail.com (K. Melnyk); borysova.n.v@gmail.com (N. Borysova); aliseiko@gmail.com (Z. Kochuieva);
dgulieva@ukr.net (D. Huliieva)
ORCID: 0000-0001-9642-5414 (K. Melnyk); 0000-0002-8834-2536 (N. Borysova); 0000-0002-4300-3370 (Z. Kochuieva);
0000-0001-8310-745X (D. Huliieva);
© 2021 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
psychological techniques for recruiting software development teams as additional stage of appraisal.
This approach allows realizing the intellectual and creative potential of employees in accordance with
the strategic goals of the IT-company. Many authors have researched the problem of team building
based on psychotypes, for example, Robert Hogan, Bruce Tuckman, Dr. Meredith Belbin, D. Keirsey
[5-7]. The task of automation of the process of recruiting software development team based on their
members’ psychological compatibility is highly relevant. There are many benefits of this approach.
Firstly, it will speed up the decision-making process on this issue, because a computer takes much less
time to process large amounts of information than a human. Secondly, automating this process will
free up the PM’s and HR-manager‘s time for other tasks. Therefore, the main point of this research is
dedicated to the development of a recommendation system with particular attention to the function of
the recruiting of software development team based on their psychological compatibility.
2. Formal problem statement
Let’s consider the formulation of the recruiting task of software development teams. Input data for
this task is information about the psychotypes of potential candidates. Designate C C1 ,..., Cn as a
set of potential candidates with suitable technical skills for the software development teams from
employees and non-employees. The set has formed as result of resolving the personnel selection task
according to [1]. The psychological portrait of a potential candidate is a set of competencies that was
built based on the model for superior performance developed by L. M. Spencer, such as result
orientation, analytical thinking, search for information, etc. [8].
Let T T1 ,..., Tm is a set of possible teams. Therefore, the task of team building based on
psychotypes is a mapping of one set to another f : C T . If cardinality of the T 1 , then the
candidate can belong to several variations of one team; if cardinality of the T 2 , then the candidate
can belong to several variations of every team. The final decision depends from the goals of the IT-
company. For instance, the team Tm is more important for the IT-company compared to the team
Tm 2 , that means the best candidate, who fits both teams, has to belong to the team Tm .
So, the purpose of the article is to resolve the recruiting task of software development teams using
the recommendation system.
To achieve the goal of the given research, it is necessary to:
conduct analytical review of methods for creating recommendation systems;
choose a method for recommendation system for team building;
develop a method for the recruiting task of software development teams;
conduct numerical researches;
assess the proposed approach of team building.
3. Literature review
There are some traditional approaches to creating of RS [9, 10]:
content-based filtering approach;
collaborative filtering approach;
hybrid approach.
Content-based filtering approach provides recommendations based on comparing information
about the user and the object of recommendations. Firstly, a profile of users and recommendation
objects are built using the same criteria and the similarity measure of these profiles is estimated; then
the closest objects to the user’s profile are selected for the recommendation.
Collaborative filtering approach provides recommendations based on comparing information about
the preferences of different users with different characteristics and makes recommendations to new
users with similar characteristics. The characteristics of recommendation objects are not taken into
account in this approach.
Hybrid approach combines both of the previous approaches, trying to use their advantages and
neutralize disadvantages.
To solve the given problem, it is efficient to use a content-based filtering approach. A fully formed
profile of the employee is compared with the profile of the “ideal” team member for this project or
with a set of requirements for a vacant position. Such problem statement moreover allows avoiding
the disadvantages of the content-based filtering approach: the limited content analysis and over-
specialization, because the profile of the “ideal” team member and real candidates should not contain
any gaps.
An analysis of various sources of information has shown the methods used for content-based
filtering have been divided into two main groups [9, 10]. The first group of methods calculates
different similarity measures between the considered objects. The second group of methods is data
mining methods, which involves the use of different machine learning methods. The choice of the
method in this case depends on the domain area, the available statistical material and the power of the
computing system. Input data for solving the task of recruiting of software development teams is the
psychological portrait of some potential candidate to appropriate job. To resolve the given task, it is
necessary to compare “ideal” portrait and information about a real candidate. It means to calculate the
similarity between two objects. Thus, the next stage is choosing the similarity metric corresponding to
the input data of the considered domain. There are many such metrics [11-12]. To find a degree of
similarity between quantitative data, there are many metrics: the Euclidean measure, the Manhattan
measure or City block distance, the Mahalanobis measure, the Chebyshev measure, and others. The
presented metrics can also be used for interval data. For qualitative information, we can use the
Hemming similarity measure, the measure of Rogers-Tanimoto, Rao coefficient. The distance
between objects described by binary or dichotomous data can be measured using the Pearson metric
or square contingency coefficient, the Jacquard coefficient, and the metrics used for qualitative data:
the Hamming measure and the Rogers-Tanimoto measure. For mixed data, it is recommended to use
Gower’s coefficient, Zhuravlev’s metric, Voronin’s similarity measure, Mirkin’s metric.
According to the L. M. Spencer’s model, each characteristic from the psychological portrait of
some potential candidate is measured from one to ten, depending on the degree of manifestation of a
particular characteristic. Therefore, it is proposed to calculate only the similarity measure for
quantitative data. Let’s consider the most commonly used metrics [11-12], where Ck {x1 ,...xn } is a
k -th candidate with psychological portrait {x1 ,...xn } and Cideal {x1 ',...xn '} is a “ideal” portrait
accordingly:
1. Euclidean distance. It is the most common distance function, that represents geometric
distance in multidimensional space:
n
(Ck , Cideal ) ( x x ')
i 1
i i
2
(1)
2. Squared Euclidean distance. It is used to give more weight to objects that are more distant
from each other:
n
(Ck , Cideal ) ( xi xi ') 2 (2)
i 1
3. Manhattan distance. The distance between two points measured along axes at right angles.
For this measure, the influence of individual large differences (outliers) decreases (since they are
not squared):
n
(Ck , Cideal ) xi xi ' (3)
i 1
4. Chebyshev distance. This distance can be useful when we need to define two objects as
“different” if they differ in any one coordinate:
(Ck , Cideal ) max xi xi ' (4)
i
An analytical review of the proposed similarity metrics (1)-(4) from the entire arsenal of existing
metrics and features of the domain area allows choosing the Euclidean distance as a measure for this
research.
4. Materials and methods
4.1. Formalization of the team building process
The task of recruiting of software development team for the particular project is a multi-criteria
task with a large number of restrictions and requirements. Any client always wants to receive the end-
product as soon as possible with maximum benefit and with limited resources, despite the
requirements of the stakeholders for the product can be contradictory. That is why the project
planning and recruiting task of team for this project are very important components of the successful
implementation of a quality product. To implement an effective business process in order to achieve
the maximum result in team recruiting, it is necessary to create a model of this process. There are
many techniques for modelling different business process, such as methodology IDEF0, BPMN
Specification, Flowcharts diagrams, Data Flow diagram, notation eEPC. To formalize given business
process, it is proposed to use Business Process Modeling Notation (BPMN). The BPMN-diagrams
allow describing business processes in a unified standardized language that is understandable to all
stakeholders, regardless of their level of technical knowledge. Let’s consider the process of team
recruiting in more detail way (Fig. 1).
The team recruiting process begins with the formation of a set of requirements that apply to the
qualitative and quantitative composition of the team. In order to draw up a list of requirements for
potential candidates for the software development team, it is necessary to hold a kickoff meeting.
Project Scope and Project charter should be formed during this meeting. These documents are the
basis not only for team requirements, but also for software requirement specification. The
requirements to technical skill for the competence profiles of IT-specialists for this project are drawn
up according to the wishes of the PO and the existing working conditions in the IT-company. The
next step is the recruiting process of potential members of software development teams. The HR-
manager firstly checks the resumes of those employees who are currently in the personnel reserve.
Figure 1: The functional model of the team building process
Next step is checking the employees who are currently included in projects from clients.
Sometimes there is a need for a particular IT-specialist who has been already allocated to another
project. In this case, this employee can be hired like a consultant. If a proper candidate were not find
in the company, HR-manager would search for potential candidates in the Internet. The process of
evaluating an IT-specialist can also be initiated due to the immediate needs of the project, for
example, in case of dismissal or illness of one of the team members. The task of evaluating potential
candidates for the software development teams is considered in [1, 13-15]. As a result of this stage,
potential candidates are screened and set of candidates are formed for each position in the project
team. The next step is to solve the problem of team recruiting for new project. Depending on the
needs of the PO, it could be recruited one or more teams with different team members.
Let’s consider the task of team recruiting in more detailed way.
4.2. The method of resolving the recruiting task of software development
teams
The team building task begin with defining the team structure. The management of the IT-
company assign the PM for a particular project. The PM and the Product Owner (PO) further
determine the number of people in the team and the distribution of roles between them according to
the project budget.
Let’s denote R as a set of different roles in a team, for example, ri – development leader, rk –
back-end developer, rl – quality assurance engineer. Then Ri , i R is the number of team members
of ri -th role in the team. So, all values of Ri are the input data from PM and PO.
The result of solving the personnel recruiting task problem is group of sets with potential
candidates for each role in the project team [1]. If C C1 ,..., Cn is a set of potential candidates for
the software development teams from employees and non-employees, than Ci – is a set of potential
candidates for the ri -th role in the team.
Let denote K as set of competencies for the psychological portrait of an IT-specialist, then xk – is
a value of the k -th competence. This variable has view xkpi for a particular p -th p Ci specialist
of ri -th role in the team.
The algorithm for solving the recruiting task of software development teams is a step-by-step
filling out the template of the hierarchical structure of the team based on proper candidates according
to the results of calculating the similarity measure. Therefore, it is necessary to compute the similarity
metric between potential candidates. Designate sijpq as the similarity measure of p -th and q -th
potential candidates between the ri -th and the rj -th roles in the team, p Ci , q C j . If i j , than
siipq is a similarity measure between people with similar role in the team. The Euclidean measure as a
similarity measure between candidates has been proposed in this study. It means the similarity
measure can be found according to (1) as follows:
sijpq x x , p C , q C .
kK
pi
k
qj 2
k i j (5)
Let’s consider the common method of team building for a project under sufficient financial, human
and time resources.
Step 1. The choosing of PM.
The Project Manager is the most important and responsible specialist for the project, who is the
connecting link between the client and the team members. Therefore, he/she is the basis for the
selection of the rest of the team. The formation of the team is carried out according to the principle of
choosing the candidates who are closest to the Project Manager in terms of psychological
characteristics.
Step 2. Appointment of specialists of the next management level: Development Leader, Quality
Assurance Leader, Architect.
There are three variants:
There are proper IT-professionals who are immediately assigned to selected roles.
This option provides several potential specialists for one position. At the same time, the PM is
ready to work with any of them because of the past positive mutual experience and
communication. In this case, there is no need to calculate the similarity measure. The decision
about the appointment to the position is made after the possible teams have been selected and
analyzed.
If employees and non-employees apply for the position, then the choice is based on
calculating the similarity measure between the PM and possible candidates using the formula (5).
The obtained set of values of the metric sijpq is sorted in ascending order. The best candidate is
determined as follows:
candidatebest minsijpq (6)
Cj
where p is the Project Manager in this case, q is any person from the set C j .
In the face of uncertainty, three lists of candidates are formed: the first list corresponds to the
group of people applying for the Development Leader role, where the top names from the list belong
people with the highest priority, the second list being candidates for Quality Assurance Leader, and
the third one for Architect.
Step 3. Assessment of the set Ck Ck C .
Let’s consider the set of potential candidates Ck for the rk -th role in the team. All following
actions depend on the results of comparing the cardinality of the set with the number of required
employees:
Ck Rk go to HR manager
action Ck Rk Ck Td d 1, m (7)
Ck Rk go to step 4
The number of potential candidates is not enough to fill the vacancy in the first case, so it is
necessary to return to the previous stage to solve the problem of personnel selection. The second
option describes a situation when the candidates’ set of rk -th role is a part of all variations of the
future team. The third case implies the resolving of the recruiting task.
Step 4. Recruitment of specialists for the rk -th role in the team.
Similarly to step 2, the similarity measure between the top-level manager (Development Leader,
Quality Assurance Leader, Architect) and the corresponding candidates from the set Ck is calculated
using formula (5). The choice is also carried out from a ranked list according to the formula (6). If
there is a need to select several candidates, then they are selected with the smallest values of the
similarity measure.
Step 5. Checking the team.
If the team is not formed, then go to step 3. If the team is fully completed, then go to step 2 to form
an alternative team. If all possible options are considered, then RS has to provide a list of possible
teams for evaluation and analysis to PM.
Thus, the method of forming of project’s team based on the psychological portrait of candidates
has been considered.
4.3. Architectural solution of the recruiting task of software development
teams
The software design always starts from the defining of the functional and non-functional
requirements. Let us look more closely at the functional requirements for RS. The main user of RS is
PM. He or she has the ability to view the lists of candidates for each vacant position in the project.
These lists are formed by IS after solving the task of personnel selection [1]. PM can view the
information about each candidate available in the system; can find a group of people according to
proximity based on psychological characteristics. There is a possibility to assign a Development
Leader, a Quality Assurance Leader and an Architect. The RS allows recruiting or assigning
candidates to teams for the selected Development Leader, Quality Assurance Leader and Architect.
The PM can assign a particular team from the list to a project. The RS can create a report with list of
generated teams. Non-functional requirements include an intuitive and user-friendly interface,
reliability of data transfer and storage, usability, and high performance. An important requirement is
the fact that the forming task of the software development teams uses the results of solving the
recruiting task [1]. Thus, RS has to coordinate its actions with IS from [1] or be a component of it.
The research [1] has shown the feasibility of using the microservice architecture as a distributed
system architecture based on the analysis of existing IS for the solving various tasks related to the
assessment and selection of personnel. This approach shows how to build the software as a set of
small, asynchronously interacting services, which implements one of the tasks from the list of ones,
which are formulated in the analytical review of the subject area and solved using the developed
models. The architecture of IS from [1] consists of several microservices:
1. Microservice Employee – this component allows processing information about employees in
the IT-company.
2. Microservice Candidate serves for finding and monitoring potential candidates for vacant
positions.
3. Microservice Estimation allows assessing the competencies of employees and candidates.
4. Microservice Identify – a database that contains authentication information about each user of
the system.
To solve the problem of forming the software development team, it is necessary to add other
microservice, which will form teams with potential candidates. Microservice architecture allows using
different technologies to solve each problem. Therefore, the following architecture of the
microservice for the recruiting task has been proposed. It is presented in the form of a deployment
diagram (Fig. 2).
Figure 2: Deployment diagram for the software component
A three-tier architecture to deploy the software component has been chosen. The advantages of the
chosen architecture are scalability, high level of security and reliability, low requirements for network
speed between clients and the application server, and low requirements for performance and technical
characteristics of clients. Let’s consider the presented architecture in more detail.
The Client node includes the following components: Web browser and JQuery – JavaScript
framework. The FormTeamApp Server node is an Internet Information Services server, which allow
the application to work. There are many components on this node: FormTeam Application is a web
application, Business logic module is a component for implementing the logic of work between
domain objects, FormTeam DB is a component that contains information about formed teams with
potential candidates, IFormTeam Application is an application interface. The Remote Data Server
node is a system database location node that uses the SQL Server database. This node deploys the
database, which is used by the IIS. The FormTeam DB database for storing, changing and processing
information about the subject area, that is, about employees, potential candidates, projects, etc., was
developed. The model of this base is represented using the IDEF1x notation (Fig. 3).
Database consists of 13 entities:
the entity «Employees» contains information about all employees of the company;
the entity «PMs» contains information about all employees of the company in the position of
PM; these employees are placed in a separate table, because of the company’s management should
be able to choose particular PM from the list to manage a new project;
the entity «IT_specialists» contains information about all employees in the positions of IT-
specialists, except for project managers; these employees are placed in a separate table, because
there are many employees who are not involved in the implementation of IT-projects;
the entity «Candidates» contains information about employees and non-employees who took
a part in a job interviews;
Figure 3: Model of database
the entity «Job» contains information about all positions in the IT-company;
the entity «Competencies» contains the competencies from job competency profiles;
the entity «Projects» contains information about all projects of the IT-company;
the entity «Clients» contains information about all clients of the IT-company including the
country and the time zone;
the entity «Characteristics» contains information about all characteristics of the projects;
the associative entity «Job_Profile» shows the competencies corresponding to a particular
position; it contains a description of the ideal profile of each position;
the associative entity «IT_specialists_Projects» shows IT-specialists who were involved or
are currently working on the company’s projects;
the associative entity «Projects_Characteristics» shows the characteristics corresponding to
each project of the company;
the associative entity «Employees_Competencies» shows the levels of competence of the
employees.
Thus, the RS architecture has been presented to solve the task of forming of software development
teams.
4.4. Results and Discussion
Let’s consider the using of the developed RS for recruiting of software development teams. One
foreign company has ordered a software from an IT-company. The names of the candidates, name of
project and IT-company have been hidden. Management of the IT-company has assigned particular
PM to this project. According to the proposed method of the team building, first step is done. Project
Manager and HR-manager with Product Owner have chosen the following structure of the team from
10 IT-specialists: Project Manager (1 person), Development Leader (1 person), Quality Assurance
Leader (1 person), Architect (1 person), Middle Front-end Developer (2 persons), Middle Back-end
Developer (2 persons), Middle QA Engineer (1 person) and Junior QA Engineer (1 person). Next step
is an appointment of specialists of the next management level: Development Leader, Quality
Assurance Leader, and Architect. The input data is the information about employees of the IT-
company and non-employees with suitable technical appraisal. The result of solving the task of
selecting potential candidates according to [1] is following: five people claim for the vacant position
of Development Leader in the project; four people claim for two vacant positions of Middle Front-end
Developers, and three people for two vacant positions of Middle Back-end Developers. Each from
group of five Development Leaders can be assigned on this role. So, final decision depends from
different variants of possible teams. The Table 1 presents the input data of potential candidates based
on the L. M. Spencer model for superior performance.
Table 1
The psychological picture of candidates to software development team
Development of others
Conceptual thinking
Information search
Focus on achieving
Analytical thinking
Customer service
Self‐confidence
Teamwork and
cooperation
Leader skills
orientation
Influence
Employees results
DevLead 1 8 10 7 8 9 8 7 8 9 10
DevLead 2 9 9 6 9 8 7 6 9 8 7
DevLead 3 8 6 10 9 8 7 8 9 5 6
DevLead 4 6 8 8 6 7 8 9 10 6 7
DevLead 5 8 6 7 7 6 7 8 9 7 8
MidFront‐end Dev 1 7 8 9 8 7 9 9 9 10 9
MidFront‐end Dev 2 8 7 9 9 8 7 8 9 8 9
MidFront‐end Dev 3 9 8 9 10 6 8 7 9 8 9
MidFront‐end Dev 4 6 5 8 9 7 6 8 9 8 7
MidBack‐end Dev 1 10 9 6 8 7 5 6 5 9 8
MidBack‐end Dev 2 9 8 6 9 8 7 6 8 9 9
MidBack‐end Dev 3 9 10 9 7 8 10 10 9 8 7
Next step is calculating the matching between candidates. The results of calculating the similarity
measure for five employees from the Development Leaders’ list (designated DevLead 1-5) are
presented in the Table 2. It is the base for forming the development teams. The best candidates for the
team are highlighted in bold.
Table 2
An example of calculating the similarity measure for recruiting of software development teams
Employees from DevLead 1 DevLead 2 DevLead 3 DevLead 4 DevLead 5
second and third lists
MidFront‐end Dev 1 4,582575695 6,082762530 6,855654600 5,291502622 5,099019514
MidFront‐end Dev 2 4,472135955 4,690415760 4,472135955 5,196152423 3,872983346
MidFront‐end Dev 3 5,000000000 4,582575695 5,567764363 6,324555320 4,690415760
MidFront‐end Dev 4 7,141428429 5,916079783 4,582575695 5,291502622 3,741657387
MidBack‐end Dev 1 5,744562647 5,000000000 8,660254038 8,831760866 6,633249581
MidBack‐end Dev 2 3,316624790 2,645751311 7,141428429 7,071067812 4,898979486
MidBack‐end Dev 3 5,567764363 6,244997998 6,708203932 5,099019514 6,324555320
The analysis of the Table 2 allows seeing three candidates from the second list instead of the two
required one for DevLead 4. The reason is equal similarity measure of two candidates. Therefore, RS
has formed two alternative teams for this employee. Thus, the result of using the RS is six alternative
software development subteams.
The rest of the team is formed in a similar way: the similarity measures between the Quality
Assurance Leader and his subordinates are calculated, and the Architect is selected. The RS forms a
set of recommendations. It is a framework for the PM, who forms the final structure of the team for
the project.
For checking the adequacy of working of RS with the opinion of experts, it is necessary to choose
special metrics. The Recall and Precision metrics are inapplicable to assess the efficiency of the work
of the developed RS, since the RS does not produce irrelevant responses. The metrics will always be
equal to 100%. However, there is another useful metric. It is the AP metric – Average Precision. It
takes into account the order of the recommendations of proposed software teams. It can be used for
evaluating the efficiency of RS if the recommendation task is considered as a ranking task [16]. AP
is calculated using the following formula:
1 N
Pk if k th item was relevant 1 Pk rel k ,
N
AP @ N
m k 1 m k 1
(8)
where rel k is an indicator that shows when k th item was relevant rel k 1 or not rel k 0 ;
N is a number of recommendations made by RS; m is the number of relevant items from list of
recommendations made by RS; P k is Precision at cutoff k , that is simply the precision calculated
by considering only the subset of RS’ recommendations from rank 1 through k .
Let’s consider an example of calculating the AP -metric. The RS formed six alternative
development subteams and arranged them in a certain order. An expert arranged formed six teams by
RS in a certain order as well. The results of the allocation of teams are presented in the Table 3.
Table 3
Results of the allocation of software development teams
Number in the list Expert RS Correct or not
1 Team 5 Team 5 Correct
2 Team 1 Team 1 Correct
3 Team 4 Team 4 Correct
4 Team 2 Team 3 Incorrect
5 Team 3 Team 2 Incorrect
6 Team 6 Team 6 Correct
Thus, the metric AP according to (8) is equal to:
1 1 2 3 3 3 4
AP @ 6 0,84.
6 1 2 3 4 5 6
The obtained value characterizes the adequacy of the RS operation, since the order of the formed
teams reflects the priority of the command. If the importance of the commands is approximately
equal, the expert can arrange in a different order than the recommendations from the RS. Thus, the
obtained results indicate the results of the RS operation reflect the stated requirements.
In addition, it is possible to assess the quality of the RS work based on a set of characteristics,
which shows the degree of achieving the goal.
To assess the quality of the forming process of the development teams, it is necessary to match the
results of work of the HR-manager and PM before applying the RS and after. Process quality is a set
of process indicators that reflects the degree of achievement of certain goals [15, 17]. To assess the
quality of team building process, the following criteria or features can be used:
f1 – difficulty level of data identification;
f 2 – difficulty level of processing data;
f 3 – time for data processing;
f 4 – level of objectivity in team building;
f 5 – level of adaptation to changing conditions;
f 6 – degree of consideration of psychological compatibility.
Each feature should be evaluated by 7-point scale, where 1 is the minimal value for metrics and 7
is the maximum accordingly. To obtain the numerical values of the criteria, the experts have
evaluated the complexity of solving the given task manually and using RS according to the proposed
scale. In this study, a weighted average quality metric PQ is proposed in [15] as a convolution
criterion, where f u ' is a normalized value of f u -th metric and wu is a weight of it:
f 'w
6
PQ u 1 u u
(9)
w
6
u 1 u
In order to normalize the selected criteria, it is necessary to determine f u – set of criteria that
make a positive contribution to the quality assessment, and f u – negative criteria. Normalization is
carried out according to the formulas (10) and (11) for negative and positive features, respectively:
f
fu ' 1 u (10)
6
f
fu ' u (11)
6
Analysis of the domain area shows that an increasing of the values of features f1 f 3 negatively
affects the quality of the process of forming the project team, and the criteria f 4 f 6 are positive. The
result of the assessment of the quality of forming the software development teams by (9)-(11) is
shown below (Table 4).
Table 4
The quality assessment
Criterion Expert RS Weight of Normalized values
criterion Expert RS
f1 6 1 0,4 0,14 0,86
f2 5 1 0,8 0,29 0,86
f3 7 2 0,9 0,01 0,71
f4 4 5 0,5 0,57 0,71
f5 6 3 0,4 0,86 0,43
f6 2 7 0,9 0,29 1,00
PQ 0,30 0,79
The results from the Table 4 allows to notice that the quality of the process of forming the
project’s team based on the use of RS has increased by 49%, which confirms the research hypothesis
stated earlier. Thus, the decision-making process of recruiting of software development teams can be
improved using recommendations that allow taking into account the psychological profile of potential
candidates.
5. Conclusion
In the course of this study, the approach for automation of the performance of one of the main
functional responsibilities of a PM was presented. It is the formation of software development teams
according to recommendation from the RS. Based on a comprehensive analysis of the domain area for
the implementation of the RS, the content-based filtering approach and Euclidean distance were
chosen. This measure has been used for calculation the proximity of candidates according to the
psychological portrait. The algorithm for solving the task of forming of software development teams
and a functional model of this business process are presented. The architecture of the RS was
proposed and the database model was provided.
The scientific novelty of the given research is to improve the process of recruiting of software
development teams using the proposed method, which allows taking into account the psychological
compatibility of team members. The similarity measures between candidates are calculated. It allows
taking into account results of calculations for forming the development team. Since the Recall and
Precision metrics are inapplicable for assessing the effectiveness of the developed recommendation
system, the Average Precision metric was calculated. It allows evaluating the ranking order of the
items in the recommendation list. The assessment of the quality of the RS work according to the
indicators proposed by the authors is calculated. Numerical studies have shown the possibility of
using the proposed method in real conditions in IT-companies to improve the efficiency of
management decisions.
6. Acknowledgements
This paper and the research behind it would not have been possible without the support of Vitalii
Hlushko – Senior Solution Architect, GlobalLogic, Kharkiv, Ukraine. We thank Vitalii Hlushko for
assistance with development of proposed method and for comments that greatly improved the paper.
7. References
[1] К. V. Melnyk, N. V. Borysova, Rozrobka informatsiinoi systemy dlia vyrishennia zadachi
avtomatyzatsii protsesu pidboru personalu, in: V. Vychuzhanin (Ed.), Aktualni problemy
informatsiinykh system i tekhnolohii, NU ”OMA”, Odesa, 2020, pp. 129-142.
[2] The Top 12 Best Applicant Tracking Systems (ATS) – February 2021, 2021. URL:
https://www.selectsoftwarereviews.com/buyer-guide/applicant-tracking-systems.
[3] A. Stepanov, 10 Sistem dlya Upravleniya Proektami: Moshhno, Nadezhno, Effektivno, 2021.
URL: https://blog.ganttpro.com/ru/sistemy-instrumenty-servisy-upravlenie-proektami/.
[4] The Best Project Management Software for 2021, 2021. URL:
https://www.pcmag.com/picks/the-best-project-management-software.
[5] R. Hogan, Personality and the Fate of Organizations, 1st. ed., Routledge, London, 2006.
[6] B. Tuckman, The Forming, Storming, Norming & Performing Team Development Model, 2021.
URL: https://www.coachingcultureatwork.com/bruce-tuckman-team-development-model/.
[7] D. Keirsey, Personology, Prometheus Nemesis Book Company, Del Mar, 2010.
[8] L. M. Spencer, S. M. Spencer, Competence at Work: Models for Superior Performance, Wiley
India Pvt. Limited, Delhi, 2008.
[9] Ch. Belhekar, Recommendation Systems – A walk through, 2020. URL:
https://medium.com/@chaitanyarb619/recommendation-systems-a-walk-trough-33587fecc195.
[10] F. O. Isinkaye, Y. O. Folajimi, B. A. Ojokoh, Recommendation systems: Principles, methods and
evaluation, Egyptian Informatics Journal, 16.3 (2015) 261–273. doi:10.1016/j.eij.2015.06.005.
[11] S. Prabhakaran, Mahalanobis Distance – Understanding the math with examples (python), 2019.
URL: https://www.machinelearningplus.com/statistics/mahalanobis-distance/.
[12] B. S. Everitt, S. Landau, M. Leese, D. Stahl, Cluster Analysis, 5th. ed., Wiley, Chichester, 2011.
[13] K. Melnyk, N. Borysova, Integrated Technology for Personnel Assessment Based on the
Competencies Model, in: Proceedings of the 9th International Conference “Information Control
Systems & Technologies”, ICTS-2020, Odessa, 2020, pp. 343-357.
doi:10.13140/RG.2.2.26024.60169.
[14] N. V. Borysova, К. V. Melnyk, I. V. Olifenko, Tekhnolohiia pidboru komandy it-fakhivtsiv dlia
vykonannia proektu, in: Proceedings of the V International Scientific-Practical Conference
”Information Technologies in Education, Science and Technology”, ITEST-2020, ChSTU,
Cherkasy, 2020, pp. 40–41.
[15] K. V. Melnyk, N. V. Borysova, V. I. Melnyk, Information system development for teaching staff
performance evaluation, in: S.V. Kotlik (Ed.), Informatsiini tekhnolohii ta avtomatyzatsiia,
Astroprynt, Odesa, 2020, pp. 46-54.
[16] S. Sawtelle, Mean Average Precision (MAP) For Recommender Systems, 2016. URL:
http://sdsawtelle.github.io/blog/output/mean-average-precision-MAP-for-recommender-
systems.html.
[17] What is process quality?, 2021. URL: http://www.taskmanagementguide.com/glossary/what-is-
process-quality.php.