Talent acquisition is an important, complex and e ort-intensive function within Human Resources (HR) management. We describe a system called TEAC that processes a set of given free-form textual resumes (in English for IT domain), creates a standardized pro le for each candidate, and for a given job description, identi es a ranked shortlist of k candidates, along with a matching score for each. The resume scoring function is hand-crafted, hierarchical, and uses domain-knowledge from recruitment experts. We describe a simple neural-network system that automatically learns some weights used in the scoring function, based on feedback about whether the candidate was SELECTED or REJECTED. The TEAC system is deployed in a large multinational IT services organization.
Copyright c by the paper's authors. Copying permitted for private and academic purposes.
In: Joint Proceedings of the First International Workshop on Professional Search (ProfS2018); the Second Workshop on Knowledge Graphs and Semantics for Text Retrieval, Analysis, and Understanding (KG4IR); and the International Workshop on Data Search (DATA:SEARCH18). Co-located with SIGIR 2018, Ann Arbor, Michigan, USA { 12 July 2018, published at http://ceur-ws.org involved in our resume scoring function based on feedback about whether the candidate was SELECTED or REJECTED. In Section 5 we empirically evaluate the proposed system. Section 6 contains related work; Section 7 gives conclusions and future work. 2
Resumes of IT professionals have a somewhat common conceptual structure. Such a resume typically contains several sections, such as personal information, education, employment history, project details, trainings and certi cations, awards and achievements, publications, professional summary etc. Each resume need not contain all these sections and some resumes may contain additional sections. The order of sections may vary across resumes, and there is a great variety in the contents and expression of each section.
We have built a machine-learning based information extraction tool called RINX to identify various sections and to extract various types of entities, their attributes and relations among these entities from the given resume. Entity type SKILL refers to IT technology platforms, systems, products, frameworks, programming languages, operating systems etc.; examples of mentions of this entity type are Java, ASP.NET, SAP, Linux, Oracle, J2EE. Entity type ROLE describes the type of tasks handled and work done in an IT project; examples of mentions of this entity type are developer, technical architect, test engineer, project leader. Mentions of the entity type PROJECT typically refer to the title of an IT project in a resume; e.g., Utilities and scripts for production support. A binary relation type connects two entity types; e.g., the relation used in connects an entity of type SKILL and an entity of type PROJECT. Thus, if a speci c project p uses a particular skill s, then RINX will extract a relation mention used in(s; p). Some attributes of an entity type are directly extracted by RINX and some are derived from relations between two entities. For example, RINX extracts mentions of the relation used in and the attribute #skills for a particular project is the count of these relation mentions within that speci c project section. This extracted information about a candidate is arranged as a set of sub-pro les. Fig. 2 shows 4 sub-pro les for a candidate; e.g., the skill sub-pro le for a candidate has one row for each of her skills, with the columns as listed. Using simple rules, each actual role mentioned in a resume is mapped to one role in a standard set of roles (along with a seniority level - a number between 1 to 15 - for each role) in our organization; e.g., EDI-SAP Analyst is mapped to the standard role ANALYST.
We have de ned a scale for assigning pro ciency level to a particular skill in a particular candidate: E0; E1; E2; E3; E4, with E0 denoting a basic familiarity and E4 denoting the highest level expertise; e.g., a candidate may be assigned E2 in SAP, E1 in Java, and E1 in Oracle RDBMS. We have de ned a simple rule-based model to compute the pro ciency level of a candidate in a given skill, using values of the corresponding attributes of that skill in the skill pro le (Fig. 2). Tier of an employer company c is derived using simple rules (higher values are better); an example of such a rule could be: IF c 2 F ortune1000 _ c is a government department _ c.turnover > USD 5 billion THEN tier = 5. Tier of a university is derived using some public-domain ranking. Trainings refer to internal trainings within an organization; certi cation is based on an external examination, such as Microsoft Certified Technology Specialist (MCTS) 3.0 Web application.
RINX has 3 di erent methods of extracting entities, attributes and relations. In a supervised method, RINX uses labeled training data to learn a binary classi er model (e.g., CRF or SVM) which can then be used for extracting mentions of a particular entity type from an unlabeled resume. In a gazette-based method, RINX starts with a small seed list of mentions of a particular entity type and iteratively discovers a list of other mentions in an unlabeled corpus of resumes. Thus starting with 5 seed examples of the entity type DEGREE, RINX discovered about 400 other degree names in a corpus of resumes. Extraction of mentions of an entity type using a gazette G is straightforward and involves identifying all strings which approximately match any mention in G. RINX also contains hand-crafted rule-based methods to extract entities such as ROLE and DATE. After RINX extracts all the entities, attributes and relations from a resume, the MAPPER component in TEAC creates the various sub-pro les for that candidate. 3
After the sub-pro les are created from the uploaded resumes of some candidates, TEAC accepts a job description (JD) from a user and creates a shortlist of k candidates, sorted on a similarity (or matching) score for each candidate, indicating how well the candidate matches the JD. A JD consists of a list of skills (e.g., SiteCore, ASP.NET, intended role (e.g., DEVELOPER), total experience (e.g., 3 6 years) etc. We now describe a handcrafted hierarchical matching function that computes a similarity score between a JD and a candidate (i.e., her sub-pro les). We have de ned an unsupervised algorithm that computes similarity score sim1(x; y) between any two given skills x and y, based on, say, the similarity between their Wikipedia pages. sim2 computes the similarity between a skill x from a JD and a row for skill y from the skill sub-pro le for a given candidate. Here, w1; : : : ; w7 are user-de ned non-negative weights for the 7 variables in the skill sub-pro le. sim2(x; (y; y1; y2; y3; y4; y5; y6; y7)) = sim1(x; y) (w1y1 + : : : + w7y7) (w1 + : : : + w7) (1)
The function sim3 gives the overall similarity between a skill x in the given JD and all the skills in the skill sub-pro le of the candidate. sim3 takes the top match similarity as it is and adds the average of sim2 values of the remaining skills in the skill sub-pro le of the candidate whose similarity with x is above some user-speci ed threshold 0. Function sim4 computes the total matching score between all skills in the given JD and all skills in the skill sub-pro le of the given candidate. Suppose there are at most K1 skills speci ed in a JD (we use K1 = 5). Let u1; : : : ; u5 denote the user-de ned weights of the rst, second, third etc. skills in a JD. Then sim4 computes the weighted average of the n sim3 values, where n is the no. of skills in the JD.
Similarity between a role in JD and a given role sub-pro le is computed as follows. Similarity function rsim1(a; b) two roles a (from JD) and b (from role pro le) is as follows, where sen(x) denotes the seniority of role x and maxsen is 15.
rsim1(a; b) = 1
if sen(b) > sen(a) = sen(a) sen(b)
maxsen sen(a) sen(a) sen(b) otherwise (2) Function rsim2 computes the similarity between a role x from a JD and an entire row for role y from the role sub-pro le for a given candidate. The formula is similar to that of sim2. Function rsim3 gives the overall similarity between the role in a given JD and all the roles in the role sub-pro le of the candidate. Its formula is similar to that of sim3. Function expsim computes the similarity between the total experience of a candidate and the experience range in a JD using a triangle function .
In addition to how well the skills, roles and experience of a candidate match those speci ed in a JD, we consider the quality of a candidate when preparing the shortlist, so that higher quality candidates come up. We measure the quality of a candidate (independent of any JD) in terms of the quality of her education and quality of her work experience. We have de ned a set of rules for assessing the quality of a candidate's education pro le. For example, the performance of a candidate during a degree is typically given in terms of % of marks, GPA, or CGPA (averaged over all semesters). After bringing the performance number from a resume to a common range (0 10), we de ne a simple rule that gives an assessment score of performance: IF perf ormance 6:0 THEN score = 2 ELSE IF 6:0 < perf ormance 8:0 THEN score = 3 ELSE IF perf ormance > 8:0 THEN score = 4. The overall quality score for the education sub-pro le of a candidate is the average of the assessment scores for each of the columns in it.
We have also de ned a set of rules for assessing the quality of a candidate's work experience sub-pro le. A rule like IF company tier == 1 THEN score = 1 ELSE IF company tier == 2 THEN score = 2:5 ELSE IF company tier == 3 THEN score = 4 gives preference to a candidate who has worked for a high tier employer. In a similar manner, rules compute the score for other attributes in the work experience sub-pro le, such as #awards, #certif ications trainings etc. The overall quality score for the work experience sub-pro le of a candidate is the average of the assessment scores for each of the columns in it.
Let Q = (S; r; (t1; t2)) denote a JD, where S = hs1; s2; ; sni is an ordered list of skills, r is a role, and t1; t2 denote the experience range. As an example, Q = (fsitecore; ASP:NETg; DEV ELOP ER; (3; 6)). Let (skill); (role); (edu); (exp) denote the sub-pro les for a particular candidate. Each sub-pro le is a set of vectors. (role); 2(role)g, where by i; e.g., in Fig. 2, (role) = f 1 We denote i-th element of a sub-pro le (which is a vector)
1(role) = hT ECHN ICAL LEAD; 4; 2:915; 1; 1; 7; 0; 0; 0i and 2(role) = hDEV ELOP ER; 2; 3:5; 4; 2; 11; 0; 0; 0i. Our skill similarity function gives sim1(Sitecore; ASP:NET) = 0:4667, sim1(Sitecore; C#) = 0:363 etc. Using the weights w1 = 0:5; w2 = 0:2; w3 = 0:02; w4 = 0:04; w5 = 0:04; w6 = 0:1; w7 = 0:1 for the 7 variables in each row of the sill sub-pro le, we compute, using Eq. (1):
sim2(Sitecore; 1(skill)) = sim2(Sitecore; (Sitecore; 2:915; 1; 1; 0; 0; 1; 1)) = sim1(Sitecore; Sitecore) (0:5 2:915 + 0:2 1 + 0:02 1 + 0:04 0 + 0:04 0 + 0:1 1 + 0:1 1) = 1:0 (0:5 2:915 + 0:2 1 + 0:02 1 + 0:04 0 + 0:04 0 + 0:1 1 + 0:1 1) = 1:8775
Similarly, we get sim2(Sitecore; 2(skill)) = 0:87622925, sim2(Sitecore; (skill)) = 0:66792, 3 sim2(Sitecore; 4(skill)) = 1:86289, sim2(Sitecore; 5(skill)) = 1:604138, sim2(Sitecore; 6(skill)) = 0:729295, and sim2(Sitecore; 7(skill)) = 0:06909.
Then the similarity between the rst skill in the JD and the candidate skill pro le is: sim3(Sitecore; (skill)) = 1:8775 + ((0:87622925 + 0:66792 + 1:86289 + 1:604138)=4) = 3:1302943125
We did not consider Oracle PL/SQL and Oracle Forms in computing sim3, because sim1(Sitecore; OraclePL=SQL) = 0:2345 < 0 and sim1(Sitecore; OracleForms) = 0:047 < 0, where we assume 0 = 0:3. In a similar manner, we get sim3(ASP:NET; (skill)) = 3:63678845. The matching score function sim4 between all the skills in the given JD and the skill sub-pro le of the given candidate is: sim4(Q; (skill)) = (1:0 3:1302943125 + 0:8 3:63678845)=(1:0 + 0:8) = 3:3554028
Assuming sen(DEV ELOP ER) = 2 and sen(T ECHN ICAL LEAD) = 4, using Eq. (2), we get rsim1(DEV ELOP ER; DEV ELOP ER) = 1 and rsim1(DEV ELOP ER; T ECHN ICAL LEAD) = 0:867. We assume the weights u1 = 0:6; u2 = 0:1; u3 = 0:05; u4 = 0:1; u5 = 0:05; u6 = 0:05; u7 = 0:05. Then, the similarity between the JD role (DEVELOPER) and one of the roles (TECHNICAL LEAD) in the role sub-pro le is: rsim2(DEV ELOP ER; 1(role)) = rsim2(DEV ELOP ER; (T ECHN ICAL LEAD; 2:915; 1; 1; 7; 0; 0; 0))
Similarly, we get rsim2(DEV ELOP ER; 2(role)) = 3:7. Then the similarity between the JD role and the role sub-pro le of the candidate is: rsim3(DEV ELOP ER; (role)) = (2:253333 + 3:7)=2 = 2:9766665. The match score for her experience is expsim = 0. The total match score for the example candidate with respect to the example JD is: 0 + 3.355 + 2.977 = 6.332. Using various rules, we get the quality score for this candidate's education sub-pro le as 6.5, for her work experience pro le it is 1.396, giving the overall quality score as 6.5 + 1.396 = 7.896. The overall matching score for this candidate with the given JD is: 6.332 + 7.896 = 14.228. 4
Our shortlist creation function is a hierarchical hand-crafted knowledge-rich function for ranking resumes with respect to a given job description. We came up with this candidate scoring function in close cooperation with HR professionals, who gave examples of candidates and their scores as well as guided us on the detailed aspects that they included when assessing candidates. Intuitively, their requirements for candidate ltering (with respect to a speci c JD) were about close \match" with one or more skills, and roles. In addition, they often preferred \high quality" candidates. The problem was that, although there is very good understanding with each HR professional about matching and quality, this understanding is not quanti ed formally. We noted with interest that the HR professionals often had somewhat di erent evaluation criteria for matching and quality, and the candidate scoring function reported here is our attempt to consolidate the di erent subjective ways into a more objective automated scoring function. After the TEAC system became operational, they monitored the scores generated by the system for di erent candidates and satis ed themselves that the these scores were reasonable. We have not done a formal evaluation of the scores produced by TEAC for various candidates. However, we have evaluated the quality of the scores in another way which is more relevant to the HR professionals, and report the results in section 5.
The candidate scoring function involves many weights that are set initially by the users. We now report a simple learning algorithm that automatically adjusts these weights by observing the end result of recruitment (SELECTED or REJECTED). For a JD Ji and the corresponding resume set Si of size Mi, we receive a feedback from the HR executives, which is in the form of a ag Fj 2 f0; 1g for every resume Rj 2 Si; 1 j Mi (0 indicates REJECTED, 1 indicates SELECTED). We translate this into a pairwise preference data as follows: for every pair of resumes Rj ; Rk 2 Si; 1 j; k Mi, such that their ags are di erent (Fj 6= Fk), we generate a preference record Rj Rk if Fk > Fj , and Rk Rj if Fj > Fk. We represent each resume Rj as a vector vj of 5 score values: scores for skill sub-pro le, role sub-pro le, education sub-pro le, work experience sub-pro le and total experience. 3 of these 5 scores are with respect to a given JD. A resume pair Rj ; Rk is presented as the tuple (vj; vk; gjk), where gjk = 1 if Rk Rj , and 0 otherwise. We use such tuples to train a neural network. The goal of this ANN is to learn the weights in a linear score combination function a1v1 + : : : + a5v5, where the 5 weights ai are unknown and v1; : : : ; v5 are the sub-pro le scores for a resume. During the training phase, the ANN starts with some initial values for ai and keeps adjusting the weights ai to conform to the relative preference rank of the given pair of resumes. After the training phase is over, the nal values of ai are used to compute the score for any resume, and to create a shortlist by selecting k0 resumes having the highest scores. 5
We do not report a direct evaluation of the quality of the shortlists created by TEAC. We report an indirect evaluation, which is more useful for HR executives. In this experiment, we collected sets S1; : : : ; SN of resumes, and corresponding J1; : : : ; JN JDs. Candidates with resumes in set Si were interviewed and the HR executives also gave us the result ag (SELECTED or REJECTED) for each candidate in each Si. To evaluate TEAC, for each set Si, we used TEAC to create a shortlist of top k0 resumes for each Si, ordered in terms of their matching score for JD Ji. The e ectiveness of TEAC is measured in terms of the fraction of SELECTED candidates in the TEAC shortlist of size k0. We de ne a measure avg precision@k0, which is simply the average value of this fraction for the N sets. Note that the sizes of the sets Si are di erent and the numbers of SELECTED candidates in the sets Si also vary from set to set. For testing, we considered 21 resume sets, containing a total of 352 resumes, and a JD for each resume set. For each resume in each set, we also had a SELECTED/REJECTED ag (130 SELECTED, 222 REJECTED). Fig. 3 shows the avg precision@k0 for various values of k0, using both the default values of these 5 weights in TEAC (all values are 1) and the new weights learned by the ANN (0.946, 0.833, 1.019, 0.841, 0.865). The ANN shows an improvement in the quality of shortlists created. 6
Recruitment is a rich source of problems for applying data mining and machine learning algorithms [
The TEAC system discussed here extracts information from free-form text resumes, and creates a shortlist for a given job description. The resume scoring function in TEAC is hand-crafted, hierarchical, and uses domainknowledge from recruitment experts. As mentioned earlier, HR professionals often had somewhat di erent evaluation criteria for matching of a candidate with a JD and about measuring candidates' quality, and the candidate scoring function reported here is our attempt to consolidate the di erent subjective ways into a more objective automated scoring function. While we have described its application for candidates in the IT domain, this scoring function can be tailored for candidates in other domains. For example, when scoring non-IT candidates in the banking domain, we can remove the scoring component related to projects (since banking work is typically not project-oriented). We are working on devising a more general scoring function for non-IT candidates, which includes a component related to speci c tasks. We also described a simple neural-network that automatically learns some of the weights used in the scoring function, based on feedback about whether the candidate was SELECTED or REJECTED. The TEAC system is deployed in a large multinational IT services organization. The ANN learns only some of the weights in the scoring function. We are working on extending the approach to automatically learn all the weights used in the scoring function. Firms specializing in recruitment usually have some software components to aid them, mostly focusing on generic search and retrieval of resumes. Our candidate scoring function is more transparent than such systems and is able to give concrete reasons for why a particular candidate is (or is not) included in the shortlist for a given JD. This function can also be customized in a limited way; e.g., by changing the weights.