Using machine learning techniques to filter and sort job candidates has been done for more than two decades; however, there are always humans involved in the final hiring decision. One primary reason is that rarely are two hiring decisions made with the same information and in the same context. Many experts believe that any information that can be passed from one human decision-maker to another can also be passed to a machine. Through empirical experiments, we look at ways in which this human feedback can be used to better train machine learning algorithms with special attention to the inherent risks, such as overfitting data and avoiding bias.
Companies have long understood that hiring the best employees produces a
competitive advantage which is hard for its competitors to duplicate. One of the more
significant challenges in finding the most appropriate applicant for a job listing is its inexact
nature - one that is influenced by “feel” as much as by skills and talent.
For over two decades information retrieval systems have been used by human
resources (HR) departments and external headhunting services to filter and sort
candidates based on a set of weighted features gathered from the cover letters and résumés
or curriculum vitae (CVs), interviews with the candidate, letters of recommendation,
as well as other supporting materials such as transcripts or certifications held. These
systems are becoming ubiquitous; with 74% of large U.S. organizations using some
form of electronic selection tool to help with the hiring process [
In data mining tasks, classification and prediction is among the popular task for KDD
and making future predictions. The classification process is known as supervised
learning, where the classification target is already known. The decision tree technique
has its advantages such as it can produce a model which may represent interpretable
rules or logic statement; it is more suitable for analyzing categorical outcomes; it is
non-parametric which is suited to capture a functional form relating independent and
dependent variables; easy to interpret, computationally inexpensive, capable in
dealing with noisy data, its prediction model is intuitively explainable to the user, it has
automatic interaction detection to find the significant high-order interactions quickly,
and it can produce more informative outputs [
One challenge in the job hiring process is that nearly all job searches are unique. Even if two job searches for the same job title in the same department of the same company require the same qualifications and use the same hiring committee, the hiring for each will be done in different contexts. They will draw in completely different candidates, each with their own unique set of strengths and weaknesses. Often a company or department is likely to have different objectives to meet from one quarter to the next. The work teams looking at the “fit” of a new hire are comprised of a different mix of people over time due to other recent hires, job transfers, and employee departures. For this reason and many others, job searches remain difficult for a human to do effectively; for a machine, the task is even more daunting.
Even with the assistance from machines to filter and sort candidates, selecting the
most appropriate job applicants for a job position is not only subjective but also
requires years of experience to accomplish well. HR and executive recruitment and
search firms typically undertake several approaches to select the most appropriate
résumés or CVs, such as completing a checklist or rubric for each submitted résumé
or CV and performing a keyword search on a collection of application materials.
Utilizing human HR experts is generally viewed as the most effective method of
search, yet it is not without significant disadvantages – it is resource intensive and
does not scale well (i.e., a human expert can only review a limited set of applicants
per day). Companies such as Johnson & Johnson, a consumer goods company,
receive 1.2 million applications for 25,000 positions annually [
In this paper, we conduct an empirical evaluation of how humans in the loop can be used to better train AI systems. Since the algorithm used to match a candidate to a job is not consistent from one job search to the next, the weights and features evaluated by the machine learning algorithm change too, limiting the transfer of information. We seek to find answers to the following research questions: 1. How consistent are HR experts in determining the best features of a candidate’s materials to evaluate? If there is little to no consistency between experts, it is challenging to develop an algorithm to replicate the human expert. Moreover, it is easy to introduce bias into the hiring process. 2. How can human experts provide input to better train a machine, particularly on the softer skills? The job requirements for softer skills are often vaguely written. Also, companies want to make it easy for candidates to apply; therefore, they accept a cover letter and CV/resume only and don’t use standardized questionnaire to obtain this information. It is up to the human or machine to determine if the candidate has the required skills from the materials they have provided. 3
To examine the first research question, we began with the set of 5 job descriptions in
English for management-level job positions (see Table 1). We chose management
level job descriptions since they are more likely to have a mix of both language
specific and soft skills. These job descriptions were taken from actual job searches
conducted in 2016-17. We followed many of the anonymity procedures for both
candidates and companies as mentioned in [
Integration 4. Unit Manager / Business Development
Manager 5. Operations Manager
Job Location Hong Kong Singapore New Jersey, USA Guangzhou, China California, USA
Total # of Applicants From each of the 5 pools of applicants, 20 applicants were randomly selected from the pool of actual received submissions. All non-standard acronyms in each job posting’s description and in the job application information was resolved (expanded) for clarity. All personally-identifying data were obscured or genericized to make all candidates and companies non-identifiable.
To answer our first research question, we asked a group of 13 HR personnel (average number of years of experience in HR = 9.0) to weigh which of the identified features from the pool of 100 résumés and cover letters were most relevant to make hiring decisions. These features and their relative importance are important inputs to the machine learning algorithms. A list of the 10 features that experts ranked highest is provided in Table 2.
From Table 2, we can see the ranking of features differs greatly between experts. To
represent this numerically, we use rank-biased overlap (RBO) as our metric [
In addition, to examine if there was some form of possible agreement on features possible between our human experts, we presented all with the overall ranking from Table 2. We asked if this ranking, converted into a score, could be used for the pool of candidates. This provided considerable discussion between them with a majority (9 of the 13) disagreeing with the utility of using the features in the rank order that was collaboratively determined. Most came up with specific examples why this ranking of features would not work from the pool of 100 resumes they examined. There are several important implications from this. First, it makes training a machine learning algorithm to screen and select job candidates challenging, since human experts are the oracle these algorithms seek to replicate. If human experts cannot agree on the weights, training an algorithm to match them becomes a nearly impossible task. Second, it can introduce bias into the job search process. Some countries, such as the United States, regularly require companies to prove the job search process is absent of any racial or gender bias. Some of the features identified, such as “no notable gaps in employment”, “university attended”, and “languages spoken”, could easily introduce bias if left unchecked.
It may appear that selecting candidates is too nuanced for even the more sophisticated
classification algorithms to perform well. However, algorithms can benefit both
short- and long-term if humans are an integral part of the process. First, by filtering
out the candidates who are clearly a mismatch for the advertised job position, the pool
of candidates can be quickly restricted; thus, more attention can be put on evaluating
those candidates that remain. Second, if the algorithm can learn which rules and
patterns are absolute (“hard”) and which can be have a probabilistic weight assigned
(“soft”) through human input, the algorithm can be trained to incorporate this
feedback into the selection and ranking process. While algorithms such as learning to
rank [
Using a C4.5 classifier, we examine how this improvement can be made for the 5 job descriptions mentioned earlier. We wish to examine how the use of the set of applicants for each position and the set of applicants for all positions would improve our algorithm when humans were added to the loop. This approach attempts to answer our second research question. 3.1
The list that most closely resembles the ranked list provided by our oracle is the best match. We use the RBO metric to determine a similarity score with the ranked list provided by our oracle. In addition, we have the experts provide a binary relevance (relevant/not relevant) for each job applicant. This allows us to also evaluate precision and recall and the F-measure, which is the harmonic mean of precision and recall. We asked a different set of 3 human HR experts collaborating as a team to rank the candidates for each job position. Rarely are more than 10 candidates brought in for interviews, so limiting the number of candidates to 10 is reasonable. These experts also evaluated each candidate’s binary relevance (ether as relevant or not relevant) to the position, and the majority relevance label from the 3 experts is used as our oracle. 3.3
For our baseline, we first eliminate candidates that don’t meet the stated criteria for each position. For the candidates that remain, we randomly select a third of them and ask 3 human HR experts to independently rank and score relevance for each candidate for each of the 5 job positions. The two thirds not randomly selected are used as the test set. A C4.5 decision tree algorithm determines a final ranked set of candidates for each position and the binary relevance.
Our first treatment (T1) is to use all the candidates who applied for each job description and randomly divide them into training and test sets with a ratio of 1:2. Rules are derived by the C4.5 algorithm based on training set. In addition, relevance is scored for each candidate. There is no human involvement.
Our second treatment (T2) is to use all the candidates who applied for all job description and randomly divide them into training and test sets with a ratio of 1:2. Rules are derived by the C4.5 algorithm based on training set. Once the test set is ranked, those who did not apply for the job are removed from the final candidate ranking. In addition, relevance is scored for each candidate. As with T1, there is no human involvement.
Our third and fourth treatments (T3 and T4) are similar to the first and second treatments, respectively, but they incorporate humans-in-the-loop inputs with the training sets by providing human feedback into the rules used in the C4.5 algorithm. Three human HR experts highlight the keywords and phrases that contributed to their ranking decision. The algorithm ranks the test set. In the case of T4, once the test set is ranked, those who did not apply for the job are removed from the final candidate ranking. 4
The overall RBO averages, and the precision, recall and F-measure for the baseline
and 4 treatments are provided in Table 3.
From Table 3, we see that treatment T4 provided the best average RBO scores,
precision, recall, and F-measure scores. Treatment T2 provided better results than T1,
particularly with RBO scores and recall, indicating the strength of having more data
from which to train our algorithm. More impressively, RBO, precision and recall
scores for treatments T3 and T4 improve upon treatments T1 and T2, illustrating the
benefits of human-in-the-loop involvement in setting rules for our C4.5 classifier.
Comparing these treatments with our baseline in average RBO score, one can observe
that the baseline falls in the middle of the pack; it outperforms treatments T1 and T2
but underperforms treatments T3 and T4. One can see that human-in-the-loop
involvement to assist with establishing rules (as opposed to selecting candidates and
letting the machine algorithm establish the rules from the selected candidates) can
improve the rankings made by the machine algorithm. One also can observe that
using a broader set of candidates for rule creation, then later eliminate those who did
not apply for that position, can improve the rule set and subsequent candidate ranking.
This supports the findings in [
We note that using a single decision tree algorithm for different job positions can lead to poor classification decisions if the jobs descriptions have little in common with each other. In our case, all were for middle-management jobs supervising technical people; therefore, the variance in our RBO rankings between positions was a very reasonable 0.047. 5
Our study examined the role of humans in selecting candidates for 5 middlemanagement job positions that relied on a combination of technical and soft skills. We used a C4.5 decision tree classification algorithm. The first part of our experiment examined how consistent are HR experts in determining the best features of a candidate’s materials to evaluate. With respect to feature selection, we found little consistency through our group of 13 HR experts. This implies that one HR personnel could arrive at a very different list from another, bringing up concerns of potential bias and demonstrating the difficulties of coming up with an algorithm to completely replace humans. For the foreseeable future, machines will still need humans to be a part of the process.
The second part of our experiment examined how human experts might provide input to better train a machine and how the results might be further improved in job candidate selection able to improve the results. Our evaluation looked at similarity with separate machine/algorithm approaches with a set of human experts, finding when humans help establish the rules (as opposed to only selecting and ranking the relevant candidates), the ranked list, recall and precision scores improve. We also added other candidates who did not apply for the job position as noise (and removed these nonapplicants at a later step), which helped improve the overall results and minimize overfitting. Thus, it is not only having humans in the loop, but having humans perform the most beneficial tasks, that best replicate our human experts.
In future work, we plan to examine the role of longitudinal information (the white box in Figure 1) as inputs to the decision process. Unfortunately, obtaining this information is challenging. Second, because most hiring is for non-management positions, we wish to see if our process can be replicated for blue-collar jobs as well. we wish to see better ways to test the effectiveness of our method’s hiring suggestions; in other words, we wish to detect the best way to measure if the best person was recommended from the pool of candidates. One possibility is to examine internal hires in a longitudinal study (assuming data collection is possible), since in theory we can track the long-term career progression of the candidate within the firm offered the position as well as those that weren’t. Third, we also plan to look at a wider variety of job positions and see how bias might potentially become part the algorithm. If we can detect bias early on, we can set some type of alarm to involve humans to correct for this. Fourth, we also wish to determine ways in which the algorithm can assist humans to do their job more effectively. One possibility is to explore how warnings can be provided for candidates that are either underqualified or overqualified based on some criteria established in advance. Another is to provide better graphical information for each candidate to illustrate their strengths and weaknesses relative to the candidate pool.