Recommendation systems based on AI are widely used by technology companies, especially in the retail and entertainment domains. When you buy products on Amazon, for example, it recommends various other products based on what you have purchased in the past. Similarly, when you watch movies on Netflix, it recommends other movies based on your past viewing history. Recommendation systems are a win-win for both consumers and service providers. For consumers, they are beneficial because they give them personalization based on their interests and affinities. For service providers, it creates an advertisement opportunity and brings their customers closer to the next items they may purchase on their platform. While there are well-known methods for making product recommendations in domains like retail and entertainment, the recruiting industry has historically lacked well-known methods of providing similar recommendations for candidates and jobs.
Recruiters are looking to select the best candidates from a typically large pool of candidates. Many recruiting technology vendors offer a way to filter candidates based on their skills and experience, but even then, the pool of candidates is often unmanageably large. Previously, there was a gap in the hiring software market for narrowing down the applicant list by more relevant criteria, such as differentiating between ordinary candidates and highly motivated talent. iCIMS identified the recruiters need and addressed it in the iCIMS Talent Cloud. In this paper, we explore how iCIMS could use intent and sentiment detection to build a recommendation system for recruiters.
Let us consider a few sentiment examples in the recruiting domain. In talent acquisition, certain events—such as a candidate looking for a job, trying to schedule an interview, inquiring about job requirements, or asking for the next steps—are considered positive events. On the other hand, a candidate’s lack of interest in the job, stalling the process, indecision on what they need to do, or canceling the interview are considered negative events.
Below is a table that outlines examples of sentiment and intent from candidate messages. 2.1 iCIMS Talent Cloud The iCIMS Talent Cloud can help you stay engaged with top talent along every step of the talent acquisition lifecycle, deepen your talent pool, and fill positions faster by building strong connections with candidates at scale.
The iCIMS Talent Cloud helps recruiters to: • • • •
Continually engage with candidates in their talent pipeline through robust talent pools.
Send targeted, automated text and email campaigns that encourage candidates to apply faster.
Host virtual career fairs to cut costs and expand their reach Get powerful recruitment marketing reporting and source analytics that helps measure ROI and efficiency Using the iCIMS Talent Cloud, recruiters can make use of this recommendation system to engage with quality talent.
At iCIMS we provide tools to recruiters that help them communicate with candidates via various channels, such as email, SMS, WhatsApp, Facebook, and more. Training data examples described below occur between candidates and recruiters mostly via text messages. With the iCIMS platform recruiters can contact candidates via text messages and corresponding communication occurs between them via SMS.
After analyzing a vast number of conversations happening between recruiters and candidates on the iCIMS platform, we observed that conversations could be broadly classified into the below categories.
These are messages from candidates who are very eager and highly motivated to find a job. Consider a few examples below: • Ok, great, I'm excited! How soon can I start if
I get hired? • I have a construction background, and this is exactly what I have been looking for. • My name is John, and I would love to be a delivery driver for ABC Pizza Company. • I have already completed the application. I am a very dedicated employee, and I am interested in taking the next steps.
These are messages from a candidate who is seen as somewhat interested in finding a job with the company. Consider these examples: • I want to apply for the HR team member position. • Can I apply online or turn in my resume in person? • • • • • • • • • •
I am a dot net developer with more than 10 years of experience in web development.
These types of conversations are where candidates are either engaging in small talk with the Recruiters or asking basic exploratory questions about the company, its policies, benefits, etc. These candidates are still trying to figure out if there is a good match and whether they want to apply for job.
Does your company offer good benefits? Who is part of your leadership team?
Do you hire people with disabilities?
These types of conversations are where a candidate is mildly frustrated at the service and/or is experiencing some difficulties in the application process.
Why haven’t you contacted me to follow up? Why are there so many steps involved in the application process?
I need to cancel my interview.
These types of conversations are where a candidate is not interested in finding a job and/or is extremely dissatisfied with the hiring process.
I have decided to work at home. I’m not looking for a job now.
I already have a job. Thank you for the opportunity.
You could add more granularity to the above sentiment classification scheme based on your needs. Instead of five levels of granularity for sentiment, you could have seven. iCIMS eventually used seven levels of granularity in the final model.
We annotate messages coming from candidates and identify the intent and sentiment behind those messages. Intent is helpful to the model because it can learn that some intents like “amountOfExperience” are associated positive sentiment while other intents like “cancelInterview” and “notInterested” are associated with negative sentiments.
Below are some examples of training data we generate:
I'm not having any interviews.
This is the end of the conversation.
Sentiment Intent 0 (very notInterested negative)
fill out an application, but it keeps redirecting me I can't come to the interview
my dog to work? Can you provide me with some information about the company? I have 10 years of experience in software development. I saw the job description. It is a perfect fit for my profile. How can I apply? 1 (negative) 1 (negative) 2 (neutral) 2 (neutral) 3 (positive)
applicationProblems cancelInterview petToWork companyInformation ammountOfExperience applicationProcess iCIMS’ models are trained on the joint features of intent and sentiment classification. We have identified dozens of intent categories and seven levels of sentiment in our training data. This way, sentiment classification, and intent classification tasks are helping each other to reduce the overall loss of the model. We are using transformers to generate embeddings of the incoming text and then use dense and softmax layers in the final layers of it to achieve classification. Although we are making use of transformers, this could also be also achieved with an LSTM or GRU-based RNN model.
Shown below is output of the model summary to give an idea of the joint model inference: The model is then run on a test dataset to evaluate the results. As you will observe from the above output, the model has intent accuracy of 88% and sentiment accuracy of almost 91% on test dataset. Thus, we have a model which could predict a sentiment score for most text messages from a candidate.
There are few other things we need to consider apart from the sentiment score described earlier.
Response time is the median time taken by a candidate to respond to a recruiter’s messages. A candidate who is engaged and interested in finding a job will often be very prompt in responding to recruiter messages and have a much shorter response time. On other hand, candidates who are not interested will likely have longer response times.
Response rate is the ratio of the number of messages which a recruiter sends to the candidate in relation to the number of messages the candidate sends to the recruiter. A high response rate is an indication that the candidate is very engaged and responsive.
Response Rate for Candidate = (total incoming messages to candidates / outgoing messages from the candidate)
We need to be mindful of the fact that candidates’ recruiting needs can change over time. A candidate who was not interested in finding a job in the past may need to relocate ,or their employer may have cut down on their hours, so is, therefore, is back in the job market. Alternatively, there may be a candidate who has already found a job but is now happy with their current employer and is no longer interested in looking for a job. In those cases, we need to apply a time decay to the sentiment score. Time decay provides a way to give more weightage to recent messages from a candidate.
Sentiment Score = ∑-# !"#$%&"#$ !()*"
# + , Where n = number of months elapsed
Candidate Message to Recruiter (2 months ago) Sorry, I am not interested in looking for a job. (very negative, sentiment score -2) Candidate Message to Recruiter (1 month ago) What kind of benefits do you offer? (Positive, sentiment score 1) Candidate Message to Recruiter (this month) I am interested in looking for a job at your company. (very Positive, sentiment score 2)
, + 1 ∗ Sentiment Score = −2 ∗ .+, 1.86 , + 2 ∗ , = , + , - + , The final score of a candidate is weighted more towards sentiment of their recent messages than past messages.
Similarly, time decay can be applied to response rate and response time as well. For example, if a candidate became responsive after a period of unresponsiveness, it could indicate that they are back in the job market.
We can calculate response time and response rate on a monthly basis, and then we could apply time decay.
Finally, using the sentiment score, response rate, and response time, we can calculate an engagement score for the candidates.
Each of the factors, i.e., sentiment score, response rate, and response time, could be given some weights to calculate engagement score. = (1 × ) + (2 × ponse Rate) + (w3 × /01234,10 5670) We could set these weights, w1, w2, and w3, manually based on what is important for the business goals. For example, we can give more weight to sentiment score and less weight to response rate and response time. We can even use machine learning to automatically compute those weights.
Let’s illustrate using an example of Engagement Scores in surfacing candidates. Assume recruiters are looking for candidates with “marketing skills” and there are numerous candidates in their database with those skills. The recruiters need some means of prioritizing the candidates with which they want to first engage. They could use Engagement Score as the criteria to engage with the top 10 or 20 candidates and work down the list in that order. Engagement score bubbles the engaging candidates who are currently looking for a position at the top and those who are not interested to the bottom.
Note that no personal data is used in the calculation of engagement scores or their components. No automated actions are performed with these scores.
There are some limitations in surfacing candidates to recruiters using the above methodology. This system suffers from a cold start problem which means that if the candidate is new, the system would not have sufficient data to derive any kind of sentiment for that candidate. So even if the candidate might be good, we would not be able to recommend his candidacy to recruiters until there are few messages exchanged. Also, sometimes weak candidates might show high engagement or good candidates may not come across as very engaging in their initial conversations with recruiters.
At iCIMS, our focus is to help our customers find the most qualified talent. Spending time on candidates that are not that interested in the job can be a barrier to that goal because it takes time away from engaging with those that are invested. Plus, it’s often time intensive for talent acquisition professionals to surface great-fit candidates without having to sift through hundreds of profiles, which ultimately increases time-to-fill. The iCIMS Talent Cloud provides solutions to both challenges.
To fill a gap in the recruiting software industry, iCIMS clients can use sentiment scores along with response time and response rate to come up with a highly accurate and valuable system for recruiters. Our clients can use engagement score and skills tags to filter quality candidates for their open roles. If a candidate is responsive and asking relevant questions, our system will give a higher engagement score to that candidate. On the other hand, if the candidate is not expressing much interest, stalling, or not truly searching for a job, our system will give a lower score to that candidate. That way, talent acquisition professionals can identify qualified talent more quickly than with the traditional candidate engagement methods.
By surfacing candidates that have relevant skills and are motivated to discuss their jobs, our clients can spend less time on back-and-forth communication and more time on strategic initiatives.
ACKNOWLEDGEMENTS I would like to acknowledge Talent Cloud AI Team at iCIMS for providing their valuable feedback in writing this paper. I would also like to acknowledge the Marketing, Product, and Legal teams at iCIMS for helping me put all this together.