This study explores the potential of reinforcement learning algorithms to enhance career planning processes. Leveraging data from Randstad The Netherlands, the study simulates the Dutch job market and develops strategies to optimize employees' long-term income. By formulating career planning as a Markov Decision Process (MDP) and utilizing machine learning algorithms such as Sarsa, Q-Learning, and A2C, we learn optimal policies that recommend career paths with high-income occupations and industries. The results demonstrate significant improvements in employees' income trajectories, with RL models, particularly Q-Learning and Sarsa, achieving an average increase of 5% compared to observed career paths. The study acknowledges limitations, including narrow job filtering, simplifications in the environment formulation, and assumptions regarding employment continuity and zero application costs. Future research can explore additional objectives beyond income optimization and address these limitations to further enhance career planning processes.
1. Introduction vidual’s professional journey cannot be overstated. It involves strategic decision-making related to one’s career goals, which may be as diverse as the individuals themselves. However, despite varied ambitions, a proacifts all, allowing individuals to align their career trajectory with their objectives, such as maximizing lifetime income. Recognizing that reality often presents multifaceted goals and constraints, this paper aims to simplify the career planning process using the power of artificial intelligence.
on the insight one has into potential career paths and vide such insights to employees. Collaborating with
RecSys in HR’23: The 3rd Workshop on Recommender Systems for Human Resources, in conjunction with the 17th ACM Conference on †Work done while on internship at Randstad Groep Nederland.
their expected rewards. This study leverages AI to pro- reliable. It is important to note that the suggested career career progression.
Several studies have taken varied approaches to this
issue. For instance, Paparrizos et al. [
The advent of more complex models has also been
witnessed. Meng et al. [9] used a hierarchical neural network
with an embedded attention mechanism, and Xu et al.
2.1. Reinforcement Learning [10] performed a talent flow analysis for predicting the
Reinforcement Learning (RL), as characterized by Sutton increments in a dynamic job transition network. Other
and Barto [
This has proved valuable for both employers and employees, facilitating strategic decision-making in hiring and
given time is characterized by his current job, which is defined as a combination of occupation and industry, We consider the problem of recommending a sequence of along with their work history. This concept forms the jobs — a career path — to the candidates that, if followed, basis of the state space defined by this module, which would maximize their earnings during their foreseeable comprises the set of available jobs and industries the future. agent can occupy. However, due to the constraints im
Formally, given = { 1, ..., } as candidates and posed by the dataset and to ensure a computationally = { 1, ..., } as jobs, we define as the recom- feasible environment, our experiments are restricted to mended career path generated for candidate . We denote the 142 most prevalent jobs present in our dataset. = { ,1 , ..., , } as the work experience of candidate , diferent jobs that worked in the past with ,1 being Transition Module Job applications do not always her first job and , being her last (current) job. Each have deterministic outcomes; similarly, the actions taken work experience contains information about the period by the agent should not always have deterministic out(start date and end date) and the area (or role) that the can- comes. When the agent applies for a job and succeeds in didate worked on in that job. The work area represents a being hired, it transitions from the current state s to a new high-level categorization for the jobs. In our experiments, state s′. If unsuccessful, it remains in the current state we define it as a combination of an occupation and an s. This transition occurs with probability ( s′|a, s). A industry. Examples are a Data Science role in Insurance Random Forest binary classifier is trained on the Job Apor a Data Science role in Banking. Refer to Section 4.1 for plication data and is used to predict the aforementioned more details. We also denote = { , 1, ..., , } probabilities. In other words, this module computes the as jobs that candidate applied for in the past in which transition probability between diferent jobs within the their outcomes are either hired or rejected. Finally, we environment. denote as all the vacancies posted on the market. We consider the following approaches for computing
Given these three input data (work experience , job the transition probabilities: applications , and vacancies ), our career path recommendation model comprises four distinct modules, as depicted in Figure 2.
The first three modules— Plausible Jobs, Transitions, and Rewards—simulate the job market environment. The fourth module employs Reinforcement Learning (RL) to learn an optimal strategy for navigating these environments. • Last Job State Representation: We assume that state s′ only contains information about the last job of a person, implying that the probability of being hired depends solely on their latest job. • Full History State Representation: Conversely, in the alternative approach, we assume that the state contains information about a person’s entire work history. This second approach is closer to reality, but it also greatly increases the size and complexity of the state space, which could make learning more challenging and could potentially sufer from a lack of data.
used to compute the reward earned from that transition. We define the reward in the form of the estimated salary that the individual earns after the transition. We use a Random Forest regressor trained on (i.e. all vacancies in the market) to predict the salary corresponding to each job. Given each ∈ consists of the textual job description, and annual salary information, for our experiments we perform this prediction on a yearly basis for each pair of job role and industry.
Reinforcement Learning Module Lastly, the RL module uses RL algorithms to learn policies that can yield optimal rewards. After training, these models can be used to recommend high-income-yielding career paths to employees. We experiment with and compare multiple algorithms during the training of the RL module. The details of the algorithms are described in section 4.3.
missing data, jobs with durations less than a week, and employees with more than fity work experience items from the work experience dataset, yielding 200K employees with 400K work experience items.
In line with Randstad’s business model, most placements are short-term or temporary jobs common in stafing, resulting in a mean job duration of 161 days and a median duration of 95 days.
The average annual salary in the vacancies dataset is approximately 42K euros, with a median salary of 38K euros.
4.3.1. Tabular Methods
We conducted our experiments on anonymized data pro- well with a discrete, small state-action space. They mainvided by Randstad as follows: tain a table of values, with each entry in the table representing the value of each possible state-action pair.
Work Experience Dataset This tabular dataset consists of work experience items that employees may sub- State–action–reward–state–action (Sarsa) Intromit to Randstad either online or through consultants, or duced by Rummery et al. [20], Sarsa is an on-policy, are directly taken from the administration of job place- tabular, temporal diference (TD) method. TD learning, ments made through Randstad. Relevant attributes for which is a hybrid of Monte Carlo and dynamicthis research include: 1) Employee ID, 2) Job start and programming ideas, can learn directly from raw end dates, 3) ISCO code1 (occupation identifier), and 4) experience without a model of the environment’s SBI code2 (industry identifier). dynamics. Like dynamic programming, TD methods
Almost all the work experience items (99.99%) per- update estimates based in part on other learned estimates, tain to Randstad placements, as these are jobs employees without waiting for a final outcome (they bootstrap). secured through Randstad. Most of the previous expe- The Sarsa algorithm aims to learn an action-value riences (before using Randstad’s services) are missing function (, ) , providing the expected reward starting essential attributes. from state , taking action , and following the policy . Vacancies Dataset This dataset includes salary ranges Q-Learning Introduced by Watkins and Dayan [21], for about six million vacancies, including their ISCO and Q-Learning is another tabular TD method. However, SBI codes, posted on various Dutch websites. We use this Q-Learning is an of-policy method, where the learned dataset to estimate expected salaries for each occupation. action-value function, Q, directly approximates ∗, the optimal action-value function, regardless of the policy followed (behavior policy).
tional_Standard_Classification_of_Occupations 2SBI oficial website: https://www.kvk.nl/overzicht-standaardbedrijfsindeling/ 4.3.2. Approximate RL Methods While tabular methods perform well in environments with a small number of state-action pairs, they face challenges when the state-action space becomes large or continuous. They are not able to eficiently store the value of every possible state-action pair, nor can they generalize the value of unvisited state-action pairs efectively. This is where approximate RL methods come in to help with the Full History State Representation. These methods use function approximation, typically employing neural networks, to estimate the value of state-action pairs, allowing them to handle environments with larger or more complex state spaces more efectively.
Deep Q-Learning (DQN) DQN is an of-policy approach introduced by Mnih et al. [22]. DQN is the first successful deep-learning model to learn control policies directly from high-dimensional sensory input using reinforcement learning. It utilizes a convolutional neural network trained with a variant of Q-learning, taking raw pixels as input and estimating future rewards through a value function.
Advantage Actor-Critic (A2C) mate solution RL method that utilizes deep reinforcement learning for function approximation. Unlike DQN, A2C is an on-policy method. Introduced by Mnih et al. [23], A2C is an actor-critic method, where the policy function is represented independently of the value function. The “critic” model estimates the value function and the “actor” learns the target policy. Both the Critic and Actor functions are parameterized with neural networks. As explained by Mnih et al. [23], the main advantage of A2C over DQN is its faster training speed.
Greedy Most Common Transition the agent always applies for the job with the highest transition probability, that is the most likely job to be hired. In the case of multiple jobs with the same ranking, a random selection is made.
Greedy Highest Expected Reward In this strategy, if she had followed the recommendations of our system. the agent applies for the job with the maximum expected reward, defined as the product of the transition probability and the immediate salary after the transition. In reality, this signifies the job with the highest likelihood of both being attained and yielding the highest immediate
In this approach, the month . Similarly, the counterfactual income is where ′ is the job that employee would have performed
the percentage change between the factual and counterfactual means. To determine the statistical significance income. As before, in the case of multiple top-ranking jobs, a random selection is made.
income diference between observed career paths (factuals) and recommended career paths (counterfactuals). Observed Career Paths Using the Work Experience dataset, we generate a list of observed career paths and their corresponding income. Given that workers can hold multiple jobs simultaneously or have periods of unemployment, the dataset requires processing to align with the requirements of our simplified environment. Our models assume people only have one job at a time and there is no unemployment. Therefore, in cases of simultaneous employment, we estimate each job’s monthly salary and assume the worker earned the mean salary.
Counterfactual Career Paths
Reported metrics
tion, we report two primary quantities - the Mean Factual and Mean Counterfactual accumulated rewards. These metrics represent the mean income accumulated by employees in reality versus the projected income they would have earned in a counterfactual scenario respectively.
For an employee their factual income denoted as , over their career of
months is calculated as () = ∑ ( , ) =1 () = ∑ ( ′, ) =1 A2C is an approxi- their counterfactuals. These are the paths each model (1) (2) where ( , ) is a function that returns the salary that the employee had earned by performing job during calculated as Baseline: Most Common Baseline: Highest Exp. Reward Q-Learning Sarsa Model Baseline: Most Common Baseline: Highest Exp. Reward Deep Q-Learning A2C of the observed diference, we calculate a p-value using a two-sided permutation test with an alpha level of 0.05.
Furthermore, we detail the proportion of employees experiencing an income rise in the counterfactual world, referred to as Gainers, along with the average magnitude of their income change. Similarly, we present data for those experiencing a decline, termed as Losers, including the mean change in their income.
This subsection presents a detailed analysis of the experiment results. We juxtapose our factual and counterfactual career paths in terms of the mean income they generate. Additionally, we assess the efectiveness of our models by examining the percentage of gainers and losers as well as the magnitude of their respective income changes.
Table 1 presents the results for the Last Job State Representation, where the job seekers’ state depends only on the last held job. From this table, we can observe that baselines do not perform significantly diferently than the factual career paths, with diferences under 1% (at 0.7% and -0.94% for Most Common and Highest Expected Reward baselines, respectively). However, Q-Learning and Sarsa models perform well with a notable percentage of income gainers (27.53% and 32.84% respectively) and a reasonable mean gain percentage of 13.81% and 11.5% respectively.
Table 2 exhibits the outcomes for the Full History State Representation, where the state of a job seeker contains their full work history. The Highest Expected Reward baseline model stands out with a significant mean income change (79.18%) and a large percentage of gainers (96.01%). As we will discuss later, this is caused by Transitions module biases. Deep Q-Learning and A2C also show promising results but fail to outperform the baselines.
In this chapter, we delve into the outcomes derived from our experimental setup featuring two unique versions of the transitions module - the Last Job State Representation and Full History State Representation. We start our discussion with findings from the two baseline methods outlined in Section 4.6. Subsequently, we elaborate on the results achieved through our eforts to learn an eficient policy.
Last Job State Representation In the Last Job State Representation, our RL approaches learned policies that resulted in career paths with higher incomes than the observed career paths. In both the last job state representation and full history state representation, the learned policies improved the mean accumulated income by around 5%. While not a drastic increase, this change is significant over longer time scales, such as an individual’s career. Notably, these improvements surpassed those of the baseline models. However, there was also a significant amount, approximately 12%, of agents for which the recommended paths performed worse than the observed.
Full History State Representation However, the Full History State Representation demonstrated a diferent pattern. While the DQN and A2C models also found policies improving counterfactual incomes, the baselines showed significantly larger improvements, particularly the Highest Expected Reward baseline. This raises ques- Comparison Between Environments It’s importions about the validity of the environment. After careful tant to note that the results from the Last Job State Repreinvestigation, we found out that this environment can be sentation and Full History State Representation cannot be easily exploited by the Highest Expected Reward baseline directly compared. Each model learns a policy to exploit due to the small-but-substantial transition probabilities the unique dynamics of the environment it is trained on, predicted by the Transitions module. As we can see in therefore, the ground truth difers for each environment. Figure 3, regardless of the agent’s starting state, it al- As such, results should be compared within the specific ways applies and eventually succeeds to be hired for the environment they originated from. highest-paying job of our dataset. By looking deeper into the classifier trained to predict the transition probabili- 5.2. Limitations of the Study ties of the Full History State Representation, we see that regardless of the agents’ prior experience there is always Job Filtering The experiments relied on a narrowed a small but significant probability of employment. After ifeld of 142 most common jobs. The decision to do so was analysis of the training data, we believe that this is due driven by the challenges posed by the vast state space to missing data in the prior experiences of employees and unreliable transition probability prediction for less hired in senior positions. Therefore, the training dataset common jobs. is incomplete and depicts a world where someone can be hired, for example, as a Senior Finance professional with no experience in Finance.
The Cost of an Action The research assumes no monetary cost for applying to a job, which is typically not the case in reality, where applications cost both time in interviewing or preparing, and perhaps other forms of preparation (e.g., studying). Considering the real-world incomes of individuals. In the Last Job State Representacosts in future studies could bring the environment for- tion, the mean accumulated income increased by around mulation closer to reality. 5%, surpassing the performance of the baseline models. However, it is important to acknowledge the limitaContinuous Employment The assumption that indi- tions and failures of the Full History State Representation. viduals are always working doesn’t account for potential The baseline models exhibited greater improvements in breaks in employment. These breaks could result from counterfactual incomes compared to the reinforcement various factors, including vacations, relocation, or further learning models. This discrepancy was due to inaccueducation, and should be considered in a more realistic racies in the transition probability predictions, which formulation of the job market as an MDP. allowed the Highest Expected Reward baseline to exploit the system.
Rivalrous market Another notable limitation is that These limitations indicate the need for further research the real-world job market is inherently rivalrous — a to refine and improve the Full History State Representajob ofered to one applicant becomes unavailable to oth- tion. Future studies could explore alternative methods ers. Recommending the most highly paying jobs to all for estimating transition probabilities and address the users could potentially lead to an overwhelming influx issue of missing prior experience data. By addressing of applications for those positions, resulting in many dis- these challenges, future research can work towards creappointed users due to the increased competition. Our ating more robust and accurate environments that better approach focuses on income optimization, but it’s crucial reflect the complexities of real-world career planning. to recognize that a well-balanced approach considering job availability, individual preferences, and market dy- References namics is necessary to avoid an undue concentration of applications in specific roles. Future research should delve into strategies that account for these challenges while still aiming for income optimization to create a more comprehensive and realistic career planning model.
State Space and the Markov Property The models used in this research made diferent assumptions about state space and respected the Markov Property in different ways. The Last Job State Representation model simplified the state to be a job, assuming an employee’s future depends solely on their last job. On the other hand, the Full History State Representation model considered employees’ whole work experience as part of the state.
The latter approach is closer to reality but can create states with too many dimensions, slowing policy learning. An option suggested in the literature for similar challenges is learning low-dimensional embeddings and reducing the state space size.
intelligence, specifically reinforcement learning, in the ifeld of career planning. By harnessing data on employee work experience and job applications, the research aimed to recommend career paths that maximize long-term income for individuals.
The findings of this study showed promising results in both the Last Job State Representation and Full History State Representation approaches. The reinforcement learning models, particularly Q-Learning and Sarsa, were able to learn policies that improved the counterfactual panion - WWW ’17 Companion, ACM Press, New reer path recommendation, IEEE International York, New York, USA, 2017, pp. 505–513. doi:1 0 . Conference on Data Mining Workshops, ICDMW 1 1 4 5 / 3 0 4 1 0 2 1 . 3 0 5 4 2 0 0 . 2018-Novem (2019) 328–333. doi:1 0 . 1 1 0 9 / I C D M W . [9] Q. Meng, H. Zhu, K. Xiao, L. Zhang, H. Xiong, A 2 0 1 8 . 0 0 0 5 4 .
hierarchical career-path-aware neural network for [17] N. Dawson, M. A. Williams, M. A. Rizoiu, Skilljob mobility prediction, Proceedings of the ACM driven recommendations for job transition pathSIGKDD International Conference on Knowledge ways, PLoS ONE 16 (2021) 1–20. URL: http://dx. Discovery and Data Mining (2019) 14–24. doi:1 0 . doi.org/10.1371/journal.pone.0254722. doi:1 0 . 1 3 7 1 / 1 1 4 5 / 3 2 9 2 5 0 0 . 3 3 3 0 9 6 9 . j o u r n a l . p o n e . 0 2 5 4 7 2 2 . [10] H. Xu, Z. Yu, J. Yang, H. Xiong, H. Zhu, Dynamic [18] M. Kokkodis, P. G. Ipeirotis, Demand-aware caTalent Flow Analysis with Deep Sequence Predic- reer path recommendations: A reinforcement learntion Modeling, IEEE Transactions on Knowledge ing approach, Management Science 67 (2021) and Data Engineering 31 (2019) 1926–1939. URL: 4362–4383. doi:1 0 . 1 2 8 7 / m n s c . 2 0 2 0 . 3 7 2 7 . https://ieeexplore.ieee.org/document/8478343/. [19] P. Guo, K. Xiao, Z. Ye, H. Zhu, W. Zhu, Indoi:1 0 . 1 1 0 9 / T K D E . 2 0 1 8 . 2 8 7 3 3 4 1 . telligent career planning via stochastic sub[11] R. Liu, A. Tan, Towards interpretable automated sampling reinforcement learning, Scientific machine learning for STEM career prediction, Jour- Reports 2022 12:1 12 (2022) 1–16. URL: https: nal of Educational Data Mining 12 (2020) 19–32. //www.nature.com/articles/s41598-022-11872-8. doi:1 0 . 5 2 8 1 / z e n o d o . 4 0 0 8 0 7 3 . doi:1 0 . 1 0 3 8 / s 4 1 5 9 8 - 0 2 2 - 1 1 8 7 2 - 8 . [12] H. Al-Dossari, F. A. Nughaymish, Z. Al-Qahtani, [20] G. Rummery, M. Niranjan, U. of Cambridge. EnM. Alkahlifah, A. Alqahtani, CareerRec: A Ma- gineering Department, On-line Q-learning Using chine Learning Approach to Career Path Choice for Connectionist Systems, CUED/F-INFENG/TR, UniInformation Technology Graduates, Engineering, versity of Cambridge, Department of EngineerTechnology & Applied Science Research 10 (2020) ing, 1994. URL: https://books.google.de/books?id= 6589–6596. doi:1 0 . 4 8 0 8 4 / e t a s r . 3 8 2 1 . JdyRPgAACAAJ. [13] Y. Lou, R. Ren, Y. Zhao, A Machine Learning Ap- [21] C. J. C. H. Watkins, P. Dayan, Q-learning, Machine proach for Future Career Planning (2010) 1 – 4. URL: Learning 1992 8:3 8 (1992) 279–292. URL: https: http://citeseerx.ist.psu.edu/viewdoc/download? //link.springer.com/article/10.1007/BF00992698. doi=10.1.1.375.3061&rep=rep1&type=pdf. doi:1 0 . 1 0 0 7 / B F 0 0 9 9 2 6 9 8 . [14] R. J. Oentaryo, R. J. Oentaryo, X. Jayaraj, S. Ashok, [22] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, E.-p. Lim, P. Kokoh, JobComposer : Career Path Op- I. Antonoglou, D. Wierstra, M. Riedmiller, Playtimization via Multicriteria Utility Learning (2018). ing Atari with Deep Reinforcement Learning [15] B. Shahbazi, A. Akbarnezhad, D. Rey, A. Ahma- (2013). URL: https://arxiv.org/abs/1312.5602v1. dian Fard Fini, M. Loosemore, Optimization of Job doi:1 0 . 4 8 5 5 0 / a r x i v . 1 3 1 2 . 5 6 0 2 .
Allocation in Construction Organizations to Max- [23] V. Mnih, A. P. Badia, L. Mirza, A. Graves, T. Harley, imize Workers’ Career Development Opportuni- T. P. Lillicrap, D. Silver, K. Kavukcuoglu, Asynties, Journal of Construction Engineering and Man- chronous Methods for Deep Reinforcement Learnagement 145 (2019) 04019036. doi:1 0 . 1 0 6 1 / ( a s c e ) ing, 33rd International Conference on Machine c o . 1 9 4 3 - 7 8 6 2 . 0 0 0 1 6 5 2 . Learning, ICML 2016 4 (2016) 2850–2869. URL: [16] A. Gugnani, V. K. Reddy Kasireddy, K. Ponnalagu, https://arxiv.org/abs/1602.01783v2. doi:1 0 . 4 8 5 5 0 / Generating unified candidate skill graph for ca- a r x i v . 1 6 0 2 . 0 1 7 8 3 .