A hierarchical occupation taxonomy helps Indeed better match job seekers and jobs. Historically, scaling this hierarchical taxonomy in international markets was a time intensive and highly manual process. Leveraging the strengths of both machine learning models and subject matter experts led to the creation of an improved human-in-the-loop system that met the needs of a growing business without sacrificing quality. This paper describes this system and discusses the challenges and insights from its implementation. Specifically, it highlights the value of involving subject matter expertise beyond just labeling and therefore ofers the application of an expert-in-the-loop framework for scaling taxonomy. Human-in-the-loop, Expert-in-the-loop, taxonomy, subject matter expert, natural language processing, scaling matching between jobs and job seekers and enables per- Figure 1: Example of an occupation taxonomy branch omy is granular, precise, and customizable by market. leverages broader and narrower concept relationships to RecSys in HR'22: The 2nd Workshop on Recommender Systems for
volume of unstructured data in the form of millions of job descriptions and resumes. Classifying a job or work experience as an occupation is one way of extracting structured data from these documents. This improves sonalization.
The Indeed Taxonomy team has spent years thoroughly researching and developing a hierarchical occupation taxonomy and a hand-curated rule system to classify jobs for core strategic markets. The occupation taxon
investment for international scaling and ongoing maintenance. As demand for Indeed’s presence to expand into more international markets rapidly grew, the team needed new ways to scale this work faster, with fewer resources, and without sacrificing data quality.
This paper reflects on the challenges and successes encountered while introducing a human-in-the-loop (HITL) approach to scaling Indeed’s occupation taxonomy development and classification to support its rapidly expanding international business strategy. In particular, it discusses a core learning: that targeted combination of domain expertise and broad application of machine learning technology has been the key to success. The scope of this combination exceeds the level of human-augmentation generally implied by HITL, and we therefore describe it more precisely as expert-in-the-loop (EITL). Human Resources, in conjunction with the 16th ACM Conference on [Hierarchy visualization of part of food & beverage occupation taxonomy] 2.
enrich understanding of occupation types. For example, Bartenders is narrower than its broader taxonomy concept, Food & Beverage Servers, and Food & Beverage Servers is narrower than its broader concept, Food & Beverage Occupations. It is therefore understood that Bartenders is a type of Food & Beverage Occupation. This broadest grouping is referred to as a Top Level concept. Given a job or resume document, our classification system assigns the most specific occupation concepts possible.
60 countries. Occupation taxonomy design in diferent markets may vary due to factors such as employment landscape, economy, and the volume of jobs hosted on
team has found that occupation taxonomy design is typically more similar than it is diferent across markets.
Compared with occupation taxonomy concepts that exist in the US, the most similar International market with complete taxonomy coverage has over 90% of occupation concepts in common, while even the least similar International market has roughly 50% in common. Having a monality and distinction allows for sharing knowledge nEvelop-O among markets, designing market-specific features, and avoiding excessive granularity where it is not relevant or required.
The Taxonomy team at Indeed is led by taxonomists who are highly skilled professional researchers and possess graduate degrees in Information Science, Philosophy, Linguistics, or related disciplines. Teams of Taxonomy Analysts are hired specifically to support strategically significant markets, and these individuals typically possess work experience in data analysis or professional research.
They must have strong expertise in a market’s employment landscape as well as spoken and written fluency in the oficial languages. They either possess a formal education in Information Science with hands-on experience with Taxonomy, or receive extensive on-the-job training in Information Science concepts and Taxonomy [End-to-end workflow of the expert-in-the-loop system] best practices. Their work is informed by consulting Figure 2: High-level overview of the expert-in-the-loop sysrelevant quantitative and qualitative data, competitive tem. Circled steps are heavily SME involved; Squared steps analyses, and applying many of the same user-centered are fully automated. design principles seen in UX Research or Content Design.
Indeed’s occupation taxonomy expansion for the first few international markets was based on a rule-populated can refer to diferent model architectures in the various manual approach and took years to achieve minimal vi- development phases and markets. This section discusses able state in each new market. As international business model selection and evolution in detail. exponentially expanded, the human-intensive process of When entering a new market, instead of manually crecrafting separate rule systems for each new market and ating a rule system from scratch for occupation classificakeeping them up-to-date became insurmountable. tions, we employ machine learning. We start by selecting
There is plenty of research on applications of machine a model for training that makes the best use of the
exlearning techniques to large-scale taxonomy develop- isting data. This existing data may include SME-curated
ment and document classification in diverse fields with occupation hierarchies and high-quality, rule-generated
minimal human intervention [
After the selected model generates predictions, SMEs Scaling conduct research and build out the market-specific occupation taxonomy iteratively. For example, SMEs identify novel market-specific occupations like Judicial Scriveners We represent a simplified overview of our EITL system in Figure 2. Note that the “model” specified in the figure in Japan, Pizza Chefs in Italy, Hostel Wardens in India, or 4.1. Transfer subject matter expertise to removing occupations that do not apply to the specific understanding in data market. The ongoing research is reflected by a dynamic layer at the inference time, which is used to process pre- Shadowing taxonomy SMEs and listening to their indictions into the market-specific taxonomy. This step is sights and challenges ofered valuable information that referred to as ”Predictions fallback” in Figure 2. would not otherwise have been obvious. Incorporating
To determine deployment eligibility, Taxonomy SMEs this information into the modeling process resulted in label the model predictions for evaluation on a concept- significant impact. by-concept basis and push concepts to production based For example, we learned that although taxonomy hion a predetermined quality threshold. Since the top level erarchies may vary in structure from market-to-market, taxonomy concepts are typically shared among all mar- there are occupation concepts across these markets that kets, it takes a short amount of time for a new market to are conceptually identical but have diferent IDs due to deploy top level concept predictions in production. those structural discrepancies. This posed an issue for
For novel, market-specific concepts, or for concepts cross-market training and prediction. By incorporating where prediction quality is below threshold, SMEs pro- the mapping into model training, we saw gains in both vide training labels for retraining the model. For de- precision and recall in all the locales, with an average ployed models and concepts, we follow a monitoring increase of 5% in precision and 12% in recall. process to discover and act on issues. Learning about diferences in taxonomy concepts
After a market-specific taxonomy is built out, our re- across markets also prompted us to explore and apply search shows that a market-specific CNN model utilizing market-specific models after the initial set of marketthis knowledge with translated training data often outper- specific hierarchies are built out. Experiments in the forms the global M-BERT model. Therefore, the “model” initial adopted market showed an average increase of in Figure 2 will be replaced by a market-specific model 12% in precision when the threshold is set to keep recall for subsequent loops. the same.
This EITL approach leveraging machine learning models eliminates the most expensive task in the process: 4.2. Quality training and evaluation data the need to curate and maintain a rule-based system in at scale pursuit of capturing 100% of all possible classifiable syntaxes. It allows the Indeed Taxonomy team to divert Quality data is key for training and evaluating any auresources to more SME-critical tasks, such as research- tomated systems. With limited resources, we explored ing and creating market-specific occupation hierarchies outsourcing labeling and using user feedback data to and SME-annotated datasets, expediting deployment, and directly approach the problem, and also revisited prioriother metadata projects. We also benefit greatly by in- tization of the tasks based on our learning. volving SME knowledge beyond just labeling like most traditional HITL systems, which we’ll discuss in detail 4.2.1. Outsourcing labeling in the next section. As a result of this approach, we observed a six times faster deployment with similar SME resources was achieved in markets adopting this system (compared to the fully manual process), and this system is now in production in dozens of markets.
is needed has been known to be challenging [
After working with external labelers on multiple tasks with mixed outcomes in the past years, we learned that significant work needed to be put in upfront in order to gain high quality outcomes–underscoring the diference 4. Challenges and Insights between EITL and HITL. Certain tasks need strong domain specific subject matter expertise which is hard to We learned a lot from implementing this approach. In gain with short-term training sessions, particularly when this section, we will discuss our success in leveraging much of the communication is asynchronous. Those subject matter expertise in the modeling process, adding tasks should not be outsourced in the first place. Guidtransparency to a blackbox system, and establishing a ing labelers to evaluate predictions required ongoing monitoring process. We will also share insights from ex- support in applying consistent classification heuristic to ploring diferent data sources and identifying the unequal their labeling tasks. For example, coaching labelers to nature of diferent errors. understand the diference between occupation and work environment was a unique challenge. Frequently, jobs for School Custodians would be labeled as a type of Education & Instruction Occupation due to their workplace environment. Occupationally, however, this should be a Cleaning & Grounds Maintenance Occupation. Moreover, our close collaborations with SMEs on labeling tasks, we if a task involves distinguishing between specific licenses learned that there is usually a certain level of subjectivity or specialties in diferent industries, significant research associated with decisions, which naturally brought us and domain knowledge are needed in order to produce this question: How bad are the errors? For example, if the high quality labels. model makes the same error as a non-SME user, it likely
After identifying tasks suitable to outsource, the next does not hurt user experience as much as a completely steps are to work with SMEs to scope the task, provide out-of-place error. After examining diferent definitions, clear guidelines, and set up monitoring-QA-calibration considering trade-ofs between information granularity loops to help ramp up the quality. In a labeling task and interpretability, we ended up defining three levels where the goal is to generate evaluation data for top of error severity with easy interpretations: 1) the mild level occupation predictions in a new market, we found errors are within 2 hops of the correct labels, typical it to be more eficient to scope the task in the binary examples include siblings, parents and children nodes; fashion: During the labeling task, labelers are asked to 2) the moderate errors are at the same top levels with the label the predictions such as “Job A is a type of Education correct labels, usually due to granularity issues; and 3) Instruction Occupations job” as correct or incorrect. The the severe errors are in diferent top levels, which will incorrect predictions are then sent to SMEs to assign result in bad user experience if not fixed. correct labels. This second step is essential to improve Having this insight allows the SME team to conduct model quality but extremely hard for non-SMEs since more nuanced error analysis such that they can better it requires familiarity with definitions of a few dozen prioritize severe errors for quality improvement tasks. It occupation concepts. also allows the data science team to measure quality in a more practical manner, and utilize data from multiple 4.2.2. User input data sources which would be considered too noisy otherwise.
noisy [
but this technical opacity can lead to ineficient crossfunctional collaboration. For example, during the initial collaboration period, SMEs were tasked to provide labels for a model in a market with around a thousand concepts.
After a significant amount of work was put in, we realized 4.3. Understand what error matters the evaluation metrics were inaccurate due to nonrandom In the classic machine learning model measurements, selection of labeled documents. In addition, the initial error is defined to be a binary concept. However, with instruction of providing a large fixed amount of labels per concept could be improved since concepts that perform [Example of the token importance function applied to a job in
the internal tool] well do not need additional training data, and the ones with fewer jobs struggle to find enough documents to be labeled and have little efect on the overall performance.
We started with knowledge sharing, Q&A, and
discussion sessions to connect common operation actions
with implications on the models, and gradually built out
resources and documentation for FAQs. In addition, we
built a function to visualize token importances in a given
training sample with the Integrated Gradients [
These insights and successes could not have been achieved without close collaboration and communication. It enabled us to efectively distribute SME resources to the highest priority task, which resulted in the research, development, and publishing of over 700 existing or new occupation taxonomy concepts in strategically significant new markets within six months.
In this paper, we presented a human-in-the-loop system at Indeed that scales occupation taxonomy development and application to international markets with high quality outcomes. Over the course of developing the system, we found tremendous value in leveraging subject matter expertise throughout the process, ranging from data collection and processing, model training and evaluation, to performance monitoring and diagnosis. Therefore we propose calling this system expert-in-the-loop.
In ongoing work, we are looking to partner more closely among functions to allow for combined SMEs, explore further improvements to the current machine learning models, and build better functionality for overriding training and production data.
Taxonomy team that contributed to this system for their diligent work and thoughtful collaboration. We are grateful to the reviewers of this paper for providing valuable feedback and comments.