Business processes that involve AI-powered automation have been gaining importance and market share in recent years. These business processes combine the characteristics of classical business process management, goal-driven chatbots, conversational recommendation systems, and robotic process automation. In the new context, prescriptive process monitoring demands innovative approaches. Unfortunately, data logs from these new processes are still not available in the public domain. We describe the main challenges in this new domain and introduce a synthesized dataset that is based on an actual use case of intelligent process automation with chatbot orchestration. Using this dataset, we demonstrate crowd-wisdom and goal-driven approaches to prescriptive process monitoring.
As IPA platforms become more powerful, we can expect a new generation of conversationoriented digital employees that can handle complex and dynamic processes. The duration of these processes can span from seconds and minutes to weeks and months.
Prescriptive process monitoring (PPM) techniques are used to improve business processes by triggering interventions at runtime to optimize the process towards a goal such as a key performance indicator (KPI). Traditionally, PPM looks at case ID, activity timing and sequences, resources, and business attributes to predict the progress towards the goal and introduce interventions. Because IPA has many unique elements when compared to traditional BPM, PPM requires adaptations to support prescriptive tasks in IPA.
One scenario of these adaptations occurs when there is a need to prescribe a recommendation in the form of a human-to-bot interaction, such as a button or textual utterance. There are four possible situations to consider: First, the discovery of skills and utterances for new users who are not yet aware of what the bot can do and which utterances will trigger those skills. Second, a recovery from interaction failure. Occasionally the bot will not understand the user or an error may occur that will cause the interaction to derail from its original context. Third, goal-oriented actions that help or remind the user to perform some activity in order to achieve the process goals. Fourth, the wisdom of the crowd and personalization that can recommend possible actions based on the behavior of other users, or on personal preferences.
Aside from the type of prescription, there are additional characteristics that are diferent from classical PPM, such as how process and session identifiers in human-to-bot interaction map to the concept of case ID, how disambiguation and errors are modeled as activities, how to treat or leverage human feedback, and the types of intervention that can achieve user engagement with the bot.
IPA is a relatively new domain and real-world deployments are not available for researchers due to confidentiality. Therefore, we believe that a synthetic dataset, inspired by actual IPA deployment, will be of value in terms of understanding the data model, testing existing algorithms, and developing new ones that can later be validated on real-world datasets.
The contribution of this paper is twofold. First, we introduce and share a first-of-a-kind synthetic dataset based on a real-world use case for an HR Management Incentive Program (MIP). This dataset presents a new type of data from IPA-based processes. Second, we demonstrate an implementation of crowd-wisdom and goal-oriented prescription tasks for this new dataset and explain the IPA-related adaptations to traditional algorithms.
The remaining part of the paper is organized as follows. Section 2 discusses the related art. Section 3 introduces the MIP dataset. Specifically, Section 3.1 explains its business use case, Section 3.2 presents parameters of the dataset simulation and Section 3.3 provides the dataset schema. The following sections apply two approaches for prescriptive process monitoring to the MIP dataset. Section 4 describes the crowd-wisdom approach based on the prediction of the next activity and Section 5 explains the goal-driven approach based on lateness prediction. Finally, Section 6 summarizes the paper and future research challenges.
The basic goal of prescriptive process monitoring (PPM) is to optimize a process at run-time
[
The recent emergence of trustworthiness in AI systems, has brought explainability and causal
papers to the PM domain. Galanti et al. [
Our paper is also related to recommendations for goal-driven chatbots [
Datasets: There are many open datasets that are process-based. Examples include the ones
created for the BPI 2011-2020 challenges [
Datasets from IPA systems with chatbot orchestration are still unavailable in the public domain.
Thus, we strive to partially close this gap by providing a synthesized dataset based on a real-life
use case. Specifically, the use case focuses on human resource (HR) automation systems, which
constitute an important application domain for the new technology. Guenole and Feinzig [
As a source of inspiration for our dataset, we used an internal IBM application in the domain of compensation and promotion. The implementation is based on Watson Orchestrator1. This use case satisfies several basic prerequisites for an IPA dataset. It is based on a multi-person business process, contains chatbot conversational interactions, and gives rise to two embedded process identifiers: case ID defines an instance of a business process and session ID defines a chatbot conversation instance.
We consider a large software engineering organization. The Management Incentive Program (MIP) process is run in the organization twice a year. The goal of the process is to determine which employees will get a salary increase.
There exist two roles in the MIP process: first-line team leaders and second-line department managers. The team leaders initiate a case instant to select team employees who are eligible to participate in the program. Then they provide the names of nominated candidates to the department managers. For each candidate, a department manager decides whether the candidate nomination should be approved, approved with correction (e.g., the amount of salary increase is changed), or rejected. Then, the department manager submits the final decision to the HR system, completing the case instance for a specific team. In summary, each case instance consists of two sequential tasks, nomination and approval, which are performed by two diferent users.
In order to make informed decisions, the team leaders and department managers go over a number of reports that contain diferent employee performance metrics and summarize employee activities and feedback. The list of 20 available reports is provided in Table 1. The table also contains probabilities that a user will look at the reports at least once during the process. These reports difer in their importance and the users view them in a random order with diferent frequencies. MIP criteria and yearly assessment reports are always viewed by the team leaders before initiating nomination actions. We also observe from Table 1 that the department managers view reports with a lower frequency than the team leaders. The activities that the team leaders and the department managers can perform are as follows. The team leader can: view report, add nomination, view nomination, submit nomination, and provide candidate name. The department manager can: view report, select candidate name, review nominated candidate, approve nomination, approve nomination with correction, reject nomination, and submit final nominations. We assume that most of these activities are initiated via free text chat. The system recognizes a specific intent of a user and performs the activity that corresponds to this intent. Two name selection activities are performed via a slot-filling mechanism that is frequently used by the goal-driven chatbots.
The use of free text to trigger activities gives rise to the two additional scenarios. First, sometimes the IPA system cannot detect a user intent with suficient confidence. In this case, the user utterance is followed by a fallback: a user is asked to provide additional input. Second, a user utterance can potentially correspond to more than one intent. A standard solution for this problem is disambiguation - the widely used prescriptive technique in the chatbot domain. The user is asked to select between several competing intents via the corresponding buttons. In the MIP dataset, we consider four disambiguation scenarios that are shown in Table 2. Like
Report MIP criteria Yearly assessments Project assessments Learning activities Client feedback Internal feedback Compensation report MIP history Overtime Innovation and patents Product defects Sprints velocity Bugs fixed Pull requests Features shipped Defects repair time Lead time Project costs Code churn Absence
Viewing by team leader, %
Viewing by department manager, % view client feedback report / internal feedback report view project assessment / project cost report view MIP criteria report / MIP history report view product defects report / defects repair time report
Example show feedback report view project data
MIP data I need defects report many business processes, the MIP process has time constraints that include a regular and a “hard stop” deadline. It is technically possible but undesirable to violate a regular deadline. On the date of a “hard stop” deadline, process participants are forced to complete the MIP process within several hours.
Finally, we assume that both team leaders and department managers are divided into two groups with diferent statistical properties: struggling users and successful users. Struggling users have, on average, larger intervals between conversation sessions, a higher number of fallbacks, and a higher probability of abandoning a session without successful completion of the Average interval between sessions (working days)
Average number Probability of fallback of conversations per utterance
In this section, we describe some deterministic and statistical properties of the MIP dataset simulation. We assume that the process starts on Monday, Mar 7, 2022 and it is desirable to complete it until the regular deadline: end of Monday, Mar 28, 2022. A second “hard stop” deadline takes place on Monday, Apr 11, 2022. Users can interact with the IPA system during a Monday to Friday working week, with working hours between 8am and 5pm, although sometimes conversation sessions take place later in the evening.
There are 250 department managers, and 4 first-line team leaders under each department manager. Overall, there are 250 · 4 = 1, 000 cases of the process, which should provide a suficiently large data sample for the analysis in Sections 4 and 5. On average, there are 10 employees in each team. The MIP nomination rate is 20%.
We assume that the struggling users constitute 1/3 of team leaders and department managers. The main statistical characteristics of successful and struggling users are presented in Table 3.
To generate user free-text we applied Lambada [
The MIP dataset is provided in csv format2. Table 4 presents the names of the dataset columns with brief descriptions and examples.
Each row of the file corresponds to the turn of a conversation with a chatbot. During all turns, except those corresponding to the chatbot welcome message, a user utterance triggers an activity in the IPA system. For some turns, the system recognizes a user intent, in which case the name of the activity coincides with the intent name. Additional activities are responsible for slot filling and user utterances with unrecognized intent (fallbacks and disambiguations).
We assumed that the activities are orchestrated using an agent orchestration concept similar
to the one used by Rizk et al. [
prescription methods, brief stub utterances (for example, “Welcome Message”) replace them in the dataset.
New users of IPA systems are frequently not fully aware of the actions that they can perform. They need a straightforward and intuitive way to flatten the learning curve. Crowd-wisdom methods help achieve this goal. Experienced users can implicitly guide inexperienced users to “happy paths”. For example, in the MIP use case, the path statistics for advanced users can be used to recommend the most important performance reports.
Prediction of the next activity or the next several activities is the key stage of the crowdwisdom approach. Such prescription systems typically present several options for the user’s next actions. As a result, the underlying predictions should be probabilistic and not single-activity ones. At the same time, undesirable activities, such as fallback or disambiguation, should not be mentioned in recommendations, even if they are performed frequently by other users.
In this section, we focus on predicting the next activity for the MIP dataset, while emphasizing several diferent feature generation approaches. There are 36 activities overall: • viewing of 20 reports, 4 disambiguation activities, and fallback for both roles; • candidate nomination, candidate name selection, viewing nomination, and submitting nomination for the team leaders; • reviewing nomination, nominee name selection, selecting one of three possible decisions, and final submission for the department managers.
• session end should be considered an additional activity in the prediction problem. For the prediction task, we do not filter out undesirable activities and compare diferent prediction methods based on the overall goodness-of-fit.
Feature generation is especially important in IPA processes that involve chatbots, where conversation sessions constitute sub-processes with special characteristics. For each prediction technique, we consider two feature generation dilemmas. The first dilemma is how to extract the process features: is information on the previous turn suficient for prediction or should it be complemented by process-aware features? In the non-process-aware (npa) approach, features include the previous activity and several attributes of the previous conversation turn. The list of the attributes includes role, intent confidence, score, expecting_response, and the number of turns in a session. The process-aware (pa) approach adds process path statistics to the feature vector. In our case, this path statistics includes the number of occurrences of each activity during the current session until the current conversation turn.
The second, more subtle dilemma, is related to the session definition. Should it be based on conversation (option conv in Table 5)? Or should we base it on the overall path of a user, even if a user was engaged in several conversations on diferent days ( user option in Table 5)? This is an important question since a user can behave diferently over diferent sessions or forget details of the previous ones. In the conv process-aware setting, we count activities from the start of conversations and add a sequential conversation number of a specific user to the feature space. In the user process-aware setting, we count activities from the first user login into the system. Finally, we combine the two approaches and use the union of features from the two session definitions ( conv+user option in Table 5). In Table 5, we compare the three process-aware approaches described above and an implementation of a non-process-aware approach.
We implemented three prediction techniques: logistic regression, CatBoost, and XGBoost. Our goodness-of-fit metrics included accuracy, weighted Top-3 recall, average Top-3 recall, weighted 1 score, and average 1 score. We computed the averages between 35 prediction classes, while leaving out the “end” class since it is defined diferently for diferent session definitions. Weighted averages were weighted by volume in the testing sets. The Top-3 recall can be interpreted as a fraction of samples where the actual activity belongs to the Top-3 predicted activities, ordered by predicted probabilities. This metric is important because crowd-wisdom recommendations typically provide several alternatives. We performed 5-fold cross-validation and averaged metrics over 5 testing datasets.
Table 5 indicates that process-aware features significantly improve the goodness-of-fit for all prediction settings under consideration. XGBoost, which uses the union of features from the two session definitions, implies the best goodness-of-fit. The conversation-based session definition shows better results than the user-based definition for all prediction techniques. Given the large number of classes overall and the fact that 20 performance reports had very significant variance in their sequences, the accuracy and Top-3 recall numbers are satisfactory. For example, a random class selection would imply approximately 0.03 accuracy.
The crowd-wisdom approach, presented in Section 4, is a useful one but, in many circumstances,
should be complemented by a goal-driven prescription setting. Processes in the IPA domain can
have goals related to process time, cost, quality, and outcomes. One of mainstream approaches
in business process management uses a two-step method [
For the MIP dataset, we address the binary prediction problem of the Mar 28, 2022 deadline violation. In the MIP dataset, 32% of the process instances violated the deadline. We do not explicitly simulate a corrective action assuming that it could be a reminder mail to a user.
We apply the same feature generation methods and prediction techniques as in Section 4 with a single change: the timestamp of the current turn is added to the features in all settings. The timestamp is transformed into the number of working days since the start of the process. We use the standard metrics for binary classification problems to compare prediction methods.
Table 6 summarizes the prediction results. The process-aware approach performs better than the non-process-aware one. In contrast to Section 4, the definition of our user-based session is preferable to the conversation-based method and implies a better balance between precision and recall. Applying a combination of the two approaches does not improve the goodness-of-fit. Although the results for the three prediction techniques are relatively close, CatBoost produces the best predictions.
An analysis of the feature importance values for XGBoost provides insights into the results above. In addition to the user role and timestamp, which clearly afect lateness predictions, the number of fallbacks during the session is also in the Top-3 features based on importance. The possible reason is that struggling users have both a significant number of fallbacks and a high probability of being late for the deadline. It is reasonable to assume that the user-based session definition provides more reliable fallback statistics than a conversation-based one. In addition, observation on the number of fallbacks demonstrates that a language-based conversation feature can be important for long-term process prediction: users with many fallbacks can be identified as risk-prone ones at an early stage of the process and provided with assistance.
We presented the emerging domain of intelligent process automation bots with a chat interface. We highlighted unique aspects of PPM for this new domain, such as the type of prescriptions and data model mapping, which require adaptations of traditional prescriptive approaches to BPM. We further introduced HR MIP - a synthetic dataset that is inspired by real-world IPA deployment. This first-of-a-kind dataset can be used by researchers to develop and test new algorithms in this domain. In addition, we presented an implementation of crowd-wisdom and goal-oriented prescriptive tasks and used it to illustrate the necessary adaptations to data mapping and feature generation steps. We hope that these contributions and the availability of an open dataset will be of value to other researchers in the community.
Prescriptions in intelligent process automation with a chat interface include many additional challenges for future research. The dynamic nature of AI-based digital employees and human-tobot interaction may entail continuous concept drift. In this setting, explore-exploit techniques such as reinforcement learning could be applied to leverage implicit and explicit user feedback. Another challenge is how to deal with more complex utterances, e.g., that are handled by an AI planner to dynamically orchestrate robotic process management tasks. Such use cases may require deep learning, NLP, and program synthesis approaches to map from user utterances to activities and then back to textual recommendations. We plan to address some of these challenges in the next version of the HR IPA system that inspired the MIP use case.
The authors thank Slobodan Radenković, Tomáš Bene, Yara Rizk, Vatche Isahagian, and Vinod Muthusamy for fruitful discussions on methodology and use cases in intelligent process automation.