The workflow management process should be under control of a specific service that is able to forecast the processing time dynamically according to the status of the processing environment and workflow itself, and to react immediately on any abnormal behavior of the execution process. Such situational awareness analytic service would provide the possibility to monitor the execution process, to detect the source of any malfunction, and to optimize the management process. The stated service for the second generation of the ATLAS Production System (ProdSys2, an automated scheduling system) is based on predictive analytics approach. Its primary goal is to estimate the duration of the data processings (in terms of ProdSys2, it is task and chain of tasks) with possibility for later usage in decision making processes. Machine learning ensemble methods are chosen to estimate completion time (i.e., “Time-To-Complete”, TTC) for every (production) task and chain of tasks, and “abnormal” task processing times would warn about possible failure state of the system. This is the primary phase of the service development that also includes the strategy for its precision enhancement. The first implementation of such analytic service is designed around Task TTC Estimator tool and it provides a comprehensive set of options to adjust the analysis process and possibility to extend its functionality.
The key point of any control system is its ability to process a large amount of data and to
extract the most valuable set of parameters and corresponding connections between them that would
describe the behavior of the controlled system and forecast its future state with a certain level of
confidence. The complexity of such control systems encapsulates a comprehensive model it is based
on. A theoretical model of situation awareness contains workflow description and corresponding
procedures regarding all above requirements. This model is viewed as “a state of knowledge” [
The formal definition of situation awareness is presented as “the perception of the elements
in the environment within a volume of time and space, the comprehension of their meaning and the
projection of their status in the near future” [
The original scope of this model of situation awareness was the dynamic human decision
making (in a variety of domains), and its structure is presented at the Figure 1 [
Specifically, this schematic model is considered as a workflow for the control service of the
ATLAS Production System (2nd generation, i.e., ProdSys or ProdSys2) [
The system contains such core components / layers as Ref. [
AMI pyAMI Central, physics group production requests
Distributed Data Management
Rucio
Execution and Definition Interface, which is an intelligent component in the PanDA system [
The possibility of controlling / tracking the state of a particular system requires being aware of its crucial parameters and processes, and their regular behavior. Such parameters and processes should be categorized following the potential failures of the controlled system. List of potential failures: i) overload in the system (that can cause for stuck handling processes); ii) malfunctioning of computing resources (that can cause failure of data processing); iii) components misconfiguration (that can cause improper operational processes); iv) general malicious activities (that can cause incorrect data results).
The following parameters, which are actual ProdSys parameters or derived ones, reflect the specific state of the system and indicates a strong correlation with its failures of a particular category presented above. Controlled parameters are: Duration of task (chain of tasks) execution (forecast and control Time-To-Complete); Rate of task submission over processing during a limited period of time; Cumulative number of failures or/and errors occurred; Metrics of resources utilization (which are compared with optimal values).
These parameters are considered as the initial set that has to be maintained, and should be extended later to improve the operational process.
The goal of the designed control system (service) is being able to track the defined set of parameters of the controlled system (ProdSys), to distinguish their non-valid values according to the running state of the system, to forecast their next values considering the state of the computing environment and system itself, and base on corresponding metrics and controlled parameters to detect the source of malfunctioning. The current project is focused on description of the task TTC estimation as a part of the control service infrastructure (i.e., as a part of a proof-of-concept).
Predictive analytics is a form of conjunction of advanced analytics and decision optimization that uses both new and historical data to forecast future activity, behaviour and trends. Advanced analytics includes in itself: statistical analysis, machine learning, modeling, data visualization, and reporting; while decision optimization is represented by: scoring engine, rules engine, and recommendation engine. Thus, multiple variables are combined into a predictive model that is capable to assess future probabilities with a certain level of confidence. These predictive models (modeling) are used to look for correlations between different data elements (features). Predictive models are presented by two types: i) classification (predict the class membership for the considered object); ii) regression (predict a number for the specific feature of the object). Three of the most widely used predictive modeling techniques are decision trees, regression and neural networks.
These techniques (or a combination of them) is a foundation for the Level 3 of the model of situation awareness. Predictive analytics should be applied to the parameters which are under the control, thus, the forecasting of their next value (state) will trigger the control system to follow the predefined actions.
Apache Spark [
The initial focus in controlling crucial parameters was set on estimation of the duration of task (chain of tasks) execution. The ability of being aware of the task execution process assists in forecasting the near-future state of the controlled system (as a basic representation of the system’s state). Three core stages for estimation of the task execution duration were highlighted, where each of the stage is applied at a certain time of the task lifecycle with available set of parameters (and applied corresponding techniques) and with a certain level of estimation accuracy: i) thresholds definition (based on statistical analysis); ii) initial / static predictions (based on descriptive data of task); iii) dynamic predictions (based on static and dynamic data of the running task).
The general workflow of the designed control system with its main components is presented at the Figure 4. It shows that the management process is concentrated in the “Control node”, while DEfT/JEDI database ShRBD M it w gaen
Sqpoo tcxaaehD data processing is executed in the “Analytics cluster”. (Blocks “Collector” and “Model Handler” are responsible for the task TTC estimation.)
The set of thresholds of the task execution duration is calculated based on the statistical analysis of the historical data for the last several months (6 months was used for the conducted research, see Figure 5). This stage is considered as a rough estimation of the duration of task of a particular type for the possible and valid values. The type of the task is a combination of its features such as <projectType> (e.g., “mc” – Monte-Carlo simulated data, “data” – real data collected from the physical experiment) and <productionStep> (e.g., evgen, simul, recon, merge, etc.).
At Figure 5, there are plots with corresponding thresholds and columns per productionStep that are represented by: i) red vertical lines, which correspond to the values of maximum task execution duration for 95% of tasks that are finished during the defined period of time; ii) blue vertical lines are the same as red ones, but for 80% of finished tasks; iii) green vertical lines represent mean values of the task execution duration among all tasks of the defined type; iv) columns of a particular color represent the number of tasks that were started in a particular month (i.e., number in the legend corresponds to the number of the month) finished within a certain period of time (i.e., task duration). Thereby, the threshold for execution duration of 95% of tasks of a particular type was chosen as a rough estimation (other 5% of tasks were pruned as exceptions, which evaluation should be done separately), and which defines the range of valid values (i.e., “no warning” values).
The process of predictive modeling assumes to solve the problem (forecasting the value of a particular parameter) by creating a model from the sample data (training data based on historical data), thus take data with known relationships, and using learned relationships to make predictions on new data (to discover a certain outcome, e.g., numerical value of task execution duration).
There are several categories of prediction approaches (where sub-categories are named according to temperature states to represent the time point of the task lifecycle when it is applied): Initial / static predictions o “Cold”-prediction is based on task definition parameters that categorize the average execution process for the defined task type (with particular conditions).
It gives the estimation of the task duration during its definition. Dynamic predictions o “Warm”-prediction is based on description and state of scout jobs that are used to check the processing environment. It gives the prediction for the task duration immediately after task is launched. o “Hot”-prediction is based on the current state of the task processing (states of environment and corresponding jobs). It gives the prediction adjustment during the task execution (this prediction is calculated multiple times during the task execution process).
The example of the static prediction based on Random Forest regression method is shown at the Figure 6. The following data was used for the prediction process: i) predictive models are based on “training” data (“mc-all” + “mc-recon” types) for the defined 3 months; ii) “test” data (“mcrecon” type) for the next month (after training data is collected) was used to generate predicted task durations and compare it with real values. This result shows that predictions for tasks (execution duration of 95% of tasks of the defined type is under 27d) are actually following the real values of task durations (the accuracy for the defined example is presented with the following absolute error values: mean=1.57d, std=6.67d, RMSE=6.85d). Since the prediction approach of this category does not consider dynamic parameters and the state of the environment, the accuracy of predictions is low and can be considered as a rough estimation, same as for thresholds (further research will explore generated predictions based on different set of parameters for the training data). Predictions of “dynamic” category will increase the final accuracy (this is under the development).
We thank ProdSys/PanDA team for providing the data for the conducted analysis and for their continued support. This work has been carried out using computing resources of the federal collective usage center Complex for Simulation and Data Processing for Mega-science Facilities at NRC “Kurchatov Institute”, http://ckp.nrcki.ru/. Also, this work was funded in part by the Russian Ministry of Science and Education under contract No. 14.Z50.31.0024.
Techniques and methods of predictive analytics would benefit the control and monitoring processes for the controlled system. The situational awareness analytic service (based on predictive analytics techniques) would provide the possibility to detect the source of any malfunction, and to optimize the whole management process.
The obtained results facilitate filtering the regular processes of tasks execution (with certain level of confidence) and form the basic layer for further sophisticated processes of highlighting abnormal operations and executions. The further improvement of the prediction process will increase the accuracy of the selection process of potentially failure task (chain of tasks), and the extension of this approach to other system’s computing blocks and data will keep the system aware of potential malfunction.