The development lifecycle of Deep Learning (DL) models requires humans (the model trainers) to analyze and steer the training evolution. They analyze intermediate data, fine-tune hyperparameters, and stop when a resulting model is satisfying. The problem is that existing solutions for DL do not track the trainer actions. There are no explicit data relationship between trainer action with the input data and hyperparameters to the output performance results, throughout the training process. This jeopardizes online training data analyses and post-hoc results reproducibility, reusability, and understanding. This paper presents DL-Steer, our first prototype to aid trainers to fine-tune hyperparameters and for tracking trainer steering actions. Tracked data are stored in a relational database for online and post-hoc data analyses.
Training Deep Learning (DL) models is an iterative process that can last for weeks. The
training can be seen as a workflow with chaining activities like loading input data,
combining hyperparameter variations to train several models, tuning hyperparameters for
model fitting, and evaluating resulting models. Based on hyperparameters (e.g., number
of layers and neurons, loss functions, dropout rates, learning rate, number of epochs, etc.)
and model performance, the training workflow iterates until performance is good enough.
Users (DL trainers) need to steer the workflow, for example by incrementally fine-tuning
the model hyperparameters. User steering decisions are based on the intermediate data
analysis and the model performance. Users often work in a time-consuming
trial-anderror manner following the DL training evolution via monitoring tools that plot model
performance results. Basic monitoring is limited for user steering. There should be
interactive tools to put the DL trainers active in the training execution loop
(“Human-inthe-Loop”) [
There are some “Human-in-the-Loop” approaches for DL trainers, but we did not
find any work that tracks user steering actions. In a survey, Kumar et al. define an
architecture for Model Selection Management Systems (MSMS) to iterate on three
activities: feature engineering, algorithm selection, and parameter tuning [
We present a first prototype of DL-Steer, a library that has three main features. First, it collects input values for deep learning model hyperparameters and relates them to the trained model performance output results (e.g., accuracy and others). These input and output values can be monitored and queried by the user during the training. Second, based on this online data analysis, DL-Steer enables users to steer the training by finetuning hyperparameter values. Third, during the training, it captures the user steering actions and correlates them with intermediate data, input hyperparameters, and resulting model metadata and stores them in a relational DBMS, as they are being generated. These three features allow for joint analysis of steering actions, input and output; and facilitates understanding the impacts of specific user-steered model fine-tunes on the results during and after the training. All these data are stored and related in a provenance database that extends W3C PROV concepts. In our case study, DL-Steer supports users to fine-tune the model and evaluate the impact of steering actions during the training. Queries show the impact on the trained models with the tunings of specific DL hyperparameters.
DL-Steer follows the concepts of Computational Steering, which have been under
investigation by multidisciplinary research fields for years [
(i) Instrumentation of scripts. In this first prototype, we aim at users that write
Python scripts with function calls to TensorFlow [
(ii) Data capture and steerable points. Data capture points identify where to
capture the intermediate training results during the workflow execution. DL-Steer
captures the training data and correlates them to input hyperparameter values and models
results, and inserts all of them in a provenance database online during the training. The
provenance database is managed by MonetDB, a relational column-oriented DBMS,
efficient for data analytics. Data steerable points determine where the user can
dynamically fine-tune hyperparameter values that will be used for training. They are
implemented as a queue data structure. The queue interfaces to Redis, a Key/Value
lightweight in-memory store. The queue data managed in Redis are accessible both by
the running script and by external steering actions. Each item in the queue contains a
combination of hyperparameter variations that is iteratively evaluated (while there are
items in the queue to evaluate) as a model fit. During the training, the user can modify
how hyperparameters should vary inside an item. To prevent inconsistencies, DL-Steer
guarantees that only the items that have not been evaluated yet can be modified by the
user. When the user decides to modify the queue, the steering action takes place only in
the next iteration. Provenance data of the steering action are captured, related to current
training data being generated, and stored in the provenance database in MonetDB. The
DL-Steer data schema follows W3C PROV provenance data model specializations, such
as our previous works [
(iii) Data steering during training. During the DL training workflow execution, the user can either run ad-hoc analytical queries on MonetDB directly (via its SQL query interface) or use a graphical interface that queries it at specified time intervals and plots evolution of training results. DL-Steer has a graphical interface and a command line interface for steering actions like modifying queue items. Hyperparameters in each item are represented as keys and values, where the keys are hyperparameters to be evaluated in the training and the values represent how they vary. In case of numeric lists of variations, the user specifies the initial and final interval values and the incremental step. For example, the key/value structure {"learning_rate": [0.1, 1.0, 0.1], "epochs": [10, 50, 10]}, sets the values of the hyperparameters learning rate and number of epochs.
In this preliminary DL-Steer case study, the user identifies data capture and steerable points to instrument a Python script that uses TensorFlow for parallel Deep Learning training of an image classifier in GPU. The data capture points are given by a queue of hyperparameter variations with initial values defined by the user and by the trained model performance results. The performance function is defined by the user as a "Gain" function. The steerable point is the queue of hyperparameter variations. At each iteration, an item in this queue is processed in a model fitting and DL-Steer captures the input values used and model output results generated. Moreover, at each iteration, DL-Steer’s dashboard queries the provenance database to plot performance results, as shown in Figure 1. In this case study, at iteration 20, the user observes that the Gain function moving average (i.e., the average of the last k iterations) is 0.28 and the hyperparameter learning rate is varying in the interval [0.01, 0.05] with step 0.01. Then, at iteration 21, the user does the first steering action to modify the learning rate interval to [0.03, 0.07], with the same step 0.01. This action impacts the gain, as observed in the moving average at 10 iterations later: 0.48 (an improvement of 0.2). At the iteration 54, the moving average is 0.51 and the hyperparameter batch size is 100. Then, the user decides for another action at iteration 55 to modify the batch size to 1000. This also impacts the gain, as the moving average 10 iterations later increased to 0.55 (improvement of 0.04). Finally, after 75 iterations, the user observes that the iteration 67 generated a model with a satisfying gain function value of 0.84. The user then decides to stop the training. Input hyperparameters and performance model results of all models, including the best model, and the related user steering data are all stored in the DL-Steer’s provenance database.
We presented DL-Steer, our first prototype of a library for tracking user-steered hyperparameter tuning during the DL training workflow. It keeps track of steering actions in the models with an explicit data relationship with their results, intermediate data, and input hyperparameters, throughout the training process, all related in a provenance database. This not only contributes for experiment reproducibility, but also helps the user to understand the impact of steering actions in the results. For future work, we plan to explore our solution to analyze variations of different network architectures, loss functions, dropout rates, and other hyperparameters in a large-scale dataset.