Finding a performant machine learning model usually requires exploring diferent combinations of model hyperparameters, preprocessing steps, data generation and train logic. To facilitate a clear analysis of the factors that determine accuracy, it is useful to make the data processing and train pipeline highly configurable such that a combination of a code version and configuration file uniquely determines the behaviour of the system. A poor configuration system can lead to repetitive code that is hard to maintain, understand, and brittle due to insuficient configuration validation logic. This paper outlines the design and usage of confr, a concise and flexible configuration system geared towards Python-based machine learning projects. It combines some of the capabilities of commonly used systems (such as gin-config, OmegaConf, and Hydra) into a library which aims to reduce repetitive code and maintenance overhead. It can be used both as part of a notebook-based and script-based workloads, and can be used for ensuring that there is no accidental diference between inference-time and train-time behaviour.
The goal of machine learning (ML) practitioners is to find
a ”good model”. This is broadly determined by three
facmodel implementation, train loop), and
hyperparameters (configuration). By hyperparameters we mean any
non-learnable parameter/configuration that influences
how the code gets executed, which may be in the data
processing, model intialisation or inference, or train loop.
When running experiments, exact and immutable
versions of the three should always be stored, so that we
can (
Early in developing a ML system, code tends to hardtation and hyperparameters. Experimentation in this setting would require changes to the code, which means that to compare two experiments one needs to find the diferences of code used in each train run, or rely on the experimenter’s description of the hypothesis that was tested. Given that most ML experiments are done in notebooks where comparing code with version control is dificult, comparing large numbers of such experiments is not feasible.
A better approach is to make the code highly configurable, so that alternative behaviours can be achieved by using diferent hyperparameter values. Now triggering many train runs (on the same code and data version) creates a dataset of hyperparameter sets and corresponding evaluation metrics. The relationship between hyperparameters and metrics can be analysed for insights that A highly configurable train pipeline also lends itself for automatic model tuning approaches such as brute-force grid/random search, or methods like Bayesian optimisation that use ML to find values for hyperparameters which maximise validation accuracy.
There are many ways to make a system configurable, such as creating an ad hoc solution from scratch or using a 3rd party config 1 system. The following is a list of qualities we would expect from a config system, which we will later use to evaluate our proposed system confr urable system.
ifg values. 1. Minimise boilerplate code. There should not be much repetitive code to have a highly config
There should be ways to reuse, rather than repeat, individual con3. Composability of config objects. It should be possible to reuse and compose diferent configurations, which is crucial for large systems.
should reduce (rather than add to) the dificulty of refactoring and developing the code base.
make it easy to understand which variables are code most choices for data processing, model implemen- against alternatives. configurable and where that configuration comes from. It is highly related (but not fully determined) by the following three qualities. In this section we provide a brief overview of existing 6. Clearly identifyable configurable arguments. configuration systems. We first describe what typical It should be clear just by looking at a function if its ad hoc solutions look like, and then look at three config argument is a configurable hyperparameter. This systems which are commonly used4 for ML workloads: improves readability and decreases the chance of OmegaConf [2], gin-config [ 1], and Hydra [3]. gin-config accidentally forgetting to provide a config value. is explicitly designed for ML work, while OmegaConf 7. Consistent mapping between config keys and Hydra are generic config systems. and variables / arguments. The system should encourage a one-to-one mapping between keys2 2.1. Ad Hoc Systems in the config file and configurable arguments.
Such mapping alleviates the cognitive load in Machine learning projects that do not use a specialised
translating diferences between config keys and config system tend to consist of Python scripts where
hyvalalrpialabcleesnwamheerse, aancdonmfigakkeeys iist euasseyd.to search for Lpiebrrpaarriaesmlei kteerasragrpeadrseefin5eadnads ccloicmk6mcaanndbleinuesaerdgfuomreenastise.r
parsing of command line arguments or providing
argu8. Global config values. In most cases we expect ments via environment variables. The individual config
the value of a config key to be the same across the values get passed to downstream functions where
necescode base, and a config system should encourage sary.
this. For example, if a preprocessing function Alternatively, a regular Python dictionary containing
assumes max_img_dimensions = (128, 128), the configuration might be read from a YAML/JSON file,
then so should the function which builds the neu- and passed along to functions that depend on it. Either
ral network whose input tensor would have the the full configuration object or individual config values
shape (
Validation of config values should be centralised
in a single place, rather than scattered around the
project in an ad hoc manner. This (
Attributes with default value of gin.CONFIGURED need to have a config specified in the .gin config file; other arguments can (but do not have to be) configured with gin. object as an argument, which can be passed to downstream functions. Therefore the config files support all the syntax and functionality of OmegaConf, and some features that Hydra adds. For detailed examples, refer to Hydra documention7.
In this section, we show the basics of how to use confr. For complete and up-to-date documentation see https://github.com/mattiasarro/confr. 3.1. Basic Usage In confr, configs are initialised similarly to gin-config: functions and classes can be decorated with confr.bind, which ensures that function and class initialiser argu
The config file is a custom text format, which is a sim- ments will be substituted from the currently active config. plified subset of Python. Using dnn.num_outputs = 10 The following example shows a function that expects at would ensure all functions named dnn will have least the num_outputs key (and optionally layer_sizes) the value of num_outputs substituted as 10. Us- to be defined in conf/base.yaml. Before calling any confring path.to.mymodule.dnn.num_outputs = 10 would configured functions, confr.init must be called, which ensure it happens only to the function in the creates an implicit global config object. path.to.mymodule module. As special syntax, values that start with ”@” refer to other gin-configurable functions or classes; values that start with ”@” and end with ”()” first get called before being passed as arguments. 2.4. Hydra Hydra is a feature-rich config system. The entrypoint function of the program using Hydra should be annotated with a @hydra.main decorator, which defines the directory where config files are stored, and config name (filename without the .yaml extension in the directory).
When called, the function receives an OmegaConf cfg 7https://hydra.cc/docs/intro/
All config files use the special form of YAML used for this is to make a preprocessing or augmentation funcby OmegaConf (with a few special cases described in tion as a configurable argument, so that you can try out the next subsections). In the default case, the argu- diferent preprocessors without changing the code. For ments get bound to top-level config keys of the same example: name in the YAML file. So in the above example, our conf/base.yaml could look like this: 3.2. Custom Config Key to Argument
Mapping In some cases, a one-to-one mapping of config keys to function argument names can be limiting, so we ofer two ways to customise this. Assume we have the following config:
When wrapping a function with confr.bind, we could tell it to only map keys under the neural_net config key:
The other option is to pass the full path in the config to confr.value: 3.3. Python References and Singletons confr adds special syntax to the YAML format supported by OmgaConf, which can be used to load Python object or initialise global singletons. Config values which start with a ”@” are ”Python references”8. A common use 8References to Python objects and references to other config values in the file look quite similar, since they both use dot notation. Python references start with a ”@”, and they refer Python modules like in absolute imports (which usually correspond to folder structure).
Referencing other parts of a config file start with a ”$” and refer to the ”path” in the YAML file, which follows the nesting of config keys.
When predict is called, the Python module my.module is imported and the resize_and_crop attribute is read from it. Any Python object could be referenced in config files - function, class, variable, constant - as long as its module is available on PYTHONPATH, which usually includes all modules in the project root as well as installed libraries such as tensorflow.
Config values which start with a ”@” and end with ”()” are singletons - Python references which get called before becoming part of the current active config and being passed as keyword arguments. For example you might define an encoder: @my.module.my_encoder() key-value pair in the config. Now if a function defines an argument as encoder=confr.value, then @my.module gets imported and its my_encoder() callable gets called before being passed as the argument value. Once my_encoder() is called, its return value gets memoized and any subsequent functions which use the encoder config will receive the same, pre-initialised object. Singletons can be referenced in other parts of the config using the familiar OmegaConf format of ${config_key.subkey.singleton}$.
Note that my_model1 and my_model2 in the following listing are the same object. 3.4. Scoped Arguments in Singletons If you would like to configure input arguments specifically for singletons, you can do the following: list of p_thresh values and calculate accuracy for each p_thresh.
Our first attempt at solving this would look like this:
Now my_model1 singleton will be initialized with location="/path/to/weights.h5" and my_model2 singleton will be initialized with location="/path/to/weights2.h5". This way they can both define an input argument called location and still receive a unique value at initialization time. This would work if precision is the only place that We call my_model1/location as a scoped argument, i.e. uses the p_thresh that is passed in. But if precision the value of location is present in only the my_model1 calls sub_function whose p_thresh value comes from singleton scope. confr, then the value of p_thresh in sub_function will
Note that you can still use the regular, non- be the same as in the config file and not the one we passed scoped arguments along with scoped ones. For ex- to precision. What we need here is to temporarily set ample, both my_model1 and my_model2 might define the value of p_thresh config key in the whole confr, like img_h=confr.value, and this value will be the same this: when initializing both singletons. 3.5. Call-time Overrides When running a Python program configured with confr, individual values can be overriden in two ways: 1. Passing command-line arguments such as
--key1.subkey1=value. 2. Setting environment variables such as
confr__key1__subkey1=value. 3.6. Run-time overrides When working in a notebook, modifying the YAML file to change the active config is cumbersome. You could instead initialize the config at the start of the notebook by selectively providing overrides to the keys you care about like this: 3.7. Config Spanning Multiple Files
If you do not want to re-initialise the whole con- Once the config is loaded, the efective final config ifg, but would like to set individual config values, use would look like this, because the _file special key tells confr.set("my_key", value). Doing this would not confr to take the configuration for neural_net subkeys re-initialise other config keys or singletons that may de- from another file. pend on my_key. But you could also override which neural net config
You may also want to provide overrides to config gets used, by passing --neural_net._file=deep when values temporarily, for the duration of calling a func- running the program. Note that there is a convention betion (and any downstream functions called by this func- tween the config keys and the folders from where _file tion). For example, you might want to iterate over a references are searched. 3.8. Accessing the Active Config Sometimes we need to explicitly fetch the value of a key in our config system. You can use confr.get and confr.set accessors to modify the current active conf:
You can also save the current active config as a YAML ifle, for example at the end of training. The code for training the model and doing inference should be in the same version control project; train-time and inferencetime pre-processing should be handled by the same function(s). This way, if code for inference is initialised from the same code revision and active config that was used during training, there would be no accidental diference between train time and test time behaviour. 3.9. Validation Two types of validations can be done with confr. Each config file with name filename.yaml can have an optional filename_types.yaml counterpart, which deifnes the datatype of all (or a subset of) the config keys.
Currently, only primitive Python types are supported, but more complex solutions will be added.
For example:
However, once the config gets complex enough, there is a need to validate diferent combinations of config values. For example, imagine we have the following config, which states that 50% of the samples come from labelled dataset, 25% come from data generator 1 and 25% come from generator 2:
In confr we can define a validator that ensures that everything in samples_per_batch sums to batch_size.
We will now evaluate the competing systems and confr on the desired qualities outlined in the introduction, giving each a somewhat subjective BAD / OK / GOOD mark. 1. Minimise boilerplate code.
• BAD: ad hoc, OmegaConf. Passing down configuration dictionaries or individual config values can be very verbose. So can be setting up CLI arguments in ad hoc systems. • OK: Hydra. Diferent functions can request a config object, and read individual keys from it. However generally there is a single config object that gets passed along. • GOOD: gin, confr. Configurable function arguments receive a value directly from the config system, which is most concise. 2. Minimise repetitive config.
• BAD: ad hoc. No way to reuse / refer to
other values in YAML/JSON files. • OK: gin-config. It is possible to reuse values, but in a cumbersome way, and the conifg file format is somewhat verbose (since all occurences of a config value need to be listed). • GOOD: OmegaConf, Hydra, confr. It is possible to reuse values and create concise config files. 3. Composability of config objects.
• BAD: ad hoc, OmegaConf, gin-config. Not
supported.
• GOOD: Hydra, confr. Supported. 4. Low maintenance overhead.
• BAD: ad hoc, gin-config. Passing along config objects/values slows down refactoring. gin-config also often requires config changes when the relevant code changes (renaming/moving functions). Ad hoc systems might need to maintain CLI argument lists. • OK: OmegaConf, Hydra. Moving code around requires changes to passing of conifg values, but this is less troublesome than changes required by gin-config or maintaining CLI argument lists in ad hoc systems. • GOOD: confr. Config files and other parts of the source generally do not need to be changed on renaming or moving functions, because the config file makes no assumptions about where the config is used and config values do not need to be propagated. 5. Low cognitive load.
• BAD: gin-config, Hydra. It is not clear, without reading the configuration file and upstream code carefully, which arguments are configured and where the value comes from. • OK: ad hoc, OmegaConf, confr. The initial cognitive load of understanding how a conifgurarion is loaded in ad hoc systems and OmegaConf is low (for example, it’s easy to understand what reading a YAML file or using argparse does). However it is much harder to reason about how the whole system behaves due to the next three qualities.
In confr, it takes more efort to think about many possible places where configurations can come from (multiple files, command line overrides), but it is easier to reason about the whole system due to the following three qualities. 6. Clearly identifyable configurable arguments.
• BAD: ad hoc, OmegaConf, Hydra. When reading code (that is ”far away” from the part that initialises config), it is not clear which arguments are configurable. • OK: gin-config. It is possible to make it explicit that some arguments should receive a value from configuration, but this is not a requirement. • GOOD: confr. It is intentionally not possible to configure an argument in confr without making it explicit in code that the value is configurable. 7. Consistent mapping between config keys and variables / arguments.
• BAD: ad hoc, OmegaConf, gin-config, Hydra. In all these systems such consistency is not encouraged, which makes it harder to read, understand and refactor. • GOOD: confr. Such consistency is enforced by default, though in rare cases it is possible to bypass this (for example when you need to use an externally-provided config ifle). 8. Global config values.
• BAD: ad hoc, OmegaConf, Hydra. It is easy to have multiple configuration objects, or to have a diferent value in diferent places for the same config key. • GOOD: gin-config, confr. There can be only one globally active configuration. In confr, each config key always has the same value.
The results are summarised in the following table. From it we can see that confr exhibits the good qualities of Hydra and gin-config, while alleviating some of the downsides one or the other. This is not a coincidence, since confr was in many ways inspired by these two, though the exact way confr achieves these qualities may be diferent.
There are features not supported by confr which other libraries provide that will be added in later versions: tab completion and more detailed type checking provided by Hydra. Hydra also has some features that were intentionally not made part of confr, such as multi-run options, custom working dir and logger config, since these were not considered relevant with respect to the evalation criteria, which we considered most useful for a ML-oriented config system.