=Paper=
{{Paper
|id=Vol-3226/paper10
|storemode=property
|title=Confr – A Configuration System for Machine Learning Projects
|pdfUrl=https://ceur-ws.org/Vol-3226/paper10.pdf
|volume=Vol-3226
|authors=Mattias Arro
|dblpUrl=https://dblp.org/rec/conf/itat/Arro22
}}
==Confr – A Configuration System for Machine Learning Projects
==
https://ceur-ws.org/Vol-3226/paper10.pdf
confr – A Configuration System for Machine Learning
Projects
Mattias Arro1
1
www.uxo.ai
Abstract
Finding a performant machine learning model usually requires exploring different combinations of model hyperparameters,
preprocessing steps, data generation and train logic. To facilitate a clear analysis of the factors that determine accuracy,
it is useful to make the data processing and train pipeline highly configurable such that a combination of a code version
and configuration file uniquely determines the behaviour of the system. A poor configuration system can lead to repetitive
code that is hard to maintain, understand, and brittle due to insufficient configuration validation logic. This paper outlines
the design and usage of confr, a concise and flexible configuration system geared towards Python-based machine learning
projects. It combines some of the capabilities of commonly used systems (such as gin-config, OmegaConf, and Hydra) into a
library which aims to reduce repetitive code and maintenance overhead. It can be used both as part of a notebook-based
and script-based workloads, and can be used for ensuring that there is no accidental difference between inference-time and
train-time behaviour.
Keywords
machine learning, configuration, experiment management, reproducibility
1. Introduction many train runs (on the same code and data version) cre-
ates a dataset of hyperparameter sets and corresponding
The goal of machine learning (ML) practitioners is to find evaluation metrics. The relationship between hyperpa-
a ”good model”. This is broadly determined by three fac- rameters and metrics can be analysed for insights that
tors: (training and validation) data, code (preprocessing, inform future hyperparameter choices or code changes.
model implementation, train loop), and hyperparame- A highly configurable train pipeline also lends itself for
ters (configuration). By hyperparameters we mean any automatic model tuning approaches such as brute-force
non-learnable parameter/configuration that influences grid/random search, or methods like Bayesian optimi-
how the code gets executed, which may be in the data sation that use ML to find values for hyperparameters
processing, model intialisation or inference, or train loop. which maximise validation accuracy.
When running experiments, exact and immutable ver- There are many ways to make a system configurable,
sions of the three should always be stored, so that we such as creating an ad hoc solution from scratch or using
can (1) analyse the factors that influence accuracy and a 3rd party config1 system. The following is a list of
(2) reproduce the results of an earlier experiment. qualities we would expect from a config system, which
Early in developing a ML system, code tends to hard- we will later use to evaluate our proposed system confr
code most choices for data processing, model implemen- against alternatives.
tation and hyperparameters. Experimentation in this
setting would require changes to the code, which means 1. Minimise boilerplate code. There should not
that to compare two experiments one needs to find the be much repetitive code to have a highly config-
differences of code used in each train run, or rely on the urable system.
experimenter’s description of the hypothesis that was 2. Minimise repetitive config. There should be
tested. Given that most ML experiments are done in ways to reuse, rather than repeat, individual con-
notebooks where comparing code with version control is fig values.
difficult, comparing large numbers of such experiments 3. Composability of config objects. It should be
is not feasible. possible to reuse and compose different configu-
A better approach is to make the code highly config- rations, which is crucial for large systems.
urable, so that alternative behaviours can be achieved by 4. Low maintenance overhead. A config system
using different hyperparameter values. Now triggering should reduce (rather than add to) the difficulty
of refactoring and developing the code base.
ITAT’22: Information technologies – Applications and Theory, Septem- 5. Low cognitive load. A config system should
ber 23–27, 2022, Zuberec, Slovakia make it easy to understand which variables are
Envelope-Open mattias.arro@gmail.com (M. Arro)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License 1
Attribution 4.0 International (CC BY 4.0). We use the words ”config” and ”configuration” interchangeably, as
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org) is common in industry.
configurable and where that configuration comes 2. Related Work
from. It is highly related (but not fully deter-
mined) by the following three qualities. In this section we provide a brief overview of existing
6. Clearly identifyable configurable arguments. configuration systems. We first describe what typical
It should be clear just by looking at a function if its ad hoc solutions look like, and then look at three config
argument is a configurable hyperparameter. This systems which are commonly used4 for ML workloads:
improves readability and decreases the chance of OmegaConf [2], gin-config [1], and Hydra [3]. gin-config
accidentally forgetting to provide a config value. is explicitly designed for ML work, while OmegaConf
7. Consistent mapping between config keys and Hydra are generic config systems.
and variables / arguments. The system should
encourage a one-to-one mapping between keys2 2.1. Ad Hoc Systems
in the config file and configurable arguments.
Machine learning projects that do not use a specialised
Such mapping alleviates the cognitive load in
config system tend to consist of Python scripts where hy-
translating differences between config keys and
perparameters are defined as command line arguments.
variable names, and makes it easy to search for
Libraries like argparse5 and click6 can be used for easier
all places where a config key is used.
parsing of command line arguments or providing argu-
8. Global config values. In most cases we expect
ments via environment variables. The individual config
the value of a config key to be the same across the
values get passed to downstream functions where neces-
code base, and a config system should encourage
sary.
this. For example, if a preprocessing function
Alternatively, a regular Python dictionary containing
assumes max_img_dimensions = (128, 128) ,
the configuration might be read from a YAML/JSON file,
then so should the function which builds the neu-
and passed along to functions that depend on it. Either
ral network whose input tensor would have the
the full configuration object or individual config values
shape (128, 128, 3) - otherwise the model’s in-
might be passed along, depending on programming style.
put dimensions would be incompatible with the
It is generally not easy to unify such CLI-based and file-
images created by the preprocessor.
based configuration styles without using a specialised
9. Centralised, multi-key validation of config. config system.
Validation of config values should be centralised
in a single place, rather than scattered around the
project in an ad hoc manner. This (1) ensures we 2.2. OmegaConf
fail quickly with a helpful message when read- OmegaConf builds on YAML file format, and adds a pow-
ing a faulty config (rather than waiting for the erful interpolation mechanism, which enables accessing
relevant code path to be reached, which may hap- config keys from other parts of the file. The below exam-
pen at a much later state), and (2) allows defining ple shows both absolute (${server.host}$ ) and relative
validation logic that sets constraints on several (${.url}$ ) references to other parts of the config file.
config keys simultaneously.
10. Usable in a notebook as well as CLI. The
system should be easy to use in an exploratory
notebook-based setting, where one might want
to dynamically (re-)define and access config val-
ues, and to ultimately write the current active
configuration to a file. It should also work in a
script-oriented CLI3 setting, where main config-
uration is loaded from a file, and certain config OmegaConf configuration is represented as a Python
values or whole sections can be overridden via object, which can be accessed as a nested object or dic-
command line arguments. tionary:
11. Configurable Python references and single-
tons. It should be possible to refer to Python ob-
jects (functions, classes, constants, objects) and
create global singletons from callable references
(functions, classes). 4
By ”commonly used” we mean systems we were able to find by
doing relevant Google searches, reading relevant discussion threads,
2
Configuration files are (possibly nested) key-value pairs. We call and looking at GitHub activity for these projects.
5
the names of configurable hyperparameters as ”config keys”. https://docs.python.org/3/library/argparse.html
3 6
CLI - Command Line Interface https://click.palletsprojects.com/
object as an argument, which can be passed to down-
stream functions. Therefore the config files support all
the syntax and functionality of OmegaConf, and some
features that Hydra adds. For detailed examples, refer to
Hydra documention7 .
2.3. gin-config
gin-config makes use of @gin.configurable decora-
tors around function and class definitions. For func-
tions (and initialisers of classes) decorated like this, ar-
guments are substituted from the global config which is
initialised with gin.parse_config_file("conf.gin") .
Attributes with default value of gin.CONFIGURED need
to have a config specified in the .gin config file; other
arguments can (but do not have to be) configured with
gin.
3. Overview of confr
In this section, we show the basics of how to use
confr. For complete and up-to-date documentation see
https://github.com/mattiasarro/confr.
3.1. Basic Usage
In confr, configs are initialised similarly to gin-config:
functions and classes can be decorated with confr.bind ,
which ensures that function and class initialiser argu-
The config file is a custom text format, which is a sim- ments will be substituted from the currently active config.
plified subset of Python. Using dnn.num_outputs = 10 The following example shows a function that expects at
would ensure all functions named dnn will have least the num_outputs key (and optionally layer_sizes )
the value of num_outputs substituted as 10 . Us- to be defined in conf/base.yaml. Before calling any confr-
ing path.to.mymodule.dnn.num_outputs = 10 would configured functions, confr.init must be called, which
ensure it happens only to the function in the creates an implicit global config object.
path.to.mymodule module. As special syntax, values
that start with ”@” refer to other gin-configurable func-
tions or classes; values that start with ”@” and end with
”()” first get called before being passed as arguments.
2.4. Hydra
Hydra is a feature-rich config system. The entrypoint
function of the program using Hydra should be anno-
tated with a @hydra.main decorator, which defines the
directory where config files are stored, and config name
(filename without the .yaml extension in the directory).
When called, the function receives an OmegaConf cfg 7
https://hydra.cc/docs/intro/
All config files use the special form of YAML used for this is to make a preprocessing or augmentation func-
by OmegaConf (with a few special cases described in tion as a configurable argument, so that you can try out
the next subsections). In the default case, the argu- different preprocessors without changing the code. For
ments get bound to top-level config keys of the same example:
name in the YAML file. So in the above example, our
conf/base.yaml could look like this:
3.2. Custom Config Key to Argument
Mapping
In some cases, a one-to-one mapping of config keys to
function argument names can be limiting, so we offer two
ways to customise this. Assume we have the following
config:
When predict is called, the Python module
my.module is imported and the resize_and_crop at-
tribute is read from it. Any Python object could be refer-
enced in config files - function, class, variable, constant
- as long as its module is available on PYTHONPATH,
which usually includes all modules in the project root as
When wrapping a function with confr.bind , we could well as installed libraries such as tensorflow.
tell it to only map keys under the neural_net config key: Config values which start with a ”@” and end with
”() ” are singletons - Python references which get called
before becoming part of the current active config and
being passed as keyword arguments. For example you
might define an encoder: @my.module.my_encoder()
key-value pair in the config. Now if a function defines an
argument as encoder=confr.value , then @my.module
gets imported and its my_encoder() callable gets called
The other option is to pass the full path in the config before being passed as the argument value. Once
to confr.value : my_encoder() is called, its return value gets mem-
oized and any subsequent functions which use the
encoder config will receive the same, pre-initialised
object. Singletons can be referenced in other parts
of the config using the familiar OmegaConf format of
${config_key.subkey.singleton}$ .
Note that my_model1 and my_model2 in the following
listing are the same object.
3.3. Python References and Singletons
confr adds special syntax to the YAML format supported
by OmgaConf, which can be used to load Python object
or initialise global singletons. Config values which start
with a ”@” are ”Python references”8 . A common use
8
References to Python objects and references to other config values in
the file look quite similar, since they both use dot notation. Python
references start with a ”@”, and they refer Python modules like
in absolute imports (which usually correspond to folder structure).
Referencing other parts of a config file start with a ”$” and refer
to the ”path” in the YAML file, which follows the nesting of config
keys.
3.4. Scoped Arguments in Singletons list of p_thresh values and calculate accuracy for each
p_thresh .
If you would like to configure input arguments specifi-
Our first attempt at solving this would look like this:
cally for singletons, you can do the following:
Now my_model1 singleton will be initialized
with location="/path/to/weights.h5" and
my_model2 singleton will be initialized with
location="/path/to/weights2.h5" . This way
they can both define an input argument called location
and still receive a unique value at initialization time. This would work if precision is the only place that
We call my_model1/location as a scoped argument, i.e. uses the p_thresh that is passed in. But if precision
the value of location is present in only the my_model1 calls sub_function whose p_thresh value comes from
singleton scope. confr, then the value of p_thresh in sub_function will
Note that you can still use the regular, non- be the same as in the config file and not the one we passed
scoped arguments along with scoped ones. For ex- to precision . What we need here is to temporarily set
ample, both my_model1 and my_model2 might define the value of p_thresh config key in the whole confr, like
img_h=confr.value , and this value will be the same
this:
when initializing both singletons.
3.5. Call-time Overrides
When running a Python program configured with confr,
individual values can be overriden in two ways:
1. Passing command-line arguments such as
--key1.subkey1=value .
3.7. Config Spanning Multiple Files
2. Setting environment variables such as
confr__key1__subkey1=value . Suppose you have the following config files:
3.6. Run-time overrides
When working in a notebook, modifying the YAML file
to change the active config is cumbersome. You could
instead initialize the config at the start of the notebook
by selectively providing overrides to the keys you care
about like this:
If you do not want to re-initialise the whole con- Once the config is loaded, the effective final config
fig, but would like to set individual config values, use would look like this, because the _file special key tells
confr.set("my_key", value) . Doing this would not confr to take the configuration for neural_net subkeys
re-initialise other config keys or singletons that may de- from another file.
pend on my_key . But you could also override which neural net config
You may also want to provide overrides to config gets used, by passing --neural_net._file=deep when
values temporarily, for the duration of calling a func- running the program. Note that there is a convention be-
tion (and any downstream functions called by this func- tween the config keys and the folders from where _file
tion). For example, you might want to iterate over a references are searched.
3.9. Validation
Two types of validations can be done with confr. Each
config file with name filename.yaml can have an op-
tional filename_types.yaml counterpart, which de-
fines the datatype of all (or a subset of) the config keys.
Currently, only primitive Python types are supported,
3.8. Accessing the Active Config but more complex solutions will be added.
For example:
Sometimes we need to explicitly fetch the value of a
key in our config system. You can use confr.get and
confr.set accessors to modify the current active conf:
However, once the config gets complex enough, there
is a need to validate different combinations of config
values. For example, imagine we have the following
config, which states that 50% of the samples come from
You can also save the current active config as a YAML
labelled dataset, 25% come from data generator 1 and 25%
file, for example at the end of training. The code for
come from generator 2:
training the model and doing inference should be in the
same version control project; train-time and inference-
time pre-processing should be handled by the same func-
tion(s). This way, if code for inference is initialised from
the same code revision and active config that was used
during training, there would be no accidental difference
between train time and test time behaviour.
In confr we can define a validator that ensures that
everything in samples_per_batch sums to batch_size .
4. Evaluation 5. Low cognitive load.
• BAD: gin-config, Hydra. It is not clear,
We will now evaluate the competing systems and confr on without reading the configuration file and
the desired qualities outlined in the introduction, giving upstream code carefully, which arguments
each a somewhat subjective BAD / OK / GOOD mark. are configured and where the value comes
1. Minimise boilerplate code. from.
• BAD: ad hoc, OmegaConf. Passing down • OK: ad hoc, OmegaConf, confr. The initial
configuration dictionaries or individual cognitive load of understanding how a con-
config values can be very verbose. So can figurarion is loaded in ad hoc systems and
be setting up CLI arguments in ad hoc sys- OmegaConf is low (for example, it’s easy
tems. to understand what reading a YAML file or
• OK: Hydra. Different functions can request using argparse does). However it is much
a config object, and read individual keys harder to reason about how the whole sys-
from it. However generally there is a single tem behaves due to the next three qualities.
config object that gets passed along. In confr, it takes more effort to think about
many possible places where configurations
• GOOD: gin, confr. Configurable function
can come from (multiple files, command
arguments receive a value directly from
line overrides), but it is easier to reason
the config system, which is most concise.
about the whole system due to the follow-
2. Minimise repetitive config. ing three qualities.
• BAD: ad hoc. No way to reuse / refer to 6. Clearly identifyable configurable argu-
other values in YAML/JSON files. ments.
• OK: gin-config. It is possible to reuse val- • BAD: ad hoc, OmegaConf, Hydra. When
ues, but in a cumbersome way, and the con- reading code (that is ”far away” from the
fig file format is somewhat verbose (since part that initialises config), it is not clear
all occurences of a config value need to be which arguments are configurable.
listed). • OK: gin-config. It is possible to make it ex-
• GOOD: OmegaConf, Hydra, confr. It is plicit that some arguments should receive
possible to reuse values and create concise a value from configuration, but this is not
config files. a requirement.
3. Composability of config objects. • GOOD: confr. It is intentionally not pos-
• BAD: ad hoc, OmegaConf, gin-config. Not sible to configure an argument in confr
supported. without making it explicit in code that the
• GOOD: Hydra, confr. Supported. value is configurable.
4. Low maintenance overhead. 7. Consistent mapping between config keys
• BAD: ad hoc, gin-config. Passing along and variables / arguments.
config objects/values slows down refactor- • BAD: ad hoc, OmegaConf, gin-config, Hy-
ing. gin-config also often requires config dra. In all these systems such consistency
changes when the relevant code changes is not encouraged, which makes it harder
(renaming/moving functions). Ad hoc sys- to read, understand and refactor.
tems might need to maintain CLI argument • GOOD: confr. Such consistency is enforced
lists. by default, though in rare cases it is possi-
• OK: OmegaConf, Hydra. Moving code ble to bypass this (for example when you
around requires changes to passing of con- need to use an externally-provided config
fig values, but this is less troublesome than file).
changes required by gin-config or main- 8. Global config values.
taining CLI argument lists in ad hoc sys- • BAD: ad hoc, OmegaConf, Hydra. It is easy
tems. to have multiple configuration objects, or
• GOOD: confr. Config files and other parts to have a different value in different places
of the source generally do not need to be for the same config key.
changed on renaming or moving functions, • GOOD: gin-config, confr. There can be
because the config file makes no assump- only one globally active configuration. In
tions about where the config is used and confr, each config key always has the same
config values do not need to be propagated. value.
9. Centralised, multi-key validation of config. There are features not supported by confr which other
• BAD: gin-config. No validation system pro- libraries provide that will be added in later versions: tab
vided, hard to add one. completion and more detailed type checking provided by
Hydra. Hydra also has some features that were intention-
• OK: ad hoc, OmegaConf, Hydra. Cen-
ally not made part of confr, such as multi-run options,
tralised validation system can be added
custom working dir and logger config, since these were
through user code to OmegaConf or file-
not considered relevant with respect to the evalation cri-
based ad hoc systems, though this is harder
teria, which we considered most useful for a ML-oriented
to do in CLI-based ad hoc systems in a con-
config system.
cise way. Hydra has a validation system,
but this is mostly limited to data type based
checks, which are insufficient. References
• GOOD: confr. Supported, built in.
10. Usable in a notebook as well as CLI. [1] Holtmann-Rice, D., Guadarrama, S., Silberman, N.:
Gin Config. https://github.com/google/gin-config
• BAD: ad hoc, OmegaConf, gin-config. Ad (2018)
hoc systems tend to work well either for [2] OmegaConf. https://github.com/omry/omegaconf
notebook-based approaches (e.g. reading (2012)
a global YAML file as a config dict), or [3] Yadan, E.: Hydra - A framework for elegantly con-
work as a sophisticated CLI script, but not figuring complex applications, https://github.com/-
both. There is no support to override op- facebookresearch/hydra (2019)
tions from the CLI in OmegaConf and gin-
config.
• GOOD: Hydra, confr. Can be used in both
settings.
11. Configurable Python references and single-
tons.
• BAD: ad hoc. Usually not supported.
• OK: OmegaConf, Hydra. Supported, but
cumbersome to use.
• GOOD: gin-config, confr. Supported, easy
to use.
The results are summarised in the following table.
From it we can see that confr exhibits the good quali-
ties of Hydra and gin-config, while alleviating some of
the downsides one or the other. This is not a coincidence,
since confr was in many ways inspired by these two,
though the exact way confr achieves these qualities may
be different.