Data management in scientific domains is more important than ever due to the increasing availability of experimental data. Automatically integrating and managing the information would significantly speed up their reuse and, in particular, the development of predictive models for a given domain. However, the diversity, ambiguity, and complexity of experimental data make it hard in practice. In this work, we propose a general approach to overcome these challenges, combining a human-in-the-loop process with a new methodology to understand automatically the semantics of experimental data, which can also be used as a data cleaning procedure. In addition, we focus on assessing the domain coverage of an experimental database using only categorical characteristics of the domain, which is essential for model validation or to understand if and where there is a need to perform additional experiments.
The collection of experimental data in many disciplines has produced a massive amount of data over the decades. However, the quality of data and the collection methodologies have changed with the evolution of the research fields and the improvements in the technology used to carry out measurements. Over the years, this progression has led to the availability of considerable amounts of data, but that are likely afected by ambiguity problems due to their heterogeneity and complexity.
At the same time, the increasing availability of experimental
data has stirred the development of predictive models to study a
domain and improve the related technologies. These data-driven
models are greedy of data, and, as a consequence, there is the
need to automatically collect, store, and manage large quantities of
information coming from diferent sources, representation formats,
and diferent quality levels. Data ecosystems address these problems
by integrating disparate or incompatible data sources, maintaining a
specific quality level [
PFR
O.C.M T 300k P 1atm Phi ... 0.5 ...
883 ... 3E-03 ... 1.2 ...
...
800 0 0.9 ...
827 0 0.9 ...
855 0 0.8 ...
883 ... 4E-04 ... 0.6 ...
... ...
Experimental Data
Simulated Data
0.0100
n ]n
io io0.0075
t t
a c
r a
ten reF0.0050
con lo
C [M0.0025
0.0000
800
[
In our case study, combustion kinetics, as shown in a simplified snapshot in Figure 1, we work with experimental data related to an experiment with associated metadata and the corresponding simulated data of a predictive model to compare them and validate and improve the model itself. The challenge is to overcome the manual management of the data: we need to automatically interpret the experiment, i.e., to distinguish the actual subject data of the experiment among all the data columns, simulate it with the right solver, and pair the experimental data with the simulated data correctly based on the content of the experiment metadata.
We propose an iterative approach to understand and store
experimental data with humans-in-the-loop, focusing on three aspects
that are critical in building scientific models: interpreting the
scientific data, assessing the coverage of the experiments in a specific
domain, and clean and improve the scientific repository. To this
purpose, we propose a rule-based interpretation of each experiment,
that enable to automatically validate and clean the data using a
similarity index. Furthermore, it is important to quantify the database
coverage within the experimental domain space. The experimental
database coverage of a domain impacts the ability to assess the
validity of numerical simulation models developed for a domain and,
if used as a training set for machine learning models, it can have a
heavy impact on the quality of the resulting model [
The paper is structured as follows. In Section 2, we discuss related work and open problems. A general approach to integrate diferent data sources with semantic heterogeneity problems is introduced in Section 3. Section 4.1 presents a rule-based approach to interpret the semantic of experimental data automatically and a methodology to quantify the coverage of an experimental database is presented in Section 4.2. An automatic analysis of a numerical simulation model using experimental data that facilitates data cleaning procedures is discussed in Section 4.3. A final discussion is debaded in Section 5. 2
In recent years there has been growing attention to the sharing
and reuse of data [
Combustion kinetics has been the subject of study for many decades. For this reason, many experimental data regarding diferent fuels in several environmental conditions have been collected over the years. The evolution of the combustion study process has led to increasingly precise measurements, enriching the experimental data with outline details that are decisive for a complete understanding of the phenomenon. Over the last few decades, data collection has been more methodical and massive, which has allowed for the development of predictive numerical models. A numerical model can simulate complex domains without the necessity to carry out expensive experiments in terms of time and price. In particular, in the case of combustion kinetics, we can predict the behavior of reactors and fuels in diferent conditions to improve their eficiency and reduce pollutants.
Even today, both the models and the experimental data are
mainly afected by two problems that make these data sources of
heterogeneous information. The first problem regards uncertainty;
the latter concerns the ambiguity of the information contained
therein. Uncertainty can be related to the experimental imprecision
or the error made by the model representing the domain. These two
types of uncertainty are thus defined as the aleatoric one, which
is related to the noise present in the data, and the epistemic one
associated to what the model does not represent precisely [
Similarly, ambiguities can be encountered both in the model and
in the experimental data. An example regards the chemical names.
Many diferent fields deal with chemical compounds whose names
are not uniquely defined, and for this reason, diverse nomenclatures
of the same compound can be found both in diferent models and
experimental data. This obstacle involves a not immediate integration
of experimental data from diferent sources and direct comparison
of diferent models [
Regarding the uncertainty in the data and the model, techniques
have been developed to separate the two types of uncertainty [
Diferent formats have been proposed to represent the
combustion kinetics experimental data to remove the ambiguity from the
experimental data. These formats contain mandatory or optional
ifelds that limit the freedom of each researcher in defining their
ifelds, thus moving towards a standardization of representation.
There are mainly two representation formats, ReSpecTh [
The diversity of a dataset is a critical aspect in many
practical applications, but it is often overlooked [
Data is critical for the development of machine learning-based
models. For this reason, data management has an increasingly
central role in these activities as the results of the models are strictly
dependent on the dataset [
In many experimental disciplines, data is collected from diferent
sources such as repositories, literature, or private communication
between research laboratories. This entails having to manage
various problems related to the heterogeneity of the data [
Experimental data are precious both for their rarity and for their
cost in collecting them. For this reason, it is essential to accept all
the experiments and then carry out a series of automatic checks to
preserve the repository’s quality. For example, a possible control
is on the consistency between the unit of measurement and the
measured property. Another quality dimension to guarantee is
completeness: Since the data comes from diferent sources, times, and
formats, it is essential to ensure that all the primary information
of an experiment, in terms of metadata, is complete. Regarding the
semantic accuracy of the experimental data, it is important that the
values of the properties are within a range of reasonableness.
However, while in the literature there has been an extensive attention to
developing techniques for managing and ensuring data quality and
consistency (see for an extensive survey [
To tackle these problems, we define the process illustrated in Figure 2, that follows the entire life cycle of experimental data to guarantee a certain level of the data quality, according to diferent quality dimensions, and at the same time, provide information to improve the predictive model.
This human-in-the-loop process is implemented within
SciExpeM (Scientific Experiments and Models), a framework that ofers
diferent services related to the management and analysis of
experimental and simulated data to speed up the predictive model
development process in combustion kinetics [
SciExpeM uses the process for diferent applications: first of all, the user enters the experimental data in the system using, for example, an interactive form. The experiment is added to the database and SciExpeM checks for syntax or detectable semantic errors. Initially the new data are tagged as new and they can be set to invalid in any of the following phases if flaws in the data are detected. In a second moment (activity Check experiment in the Figure 2), an expert has to verify each new experiment, checking for undetectable semantic errors and fill the incomplete experiment metadata. Once an experiment is verified, the status field change accordingly, and SciExpeM couples the experiment to an interpreter. Experimental data and results of simulators are records of information that we need to distinguish and pair automatically. To this purpose we propose to associate to experimental data to the concept of an interpreter for the data. This entity, in particular, can recognize the properties that are under investigation in an experiment from the others that are just auxiliary information, such as environmental conditions. For example, in Figure 1, the pressure is neither the dependent nor the independent variable (or property) under investigation, unlike temperature and concentration. Moreover, based on the experiment details, the interpreter knows which solver needs to be used to simulate it and correctly pair the experimental data with the corresponding simulated ones. Finally, when the system can manage an experiment independently with its simulations, a loop starts. The simulated data are compared with the experimental data using a similarity index that provides information to improve both the model and the repository quality. This comparison is possible because we leverage a bidirectional relationship: we use the model to validate the data and use the data to validate the model. First, using the experimental data to validate the model helps understanding which aspects or portions of the domain that an experiment represents still need to be improved. Second, we use the model to understand if the semantics of the experimental data is reliable: a model that difers strongly from the experimental data is synonymous with an error in the model or incorrect experimental data. The human-in-the-loop approach allows assessing these discrepancies and taking the appropriate actions.
Representing, collecting, and integrating heterogeneous data in a database are only the initial steps to extract value. In Section 4.1, we present our approach to interpreting the semantics of the data Add Experiment
Check Experiment
Verify Experiment
Interpret Experiment
Simulate Experiment Improve Model
and / or Data Cleaning correctly, and then in Section 4.2 we measure the coverage of a database in a given domain, while in Section 4.3 we focus on improving the repository quality. 4.1
Experiments are records of measured properties and other metadata that characterize them. Besides, among the measurements, it is not rare to find additional measured properties that specify, for example, the environmental conditions of the measures, but without being the subject of the scientific observation. This peculiarity generates ambiguity since a property could be the subject in an experiment but not in another. In practice, to manage scientific data, there is the need to distinguish automatically which, among the measured properties, is the dependent and the independent variable. In this context, we need to teach the data management system the ability to recognize the role of each property in each experiment, keeping in mind that what makes a property a subject of an experiment is a particular combination of metadata values of the experiment itself.
For this reason, it is necessary to define a flexible methodology to distinguish the subject properties from the auxiliary ones. In other words, we need to find an approach to transfer the domain knowledge into the SciExpeM to interpret the semantics of an experiment correctly and treat all the database entries with equal semantics in the same way.
Manual management of this complex database is not feasible because an experiment could contain dozens of measured properties, and, for example, we should tag each of them correctly if they are the subject of the experiment or not. Moreover, this procedure should be repeated hundreds of times, once for each experiment, making it hard to analyze a large amount of data. Accordingly, we propose a methodology automatically extracting useful information from a database model in which semantic heterogeneity is present.
We propose a dynamic interpretation of a database model based
on rules, similar to what is done for data cleaning or to ensure
consistency and accuracy in a database [
The Model Validation procedure systematically measures how good the predictions of a model are, compared to the corresponding experimental data. To consider the result of this procedure reliable, the experimental database, if possible, should cover as much as possible the domain with equal granularity. Database coverage can help in this task, providing an immediate procedure to measure the diversity and completeness of representation of the database.
We leverage categorical attributes and a multidimensional matrix to represent the domain and to define a coverage index. This approach overcomes the limitations of using patterns and thresholds that are sensible and directly afect the measurements based on the way they are defined. We create a detailed and generic representation of the database coverage that can be used to assess which part of the domain is poorly covered by data and consequently can be used to start the process of Design of Experiments.
We measure the coverage C of dataset D that regards the model M with attributes, = 1, ..., in three steps.
First, it is necessary to identify a subset of the model fields (or attributes) {1, ..., } = ˆ ⊆ , and transform them into categorical attributes. A categorical attribute of the model is a field that can only take a value from a restricted number of options. In this way, any attribute ∈ ˆ can only have diferent ordered categorical values (or possible options). If the attribute ∈ ˆ is a continuous numeric field, we take the minimum ( min) and the maximum (max) value that can be taken by in the domain, and we fix equidistant ticks from the range [, ] and associate the value of the attribute to the closest tick. Instead, if the possible values of an attribute are not continuous but with high cardinality, we can identify a subset of the possible values leveraging a EXP ID DATA NAME
VALUE
METADATA (META) NAME
VALUE EXP ID
EXPERIMENT (EXP.) EXP ID
INTER. ID
INTERPRETER (INTER.) INTER. ID
SOLVER RULE
MAPPING MODEL
NAME
VALUE INTER. ID
INTER. ID
TYPE DATTAYPNEAME
hierarchy among them or using bucketization: similar values are
associated with the same bucket [
Second, we define a multidimensional space that reflects our database’s coverage among the set of attributes with cardinality | | = . Each characteristic ∈ ˆ defines a dimension of the space of size . We then create a matrix, called coverage matrix CM , with dimension CM = 1 × ... × to represent this space.
Finally, after initializing all the matrix cells to 0, for every entry in the model M, for every possible combination of categorical values of the attributes, we update the coverage matrix using Equation (1) only if it holds the condition in Equation (2) for when ≠ 0 with ∈ [1, ]. (1)
CM [1, ..., ] += 1 C =
1, [1] == ... == , [ ] == 1 (2)
The final result is a density matrix that represents the coverage of our database regarding some given categorical attributes. Immediately, we can define a database coverage index: after examining all the entries present in the dataset D, we can count the number of cells with value bigger than a given threshold , and normalize this value on the total number of cells (Equation (3)).
Í ∈ [1,1 ],..., ∈ [1, ] 1, M [, ..., ] ≥
∈ [
CM 4.3
Data-driven applications are sensitive to data quality, but in domains where the experimental data are rare and afected by nonnegligible uncertainty, it is hard to define and measure the quality level of an experiment based on which accept or reject the insertion in the repository. As discussed in Section 3, the process that we have identified tries to mitigate three diferent data quality dimensions: consistency, completeness, and accuracy. The domain-specific automatic checks, for example, ensure consistency, examining that the unit of measurement of a property is valid. Instead, in the verification step, the scientist completes the empty mandatory metadata of the experiment. The accuracy of experimental data afected by uncertainty is hard to quantify, but the combination of a humanin-the-loop, the predictive model, and a similarity index can help in this task. The predictive model has its own uncertainty, for this METADATA (META) NAME VALUE EXP ID
EXPERIMENT (EXP.) EXP ID
INTER. ID 1 1 2 2
RULE Reactor PFR Exp. Type IDT Reactor RCM IDT Type d/dt OH MODEL META
NAME
Reactor META Exp. Type META
Reactor META IDT Type d/dt OH
VALUE INTER. ID PFR IDT RCM 50 50 60 60 reason, if we use a similarity index that quantifies the diference between the predicted data to the experimental data, we can automatically identify an experiment that has a behavior somewhat diferent from the other similar experiments. It will then be the scientist who establishes what happened case by case, invalidating the experiment, if necessary, through the metadata of the state. Once an iteration of the simulation-analysis-cleaning loop is terminated, the cycle can start over, and the attention is moved over another experiment. Section 5 presents examples on data cleaning, database coverage and semantic interpretation. 5
The backbone of automation in a scientific data management system is the ability to understand the semantic of an experiment. In our case study means distinguishing the x-axis from the y-axes and correctly pairing the experimental properties with the simulated data. In Figure 4 there is an example of the Interpreter assignment to two experiments based on rules. Interpreter with ID 50 is assigned to experiment with ID 1. In fact, all the rules specified by this interpreter are fulfilled by the experiment. Then, thanks to the interpreter, we are able to recognize the x-axis and the y-axis of the experimental data.
SciExpeM has a database of about 500 experiments, which, as described in Section 4.2, have been categorized based on two metadata as suggested by domain experts: temperature and pressure. Specifically, the temperature is tokenized from a min of 500 K to a max of 2000 K in steps of 25 degrees. Instead, the pressure goes from 0 to 40 bar with step 10. The coverage index using as threshold 1 is 0.88. Instead, if 3 and 5 are the thresholds, the coverage index is 0.55 1 2 50 60 50 60 and 0.32, respectively. Figure 5 shows the density of the coverage matrix CM , which is used to calculate these indices.
Through the interpreter, SciExpeM can simulate an experiment with diferent models and compare the results.
With model validation, given a domain-specific similarity measure, we measure the predictive performance of a model against a set of experimental data. The analyses of the similarity scores after the model validation provide essential information for the model improvement and can also be used to improve the quality of the repository itself. In fact, we can also use the predictive model capabilities to perform data cleaning. A rule-based approach for data cleaning is already implemented, and it is focused on syntax or semantic rules on attributes of the database model, but it is not powerful enough to understand if the measurements contained inside the experimental data are reliable.
We combine the use of the predictive model with some automatic
statistical investigation tools to detect outliers [
In our case study, we use Curve Matching [
In this work, we have presented the problems and proposed solutions for managing a complex database that represents an experimental domain. As in many cases, creating a scientific repository is not the final goal, but it is preliminary to extract value from the data. For this purpose, we have created a human-in-the-loop process in which the users have diferent tasks. First, to solve the heterogeneity of the data, using general metadata as additional model attributes and with the help of the users, we can categorize and distinguish which portion of the domain is precisely represented by the experiments. Second, using a rule-based procedure, we can automatically understand the semantic of experimental data. This information is essential for the following automatic analyses. Finally, as in many validation scenarios, the reliability of the prediction accuracy depends on the coverage of the test set. For this purpose, we develop a general coverage index, that given a set of model attributes that define the domain space, quantifies the domain coverage of the database. Besides, we can combine this information with a statistical investigation. Given a similarity measure and human support, we can establish if an experiment outlier is a source of information for the predictive model improvement or unreliable experimental data, thus improving the overall database quality.
The work of E.R. is supported by the interdisciplinarity PhD project of Politecnico di Milano.