Relations between the components of the data layer and application layer in the quantitative spectroscopy are schematically represented. The sequence of actions in the data quality analysis in the W@DIS information system is described. Methods for the spectral data quality analysis and assessment of trust in expert data sources and the results of their application in the W@DIS information system are presented. Along with common techniques for the primary data source quality analysis, techniques for decomposition of expert data sources and pairwise comparison of ordered data sets are reviewed. We also discuss the use of empirical data for filtering large data collections by an acceptable difference between identical energy levels in primary data sources and in an empirical data source. Two types of the user interface for viewing the results of spectral data analysis and assessment of trust in expert data are considered. Original methods for the data source quality analysis and assessment of trust in expert data are briefly described. These methods are used for finding discrepancies between different spectral data types (primary, expert, and empirical).
Assessment of the quality of information resources located on the Internet and
representation of these resources in the form of software agents convenient for work
are among currently important problems. Semantic Web (SW) technologies [
Assessments of the quality of scientific resources can be divided into two groups. In the first group, the reliability of scientific resources connected with a mathematical model of a subject domain is assessed. For the validity check, criteria are used derived from the constraints imposed by the model on the entities described in the subject domain. In the second group, trust in resources is assessed, in particular, the influence of experts, presenting these resources.
When studying the states and transitions of molecules and atoms, large-volume numerical arrays are used; some of them contain parameters of several billion transitions for a molecule.
Virtual Atomic and Molecular Data Centre (VAMDC) [
Information system W@DIS [
This paper continues the cycle of our works [
Before consideration of a data model for quantitative spectroscopy, we should emphasize the fact that the model suggested does not describe all facts related to the subject domain under study. However, it includes most data used in applied subject domains. The data model describes three data types.
Data related to measurements or calculations published in a particular work and their
brief description together are called the primary data source. Primary experimental and
theoretical data are interrelated, because not all values of the properties of transitions
and states are measurable, for example, quantum numbers, the values of which can be
determined only by solving computational problems. In addition to primary data,
composite data are also used in spectroscopy. Two composite data types are important
for applications: empirical and expert data. The former are calculated by using a
multiset of measured data, and the latter are a set of calculated, measured, and reference
data. Thus, the data layer contains three data types: measurement data and their
derivatives (empirical energy levels calculated on the basis of the Rydberg–Ritz
principle [
Figure 1 shows three groups of data and their relationship with experiment and
calculations. Each data group is marked by a rounded rectangle; and the applications
where these data are found, by rectangles. These applications are connected with six
problems of quantitative spectroscopy [
Since the sources of spectral information in the form of expert datasets are in great demand, expert data are also of interest in this work. There are three reasons for this. Firstly, the expert data on applied subject domains, the number of which exceeds several dozen, are contradictory. Note that data for a particular subject domain must meet certain requirements for their quality. Secondly, expert data are formed via informal manipulations, where the professional skills and preferences of experts play an important role. This is why different forms of publication criteria are a useful tool for the expert data quality assessment. Thirdly, the values derived from measurements and calculations are adapted when forming expert data used in solution of applied problems.
The analysis of the quality of C0 — the multiset of transitions characteristics of which
are measured and published for each molecule takes the central place in the data
analysis. This set consists of several parts. The part C0A includes incorrectly identified
transitions which are excluded from the quality analysis. The part C consists of
correctly identified transitions, but can contain conflicting values of wavenumbers and
other characteristics for identical transitions. For a number of molecules, Cantor sets of
states (sets with all elements unique) are composed based on the multiset of transitions
measured. The states in these sets are described by empirical energy levels ER (for
example, for a water molecule [
The quality analysis of data related to transitions from the set CA is of greatest interest. If this set is Cantor, then a possibility of compiling a set of empirical energy levels from it is doubtful. Therefore, it is necessary to carry out measurements to expand the number of empirical energy levels in the set CiR.
The main information resources in W@DIS are sources of data on states and transitions. Each data source includes an uploaded numerical array or plot published and the properties of this array (plot).
Let D0n and D0Em denote the primary array of measured values of physical parameters and the expert data array, respectively. Figure 3 shows the order of the quality analysis of an array uploaded (D0n or D0Em). Note that the above introduced multiset of transitions is the union of arrays: Hence, the equalities
C0 = UNn=1 D0n.
C = UNn=1 Dn, C0A = UMm=1 Dm, etc. (1) (2) are true.
The ellipses in Fig. 3 show the parts of the dataset related to the parts of the multiset. The boxes represent applications which perform calculations during the data quality analysis. The in (out) arrows to a rectangle are associated with the input (output) of these applications. Plus and minus signs indicate that the output satisfies (+) or does not satisfy (-) the constraints of an application.
The difference in the data quality analysis for expert data in comparison with the previous one is the appearance of the Decomposition application during the second stage. This application outputs two data sets: DEDn which includes physical parameter values close to published, and DAEDn which include values which significantly differ from published ones. It is possible to work with both data groups obtained from decomposition in W@DIS. CC is the set of primary computed transitions that is used during decomposition.
The data quality analysis results are part of the metadata for each data source in the W@DIS system. This metadata are represented in two forms: human readable and ontology for software agents. 3
The spectral data quality is analyzed in each paper on quantitative spectroscopy with
the use of traditional methods. The division of spectral data into data sets, which relate
to individual molecules and solutions to one of the seven spectroscopy problems in the
W@DIS information system (IS) (http://wadis.saga.iao.ru) [
Each data source is connected with a conventional set of metadata, which describes
the set of properties of this source. Data on different molecules are not connected. This
fact determines the branched modular structure of ontologies, each describing one of
the modules. Every molecular collection is described by ontologies, which
characterizes the information resources or physical quantities in this collection.
The main properties of data sources are those which describe the source quality analysis
results. Data sources in the collections are typified. The following types are used for
the classification: primary (measurements or calculations), expert, and empirical. The
first three types are traditional for natural science data collections, while empirical data
in different subject domains can be formed according to different principles. In
spectroscopy, empirical data includes data related to energy levels and other physical
quantities. Empirical energy levels are obtained from processing the complete set of
energy levels derived from the inverse problem solution accounting the Rydberg–Ritz
principle [
Authors of most publications use traditional methods for the analysis of spectral data quality: check of selection rules and calculation of standard deviations. When working with data collections, we use original methods for the data quality analysis and assessment of trust in expert data. These methods are briefly described below. 4
These methods can be divided into traditional and new [
During the analysis of spectral data quality, the states and transitions with incomplete set of quantum numbers are first distinguished. The second group of data excluded from the consideration includes transition duplicates in a data source.
The values of quantum numbers which characterize the energy levels and transitions between energy levels in a molecule are limited by the selection rules which follow from the mathematical model of a molecule. In W@DIS, forbidden transitions are marked and do not participate in any operations with data sources.
Paired relationships between data sources are estimated by the standard deviation or maximal deviation between the values of a physical quantity, identical states, or identical transitions compared. The acceptable deviation is determined by a researcher which uses the data to solve a problem. b
Along with the above described methods, there is an approach which uses the empirical data calculated and allows improvement of the quality of spectral data in the collections. Such empirical data have been acquired for several dozens of molecules. This method is currently applied to data on all isotopologues of the water molecule in W@DIS. The allowable difference determined by an expert (0.035 cm-1) between the energy levels of identical states taken from the data source, checked for quality, and from an empirical data source forms a filtering criterion. The results of application of this criterion are shown in Fig. 6.
The efficiency of this method depends on the number of states with empirical values of the energy levels ER. The efficiency is maximal when the set CA is empty.
The method [
Our recent studies have shown that the need in this method arises in the cases where traditional analysis techniques give a satisfactory quantitative result, but doubts remain about the degree of accuracy. In other words, this method should be applied in the cases where, for example, the spectral line centers should be determined with high accuracy in narrow spectral ranges.
The above methods for the data quality analysis are applied to the primary data derived from measurements and calculations. Expert data are generated according to informal criteria, including subjective criteria, expediency, etc. Therefore, these data require another method for qualitative assessment. The method of expert data array decomposition we use is based on the following assumption: expert data should not differ significantly from the full set of primary data published. The decomposition by complete consistent (rigorous) or inconsistent set (non-rigorous method) is admissible. In spectroscopy, the non-rigorous method is required during the initial stage of study of spectral properties of molecules when there is no sufficiently good model or accurate measurement data. A researcher must choose numerical values based of the requirements for the task he solves.
Two sets of interfaces are created in W@DIS for viewing the data quality analysis results. The first interface is intended for viewing metadata of a data source selected and its correlations with all other sources. The second interface is designed for description of paired relationships for the entire collection related to a molecule.
Figure 4 shows the interface for viewing metadata of expert data source [
This comparison is carried out when describing a data source selected, but it can also
be performed as refinement of the comparison (many–to–many) when describing a
collection. Figure 4a shows that there are no forbidden transitions according to the
selection rule checking; 6410 transitions were not identified. Fig. 5b shows a part of
the list (comparison was made with 171 data sources), where the results disagree with
the calculation data [
The results of comparison of all data sources can be presented in one viewing interface with quantitative characteristics or without them, i.e. qualitatively. Such a qualitative representation is shown in Fig. 6a, where pairs of sources are shown in red if the criterion chosen by a researcher is violated. This representation allows one to visually assess the number of good or bad pairs. Fig. 6b shows a fragment of the interface, where the quantitative coincidence of the data compared is obvious.
Figure 6a shows a part of a symmetric matrix with the qualitative results of estimation of the maximal difference between the frequencies in two published arrays of transitions, which are to be compared and include parameters of identical transitions, in the cells. If the maximal difference is less than 0.035 cm–1, then the corresponding cell is light gray; if the cell is white, this means that the compared arrays have no identical transitions, and if the cell color is dark gray, then the maximal difference is higher than 0.035 cm–1. Note that the number of cells with the high difference is quite large (slightly less than half of all data). The arrays of values published have passed the first stage of data filtering.
Figure 7 shows the quality analysis results for the multiset of H2O transitions measured, which have been tested by the criterion of maximal deviation between the values of wavenumbers of a pair of identical transitions Δ < 0.035 cm-1.
C - Transitions with correct quantum numbers CA - Transitions in which at least one state (upper or lower) is in ER
lower) are in ER Ai - Identical transitions for which |ωCiR – ωСR| < Δ
The number of measured transitions
Total transitions of which are
unique 312366 19932 292434 1400 291034 281561 9473 103324
837 102487 100270 5302 which differed from the corresponding wavenumbers of the identical transitions in the CiR array were rejected. The number of dark gray cells in Fig. 7 is much less than in Fig. 6a.
Table 1 presents number of transitions in different parts of the multiset C0. They are calculated for the transitions of the main isotopologue of the water molecule. In the table one can check the following equalities from Figure 2: the number of transitions in C is equal to the sum of transitions in the sets CA and CiR and the number of unique transitions in C is equal to the sum of unique transitions in the sets CA and CiR.
Details of the study on improvement of the quality of data on the water molecule are
given in [
The expert source contains 135606 transitions in total. The lines “Distrusted Vacuum Wavenumbers” characterize the number of transitions with low trust in different spectral ranges and over the entire spectral region. These transitions make 8% of the total number of transitions. Most low-trust transitions (87%) are in the near-IR and visible regions. We describe a collection of spectral data in W@DIS information system and the methods used in it for the analysis of spectral data quality. A data model in quantitative spectroscopy and a group of applications for data acquisition are presented. Three data groups are distinguished: data derived from measurements, from calculations, and their combination in the form of expert spectral data. Different parts of the multiset of transitions, which consists of numerical arrays extracted from publications, are described. The sequence of actions for the data quality analysis is explained. The quality assessment techniques commonly used in spectroscopy are briefly described, as well as three methods for the spectral data quality analysis developed by the authors for large arrays of conflicting spectral data. For better visual representation of the results, graphical user interfaces for working with the results of data quality assessment are exemplified. The percentage ratio of the parts of the multiset is shown, with the multiset of transitions of the main water molecule isotopologue as an example. In particular, we have shown that less than 1.5% of the data remains after data filtering in the automatic spectral data quality analysis, which require additional processing by experts. Acknowledgement. The main part of the work was performed within project № АААА-А17-117021310147-0. 15
16