The possibility of searching for classification regularities in large arrays of chemical information by means of machine learning methods is discussed. Tasks peculiarities in inorganic chemistry and materials science are considered. The short review of these methods applications to inorganic chemistry and materials science is presented. The system for computer-assisted inorganic compounds design based on machine learning methods has been developed. The developed system usage makes it possible to predict new inorganic compounds and estimate some of their properties without experimental synthesis. The results of this information-analytical system application to inorganic compounds design are promising for new materials search.
Throughout the centuries of its evolution chemistry and materials science accumulated huge information. In common with other experimental sciences chemistry got through several stages: information accumulation, data analysis and development of classification schemes and rules that allow classifying a new object to a particular substances class. The substances division into inorganic and organic ones, Periodic table of elements, compounds classification according to crystal structure type, etc. are examples of such classifications. Essentially, in all cases these classifications are imprecise, and classes intersect partially. For example, organic chemistry is determined as the carbon compounds chemistry but carbides and carbonates belong to inorganic chemistry objects as well as boron hydrides (boranes) or silicon hydrides (silanes) which are closer to hydrocarbons (organic chemistry objects) in many properties. In large measure, it is caused by imperfection in the classification rules which were developed by chemists. One way to get around these problems in inorganic chemistry and materials science is machine learning methods application to information analysis aimed at discovery of complicated classifying regularities that allow considering of substances to particular classes. It is noteworthy that obtained regularities include substance components properties as variables, and for this reason, their use allows us to predict the class for the substances that is not yet synthesized knowing only the well-known parameters values for chemical elements forming this substance.
Half a century ago IMET pioneered in applying such
approach to machine learning use to search for
classifying regularities that allows a prediction of new
inorganic compounds and some of their properties
estimation [
Suppose that every inorganic substance is described by a vector x = (x1(1), x2(1),… xM(1), x1(2), x2(2),… xM(2),…, x1(L), x2(L),... xM(L)), where L is the number of chemical elements that form a compound and M is the number of chemical elements parameters. Each substance is also characterized by a class membership parameter: a(x) ∈ {1, 2,…, K}, where K is the number of classes. The learning sample consists of N objects: S = {xi, i = 1, …, N}. We denote the learning sample objects subset from class aj, j = 1, 2, …, K, by Saj = {x: a(x) = aj}. The machine learning aim is to construct a classification rule that distinguishes not only different classes objects of the learning sample but also preserves prognostic ability to generate new combinations of chemical elements that were not used for learning.
Among the numerous machine learning methods, various of Artificial Neural Network (ANN) learning algorithms modifications and Support Vector Machine algorithms (SVM) are the most popular (figure 2). This is due to appropriate software packages accessibility and seeming exam score accuracy (many investigators do not take into account an influence of overfitting effect on subsequent prediction reliability that is inherent in these methods).
Notation: ANN – artificial neural network learning; SVM – support vector machine; KNN – k-nearest neighbors method; DT – decision trees learning; GPN – concept formation using growing pyramidal networks; LM – linear machine method; LDF – linear Fisher discriminant; LoReg – voting algorithm where estimations for classes are calculated by means of voting by logical regularities system; SWS – statistical weighted syndromes; DTA – deadlock test algorithm; ECA – estimate calculating algorithm.
A great diversity of chemical and materials science
tasks were solved successfully using machine learning
methods, e.g.:
theoretic tasks of prediction of:
- inorganic system phase diagram type [
- inorganic compounds crystal structure type [
- some of inorganic compounds properties (melting
point [
- functional properties of ceramic materials [
A special information-analytical system (IAS) that
allows an automation of task solution procedure in the
field of inorganic chemistry using machine learning was
developed in IMET [
2) Strong correlation within set of these attributes for each component due to their dependence on common parameter - chemical elements atomic number (it follows from the Periodic Law).
3) Individual chemical elements properties give small informative gain therefore more informative parameters of single compounds (for example, single oxides, halogenides, chalcogenides, etc.) and component properties algebraic functions are widely used additionally.
4) Blanks of attributes’ values that are filled by various methods including interpolation taking into account a periodicity in chemical elements properties variation with their atomic numbers.
5) Large asymmetry of learning set sizes for different classes (at that often the least of representative – as a rule newly obtained classes of substances – are the most interesting for chemists).
6) Errors and discrepancies in inorganic compounds experimental classification of learning set decreases the prediction accuracy drastically.
Machine learning procedure involves several stages: 1) objects selection for machine learning, 2) attribute description formation (including the most informative attributes selection and filling attribute values blanks also), 3) machine learning algorithms selection, 4) machine learning including application of algorithms ensembles and collective solution synthesis in a case of several algorithms usage, 5) machine learning quality estimation, 6) new objects status prediction and results interpretation.
Representative and reliable set formation for machine
learning preconditions subsequent prediction accuracy
in a great measure. Objects selection (known inorganic
substances examples) for machine learning is performed
by experts in subject domain by means of information
stored in data bases (DBs) on inorganic substances and
materials properties including DBs that were developed
in IMET [
Attribute description formation problem is complicated and hard-to-solve task of modern machine learning theory. There are a large number of approaches which have proved their effectiveness at various task types solution. However, it is impossible to evolve a surely optimal universal method of attributes selection. In this regard few alternative methods with subsequent collective decision synthesis are used by us for attribute selection. 2D-projections visualization tools are applied additionally for points corresponding to certain type compounds in chemical elements properties space. The parameters set includes not only initial attributes but also the algebraic functions of these attributes which are selected by user.
The IAS includes a set of machine learning algorithms
which are the most popular among chemists (figure 1).
At present time IAS involves the following software:
programs based on well-known linear machines
methods, Fisher linear discriminant, k-nearest
neighbors, support vector machine, neural-network
algorithms, and also algorithms which were developed
by the Computing Centre, Russian Academy of
Sciences and based on estimates calculation, deadlock
tests voting algorithms, logical regularities voting
algorithms, weighted statistical voting algorithms, etc.
[
Our experience in inorganic chemistry prediction tasks
solution shows [
The cross-validation on learning set objects is the most widely used universal and reliable tool for machine learning quality estimation. IAS contains special software for this procedure realization that is used in the best machine learning algorithms selection. However, an attempt of cross-validation application to machine learning accuracy estimation at use of algorithms ensembles as optimizable criterion results the loss of estimate unbiasedness. In this case, there is a certain overfitting risk. In this regard, the traditional approach to collective algorithms accuracy evaluation using examination recognition of N examples chosen randomly from learning samples and unused in learning (at the final prediction stage, reference examples are returned to the learning set) is applied. The corresponding program was included to IAS.
The learning set sizes asymmetry for different classes is an important problem at machine learning accuracy estimation. Naturally in this case the generalized examination recognition accuracy does not represent the prediction error for small classes, therefore the ROC curves application is appropriate to different algorithms prediction quality analysis. ROC curves allow recognition accuracy comparison for the targeted and alternative classes at variation of cut-offs which identifies belonging to different classes.
It should be pointed out that machine learning quality estimation procedure belongs to yet hardly unsolved machine learning task. Some algorithms (SVM, ANN, etc.) characterized by overfitting effect, show high examination recognition accuracy often but this fact does not always provide high predicting reliability for new objects.
To increase predicting accuracy in the case of learning
sets with K classes (K > 2) the following method is
used. Firstly, multi-class learning and prediction are
carried out. Next, K dichotomies are calculated: the
targeted class and all the alternative classes, followed
by subsequent K predictions. The results of multi-class
prediction and dichotomies series are intercompared,
and if the predictions are not contradictory the decision
on the object status is made. The special tools for
collective decision formation based on comparison of
multi-class prediction results and dichotomies series
were developed. The efficiency of such approach that
allows to increase prediction accuracy was approved
during numerous tasks solution [
The machine learning application allowed a search for
inorganic compounds formation regularities, a
prediction of thousands not yet synthesized substances
and some their properties estimation using obtained
regularities. This approach efficiency to inorganic
compounds design can be illustrated by comparison of
the predictions results with newer experimental data
obtained after publication of our predictions [
The table contains AB3X3 compounds formation
possibility predictions in the A2X3–B2X systems (A and
B are various elements, and X = S, Se, or Te) under
normal conditions, which could be promising for search
for new semiconductor, nonlinear optical,
electrooptical, and acousto-optical materials. Experimental
information on 117 examples of AB3X3 compounds
formed and 58 examples when no such composition
compounds were formed in the A2X3–B2X systems
under normal conditions was used for computer
analysis. To describe the compounds in computer
memory we selected A, B, and X elements properties
(the melting and boiling points; covalent, ionic (by
Bokii and Belov), and pseudopotential (by Zunger)
radii; the first three ionization potentials;
electronegativity (by Pauling); the standard enthalpies
of atomization and evaporation; thermal conductivity;
molar heat capacities, etc.), simple A2X3 and B2X
chalcogenides properties (standard entropy and
enthalpy), and some algebraic functions of these
properties (for example, the ratio of the covalent radius
to the metal radius for elements A, B, and X). The table
1 presents predictions examples for the AB3X3
compounds and their experimental verification results.
The following notation is used: 1, the prediction of
AB3X3 formation under normal conditions; 2, the
prediction of AB3X3 absence under normal conditions;
#, examples, the information on which is used for
machine learning; empty cells, uncertain prediction; ©,
the prediction of AB3X3 formation matches new
experimental data; and Θ, the prediction of compound
absence matches experimental data. All 27 tested
predictions coincided with the experimental data.
During half of the century the predictions of thousands
of inorganic compounds in binary, ternary and more
complicated chemical systems were obtained and some
their properties (melting point, critical temperature of
superconductivity, band gap energy, etc.) were
estimated in IMET [
This work was partially supported by the Russian Foundation for Basic Research (project nos. 16-0701028, 17-07-01362, and 15-07-00980). We are grateful to V.V. Ryazanov, O.V. Sen’ko and A.A. Dokukin for long-term help and collaboration.