The paper discusses the problems and prospects of implementing a hybrid classifier for career guidance testing according to the author's method «Associative color space». The authors set themselves the following goals: to increase the accuracy of determining the main classes and to identify subtypes that have visual similarities, but significant differences in terms of the characteristics of the work style. To improve the accuracy of determining the problem class of images, 11 special filters were developed and implemented in order to identify implicit features of the class. The use of these filters additionally solved the problem of distinguishing visually similar subtypes of the two main classes, which makes it possible to further automate not only the definition of the main class with optimal accuracy, but also the subtype, which will allow for a more thorough diagnosis of career guidance and provide the subjects with more specific characteristics and recommendations regarding their work style. Analyzing the results of the experiments, the authors hypothesized about the possibility of creating a three-dimensional map of types and subtypes, where there could be areas of pronounced types, subtypes, as well as transitional types with special characteristics that allow avoiding the result of a «mixed» type for the subjects.
2. To analyze the results of an experiment with a test sample of subtypes of class E «non-contrast creative» and representatives of class D with signs of a creative type on the presence of criteria that allow dividing these types of individuals automatically. 3. Identify the prospects for further research.
The fact is that in some classes, the neural network classifier works more effectively, and in some others, the filtering algorithm works better, which creates a prerequisite for the formation of a hybrid classifier as a final tool for career guidance testing. In addition, the use of a hybrid classifier makes it potentially possible to automate the division not only into main classes, but also subtypes within them. 2.
Consider the desired types (classes) of images using the «Associative color space» testing method:
type A («skeptical»), type B («medium avant-garde»), C («skeptical-creative»), D («rational»), E
(«creative»), F («radically avant-garde») [
Only individuals of type F use the «cut» tool when working with a test form, so it is determined already in the process of filling out the form and does not pass classification by processing neural network and filtering algorithms. This leaves 5 types of individuals to classify.
The A, B, C, D, and E types differ mainly in the distribution of dark and light pixels within the
planigon [
The object of research and testing of algorithms are sweeps of ready-made test forms (Figure 1). Sweeps allow us to consider all areas of the planigon as equivalent, excluding distortions present in quasi-space.
The A type characterized by a predominant (50% or more, respectively 100% – pure type) presence of elements «horizontal lines» (Figure 1b.). Lines (from 3 cells in length) can be either the colors of strictly identical coordinates, or the colors of different coordinates, but belonging to the same tone, for example, «yellow».
Tone – a range of colors that are indicated in the color circle as «yellow», «red», «orange», and so on. It is also the tone that gives the color its name.
In the current sample, elements of A type are mostly represented by colors of the same coordinates, since the neural network classifier is undergoing training and first tests, the types are mostly «bright».
Another condition for the presence of A type is a small total number of shades (no more than one tone with the number of shades of 5 or more).
There are the following signs of dominant A type: 1. 50% or more of the scan is filled with the desired «horizontal line» elements (from three cells of the same color or tone). 2. A small number of shades, namely no more than one tone with a number of shades of 5 or more. 3. The fewer shades applied by the recipient, the more pronounced type A is considered. a b
The B type characterized by the presence (3 elements per scan or more) of the «vertical lines» elements (from three cells in height) with identical coordinates (Figure 2). The search elements have a minimum of 3 cells in height, they can be up to 6, respectively, the definition area can reach 6x1.
The total number of shades can be very different, the main thing-3-fold or more presence of an element from 3 cells in height of exactly the same color (Figure 2).
There are the following signs of dominant B type: the presence of 3 or more elements of the «vertical lines».
a b
The C type characterized by the presence of the elements «horizontal lines» (Figure 3b) and «wide» or «narrow» gradients in the definition areas 33 and 23 (for wide) and 31, 41, 51, 61 (for narrow ones) (Figure 3b, Figure 5b). Lines (from 3 cells in length) can consist of colors of exactly the same coordinates, or colors of different coordinates, but belonging to the same tone (Figure 1b).
«Wide gradients» are bands of colors of exactly the same coordinates in width and colors of the same tone range in length, for example, gradients of green and red colors in the definition areas 33 and 23 in the Figure 3b.
The total number of shades may vary, but there are generally 4-5 shades for 2-3 tones. For example, in the Figure 5b, the scan shows from 3 shades for red, green, purple and orange tones.
There are the following signs of dominant C type: 1. Simultaneous presence of elements of «wide» or «narrow» gradients and «lines». 2. The minimum number of elements for determining the type of 2 lines and one wide gradient 33 and 23, or 2 lines and two narrow gradients. 3. Relatively high variety and number of shades.
a b
The D type characterized by the presence of elements «chess» or «chess-like» (Figure 4). The «chess» and «chess-like» elements are arranged by a special arrangement of dark and light cells in the area of definition 22. Moreover, «chess» elements can contain any color.
The total number of shades is usually small (the brighter the type, the fewer shades). There are the following signs of dominant D type: 1. The presence of «chess» or «chess-like» elements on 50 or more percent of the scan area. 2. A large number of cells with repeated colors of the same coordinates, especially black, white, yellow, red and blue. 3. Using a relatively larger number of colors from the «basic» section of the ColourUnique M software module compared to other types.
The E type mainly characterized by the presence of a large number of shades of selected colors, namely – from 5 shades for at least two tones.
Representatives of this type are distinguished by the presence of mainly «narrow» gradients (Figure 5b), less often there are «wide» gradients (Figure 3b).
There are the following signs of dominant E type: 1. The presence of a large number of shades, namely – from 5 shades for at least two tones. 2. The predominant presence of «narrow», sometimes «wide» gradients.
a b
As can be seen from the examples, the desired elements differ in the shape of the definition areas and their orientation in space, as well as have features of the color system. That is why the authors decided to use convolutional neural networks, which allow detecting signs of a particular class in any area of the analyzed image.
In experiment 1, the architecture of the Inception_v3 convolutional network was used. For
experiment 2, the Unet network is trained. The number of images for each experiment is 350 taking into
account augmentation [
To obtain and analyze the data, such procedures as observations and experiments with a neural network and a filtering algorithm were used, the results of machine analysis were compared with the results of the analysis of an expert (the developer of the methodology). 3.1.
Previously untrained neural networks were used to classify individual types using the «Associative
color space» testing method. Such architectures as MobileNet [
The neural network (for experiment 1) diagram for classifying images (ready-made test forms) is shown in the Table 1 the architecture of the Inception_v3 network was taken as the basis, then 4 fully connected layers were attached to it. The first layer has 64 neurons, the second has 32 neurons, and then there is the dropout parameter, which is responsible for preventing overtraining. This parameter, with a set probability of 0.1, «turns off» the neuron so that the neurons that are with it do not focus on it, but act on their own. The fourth layer contains 16 neurons. The final layer contains 5 neurons – 1 neuron is responsible for 1 specific type (class) of the image. inception_v3 (Mode 1)
flatten (Flatten) dense_one (Dense) dense_two (Dense) dropout_one (Dropout) dense_three (Dense) output (Dense) (None, 3, 1, 2048) (None, 6144) (None, 64) (None, 32) (None, 32) (None, 16) (None, 5)
The Unet architecture (for experiment 2) is shown in the Figure 6. It consists of a narrowing (left) and expanding (right) path. The narrowing path involves re-applying two 3×3 convolutions, followed by a rectified linear block (ReLU) and a maximum join operation (2×2 of degree 2) to lower the resolution.
At each step of downsampling, the property channels are doubled. Each step in the expanding path consists of a step-up discretization operation of the property map, followed by: 1. 2×2 convolution, which reduces the number of property channels. 2. Merge with the appropriately cropped property map from the shrinking path. 3. Two 3×3 convolutions followed by ReLU.
On the last layer, a 1×1 convolution is used to map each 64-component property vector to the desired number of classes. In total, the network contains 23 convolutional layers.
For training, 1470 images were used, for validation – 315 (63 images for each class). The classification accuracy of each of the networks can be seen in the Table 2.
The conducted experiments show that the maximum classification accuracy using the Unet network does not exceed 71,05 %. As a solution to the problem, the authors consider the possibility of adding new classification layers or replacing the optimizer. The Inception_v3 network has an accuracy of 91,12 %.
Class D became the most problematic for neural network algorithms and, despite the fact that according to the results of the last experiment, it was possible to increase the classification accuracy in class D to 62% against the previous 45%, the accuracy of determining samples of class D remains unsatisfactory.
Let us consider the joint operation of the filtering algorithm and the neural network algorithm to improve the accuracy of determining objects of class D, as well as to differentiate some subtypes.
The filtering algorithm is developed in parallel with the neural network algorithm in the future, both classifiers will pass comparative tests, which will result in the selection of the main algorithm, while the second one will be a supporting or alternative class, where they were shown the best result.
Consider a filter developed for the desired class D, where at the moment the neural network does not classify images reliably.
Filtering is based on the principle of comparing regions 22 of the planigon cell with samples of «chess» and «chess-like» combinations consisting of cells of 4 states (Figure 7).
To count the number of combinations, we used permutation schemes with and without repetitions. = ! ,
( + +...+ )! ,
( , , … , ) = !∙ !∙…∙ !
(2)
where n – the number of elements, Pn – the number of permutations, and P(n1,..., nk) – the number of
permutations with multiplicities n1,...,nk [
Received: 1. 24 combinations for cases of using cells of all 4 conditions for the 2x2 definition area without repetitions. 2. 144 combination for the use cases of the cells 3 conditions with a single repeat of one condition (0, 1, 2 or 3). 3. 24 combinations for cases where 2 condition cells are used for the definition area 22 with a single repeat of two conditions (0, 1, 2 or 3).
Finally, there is 192 combinations.
However, not all of the resulting combinations are «chess» or «chess-like». For example, in the Figure 7 it can be seen that most combinations form «bands» in areas 12 of the cell due to the fact that cells of the same states fall into this area.
Now, out of 12 combinations, there are 4, which means only 48 for cases of using cells of 3 states with a single repeat of one state (0, 1, 2 or 3).
Translation of an image in grayscale is due to the fact that, unlike the desired elements of other types
(classes), the elements of the desired type (class) of images D can contain cells of all possible colors of
the palette provided by the ColourUnique M program. Since «chess» or «chess-like» structures form
alternating «dark» and «light» cells, converting the image to grayscale simplifies classification [
In the process of implementing the filtering algorithm for class D, the authors predicted the possibility of losing some of the sought elements if the subject uses cells of green and red colors, the shades of which often have similar degrees of lightness, which, when converted to grayscale, gives cells of the same color (Figure 8).
Now, the problem for most of the desired elements is solved by using several «filters» to identify the desired «chess» and «chess-like» elements by increasing the contrast in the light or dark side, as well as narrowing the ranges. Currently, 11 filters are used for experiments, in the future it is planned to reduce their number if some of them do not show sufficient efficiency.
Initially, the color for each cell is assigned gray according to the following formula: color = 0.299*R+ 0.587 *G + 0.114 *B.
Next, all the gray colors (from 0 to 255) of the cells are divided into 4 ranges by 3 border colors (C1, C2, C3), and the cell is assigned a number (1, 2, 3 or 4), depending on which range (0-C1, C1-C2, C2C3, C3-255) its color fell into. The resulting cell scan is displayed in four colors, depending on the range number. Each number of the range was assigned a color in advance (1: white, 2: light gray, 3: gray, 4: dark gray).
And, finally, an analysis is performed for the presence of the desired elements of class D, identifying combinations of repetitions of 2 states, the 1st state. Combinations without repetitions remain after the exclusion of their calculation of the area of definition of 2x2 combinations where all the colors are the same, where three colors are the same and where two identical colors are next to each other.
An example of how filters work to identify the desired elements of class D can be seen in the Figure
The most effective filters are presented for this sample, in each case the set of the most effective filters is different, as can be seen from the Table 3.
In Table 3: green is the filter that showed the highest efficiency for the sample, blue is the second most effective filter, yellow is the highest result for samples identified by the expert as «non-contrast creative»
The results of the hybrid classifier (neural network algorithm plus filtering algorithm) for the class of subtypes E «non-contrast creative», showed the following: subtype E «non-contrast creative» and impure types D, having many shades and gradient elements, show similar structures but have dissimilar characteristics of types of labor activity. Therefore, it was decided to programmatically separate these types, and just the joint use of neural network and filtering algorithms made this task possible. This opportunity is especially relevant for future users of the ColorUnique Pro software package who do not have a specialized artistic, psychological or design education.
The visual similarity of the subtype «non-contrast creative» and type D with creative features can be estimated in the Figure 10.
The «non-contrast creative» subtype E also uses many shades like balanced E, but often forms not gradients, but a kind of mosaic, which visually looks like a «chess-like» structure, rather pronounced «chess» elements are also present in the «non-contrast creative» subtype E, up to 34-35 % of expression after passing the filtering algorithm through 11 filters (Table 3). The examples presented in the Figure 10 in column b cannot be considered representatives of the «pure» type D, however, the degree of their severity tends to 50%, which indicates the obvious presence and possible dominance of type D over others, whose signs are also determined in the sample.
Thus, the classification of a sample by a neural network classifier to class E or D allows us to separate the types with the visual dominance of «chess-like» elements, despite the fact that both these and other samples can show approximately equally high percentages of the presence of «chess» and «chess-like» elements.
Nevertheless, the samples defined by the hybrid classifier as representatives of class D, in general, have higher percentages of the presence of «chess» and «chess-like» elements (Table 4 and Figure 11, Figure 12).
Now, if you look at the Figure 10 again, you can see that the clearer and brighter the «chess» and «chess-like» elements are, the more the type visually approaches «pure» D, and the subtype E «noncontrast creative» can be considered a transition between «pure» type D and contrast or balanced E.
In the future, it is planned to improve a hybrid classifier and, in particular, a filtering algorithm in order to create a «map» of types on which it would be possible to mark the subject in accordance with the severity of certain characteristics of types in 3D-space, expanding and individualizing recommendations and characteristics for each individual. A schematic image of the map can be seen in the Figure 13.
At this stage of development, the map of types looks flat and has only the types and subtypes derived by the authors at the moment. However, in the process of further research, experiments and expanding the capabilities of the filtering algorithm, it is possible to derive new subtypes of the main classes, as well as to derive transitional types that are not subtypes, but have specific characteristics and features of the other two main classes, for example – C type.
Now, consider the results and prospects of the conducted research: 1. The accuracy index for determining the main classes of images by a neural network classifier based on the Inception_v3 network was 91.12 %. 2. The combined use of a neural network classifier and a filtering algorithm for class D allows to increase the overall accuracy of the classification (to output the accuracy indicator as a percentage, experiment 3 is required on a wide sample of subjects), as well as to calculate the number of required elements for class D as a percentage. 3. The developed filters for identifying «chess» and «chess-like» structures allow not only to identify non-obvious elements, but also, together with analysis through a neural network classifier, automatically divide the subtypes of classes E and D. 4. As a prospect for further research, the authors highlight the possibility of creating a threedimensional map of subtypes with areas of pronounced basic types, transitional types and subtypes that have features of the structural organization of images (scans of ready-made test forms), which characterizes their work style. 5. Creating a three-dimensional map of subtypes opens up the possibility of implementing an interactive map that the subject can rotate at different angles, estimating his location relative to the main types, which gives additional information.
The paper was prepared at the departments of Nizhny Novgorod State University of Architecture and Civil Engineering and Nizhny Novgorod state technical University n. a. R. E. Alekseev, the authors thanks the heads of departments Alexander Shapoval, Evgeny Nikolsky and Vyacheslav Andreev for their help in organizing research.