Algorithms for Automated Differentiation of Subtypes and Improving the Overall Accuracy of Image Classification in Career Guidance Iuliia Tarasova 1, Vyacheslav Andreev 2, Andrey Chechin 1 and Denis Toskin 2 1 Nizhny Novgorod State University of Architecture and Civil Engineering, 65 Ilinskaya str., Nizhny Novgorod, 603950, Russia 2 Nizhny Novgorod state technical university n. a. R. E. Alekseev, Minina 24, Nizhniy Novgorod, 603950, Russia Abstract The paper discusses the problems and prospects of implementing a hybrid classifier for career guidance testing according to the author's method «Associative color space». The authors set themselves the following goals: to increase the accuracy of determining the main classes and to identify subtypes that have visual similarities, but significant differences in terms of the characteristics of the work style. To improve the accuracy of determining the problem class of images, 11 special filters were developed and implemented in order to identify implicit features of the class. The use of these filters additionally solved the problem of distinguishing visually similar subtypes of the two main classes, which makes it possible to further automate not only the definition of the main class with optimal accuracy, but also the subtype, which will allow for a more thorough diagnosis of career guidance and provide the subjects with more specific characteristics and recommendations regarding their work style. Analyzing the results of the experiments, the authors hypothesized about the possibility of creating a three-dimensional map of types and subtypes, where there could be areas of pronounced types, subtypes, as well as transitional types with special characteristics that allow avoiding the result of a «mixed» type for the subjects. Keywords 1 career guidance, software package, neural network, hybrid classifier, algorithm, subtype 1. Introduction The «Associative color space» testing method is a classification of individuals by type of individual style of activity (ISA) [1]. This method allows us to trace the correlation with the Holland’s method [2] and provide the recipient with a sufficiently detailed characteristic, since it is possible to predict the result of the Holland’s method at the same time by studying the data from the results of the «Associative color space» method and conversely. Thus, the «realistic» type according to the method is highly likely to correspond to the «rational» type according to the «Associative color space» testing method, and the «enterprising» type – to the «skeptical» or «avant-garde» types, respectively. The purposes of this paper: 1. To analyze the results of the joint work of the neural network classifier and the filtering algorithm using special filters in terms of improving the classification accuracy. GraphiCon 2021: 31st International Conference on Computer Graphics and Vision, September 27-30, 2021, Nizhny Novgorod, Russia EMAIL: tar06@list.ru (I. Tarasova); vyach.andreev@mail.ru (V. Andreev); chechin-a@yandex.ru (A. Chechin), toskin.dv@gmail.com (D. Toskin) ORCID: 0000-0003-4770-9431 (I. Tarasova); 0000-0002-7557-352X (V. Andreev); 0000-0001-8038-9404 (A. Chechin) 0000-0002-8083- 8617 (D. Toskin) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 2. To analyze the results of an experiment with a test sample of subtypes of class E «non-contrast creative» and representatives of class D with signs of a creative type on the presence of criteria that allow dividing these types of individuals automatically. 3. Identify the prospects for further research. The fact is that in some classes, the neural network classifier works more effectively, and in some others, the filtering algorithm works better, which creates a prerequisite for the formation of a hybrid classifier as a final tool for career guidance testing. In addition, the use of a hybrid classifier makes it potentially possible to automate the division not only into main classes, but also subtypes within them. 2. Raw data of experiments Consider the desired types (classes) of images using the «Associative color space» testing method: type A («skeptical»), type B («medium avant-garde»), C («skeptical-creative»), D («rational»), E («creative»), F («radically avant-garde») [3]. Only individuals of type F use the «cut» tool when working with a test form, so it is determined already in the process of filling out the form and does not pass classification by processing neural network and filtering algorithms. This leaves 5 types of individuals to classify. The A, B, C, D, and E types differ mainly in the distribution of dark and light pixels within the planigon [4]. The complexity of determining the evaluation criteria lies in the features of the color distribution, and the complexity is also represented by types containing elements of several desired classes (C, E types). The object of research and testing of algorithms are sweeps of ready-made test forms (Figure 1). Sweeps allow us to consider all areas of the planigon as equivalent, excluding distortions present in quasi-space. The A type characterized by a predominant (50% or more, respectively 100% – pure type) presence of elements «horizontal lines» (Figure 1b.). Lines (from 3 cells in length) can be either the colors of strictly identical coordinates, or the colors of different coordinates, but belonging to the same tone, for example, «yellow». Tone – a range of colors that are indicated in the color circle as «yellow», «red», «orange», and so on. It is also the tone that gives the color its name. In the current sample, elements of A type are mostly represented by colors of the same coordinates, since the neural network classifier is undergoing training and first tests, the types are mostly «bright». Another condition for the presence of A type is a small total number of shades (no more than one tone with the number of shades of 5 or more). There are the following signs of dominant A type: 1. 50% or more of the scan is filled with the desired «horizontal line» elements (from three cells of the same color or tone). 2. A small number of shades, namely no more than one tone with a number of shades of 5 or more. 3. The fewer shades applied by the recipient, the more pronounced type A is considered. a b Figure 1: The A type: a – quasi-space, b – sweep The B type characterized by the presence (3 elements per scan or more) of the «vertical lines» elements (from three cells in height) with identical coordinates (Figure 2). The search elements have a minimum of 3 cells in height, they can be up to 6, respectively, the definition area can reach 6x1. The total number of shades can be very different, the main thing-3-fold or more presence of an element from 3 cells in height of exactly the same color (Figure 2). There are the following signs of dominant B type: the presence of 3 or more elements of the «vertical lines». a b Figure 2: The B type: a – quasi-space, b – sweep The C type characterized by the presence of the elements «horizontal lines» (Figure 3b) and «wide» or «narrow» gradients in the definition areas 33 and 23 (for wide) and 31, 41, 51, 61 (for narrow ones) (Figure 3b, Figure 5b). Lines (from 3 cells in length) can consist of colors of exactly the same coordinates, or colors of different coordinates, but belonging to the same tone (Figure 1b). «Wide gradients» are bands of colors of exactly the same coordinates in width and colors of the same tone range in length, for example, gradients of green and red colors in the definition areas 33 and 23 in the Figure 3b. The total number of shades may vary, but there are generally 4-5 shades for 2-3 tones. For example, in the Figure 5b, the scan shows from 3 shades for red, green, purple and orange tones. There are the following signs of dominant C type: 1. Simultaneous presence of elements of «wide» or «narrow» gradients and «lines». 2. The minimum number of elements for determining the type of 2 lines and one wide gradient 33 and 23, or 2 lines and two narrow gradients. 3. Relatively high variety and number of shades. a b Figure 3: The C type: a – quasi-space, b – sweep The D type characterized by the presence of elements «chess» or «chess-like» (Figure 4). The «chess» and «chess-like» elements are arranged by a special arrangement of dark and light cells in the area of definition 22. Moreover, «chess» elements can contain any color. The total number of shades is usually small (the brighter the type, the fewer shades). There are the following signs of dominant D type: 1. The presence of «chess» or «chess-like» elements on 50 or more percent of the scan area. 2. A large number of cells with repeated colors of the same coordinates, especially black, white, yellow, red and blue. 3. Using a relatively larger number of colors from the «basic» section of the ColourUnique M software module compared to other types. a b Figure 4: The D type: a – quasi-space, b – sweep The E type mainly characterized by the presence of a large number of shades of selected colors, namely – from 5 shades for at least two tones. Representatives of this type are distinguished by the presence of mainly «narrow» gradients (Figure 5b), less often there are «wide» gradients (Figure 3b). There are the following signs of dominant E type: 1. The presence of a large number of shades, namely – from 5 shades for at least two tones. 2. The predominant presence of «narrow», sometimes «wide» gradients. a b Figure 5: The E type: a – quasi-space, b – sweep As can be seen from the examples, the desired elements differ in the shape of the definition areas and their orientation in space, as well as have features of the color system. That is why the authors decided to use convolutional neural networks, which allow detecting signs of a particular class in any area of the analyzed image. In experiment 1, the architecture of the Inception_v3 convolutional network was used. For experiment 2, the Unet network is trained. The number of images for each experiment is 350 taking into account augmentation [5]. 3. Classification methods To obtain and analyze the data, such procedures as observations and experiments with a neural network and a filtering algorithm were used, the results of machine analysis were compared with the results of the analysis of an expert (the developer of the methodology). 3.1. Neural network classifier Previously untrained neural networks were used to classify individual types using the «Associative color space» testing method. Such architectures as MobileNet [5], Inception_v3 [6] and Unet [7] were chosen for experiments. Experiments 1 and 2 were conducted with the Inception_v3 and Unet networks, respectively, the stochastic gradient descent was used as an optimizer: 1 𝑄(𝑤) = 𝑛 ∑𝑛𝑖=1 𝑄𝑖 (𝑤), (1) where the parameter w that minimizes Q(w) should be evaluated [8]. The neural network (for experiment 1) diagram for classifying images (ready-made test forms) is shown in the Table 1 the architecture of the Inception_v3 network was taken as the basis, then 4 fully connected layers were attached to it. The first layer has 64 neurons, the second has 32 neurons, and then there is the dropout parameter, which is responsible for preventing overtraining. This parameter, with a set probability of 0.1, «turns off» the neuron so that the neurons that are with it do not focus on it, but act on their own. The fourth layer contains 16 neurons. The final layer contains 5 neurons – 1 neuron is responsible for 1 specific type (class) of the image. Table 1 The table of the first neural network for career guidance testing based on the Inception_v3 network Layer (type) Output Shape Param # inception_v3 (Mode 1) (None, 3, 1, 2048) 21802784 flatten (Flatten) (None, 6144) 0 dense_one (Dense) (None, 64) 393280 dense_two (Dense) (None, 32) 2080 dropout_one (Dropout) (None, 32) 0 dense_three (Dense) (None, 16) 528 output (Dense) (None, 5) 85 The Unet architecture (for experiment 2) is shown in the Figure 6. It consists of a narrowing (left) and expanding (right) path. The narrowing path involves re-applying two 3×3 convolutions, followed by a rectified linear block (ReLU) and a maximum join operation (2×2 of degree 2) to lower the resolution. Figure 6: Unet network architecture At each step of downsampling, the property channels are doubled. Each step in the expanding path consists of a step-up discretization operation of the property map, followed by: 1. 2×2 convolution, which reduces the number of property channels. 2. Merge with the appropriately cropped property map from the shrinking path. 3. Two 3×3 convolutions followed by ReLU. On the last layer, a 1×1 convolution is used to map each 64-component property vector to the desired number of classes. In total, the network contains 23 convolutional layers. For training, 1470 images were used, for validation – 315 (63 images for each class). The classification accuracy of each of the networks can be seen in the Table 2. Table 2 The comparison of accuracy for three neural networks Inception_v3 Unet 91,12 % 71,05% The conducted experiments show that the maximum classification accuracy using the Unet network does not exceed 71,05 %. As a solution to the problem, the authors consider the possibility of adding new classification layers or replacing the optimizer. The Inception_v3 network has an accuracy of 91,12 %. Class D became the most problematic for neural network algorithms and, despite the fact that according to the results of the last experiment, it was possible to increase the classification accuracy in class D to 62% against the previous 45%, the accuracy of determining samples of class D remains unsatisfactory. Let us consider the joint operation of the filtering algorithm and the neural network algorithm to improve the accuracy of determining objects of class D, as well as to differentiate some subtypes. 3.1.1. Filtering algorithm The filtering algorithm is developed in parallel with the neural network algorithm in the future, both classifiers will pass comparative tests, which will result in the selection of the main algorithm, while the second one will be a supporting or alternative class, where they were shown the best result. Consider a filter developed for the desired class D, where at the moment the neural network does not classify images reliably. Filtering is based on the principle of comparing regions 22 of the planigon cell with samples of «chess» and «chess-like» combinations consisting of cells of 4 states (Figure 7). Figure 7: Example of «chess-like» combinations of series 1102 To count the number of combinations, we used permutation schemes with and without repetitions. 𝑷𝒏 = 𝒏! , (𝒏𝟏 +𝒏𝟐 +...+𝒏𝒌)! 𝑷(𝒏𝟏 , 𝒏𝟐 , … , 𝒏𝒌 ) = , (2) 𝒏𝟏 !∙𝒏𝟐 !∙…∙𝒏𝒌 ! where n – the number of elements, Pn – the number of permutations, and P(n1,..., nk) – the number of permutations with multiplicities n1,...,nk [9]. Received: 1. 24 combinations for cases of using cells of all 4 conditions for the 2x2 definition area without repetitions. 2. 144 combination for the use cases of the cells 3 conditions with a single repeat of one condition (0, 1, 2 or 3). 3. 24 combinations for cases where 2 condition cells are used for the definition area 22 with a single repeat of two conditions (0, 1, 2 or 3). Finally, there is 192 combinations. However, not all of the resulting combinations are «chess» or «chess-like». For example, in the Figure 7 it can be seen that most combinations form «bands» in areas 12 of the cell due to the fact that cells of the same states fall into this area. Now, out of 12 combinations, there are 4, which means only 48 for cases of using cells of 3 states with a single repeat of one state (0, 1, 2 or 3). Translation of an image in grayscale is due to the fact that, unlike the desired elements of other types (classes), the elements of the desired type (class) of images D can contain cells of all possible colors of the palette provided by the ColourUnique M program. Since «chess» or «chess-like» structures form alternating «dark» and «light» cells, converting the image to grayscale simplifies classification [10]. In the process of implementing the filtering algorithm for class D, the authors predicted the possibility of losing some of the sought elements if the subject uses cells of green and red colors, the shades of which often have similar degrees of lightness, which, when converted to grayscale, gives cells of the same color (Figure 8). Figure 8: The partial loss of red-green and red-blue cells when converting an image to grayscale Now, the problem for most of the desired elements is solved by using several «filters» to identify the desired «chess» and «chess-like» elements by increasing the contrast in the light or dark side, as well as narrowing the ranges. Currently, 11 filters are used for experiments, in the future it is planned to reduce their number if some of them do not show sufficient efficiency. Initially, the color for each cell is assigned gray according to the following formula: color = 0.299*R+ 0.587 *G + 0.114 *B. Next, all the gray colors (from 0 to 255) of the cells are divided into 4 ranges by 3 border colors (C1, C2, C3), and the cell is assigned a number (1, 2, 3 or 4), depending on which range (0-C1, C1-C2, C2- C3, C3-255) its color fell into. The resulting cell scan is displayed in four colors, depending on the range number. Each number of the range was assigned a color in advance (1: white, 2: light gray, 3: gray, 4: dark gray). And, finally, an analysis is performed for the presence of the desired elements of class D, identifying combinations of repetitions of 2 states, the 1st state. Combinations without repetitions remain after the exclusion of their calculation of the area of definition of 2x2 combinations where all the colors are the same, where three colors are the same and where two identical colors are next to each other. An example of how filters work to identify the desired elements of class D can be seen in the Figure 9. Figure 9: Different filters for detecting lost cells The most effective filters are presented for this sample, in each case the set of the most effective filters is different, as can be seen from the Table 3. Table 3 Efficiency of different filters for different samples the sample 1 2 3 4 5 6 7 8 9 10 the filter Dark 80% 18 % 34% 17% 9% 19% 21% 20% 18% 13% Dark 2 100 % 72 % 32% 33% 0% 15% 13% 20% 11% 29% Dark 3 40 % 0% 30% 4% 9% 19% 17% 0% 7% 0% Light 80% 18% 29% 6% 0% 9% 23% 0% 15% 12% Light 2 40 % 0% 24% 4% 5% 27% 17% 0% 7% 0% Light 3 20 % 0% 7% 2% 11% 5% 15% 0% 11% 0% Medium 40 % 0% 26% 4% 9% 24% 17% 0% 7% 0% Medium 2 40 % 18% 22% 9% 9% 15% 20% 0% 18% 15% Medium 3 80% 73% 29% 16% 9% 17% 18% 20% 18% 24% Medium 4 80% 36% 31% 28% 9% 9% 19% 40% 15% 20% Dark 4 80% 36% 25% 14% 0% 22% 9% 40% 11% 25% The results 100 % 73% 34% 33% 11% 27% 23% 40% 18% 29% In Table 3: green is the filter that showed the highest efficiency for the sample, blue is the second most effective filter, yellow is the highest result for samples identified by the expert as «non-contrast creative» The results of the hybrid classifier (neural network algorithm plus filtering algorithm) for the class of subtypes E «non-contrast creative», showed the following: subtype E «non-contrast creative» and impure types D, having many shades and gradient elements, show similar structures but have dissimilar characteristics of types of labor activity. Therefore, it was decided to programmatically separate these types, and just the joint use of neural network and filtering algorithms made this task possible. This opportunity is especially relevant for future users of the ColorUnique Pro software package who do not have a specialized artistic, psychological or design education. The visual similarity of the subtype «non-contrast creative» and type D with creative features can be estimated in the Figure 10. a b Figure 10: a – subtype E «non-contrast creative», b – type D with creative features The «non-contrast creative» subtype E also uses many shades like balanced E, but often forms not gradients, but a kind of mosaic, which visually looks like a «chess-like» structure, rather pronounced «chess» elements are also present in the «non-contrast creative» subtype E, up to 34-35 % of expression after passing the filtering algorithm through 11 filters (Table 3). The examples presented in the Figure 10 in column b cannot be considered representatives of the «pure» type D, however, the degree of their severity tends to 50%, which indicates the obvious presence and possible dominance of type D over others, whose signs are also determined in the sample. Thus, the classification of a sample by a neural network classifier to class E or D allows us to separate the types with the visual dominance of «chess-like» elements, despite the fact that both these and other samples can show approximately equally high percentages of the presence of «chess» and «chess-like» elements. Nevertheless, the samples defined by the hybrid classifier as representatives of class D, in general, have higher percentages of the presence of «chess» and «chess-like» elements (Table 4 and Figure 11, Figure 12). Table 4 The ratio of neural network classes and percentages of the severity of the staggered structure of the filtering algorithm the sample 001 002 003 004 005 006 007 008 009 0010 the results Percentage of the 34% 33% 27% 23% 33% 31% 17% 40% 40% 17% presence of elements D (filtering) Class (neural D E E E D D E D D E network) Figure 11: Diagram of the ratio of the severity of «chess» and «chess-like» elements in classes D with creative pairs and subtype E «non-contrast creative» Figure 12: Block diagrams of subtype differentiation Now, if you look at the Figure 10 again, you can see that the clearer and brighter the «chess» and «chess-like» elements are, the more the type visually approaches «pure» D, and the subtype E «non- contrast creative» can be considered a transition between «pure» type D and contrast or balanced E. In the future, it is planned to improve a hybrid classifier and, in particular, a filtering algorithm in order to create a «map» of types on which it would be possible to mark the subject in accordance with the severity of certain characteristics of types in 3D-space, expanding and individualizing recommendations and characteristics for each individual. A schematic image of the map can be seen in the Figure 13. At this stage of development, the map of types looks flat and has only the types and subtypes derived by the authors at the moment. However, in the process of further research, experiments and expanding the capabilities of the filtering algorithm, it is possible to derive new subtypes of the main classes, as well as to derive transitional types that are not subtypes, but have specific characteristics and features of the other two main classes, for example – C type. Figure 13: Predicted view of the subtype map 4. Conclusions Now, consider the results and prospects of the conducted research: 1. The accuracy index for determining the main classes of images by a neural network classifier based on the Inception_v3 network was 91.12 %. 2. The combined use of a neural network classifier and a filtering algorithm for class D allows to increase the overall accuracy of the classification (to output the accuracy indicator as a percentage, experiment 3 is required on a wide sample of subjects), as well as to calculate the number of required elements for class D as a percentage. 3. The developed filters for identifying «chess» and «chess-like» structures allow not only to identify non-obvious elements, but also, together with analysis through a neural network classifier, automatically divide the subtypes of classes E and D. 4. As a prospect for further research, the authors highlight the possibility of creating a three- dimensional map of subtypes with areas of pronounced basic types, transitional types and subtypes that have features of the structural organization of images (scans of ready-made test forms), which characterizes their work style. 5. Creating a three-dimensional map of subtypes opens up the possibility of implementing an interactive map that the subject can rotate at different angles, estimating his location relative to the main types, which gives additional information. 5. Acknowledgements The paper was prepared at the departments of Nizhny Novgorod State University of Architecture and Civil Engineering and Nizhny Novgorod state technical University n. a. R. E. Alekseev, the authors thanks the heads of departments Alexander Shapoval, Evgeny Nikolsky and Vyacheslav Andreev for their help in organizing research. 6. References [1] M. V. Grigorieva, Psychology of labor. Abstract of lectures, Higher education, Arkhangelsk, 2006. [2] Proforientator.ru Career guidance tests, 2021. URL: https://proforientator.ru/tests/. [3] I.S. Tarasova, A.V. Chechin, V.V. Andreev, Implementation of algorithms of image analysis in the software package ColourUniquePRO with the aim of increasing the accuracy of classification types individuals, in: Proceedings of the 29th International Conference on Computer Graphics and Vision, Bryansk, 2019, pp. 189-193. doi: 10.30987/graphicon-2019-2-189-193 [4] V. A. Utrobin, Computer image processing. Information models of the understanding stage: studies.manual, NSTU, N. Novgorod, 2006. [5] G. A. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, MobileNets: Efficient Convolutional Neural Networks for Mobile VisionApplications, 2017. URL: https://arxiv.org/pdf/1704.04861.pdf. [6] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, 2014. URL: https://arxiv.org/pdf/1409.4842.pdf. [7] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, 2015. URL: https://arxiv.org/pdf/1505.04597.pdf. [8] P. E. Gladilin, K. O. Bochenina, Machine learning technologies. Educational and methodological manual, 2020. URL: https://books.ifmo.ru/file/pdf/2633.pdf. [9] V. L. Topunov, Combinatorics. Workshop on solving problems, MPSU, Moscow, 2016. [10] V. V. Andreev, I. S. Tarasova, A.V. Chechin, Problems and prospects of implementation of the algorithm of classification of test forms of the color Unique Pro software complex, in: Information systems and technologies, N. Novgorod, 2020, pp. 913-918. In Russian. URL: https://www.nntu.ru/frontend/web/ngtu/files/news/2020/05/12/ist2020/sbornik_ist2020.pdf.