Conceptual-visual metalanguage of hybrid intelligent systems

Abstract

The urgency of the metalanguage is caused by the development of visually-shaped representations and reasoning in hybrid and synergetic intelligent systems. The absence of formalisms determines the high science intensity of special environments for manipulating and processing visual models. The formalization of the metalanguage is a condition for the development of hybrid intelligent systems with a heterogeneous visual field; it ensures cooperation, relativity, the complementarity of collective natural and artificial intelligence capable of visual thinking and speaking in verbal-symbolic languages.

1. Introduction

The long experience of the Kaliningrad school of hybrid artificial intelligence confirms not only the advantages of functional hybrid intelligence systems (FHIS), but also contrasts their shortcomings, for example, interaction with users and specialists only through symbolic-logical methods of representing information that practically does not activate the visually-shaped, right hemisphere reasoning of the decision-maker (DM). Visualization and, in particular, schematization provides understanding and explanation of problems and their solutions by FHIS, activates the mechanisms of intuitive, insightful thinking, which is especially important in the context of diversity of opinions. Visualization of complex structure and dynamics of changing problem situations, the principle of "seeing the problem at a glance" will allow to act promptly.

Visually-shaped representations and cognitive modeling were considered in papers by D.A. Pospelov, A.A. Zenkin, G.P. Shchedrovitsky, Yu.R. Val'kman, B.A. Kobrinsky, O.P. Kuznetsov, G.S. Osipov, V.B. Tarasov, I.B. Fominykh, Т.А. Gavrilova, A.E. Yankovskaya, etc. Visual languages are developed for functional programming, programming by examples, finite automata, data flows and other domains [ 1 ]. The implementation of these languages requires considerable effort, development in each case of special environments for creation, manipulation and processing of visual models. To reduce them, the formalized model of visual language based on the principles of system theory and system analysis is proposed. It should be considered within scope of I.B. Fominykh’s opinion: "Solving problems of representation of right hemispheric mechanisms by left hemispheric means, research and modeling of the interaction of figurative and symbolic-logical thinking, the forming methods of figurative thinking computer support are of considerable interest".

2. Modeling of visually-shaped representations as semiotic system

The concept of the semiotic system D.A. Pospelov, G.S. Osipov [ 2 ] is applied for modeling of reasoning on visually-shaped schematic images [ 2 - 6 ]: vl   VT ,VS,VA,VP, υτ, υσ, υα, υπ , (1) where VT , VS , VA are sets of basic symbols, syntactic rules and axiom-knowledge about the subject domain (semantic rules) respectively; VP is a set of rules for inference of solutions (pragmatic rules); υτ , υσ , υα , υπ are sets of rules for changing sets VT , VS , VA , VP respectively. The sets VT,VS,VA,VP, υτ, υσ, υα, υπ in equation (1) are defined by the expressions:

VT   P, D,VR  ,

VS   VT ,VN, PRU  ,

VA   DO,GRES,GACT ,GPR ,GR  , DO   RES, ACT , PR, R , GRES : RES  P , GACT : ACT  P , GPR : PR  D , GR : R VR , VP  { AG, act, M ,W },

υτ   P, D, VR  , υσ   υτ, VN, PRU  , υα   DO,GΔRES,GΔPR ,GΔR  , DO   RES, ACT , PR, R  , GΔRES : RES  P , GΔCT : ACT  P , GΔPR : PR  D , GΔR : R  VR ,

υπ  { AG, act, M , W } , where, in addition to the previously introduced notation P is a set of visual primitives; D is a set of visual dimensions characterizing visual primitives; VR is a set of visual relations between one or more primitives [ 4 ]; VN is a dictionary of nonterminal symbols; PRU is a set of production rules; RES , ACT , PR , R are sets of concepts of resources, actions, properties and relations, respectively; AG is a set of models of native speakers (experts, elements, agents), to which the behavior norm is addressed (various social prohibitions and restrictions imposed by the community on a separate speaker); act  ACT is the action, defined on the set of actions ACT , the object of normative regulation (the content of the norm); M is a set of systems of modalities associated with the action, for example, the system of norms expressed by deontic modalities: M N  {ma, al,in, proh} , where ma is "mandatory", al is "allowed", in is "indifferent", proh is "prohibited"; W is a set of models of worlds in which the norm is applicable (the conditions of application, the circumstances in which the action should or should not be performed) [ 7 ]; P , D , VR , VN , PRU , RES , ACT , PR , R , AG , M , W are the sets of admissible changes of the sets P , D VR , VN , PRU , RES , ACT , PR , R , AG , M , W respectively; act is a set of permissible changes in the content of the norm act .

As shown in [8], the languages of professional activity are poly-languages [9]. This is due to the inherent structure of the external world and the specificity of the asymmetry of verbal-sign and visually-shaped representations of resources, properties, actions, structures, situations, states, the behavior of the control object and, taking into account the subject of management activities, of goals, tasks, plans, and assessments. In this paper, visually-shaped representations are considered as a multilayered hierarchy of semiotic systems.

3. The multilayered model of conceptual-visual metalanguage of visually-shaped representations

In [9] the family of verbal-symbolic languages for description of resources, operations, structures, situations, states, behavior of the control object, as well as goals, plans and problems is represented. In [10] eight levels of visual languages are allocated for the implementation of automated reasoning in intelligent systems: 1) conceptual and visual basis vl1 ; 2) resources, actions and properties vl2 ; 3) hierarchies of resources, actions and properties vl3 ; 4) spatial and production structures vl4 ; 5) states, situations and events vl5 ; 6) tasks and problems vl6 ; 7) experts’ reasoning models vl7 ; 8) integrated models of collective intelligence reasoning vl8 . In this case, the developer has a set of components for constructing conceptual-visual metalanguage, describing the complex problem solving by combining several interrelated processes of reasoning in different languages. Depending on the requirements of the problem some levels could be missed. Thus, the conceptual-visual metalanguage can be formally represented by the expression: where VLR is a set of relations between language elements vlk , k  Ґ , k [ 1, 8 ] .

The metalanguage is visualized by the "layered pie" (Figure 1) of semiotic systems in which the image signs (below the images) reflect reality at two image reflection levels of three distinguished in [11]: the level of representations (secondary images of objects) and verbal-logical (symbolic-logical) thinking. At the bottom of the "layered pie" are dictionaries of concepts and relations, a conceptualvisual basis on which the family of descriptive languages arranged in levels is built.

Each layer of image representation (Figure 1) is associated with a semiotic system containing: 1) the core of the base signs (terminal alphabet) VT k ; 2) derived signs VN k , formed according to the rules PRU k . The visual core of the higher layer languages can contain the signs of the core of the lower layer VT k  VT k1 and signs formed outside the core of the lower layer language VT k1 VN k   , k  Ґ , k [ 1, 7 ] .

Let's consider interrelations of the images formed by syntactic rules, at various layers of metalanguage.

The conceptual-visual basis of the language is in the first layer. It’s a set of basic concepts and forms needed to build images on higher layers.

The first layer language vl1 uses heuristic rules PRU 1 for constructing of images of derivative (composite) relations vr1 VR1  VT 1 from P1 , D1 and VR1 :

vl1(P1, D1,VR1, PRU1) {vr1}.

At the second layer language vl2 , heuristics PRU 2 are used to generate images of resources res2  RES 2  VT 2 , actions act 2  ACT 2  VT 2 and properties pr2  PR2  VT 2 without regard for their hierarchy using the definition relations VR11  VR1 :

vl2 (P1, D1,VR11, PRU 2 )  RES 2  PR2  ACT 2.

At the third layer, inclusion relations VR51  VR1 and heuristics PRU 3 formalize the hierarchies of resources res3  RES 3  VT 3 , actions act3  ACT 3  VT 3 and properties pr3  PR3  VT 3 :

The fourth layer formalizes the spatial str14  STR14  VT 4 , operational and technological str34  STR34  VT 4 structures on the basis of images defined at previous layers using temporal VR31  VR1 , spatial VR41  VR1 and cause-effect VR61  VR1 relations, as well as heuristics PRU 4 : vl4 (P1, D1, RES3, PR3, ACT 3,VR31,VR41,VR61, PRU 4 )  STR14  STR34.

At the fifth layer heuristics PRU 5 formalize images of situations sit5  SIT 5  VT 5 and signs of states st5  ST 5  VT 5 :

vl5 (STR14 , STR34 , PRU 5 )  SIT 5  ST 5.

At the sixth layer, images of homogeneous

prbh 6  PRBh 6  VT 6 and heterogeneous prbu 6  PRBu 6  VT 6 problems are specified on the basis of image images of the previous layers and heuristics PRU 6 :

vl6 (P1, D1, RES3, PR3, ACT 3,VR1, ST 5, PRU 6 )  PRBh 6  PRBu 6.

At the seventh layer, heuristics PRU 7 form images of meta 7  MET a 7  VT 7 for solving problems simulating the expert's reasoning: autonomous methods vl7 (P1, D1, RES3, PR3, ACT 3,VR1, PRU 7 )  MET a 7.

At the eighth layer, integrated methods met u 8  MET u 8  VT 8 for solving problems simulating the reasoning of the team of experts are specified, based on images of the previous layers and heuristics PRU 8 :

vl8 (P1, D1, RES3, PR3, ACT 3,VR1, ST 5, SIT 5, PRBh 6, PRBu 6, MET a 7 , PRU 8)  MET u 8.

Such multi-layer model of conceptual-visual language is a tool for a complex description of the domain of various degrees of generalization, which forms it from a set of simpler models. As an example, let us consider the use of this model for describing the alphabet of the first two layers of the metalanguage for the FHIS of the operational and dispatching regional power systems management.

4. Basic layers of conceptual-visual metalanguage

Analysis of papers on visual control, cognitive graphics, methods of information visualization [ 3, 6, 10 ] made it possible to identify the main figures underlying the conceptual-visual metalanguage (Figure 2a), and the set of pictograms [12], examples of which are shown in Figure 2b, for constructing schematized images of resources, properties and actions in the management of the power system.

(a) (b)

A point is the basis of measurements; it generates a line, a movement. A straight line is the component of all geometric figures. The circle is a universal symbol of integrity, continuity and initial perfection. The square symbolizes thing or resource category [12], the triangle symbolizes property category and the arrow symbolizes action category. These forms in combination with the plane, color, texture, the set of pictograms that represent the visual "names" of concepts, as well as syntactic rules for recording visual role relationships VR are enough for constructing of a schematic image of any complexity.

At the first layer vl1 of the metalanguage the derivative primitives, dimensions and relations are constructed with heuristic rules PRU 1 from the kernel's graphic primitives. As can be seen from Figure 2a limited, relatively small set of graphic primitives is required to represent the basic elements of the visual meta-language of power system management: straight line segment, circle, pictogram and filling (Figure 3). These primitives in turn can be described by the set of points that make up them. The relation of the definition vr11 is represented graphically by a rectangle divided into two parts: the top contains the primitive to be defined, and the lower one contains definition. Formally this relation is written with the sign "=".

The point in this paper is understood as a square with the size of one pixel with coordinates on the plane and color. For images drawn by hand fuzzy points and the methods of fuzzy geometry should be used. The primitive "point" p11 has visual dimensions: the coordinates on the plane d11 [0, d11max ] , d21 [0, d21max ] (Figure 4a), where, d1max , d2max are the maximum values of the coordinates; the color 1 1 d31 [0,100] on the scale "Grayscale" (Figure 4b). The point is displayed according to the rule: p11(d11, d21, d31)  pru11(d11, d21, d31).

When describing graphic primitives in this paper, the coordinates d11 and d21 are given in the local coordinate system of the primitive. When constructing a schematic image, the local coordinates of the primitive are recalculated into the image coordinate system, and primitive rotation and scaling can be performed. These operations are not considered in the present paper because of the absence of semantic load.

According to the rule (2) of generating the visual primitive "point" p11 , the primitive "line segment" can defined: p12 ( p11b , p11e , d31)   p11(d11, d21, d31) | d21  (d11  d11b )(d21e  d21b )(d11e  d11b )1  d21b , d11b  d11( p11b ), d21b  d21 ( p11b ), d11e  d11( p11e ), d21e  d21 ( p11e ), d11 [d11b , d11e ], d21 [d21b , d21e ], d11, d21  ў , where  x is the rounding operation of the number x to the nearest integer.

The primitive “circle” is defined by the expression: p31 ( p11c , p11e , d31)   p11(d11 , d21 , d31) |(d11  d11c )2  (d21  d21c )2  (d11e  d11c )2  (d21e  d21c )2 d11c  d11( p11c ), d21c  d21 ( p11c ), d11e  d11 ( p11e ), d21e  d21 ( p11e ).

The fill is described by the expression:

(2) p11h, p11l {p11}, d11h  d11(p11h), d21h  d21(p11h), d11l  d11(p11l), d21l  d21(p11l), d11,d21 ў.

Graphical representation of the pictogram is a set of points corresponding to its raster image; it’s generated by the rule pru2 :

p51(d41,d31, p11dl, p11tr) {p11(d11,d21,d31)| p11(d11,d21,d31) pru12(d41,d31, p11dl, p11tr)}, where d41 is the bitmap image of the pictogram , d31 is its color, p11dl , p1tr are the lower left and the 1 upper right points.

At the second layer of the metalanguage images of resources, actions, properties and relationships are formed from the visual primitives of the first layer (Figure 4).

The graphical representation of the concept of "resource" res2 RES2 (Figure 4a) can be formally represented by the expressions: res2(p11dl, p11tr,d31)  r12respr(res2, p11dl)or12respr(res2, p11tr)or12respr(res2,d31)   p12(p11dl, p11dr,d31)  p12(p11dr, p11tr,d31)  p12(p11tr, p11tl,d31)  p12(p11tl, p11dl,d31), p11dr  p11(d11(p11tr),d2(p11dl),d31), p11tl  p11(d11(p11dl),d21(p11tr),d31),

1 d11dl  d11(p11dl), d21dl  d21(p11dl), d11tr  d11(p11tr), d21tr  d2(p11tr), 1 where o is the operation of gluing concepts [8], p11dl, p11tr are the coordinates of the vertices of the element (Figure 5a), r12respr is the relation "to have a property" of the second layer of the language of the class "resource-property" [9].

The concept "property" pr2 PR2 (Figure 5b) is represented by the expressions: pr2(p11dl, p11t, p11dr, p51,d31)  r12prpr(pr2, p11dl)or12prpr(pr2, p11t)or12prpr(pr2, p51)o or2prpr(pr2,d31)  p12(p11dl, p1t,d31)  p12(p1t, p11dr,d31)  p12(p11dr, p11dl,d31)  p51,

1 1 1 p12(p11dl, p11t,d31)  p51 1, p12(p11t, p11dr,d31) p51 1, p12(p11dr, p11dl,d31)  p51 …1, where p11dl, p1 , p11dr are the coordinates of the vertices of the element, p51 is the inscribed icon, r12prpr is 1t the relation "to have a property" of the second layer of the language of the class "property-property".

The concept "action" act2 ACT2 (Figure 5c) is represented by the expressions: act2(p11dl, p11tr,d31)  r12actpr(act2, p11dl)or12actpr(act2, p11tr)or12actpr(act2,d31)  p12(p11dl, p11dr,d31)  p12(p11dr, p11cr,d31) p12(p11cr, p11tr,d31)  p12(p11tr, p11tl,d31)  p12(p11tl, p11dl,d31),

p11dr  p11(d11(p11tr),d21(p11dl),d31), p11tl  p11(d11(p11dl),d21(p11tr),d31), p11сr  p11(d11(p11dr)  0,5(d21(p11tr)  d21(p11dl)),d21(p11dr)  0,5(d21(p11tr)  d21(p11dl)),d31), where p11dl, p11tr are the coordinates of the vertices of the element (Figure 5c), r12actpr is the relation "to have a property" of the second layer of the language of the class "action-property".

The visual relation "resource-resource" vr2res res (resa2 , resb2 ) (Figure 5d) is formalized by the expression:

vr 2res res (resa2 , resb2 )  (resa2 , resb2 ) |p11bp11ahp11al (d11al„ d11b„ d11ah )  p11bp11ahp11al (d21al„ d21b„ d21ah ), p11ah , p11al  resa2 , d11ah  d11( p11ah ), d21ah  d21 ( p11ah ), d11al  d11( p11al ), d21al  d21 ( p11al ), p11b  resb2 , d11b  d11( p11b ), d21b  d21 ( p11b ), d11, d21  ў .

Formalized descriptions of visual relations "resource – property", "resource – action", "property – resource", "property – property", "property – action", "action – action" (Figures 5e-5i, 5l) are analogous to the equation (3) for the visual "resource – resource" relation, therefore, they are not given here. The visual relation "action – resource" (Figure 5j) is represented by the expression: vr2act res (acta2 , resb2 )  (acta2 , resb2 ) | p11adl  p1bdl  p1atr  p11btr , p1adl  p11dl (acta2 ),

1 1 1 p1atr  p11tr (acta2 ), p1bdl  p11dl (resb2 ), p1btr  p11tr (resb2 ),

1 1 1 and а visual relation "action – property" (Figure 5k) is represented by the expression: vr 2act pr (acta2 , prb2 )  (acta2 , prb2 ) |( p11acr  p11bt )  ( p11atr  p11bdl )  ( p11adr  p11bdr ), p1aсr  p11(d11( p11adr )  0,5(d21 ( p11atr )  d21 ( p1adl )), d21 ( p11adr )  0,5(d21 ( p11atr )  d21 ( p1adl )), d31), 1 1 1 p1atr  p11tr (acta2 ), p1adr  p11(d11( p1atr ), d21 ( p1adl ), d31), p1adl  p11dl (acta2 ), p1bdl  p11dl ( prb2 ), 1 1 1 1 1 1 p11bt  p11t ( prb2 ), p1bdr  p11dr ( prb2 ).

1

Thus, the visual primitives (Figure 5), which constitute the alphabet (core) of the second layer of the metalanguage for FHIS of the regional power systems management, are formally defined. On their basis, a set of visual primitives is defined, examples of which are shown in Figures 6a-6d. It’s the dictionary VN 2 of the second layer of the metalanguage. From the primitives in Figures 6a-6d, in turn, the schematic images of resources, properties and actions are constructed as shown in Figure 6e. (3)

These primitives and schematized images can be used as an alphabet in higher-level languages to organize visual reasoning about resource hierarchies, actions, properties, states, situations, etc.

The presence of interconnected formal and visual descriptions of the same entities allows the FHIS, depending on the degree of uncertainty in the decision-making environment, to enable the expert to participate in solving the problem, visualizing the current decision-making situation, and thus to switch between verbal-logical and visual-shaped mechanisms of reasoning.

5. Conclusion

A formal model of conceptual-visual metalanguage is proposed as a multilayered structure which is the basis for the automated solution of problems using integrated verbal-logical and visual-shaped reasoning. An example of the application of the proposed model for describing the alphabet of the metalanguage of operational and dispatching control of regional electric networks is considered.

The use of the proposed models makes it possible to implement FHIS that are able to dynamically synthesize an integrated model and method over heterogeneous model and visual fields and simulate the cooperation, relativity and complementarity of collective intelligence for finding solutions on verbal-symbolic and visual-shaped languages. FHIS of this class will be able to manage the simulation process depending on the uncertainty of the problem situation: when the domain of phenomena is formalized (partially formalized) to use expert knowledge models from a heterogeneous model field to search for solutions, and when there is a significant uncertainty not removed by accurate analysis and logical-mathematical reasoning, to activate the mechanisms of visual-spatial, imaginative thinking of FHIS users. technologies for the design of intelligent systems (OSTIS-2015): Proc. of the V Int. Scientific and Tech. Conf.] (Minsk: BGUIR) pp 25 – 42 [8] Kolesnikov A V and Kirikov I A 2007 Metodologiya i tekhnologiya resheniya slozhnykh zadach metodami funktsional'nykh gibridnykh intellektual'nykh sistem [Methodology and technology for solving complex problems using the methods of functional hybrid intelligent systems] (Moscow: IPI RAN) p 387 [9] Kolesnikov A V 2001 Gibridnye intellektual'nye sistemy. Teoriya i tekhnologiya razrabotki [Hybrid intelligent systems: theory and technology of development] (Saint Petersburg: SPbGTU Publs.) p 711 [10] Kolesnikov A V and Listopad S V 2016 Kontseptual'no-vizual'nyye osnovy virtual'nykh geterogennykh kollektivov, podderzhivayushchikh prinyatiye resheniy [Conceptual-visual foundations of virtual heterogeneous groups supporting decision-making] Gibridnye i sinergeticheskie intellektual'nye sistemy”: materialy III Vserossiyskoy Pospelovskoy konferentsii s mezhdunarodnym uchastiem [Proc. of the 3rd All-Russia Pospelov conf. with int. participation “Hybrid and synergistic intelligent systems”] (Kaliningrad: IKBFU Publ.) pp 8 – 56 [11] Zavalova N D, Lomov B F and Ponomarenko V A 1986 Obraz v sisteme psikhicheskoy regulyatsii deyatel'nosti [Image in the system of mental regulation of activity] (Moscow: Nauka) p 172 [12] Kolesnikov A V 2018 Funktsional'nyye gibridnyye intellektual'nyye sistemy vizual'nogo upravleniya [Functional hybrid intelligent visual control systems] Gibridnye i sinergeticheskie intellektual'nye sistemy”: materialy IV Vserossiyskoy Pospelovskoy konferentsii s mezhdunarodnym uchastiem [Proc. of the 4th All-Russia Pospelov conf. with int. participation “Hybrid and synergistic intelligent systems”] (Kaliningrad: IKBFU Publ.) pp 18 – 81

Acknowledgements

The work was supported by the Russian Foundation for Basic Research (project 16-07-00271a).

References

[1] Golin E J and Reiss S P 1990 The Specification of Visual Language Syntax Journal of Visual Languages and Computing 1 pp 141 - 57

[2] Osipov G S 2002 Ot situatsionnogo upravleniya k prikladnoy semiotike [From situational management to applied semiotics] Novosti iskusstvennogo intellekta [ Artificial intelligence news] 6 ( 54 ) pp 3 - 7

[3] Bowman W J 1968 Graphic communication (New York: John Wiley) p 210

[4] Lakin F 1987 Visual grammars for visual languages Proceedings of the Sixth National Conference on Artificial Intelligence (AAAI-87) (Menlo Park, California: AAAI Press) pp 683 - 8

[5] Narayanan N H and Hubscher R 1998 Visual language theory: Towards a human-computer interaction perspective Visual Language Theory (New York: Springer-Verlag) pp 81 - 128

[6] Sibbet D 2013 Visual leaders: new tools for visioning, management, and organization change . (Hoboken , New Jersey: Wiley) p 229

[7] Tarasov V B 2015 Problema ponimaniya: nastoyashcheye i budushcheye iskusstvennogo intellekta [Problem of understanding: the present and future of artificial intelligence] Otkrytyye semanticheskiye tekhnologii proyektirovaniya intellektual'nykh sistem (OSTIS2015): Materialy V Mezhdunarodnoy nauchno-tekhnicheskoy konferentsii [Open semantic