The activities of any manager are closely related to the need to make decisions on a daily basis. Decision-making is a specific type of activity that involves forming a set of decision options (alternatives) and then evaluating their relative effectiveness and resource allocation among the decision options, based on their assessments. The simpler types of decisions include accepting or rejecting an alternative, choosing the best alternative from a given set, and ranking alternatives.
In making complex decisions, there is often a need to consider numerous (tens or hundreds) interrelated factors that interact in complex ways. To ensure high-quality decision-making, the integration of knowledge from many expert specialists is necessary. However, due to psychophysiological limitations, humans are only capable of processing around 7-9 objects simultaneously [1]. Decision Support Systems (DSS) are used to overcome this limitation (Figure 1).
In solving problems in weakly structured domains, where DSS are increasingly utilized, the task of enhancing the adequacy of the domain model to improve the reliability of recommendations produced by DSS becomes relevant. An essential component of DSS is the information obtained from experts in the form of object names formulated in natural language. Therefore, it is crucial to unambiguously identify these objects in knowledge bases (KB) of DSS to adequately consider the collective expertise of experts when establishing relationships among these objects. Large KBs are created to describe complex subject areas, raising the issue of unequivocal object identification in these KBs.
ITS-2023: Information Technologies and Security, November 30, 2023, Kyiv, Ukraine andriichuk@ipri.kiev.ua(O. Andriichuk); seriga2009@gmail.com(S. Kadenko); aflorekpaszkowska@pucp.edu.pe (A. Florek-Paszkowska) 0000-0003-2569-2026(O. Andriichuk); 0000-0001-7191-5636 (S. Kadenko); 0000-0002-1249-5069 (A. Florek-Paszkowska) Figure1: Functional scheme of adecision support system Currently, when constructing domain models in DSS, researchers focus significantly on methods for obtaining and processing expert evaluations, which are extensively covered in scientific publications. Typically, attention is given to evaluating objects from already constructed KBs. As the number of objects in DSS KBs grows, especially when using group methods for KB construction, the issue of error-free identification becomes increasingly important.
During the construction and maintenance of KBs, the most accurate representation of integrated expert opinions can be achieved by eliminating the erroneous repeated input of content-identical (but differently formulated) object names. To achieve this, it is advisable to search for content-related formulations. Enhancing the efficiency of using knowledge obtained from experts is enabled by the reuse of previously constructed KBs, but the process of merging KBs needs to be automated. In sufficiently large KBs, especially those built by expert groups of various profiles, there is a significant likelihood of errors.
The application of DSSs results in recommendations (see Figure 1) for decision-makers [2][3][4]. This involves modeling weakly structured subject areas [5,6], as shown on Figure 2, using DSS tools based on the constructed corresponding KB. One of the important characteristics of such subject areas is the incomplete description of objects, which makes it difficult to create a quality training sample for machine learning. Therefore, expert knowledge needs to be utilized. Expertise, including group expertise, requires significant time and financial investments.
The properties of weakly structured subject domains outlined in Figure 2 include the absence of a formalizable functioning goal, the lack of an optimality criterion, uniqueness, dynamics, incomplete description, the presence of a subjective human factor, the inability to construct an analytical model, the absence of benchmarks, and high dimensionalities.
Let's delve deeper into the characteristics of these weakly structured subject domains. Objects within such domains are inherently unique. Management systems designed for these domains are typically tailored to address specific real-world problems, making replication on other entities costly or unfeasible.
In systems not engineered by humans, like biological systems, formalizing a functioning goal is often unattainable. While these systems aim for efficiency and parameter maintenance within defined limits, articulating a specific criterion for their functioning proves challenging due to the intricate and numerous factors at play, with complex and obscure connections that resist easy categorization. The absence of a formalizable functioning goal precludes the construction of an optimization function that could dictate the ideal operational state of an object. Optimization becomes complex as factors are interdependent, disrupting the delicate balance necessary for system stability amidst environmental changes, potentially resulting in catastrophic outcomes and irreversible system alterations.
Without a formalizable functioning goal or an achievable optimization function, constructing an analytical model for these subject domains becomes unviable. Dynamism stems from the evolving nature of these objects over time, necessitating adaptive management strategies that mirror the object's changes.
Incompleteness in description arises from data inaccuracies, incompleteness, falsities, ambiguities, contradictions, uncertainties, and unreliabilities surrounding the object, making quantitative characterization challenging. Benchmarks for object characteristics within weakly structured subject domains are inappropriate due to these issues.
The vast decision space dimension results from the multitude and heterogeneity of criteria defining the subject domain. Human subjects with free will further complicate management efforts, as predicting and controlling human behavior within a system is intricate given individual goals and interests influencing actions.
The properties of weakly structured subject domains underscore the reliance on experts as the primary, and sometimes sole, source of information within these domains.
Research was conducted in the United States to examine the distribution of knowledge types utilized in the daily operations of specific organizations [6]. The findings from the Delphi Group's investigation, as depicted in Figure 3, revealed that a significant portion (42%) of the knowledge employed is not formally documented or stored on any data mediums. This type of knowledge is exclusively held by skilled experts, posing a challenge for AI tools to leverage it effectively for providing recommendations [7,8]. Consequently, while AI serves as a valuable tool in Natural Language Processing (NLP), it cannot completely supplant the expertise of a subject matter specialist. Despite AI's proficiency in processing vast datasets and recognizing semantic relationships, it may struggle to account for the distinctive context and nuances inherent in individual scenarios. Recently, artificial intelligence (AI) tools based on large-scale linguistic models implemented using artificial neural networks with transformer architecture [9,10] have gained significant popularity due to their convenience and accessibility. These tools have rapidly and widely spread among users, finding direct applications in the practice of preparing and writing various types of works, including academic papers, particularly English texts, as well as in publishing [11]. Some researchers even use these AI tools to construct networks of keywords as conceptual models for specific subject areas [12]. However, developers of such tools caution that large language models do not guarantee the truthfulness or reliability of the output data [13].
The above considerations necessitate and highlight the importance of researching the possibilities of using AI tools in DSS to build models for weakly structured subject areas [6,11], as shown on Figure 2.
Since the main component of DSS is the KB, let us consider its structure, which is shown in Figure 4.
The main elements of the BR are objects and relations between them. KB objects can be goals and projects. The object of the BR is named in the form of a short formulation. A tuple of keywords is used to define the meaning of a KB object. The object of the KB can be quantitative or qualitative, threshold or quasi-linear. For projects, duration of execution and resources are specified. The relationship between BR objects can: be positive or negative, have a time delay, compatibility groups. It is also characterized by a private influence coefficient. On Figure 5, tasks arising when constructing KBs for weakly structured subject areas are shown [14]. When obtaining knowledge from expert groups, the decomposition of goals and the determination (evaluation) of their impact levels are conducted. When formulating goals, each expert in the group provides texts of their individual sub-goal formulations. Figure 6 shows this stage in the Consensus-2 system [15]. Subsets of content-matching formulations are identified. The best formulation is selected in each subset. When establishing influences between goals, in the KB, goals that affect a specific goal and goals affected by a specific goal should be sought. Each expert determines the importance of each sub-goals by pairwise comparisons. To achieve higher reliability and consistency of expert evaluations, it is necessary to take into account the order of alternatives in pairwise comparisons [16,17]. After group expert assessment of impact levels, aggregation of relevant assessments occurs upon reaching a sufficient level of consensus. When aggregating knowledge obtained from different expert groups, the search and consolidation in the KB of content-matching goals formulated by different groups take place. Additionally, "pre-establishing" influences on the combined set of goals with re-scaling the relevant impact level values occurs. In this process, goals affecting a specific goal and goals affected by a specific goal should be sought in the KB. When building KBs, the following issues arise: a high level of detail in the KB leads to a deterioration in the adequacy of models in weakly structured subject areas, namely: redundancy, ambiguity, and the presence of contradictions in the KB.
Figure 7 illustrates the factors that influence the quality of recommendations in DSSs. One of the most important among them is the "Adequacy of the domain model." In turn, this factor is influenced by the aforementioned redundancy, ambiguity, and the presence of contradictions in the KB of DSS.
For almost all the aforementioned tasks in constructing KBs for weakly structured subject areas, it is advisable to use AI tools. The question arises: can one fully rely on AI tool recommendations and perform these tasks in automatic mode? Below are the results of an experimental study on the accuracy of AI tool recommendations regarding improving expert formulations. These results indicate that such use of AI tools in practice is only advisable in an automated mode with the involvement of expert groups. It is not advisable to rely entirely on AI tool recommendations at this stage of its development.
During the construction of such KBs, knowledge engineers, analysts, and multidisciplinary experts consistently decompose a particular object of the subject area, step by step. This approach is applied to group modeling of subject areas within the framework of the "Consensus-2" system for distributed collection and processing of expert information for decision support systems [15].
The conducted experimental study on the accuracy of AI tool recommendations for improving expert formulations consisted of three described above stages. 1. A group of expert respondents provided texts of their English-language formulations. Most of these texts were translated from Ukrainian and Russian using machine translation and qualified English translators. Each expert in this group specializes in the field of information technology and security. The English texts containing expert formulations belong specifically to this field. It is also worth noting that English is not the native language for these experts. 2. Recommendations for improving the quality of English texts containing expert formulations were obtained using AI tools with the architecture of GPT-3.5. Special prompts were used with the AI tool for this purpose. 3. A group of expert validators assessed the accuracy and quality of the English texts containing expert formulations, as well as the recommendations obtained from the AI tool. This group of experts has competence in the field of information technology and security and is proficient in English. Additionally, this group consulted with a group of experts specializing in English language in general, translation, philology, linguistics, and English in the field of computer science. The questionnaire for expert validators contained the following questions: "Please compare the quality of the AI tool's recommendation with the quality of the original English-language expert opinion:
-AI tool recommendation is unreliable; -the AI tool's recommendation is reliable but of equivalent quality; -the recommendation of the AI toolkit is reliable and requires minor adjustment; -the recommendation of the AI tool is reliable and does not require adjustment."
In order to ensure statistical credibility of the research, we calculated the necessary number of experiment instances. Evaluation of statistical credibility was conducted based on the central limit theorem [18]. If we set the confidence probability value at Pβ = 0.95 (i.e., the probability that the random variable value falls within confidence interval β), and confidence interval size for the given experimental study is β = 0.05, the minimum necessary number of experiment instances can be calculated based on the following inequality:
where 𝐹 is the inverse Laplace function [19]; p is the frequency of repetition of value of the random characteristic under consideration. We select the value of p based on previously obtained experiment results as the "worst" probability/frequency (i.e. the one closest to 0.5). As a result of test (initial) experiment series, we collected 61 ranking of alternative pair sequences. After filtering (screening), the remaining set of test experiment series constituted 33 rankings of alternative pair sequences. The results of test experiment series are presented in table 1. Results of the experimental study on the credibility of AI tool recommendations for improving expert formulations:
56% of the recommendations provided by AI tools were accurate and did not require corrections. 65% of the recommendations improved the quality of formulations. 67% of the recommendations did not worsen the quality of formulations. 1.5% of the recommendations were found to be futile, adding no value. 33% of the recommendations were considered harmful, distorting the content of the formulations.
During the analysis of harmful recommendations, it was found that the main causes of distortions lie in the insufficient adaptability of the model to the specifics of the terminology and context used, as well as in the inadequate processing of terms that have specific meanings in the field of computer science.
It has been shown that it is beneficial to utilize artificial intelligence tools to address a range of tasks that arise during the construction of knowledge bases for decision support systems.
Ways of applying artificial intelligence tools in knowledge base construction have been proposed, particularly to enhance the quality of expert formulations.
An experimental study on the accuracy of recommendations from artificial intelligence tools for improving expert formulations has been conducted.
Corresponding empirical results have been obtained, indicating that the application of artificial intelligence tools in practice is advisable only in an automated mode, i.e., involving a group of experts. It is not advisable to rely solely on the recommendations of artificial intelligence tools at this stage of the technology's development.