The work is dedicated to the issue of applying artificial intelligence tools to enhance expert formulations in constructing knowledge bases for decision support systems. The challenges arising in constructing a knowledge base for weakly structured subject areas are considered, and corresponding approaches to the use of artificial intelligence tools are proposed. An appropriate experimental study has been conducted, the results of which indicate that such application of artificial intelligence tools in practice is advisable only in an automated version involving a group of experts.
The activities of any manager are closely related to the need to make decisions on a daily basis. Decision-making is a specific type of activity that involves forming a set of decision options (alternatives) and then evaluating their relative effectiveness and resource allocation among the decision options, based on their assessments. The simpler types of decisions include accepting or rejecting an alternative, choosing the best alternative from a given set, and ranking alternatives.
In making complex decisions, there is often a need to consider numerous (tens or hundreds) interrelated factors that interact in complex ways. To ensure high-quality decision-making, the integration of knowledge from many expert specialists is necessary. However, due to psychophysiological limitations, humans are only capable of processing around 7-9 objects simultaneously [1]. Decision Support Systems (DSS) are used to overcome this limitation (Figure 1).
In solving problems in weakly structured domains, where DSS are increasingly utilized, the task of enhancing the adequacy of the domain model to improve the reliability of recommendations produced by DSS becomes relevant. An essential component of DSS is the information obtained from experts in the form of object names formulated in natural language. Therefore, it is crucial to unambiguously identify these objects in knowledge bases (KB) of DSS to adequately consider the collective expertise of experts when establishing relationships among these objects. Large KBs are created to describe complex subject areas, raising the issue of unequivocal object identification in these KBs.
Currently, when constructing domain models in DSS, researchers focus significantly on methods for obtaining and processing expert evaluations, which are extensively covered in scientific publications. Typically, attention is given to evaluating objects from already constructed KBs. As the number of objects in DSS KBs grows, especially when using group methods for KB construction, the issue of error-free identification becomes increasingly important.
During the construction and maintenance of KBs, the most accurate representation of integrated expert opinions can be achieved by eliminating the erroneous repeated input of content-identical (but differently formulated) object names. To achieve this, it is advisable to search for content-related formulations. Enhancing the efficiency of using knowledge obtained from experts is enabled by the reuse of previously constructed KBs, but the process of merging KBs needs to be automated. In sufficiently large KBs, especially those built by expert groups of various profiles, there is a significant likelihood of errors.
The application of DSSs results in recommendations (see Figure 1) for decision-makers [
The properties of weakly structured subject domains outlined in Figure 2 include the absence of a formalizable functioning goal, the lack of an optimality criterion, uniqueness, dynamics, incomplete description, the presence of a subjective human factor, the inability to construct an analytical model, the absence of benchmarks, and high dimensionalities.
Let's delve deeper into the characteristics of these weakly structured subject domains. Objects within such domains are inherently unique. Management systems designed for these domains are typically tailored to address specific real-world problems, making replication on other entities costly or unfeasible.
In systems not engineered by humans, like biological systems, formalizing a functioning goal is often unattainable. While these systems aim for efficiency and parameter maintenance within defined limits, articulating a specific criterion for their functioning proves challenging due to the intricate and numerous factors at play, with complex and obscure connections that resist easy categorization.
The absence of a formalizable functioning goal precludes the construction of an optimization function that could dictate the ideal operational state of an object. Optimization becomes complex as factors are interdependent, disrupting the delicate balance necessary for system stability amidst environmental changes, potentially resulting in catastrophic outcomes and irreversible system alterations.
Without a formalizable functioning goal or an achievable optimization function, constructing an analytical model for these subject domains becomes unviable. Dynamism stems from the evolving nature of these objects over time, necessitating adaptive management strategies that mirror the object's changes.
Incompleteness in description arises from data inaccuracies, incompleteness, falsities, ambiguities, contradictions, uncertainties, and unreliabilities surrounding the object, making quantitative characterization challenging. Benchmarks for object characteristics within weakly structured subject domains are inappropriate due to these issues.
The vast decision space dimension results from the multitude and heterogeneity of criteria defining the subject domain. Human subjects with free will further complicate management efforts, as predicting and controlling human behavior within a system is intricate given individual goals and interests influencing actions.
The properties of weakly structured subject domains underscore the reliance on experts as the primary, and sometimes sole, source of information within these domains.
Research was conducted in the United States to examine the distribution of knowledge types
utilized in the daily operations of specific organizations [
Recently, artificial intelligence (AI) tools based on large-scale linguistic models implemented using
artificial neural networks with transformer architecture [
The above considerations necessitate and highlight the importance of researching the possibilities
of using AI tools in DSS to build models for weakly structured subject areas [
Since the main component of DSS is the KB, let us consider its structure, which is shown in Figure 4. The main elements of the BR are objects and relations between them. KB objects can be goals and projects. The object of the BR is named in the form of a short formulation. A tuple of keywords is used to define the meaning of a KB object. The object of the KB can be quantitative or qualitative, threshold or quasi-linear. For projects, duration of execution and resources are specified. The relationship between BR objects can: be positive or negative, have a time delay, compatibility groups. It is also characterized by a private influence coefficient.
On Figure 5, tasks arising when constructing KBs for weakly structured subject areas are
shown [
When building KBs, the following issues arise: a high level of detail in the KB leads to a deterioration in the adequacy of models in weakly structured subject areas, namely: redundancy, ambiguity, and the presence of contradictions in the KB.
Figure 7 illustrates the factors that influence the quality of recommendations in DSSs. One of the most important among them is the "Adequacy of the domain model." In turn, this factor is influenced by the aforementioned redundancy, ambiguity, and the presence of contradictions in the KB of DSS.
For almost all the aforementioned tasks in constructing KBs for weakly structured subject areas, it is advisable to use AI tools. The question arises: can one fully rely on AI tool recommendations and perform these tasks in automatic mode? Below are the results of an experimental study on the accuracy of AI tool recommendations regarding improving expert formulations. These results indicate that such use of AI tools in practice is only advisable in an automated mode with the involvement of expert groups. It is not advisable to rely entirely on AI tool recommendations at this stage of its development.
During the construction of such KBs, knowledge engineers, analysts, and multidisciplinary experts
consistently decompose a particular object of the subject area, step by step. This approach is applied
to group modeling of subject areas within the framework of the "Consensus-2" system for distributed
collection and processing of expert information for decision support systems [
The conducted experimental study on the accuracy of AI tool recommendations for improving expert formulations consisted of three described above stages.
1. A group of expert respondents provided texts of their English-language formulations. Most of these texts were translated from Ukrainian and Russian using machine translation and qualified English translators. Each expert in this group specializes in the field of information technology and security. The English texts containing expert formulations belong specifically to this field. It is also worth noting that English is not the native language for these experts. 2. Recommendations for improving the quality of English texts containing expert formulations were obtained using AI tools with the architecture of GPT-3.5. Special prompts were used with the AI tool for this purpose. 3. A group of expert validators assessed the accuracy and quality of the English texts containing expert formulations, as well as the recommendations obtained from the AI tool. This group of experts has competence in the field of information technology and security and is proficient in English. Additionally, this group consulted with a group of experts specializing in English language in general, translation, philology, linguistics, and English in the field of computer science.
The questionnaire for expert validators contained the following questions:
"Please compare the quality of the AI tool's recommendation with the quality of the original
English-language expert opinion:
- AI tool recommendation is unreliable;
- the AI tool's recommendation is reliable but of equivalent quality;
- the recommendation of the AI toolkit is reliable and requires minor adjustment;
- the recommendation of the AI tool is reliable and does not require adjustment."
In order to ensure statistical credibility of the research, we calculated the necessary number of
experiment instances. Evaluation of statistical credibility was conducted based on the central limit
theorem [
is the inverse Laplace function [
We select the value of p based on previously obtained experiment results as the “worst” probability/frequency (i.e. the one closest to 0.5). As a result of test (initial) experiment series, we collected 61 ranking of alternative pair sequences. After filtering (screening), the remaining set of test experiment series constituted 33 rankings of alternative pair sequences. The results of test experiment series are presented in table 1. total number of recommendations received unreliable recommendations reliable recommendations that are equivalent in quality to the original wording reliable recommendations that require minor adjustments reliable recommendations that do not require adjustment
Quantity (frequency) 797 285 10 76 426
Among the frequencies, defined based on the table {285/797≈0.358; 10/797≈0.013; 76/797≈0.095; 426/797≈0.535}, the worst one according to the specified criterion is frequency p = 0.535, which we will input into the formula for calculation.
After inputting all the respective values into the formula, we get: then: (0.95) ≈ 1.96, (0.95)
≈ 3.84, and, finally, n ≥ 382.12. It means, that in order to draw credible conclusions based on the experiment results, it is sufficient to perform at least 383 instances of the experiment.
The results of the calculation of the accuracy of AI tool recommendations for improving expert formulations obtained during the experimental study are presented in Table 2. total number of recommendations received unreliable recommendations reliable recommendations that are equivalent in quality to the original wording reliable recommendations that require minor adjustments reliable recommendations that do not require adjustment
It has been shown that it is beneficial to utilize artificial intelligence tools to address a range of tasks that arise during the construction of knowledge bases for decision support systems.
Ways of applying artificial intelligence tools in knowledge base construction have been proposed, particularly to enhance the quality of expert formulations.
An experimental study on the accuracy of recommendations from artificial intelligence tools for improving expert formulations has been conducted.
Corresponding empirical results have been obtained, indicating that the application of artificial intelligence tools in practice is advisable only in an automated mode, i.e., involving a group of experts. It is not advisable to rely solely on the recommendations of artificial intelligence tools at this stage of the technology's development.
[1] George A. Miller, The Magical Number Seven, Plus or Minus Two. The Psychological Review, 1956, vol. 63, pp. 81-97. [2] T. L. Saaty, Principia Mathematica Decernendi - Mathematical principles of decision making Generalization of the Analytic Network Process to neural firing and synthesis. RWS Publications, Pittsburg, 2010.