The paper present the problem of non-formalized expert selection in intelligent systems for reengineering tasks. It proposes a pre-panel selection methodology that standardizes candidate descriptions in a unified feature profile, sets admission thresholds, regulates thematic coverage and institutional independence of the panel, and fixes an immutable evaluation benchmark for subsequent stages. The method includes reproducible stratification of the pool into two independent subsamples with a fixed randomness parameter, clustering by competence- and topic-related features, and transparent rules for acceptance/recusal in the presence of conflicts of interest. We describe the components required for reproducibility and auditability (frozen data versions, verification protocols, decision logs) and present a plan for testing selection stability on independent subsamples and under sensitivity scenarios. The methodology is suitable for reengineering projects with limited data and elevated transparency requirements for expert-panel formation, and it is compatible with common elicitation frameworks without altering their internal mechanics.
Business-process and socio-technical reengineering relies on expert judgment under data sparsity,
multiple conflicting criteria, and tight resource constraints. In this setting, intelligent expert systems
(IES) combine formalized knowledge with independent external validation through the result
template. stagewise artefacts, and quality metrics along trajectory [
Existing literature concentrates on aggregation and validation of expert judgments assuming that
the expert panel is given [
Convenience recruitment (self-selection, affiliation-based invitations, manager nomination,
single-institution panels ) introduces systematic risks: composition bias, domain coverage gaps,
mismatch [
Therefore, the non-standardized expert selection constitutes a core methodological gap in the lifecycle of IES for reengineering. Closing this gap calls for a minimal, auditable standard: an explicit candidate feature matrix X, threshold screening by x0, stratification of the pool Ω into ΩA/ΩB for external validation, competence-homogeneous clustering, the institution and role fixation of an external commission, and immutability of eij in 0 during the project. The research is thus timely and relevant, as robust reengineering decisions in critical domains cannot be guaranteed without formalizing the selection stage itself.
The composition of the upper-level external expert commission (ULEC) for IES is to be formed and fixed so that reengineering decisions remain valid, transparent, and reproducible under data scarcity and resource constraints (τ, ε). The initial objects are the candidate pool Ω and the admission threshold x0; each candidate is represented by a profile in the feature matrix X. For independent verification, the pool is split into two non-overlapping subsamples ΩA, ΩB. The task is to select a ULEC of size mopt ⊂ Ω that meets x0, provides thematic coverage of the required roles and domains R, maintains a balanced mix of competencies and institutional independence, and yields stable results when compared across ΩA and ΩB. Decision quality is controlled against the benchmark 0 ( ) = , , approved once by ULEC at the outset and kept immutable thereafter, preventing post-hoc changes and ensuring reproducibility [12-13]. Within this formulation, resource and organizational constraints are fixed, decisions and versions of X are logged, and the ΩA/ΩB split is preserved; the outputs are the approved ULEC of size mopt, the fixed benchmark 0 ( ) , and a transparency sufficient for external auditing and procedural replication.
First, the full candidate pool Ω and describing each candidate with a coherent profile that captures domain expertise, methodological capability, independence and potential conflicts, institutional affiliation, and availability under the project’s resource constraints (τ, ε). These attributes are mapped into a feature matrix X using predefined, auditable scales: qualitative evidence is coded through rubric-based anchors; categorical variables are encoded consistently to preserve interpretability; quantitative indicators are normalized to ensure cross-source comparability. A calibration procedure aligns heterogeneous inputs before any selection decisions: reference cases are used to harmonize rubric thresholds, repeated measures are checked for stability over time, and disagreement between sources is resolved under a documented precedence rule. The admission threshold x0 is then applied as a minimal, evidence -based gate that filters out profiles failing baseline competence or independence requirements; when evidence is incomplete, the protocol mandates either targeted verification or exclusion with justification to avoid post-hoc upgrading. The outcome of this stage is a cleaned and calibrated representation of Ω and X, together with a decision log that records sources, transformations, and reasons for inclusion or exclusion. This representation becomes the fixed input for subsequent stratification into ΩA and ΩB, downstream clustering and coverage checks, and ultimately the construction of the ULEC of size mopt under the immutable benchmark 0 ( ) , in the Figure 1.
At this stage, the post-screening feature matrix X is brought to an operational form: all columns have aligned definitions, units, and interpretation bounds, and the set of features itself is “frozen” so it is not altered during selection. Two non-overlapping subsamples ΩA and ΩB are then formed using a fixed randomness parameter s0 (seed); they act as independent “mirrors” of each other and serve as the basis for external checks of decision stability. Candidate assignment is performed as constrained randomization initialized by s0: the groups are kept approximately equal in size, balance is maintained on key features from X (topic area, competence level, institutional affiliation, independence status), and safeguards are introduced against information leakage between groups.
To ensure reproducibility, randomization is executed with a fixed random seed, which is recorded in the protocol together with the RNG algorithm, software version, and timestamp. Using the same seed guarantees that rerunning the procedure under the same constraints reproduces the identical ΩA and ΩB composition. If balancing requires retries, a deterministic sequence of seeds is used, pregenerated from a base value and fully logged in the decision journal; post -hoc seed substitution is prohibited. Once approximate balance in topical coverage and key features is reached, the compositions of ΩA and ΩB are fixed and documented along with all randomization parameters and applied constraints. The result is two consistent and balanced subsamples that enable testing whether ULEC selection conclusions and subsequent aggregated assessments persist under changes in group composition, and whether intermediate results align with the immutable benchmark 0 ( ) . This sets the stage for the next step-competency-based clustering and coverage checks before forming the ULEC of size mopt.
At this stage, the profiles from the feature matrix X for ΩA and ΩB are grouped by similarity in competencies, roles, and contextual attributes to obtain competence-homogeneous clusters and to check whether the required reengineering topics are covered evenly. Before running the algorithm, all features are mapped into a distance-suitable space: numerical variables are normalized, categorical variables are encoded using a consistent scheme, and binary variables are checked for imbalance. To avoid dominance of a single institution or role, soft constraints on cluster composition are applied, and outlier profiles with very large distances to centroids are temporarily flagged as “review candidates” so they do not distort the structure.
The number of clusters is chosen as a compromise between simplicity and stability: several values of k are evaluated and the one is selected that provides low within-cluster dispersion while reproducing on both subsamples. Homogeneity is assessed by a combination of criteria: average within-cluster variance of profiles, average distance to the centroid, the silhouette index for internal separability, and a “topic coverage” indicator reflecting the share of key topics actually represented in a cluster. Thresholds for these indicators are set prior to clustering and are not changed during selection; their values are logged together with the algorithm configuration and software version.
Stability is tested crosswise between ΩA and ΩB : clusters from ΩA are matched to the nearestcentroid clusters from ΩB, compositions are compared by topics, competence levels, and affiliations, and the consistency of homogeneity metrics is evaluated. If a cluster fragments or shifts its profile sharply between subsamples, it is marked as unstable, and the procedure either adjusts the feature set for that cluster or moves part of its profiles into the “review candidates” status. The outcome is a set of stable, competence-homogeneous clusters in each subsample and a consolidated report on their homogeneity and topic coverage, which is then used to form the ULEC of size mopt with the required balance of competencies and independence under the immutable benchmark 0 ( ) .
At this stage, the final ULEC of target size mopt is assembled from the harmonized clusters in ΩA and ΩB. Selection is treated as a balance between coverage of key reengineering topics, competence level, and institutional independence, with every decision checked for stability across both subsamples. Candidates are chosen so that all required roles and domains R are represented, no single institution dominates or conflicts of interest arise, and the cluster homogeneity metrics and quality indicators remain consistent under the ΩA ↔ ΩB mapping. If gaps in coverage or excessive affiliation concentration are detected, the selection is adjusted within the previously fixed rules until an acceptable balance is achieved, without altering the definitions of features or thresholds established earlier.
Once the ULEC composition is agreed, the benchmark 0 ( ) is fixed as a rectangular matrix of requirements for the final deliverable and intermediate artifacts; its elements eij become immutable for the entire project period. This means that no subsequent elicitation rounds, reconciliations, or changes in working groups may retrospectively edit eij. Any corrections are treated as a new benchmark version with separate labeling and its own verification trajectory, not as a replacement of the initial 0 ( ) . The benchmark’s invariance, together with frozen versions of the feature matrix X, the clustering protocols, and the stratification parameters, ensures reproducibility of assessments and prevents post-hoc tailoring to a desired outcome.
Formal closure is provided by a public protocol: the list of ULEC members with roles, links to the corresponding clusters and inclusion rationales, evidence of stability checks between ΩA and ΩB, declarations of no conflict of interest, as well as timestamps, software versions, and randomization parameters used in prior steps. The ULEC composition and the benchmark are “frozen” and serve as anchor points for all subsequent elicitation sessions and external validation; in the event of a forcemajeure replacement of an individual expert, a regulated like-for-like substitution procedure is applied with a renewed stability check on ΩA /ΩB, without changing feature definitions and without editing eij.
The proposed protocol turns expert selection from informal practice into a clear, reproducible procedure: candidate profiles are represented in a calibrated form, admission decisions are made using transparent thresholds, the stability of the selection is tested on independent subsamples, and the final commission is fixed with attention to thematic coverage, competence, and institutional independence. A fixed evaluation benchmark and “frozen” data artifacts preclude post-hoc adjustments and build trust in subsequent expert judgments.
At this section, present a mini example with artificial data. Table 1 lists eight initial candidates: competence aggregate professional competence score (0–100); risk aggregated engagement-risk indicator (probability of bias/COI or organizational constraints, 0–1); domain - thematic profile; affiliation - institutional belonging; availability - readiness to participate in the project changes, and transparency and reproducibility under a fixed benchmark 0 ( ) and the target size mopt.
After applying the admission rules (competence ≥ 70; risk ≤ 0.30; availability = yes), the following candidates advance: C1, C2, C6, and C8. Exclusions are as follows: C3 and C5 - excessive risk; C4 insufficient competence; C7 - not available. The next Table 2 will show the shortened list after admission, the split into ΩA /ΩB, and the final EKVR composition with affiliation caps.
Next in Table 2 is the shortened list after admission, the stratification into subsamples(for reproducibility we fix s0=42), and the final EKVR composition subject to affiliation caps and thematic coverage requirements. For this example we set mopt=3, the required roles/domains R={Process, Data, Legal}, and an affiliation cap: no more than 2 members from the same affiliation.
ID C1 C2 C6 C8
Subsample
Domain
Affiliation
Affiliation ΩA ΩB ΩB ΩA
The resulting EKVR is {C1, C6, C8}. Thematic coverage is satisfied (Process, Data, Legal), the affiliation constraint holds (A - 2, B - 1), and balance across ΩA /ΩB is preserved. An equivalent replacement is foreseen: if C8 becomes unavailable, C2 may take the Data role, subject to re-checking the caps and selection stability.
To conclude the example, in Table 3 is record the basic checks required before approval: coverage of the required roles/domains R,compliance with independence and affiliation caps, and a brief stability snapshot across subsamples.
All criteria are satisfied, so the EKVR {C1,C6,C8} is approved for the elicitation phase. If C8drops out, an equivalent replacement is foreseen: C2 may assume the Data role, subject to re-checking the affiliation cap and the stability. 4. Applicability, Limitations and Threats to Validity The proposed protocol is suitable for reengineering tasks that require external verification and transparent decision logging under data scarcity. Its strength lies in structured admission to the expert commission with prior feature calibration, splitting the pool into independent subsamples, and controlling topic coverage and institutional independence. This makes the project’s opening moves reproducible and minimizes the risk of post-hoc adjustments.
At the same time, several factors limit transferability. The composition and importance of features are domain-dependent, and the choice of homogeneity metrics and clustering algorithms can introduce different sensitivities to local profile changes. During stratification and clustering, edge cases may occur when balancing subsamples is hard due to a limited pool or strong correlations among features; in such situations, predefined compromise rules are needed without altering the “frozen” feature definitions. Organizational factors also matter: expert availability, institutional policies on participation and conflict-of-interest declarations, and ethical requirements for handling personal data.
Threats to validity fall into three classes in the Figure 2. Internal validity suffers if some features are measured with systematic bias or if selection rules contain hidden dependencies on affiliations. External validity is limited because the feature set and thresholds are tuned to a specific domain; transferring to another domain calls for recalibration and cross-checking on independent subsamples. Construct validity depends on whether the chosen features truly reflect competence and independence; this requires periodic review of the feature vocabulary with domain stakeholders.
The following section shows how an EKVR formed under our protocol plugs into four widely used frameworks (Delphi, AHP, SHELF, and Structured Expert Judgment in Cooke’s classical model): which inputs and roles are passed to each method, which settings must be aligned with the benchmark, and which residual risks remain outside their formal remit. This format treats elicitation as the next phase layered on top of a unified, reproducible point of entry.
In this section and Table 4, the most widely used expert‐elicitation methods - Delphi, AHP, SHELF, and Structured Expert Judgment (Cooke’s method) are examined to determine the extent to which each meets the key requirements of our problem statement: a formalized selection into the ULEC with threshold x0, construction and calibration of the feature matrix X, stratification ΩA/ΩB for independent verification, robustness of results to panel composition changes, and transparency and reproducibility under a fixation of a benchmark matrix E=[eij] and target panel size mopt.
Delphi is a multi-round consensus procedure with anonymous feedback and aggregated judgments that effectively reduces group pressure and stabilizes responses across rounds [14-15]. However, in the context of our problem statement, Delphi provides little regulation of the pre-panel stage: formal selection into the ULEC from the threshold x0 is typically absent; candidate attributes are not assembled as an explicit feature matrix X with defined scales and calibrations but are gathered implicitly via questionnaires; independent stratification ΩA/ΩB for external verification is not envisaged. The method is sensitive to panel composition and facilitation, so robustness to participant changes is limited; transparency and reproducibility are described for the elicitation rounds, not for assembling the panel itself or for fixing the artifacts of selection [16].
So, Delphi is suitable as a mechanism of eliciting opinions, but it requires a complementary explicit threshold screening x0, construction and calibration of the feature matrix X, stratification ΩA/ΩB for independent verification, and prior fixation of the 0 ( ) and target panel size mopt before the rounds begin. 5.2. AHP AHP is a pairwise‐comparison method that structures a problem into a hierarchy of goals, criteria, and alternatives, collects judgments on the Saaty scale, and checks consistency through the Consistency Ratio [17-18]. Within our problem statement, AHP is useful for formalizing criteria and eliciting weights, but it provides little regulation of the pre-panel stage: there is no formal selection into the ULEC using the threshold x0; candidate attributes are not specified as an explicit feature matrix X with agreed scales and calibration; independent stratification ΩA/ΩB for external verification is not envisaged. The method is sensitive to panel composition and to scale choices, with known issues such as rank reversal and dependence on the permitted inconsistency level; consequently, robustness under replacement of participants and transfer of judgments between ΩA and ΩB is limited. Transparency and reproducibility are ensured at the level of pairwise comparison matrices and CR, but not at the level of fixing the panel selection artifacts and the benchmark 0 ( ) .
AHP should be applied after formal selection as a tool for structuring criteria and setting weights for the already selected ULEC of size mopt while being complemented by our protocol elements: threshold screening x0, explicit construction and calibration of X, stratification ΩA/ΩB prior fixation of 0 ( ) , and sensitivity analysis to panel changes and admissible inconsistency levels. 5.3. SHELF SHELF is a facilitated elicitation framework in which group sessions are organized around clear scenarios, interim calibration tasks, and stepwise reconciliation of individual judgments into a shared distribution [19]. The method defines moderator and scribe roles, uses interval judgments, seed questions, and justification of assumptions, and finally records an agreed expert distribution together with a session protocol and sources of uncertainty. Within our problem statement, SHELF’s strength lies in transparent facilitation and thorough documentation of the elicitation stage, which improves reproducibility of that stage [20].
At the same time, SHELF provides little regulation of the pre-panel phase. Formal selection into the ULEC using the threshold x0 is typically absent; candidate profiles are not represented as an explicit feature matrix X with agreed scales and calibration steps; independent stratification ΩA/ΩB for external verification outside the main group is not envisaged. The method is sensitive to participant composition and facilitation style, so robustness to replacing part of the panel is limited. To align with our problem statement, SHELF should be combined with a formal selection protocol: introduce threshold screening x0, construct and calibrate X, perform stratification ΩA/ΩB for independent validation and fix the benchmark 0 ( ) and the target ULEC size mopt before sessions begin.
Structured Expert Judgment (Cooke’s method) relies on calibrating experts using control “seed” questions with known truths, scoring them with proper scoring rules (calibration and information scores), and producing an aggregate assessment with weights proportional to empirical accuracy [21]. Its strength is explicit, statistically grounded calibration and performance weighting, which mitigates overconfidence and identifies strong experts by data rather than status [22]. In the context of our problem statement, however, a key gap remains: the method assumes the panel is already formed and does not regulate pre-panel selection into the ULEC using the threshold x0, the explicit construction and calibration of the feature matrix X or independent stratification ΩA/ΩB for external verification outside the main group [23]. High -quality seed variables that are independent of the targets are additionally required; otherwise, bias and overfitting to the control set may increase. Sensitivity to panel composition also persists because weights are computed only after the panel has been assembled. To align with our framework, Cooke’s method is best applied after formal selection: first introduce threshold screeningx0, build and calibrate X, perform ΩA/ΩB stratification with fixation of the benchmark 0 ( ) and the target size mopt, and then apply SEJ performance weighting on the selected panel as the calibration and aggregation mechanism .
In [
All four methods focus on elicitation and aggregation but leave pre-panel selection underspecified. This motivates our protocol that precedes any elicitation with threshold screening x0, a calibrated feature matrix X, stratification ΩA/ΩB, and prior fixation 0 ( ) and mopt, thereby providing the missing transparency and rep roducibility guarantees for ULEC formation. 6. Discussion and Сonclusions The paper describes the problem of non-formalized expert selection in intelligent systems for reengineering tasks, which complicates transparency, reproducibility, and robustness of decisions. A possible selection procedure is outlined in which candidate profiles are represented in a calibrated feature matrix; a minimal admission threshold is applied; the pool is split into independent subsamples for external checking; clustering by competencies is performed; and the commission is formed with attention to topical coverage and institutional independence under a pre-fixed evaluation benchmark. In this formulation, the procedure is considered reproducible and auditable; however, specific features, thresholds, and metrics remain domain-dependent.
Future work should explore a possible development path: testing portability across different classes of reengineering tasks; refining the feature vocabulary and thresholds based on applications; and creating minimal tooling for data “freezing”, controlled stratification, and coverage/independence reporting. Integration with post-selection performance weighting of experts is also considered, which may provide a coherent transition from standardized selection to subsequent aggregation of judgments.
Declaration on Generative AI The authors have not employed any Generative AI tools. [10] W. Wojcik, V. Osypenko, V. Lytvynenko, The use of inductive clustering algorithms for forming expert groups in large-scale innovation projects. Electronika, 2013. URL: https://www.theiet.org/publishing/inspec/. [11] J. Xing, S. Morrow, Guidance for Conducting Expert Elicitation in Risk-Informed Decisionmaking Activities (NUREG-2255). U.S.NRC, 2024. URL: https://www.nrc.gov/readingrm/doc-collections/nuregs/staff/sr2255/index. [12] N. Kiktev, V. Osypenko, N. Shkurpela and A. Balaniuk, "Input Data Clustering for the Efficient Operation of Renewable Energy Sources in a Distributed Information System," 2020 IEEE 15th International Conference on Computer Sciences and Information Technologies (CSIT), Zbarazh, Ukraine, 2020, pp. 9-12, doi: 10.1109/CSIT49958.2020.9321940. [13] V. Osypenko, B. Zlotenko, T. Kulik, S. Demishonkova, O. Synyuk, V. Onofriichuk, S. Smutko, , Improved Algorithm for Matched-Pairs Selection of Informative Features in the Problems of Recognition of Complex System States (April 30, 2021). Eastern-European Journal of Enterprise Technologies, 2021, 2(4 (110), 48–54, doi:10.15587/1729-4061.2021.229756 [14] H. B. Triantono and M. B. Susetyarto, Technical use of analytical hierarchy process and Delphi method in determining terminal location multi function of Merak Port, 2017 International Conference on Information Management and Technology (ICIMTech), Special Region of Yogyakarta, Indonesia, 2017, pp. 350-355, doi: 10.1109/ICIMTech.2017.8273564. [15] L. Thwaites, P. Nasa, B. Abbenbroek et al, Management of adult sepsis in resource-limited settings: global expert consensus statements using a Delphi method. Intensive Care Med, 2025, 51, 21–38, doi: 10.1007/s00134-024-07735-7 [16] W. Ji, J. Wang, X. Fang and S. Gu, Improvement and application of the Delphi method, Proceedings of the 10th World Congress on Intelligent Control and Automation, Beijing, China, 2012, pp. 4026-4029, doi: 10.1109/WCICA.2012.6359147. [17] Yang W-C, Ri J -B, Yang J-Y, Kim J-S. Materials selection criteria weighting method using analytic hierarchy process (AHP) with simplest questionnaire and modifying method of inconsistent pairwise comparison matrix. Proceedings of the Institution of Mechanical Engineers, Part L: Journal of Materials: Design and Applications 2022, 236, 69–85, doi: 10.1177/14644207211039912. [18] O. Shankar, F. Mirza, R. Khatwani, P. K. Mitra, P. Shah and R. Sekhar, AHP and Fuzzy AHP Multi-Criteria Decision Making Process: A Brief Study of Laptop Selection, 2024 International Conference on Intelligent Systems and Advanced Applications (ICISAA), Pune, India, 2024, pp. 1-6, doi: 10.1109/ICISAA62385.2024.10828604. [19] D. B. Rizzo and M. R. Blackburn, "Harnessing Expert Knowledge: Defining Bayesian Network Model Priors From Expert Knowledge Only—Prior Elicitation for the Vibration Qualification Problem," in IEEE Systems Journal, vol. 13, no. 2, pp. 1895-1905, June 2019, doi: 10.1109/JSYST.2019.2892942. [20] J.E. Oakley, A. O’Hagan, Sheffield elicitation framework. URL: https://shelf.sites.sheffield.ac.uk [21] D. Kurowicka, Aspinall, W.; Cooke, R. Foundational Aspects for Incorporating Dependencies in Copula-Based Bayesian Networks Using Structured Expert Judgments, Exemplified by the Ice Sheet–Sea Level Rise Elicitation. Entropy 2024, 26, 949, doi: 10.3390/e26110949 [22] S. French, A. M. Hanea, T. Bedford, G. F. Nane, Introduction and Overview of Structured Expert Judgement. In: Hanea, A.M., Nane, G.F., Bedford, T., French, S. (eds) Expert Judgement in Risk and Decision Analysis. International Series in Operations Research & Management Science, 2021, vol 293. Springer, Cham, doi: 10.1007/978-3-030-46474-5_1 [23] R. Cooke, Supplementary Information for Structured Expert Judgment, 2022. URL: https://rogermcooke.net/rogermcooke_files/2022.pdf