A critical aspect of organizing competitions is ensuring the objectivity and professionalism of evaluations, which depend on the composition of the jury. This article addresses the challenge of automating the selection of jury members for student research competitions. The primary focus is on using scientometric and altmetric indicators for a comprehensive evaluation of candidates. The authors propose integrating metrics for the objective selection of experts, including the h-index and other measures to assess academic productivity, altmetric indicators to analyze research impact in digital media, and online presence evaluations, parameters that describe scientific interactions of potential jury members in the scientific community. For each evaluation criterion, weighting factors are introduced, and expert selection is performed using a preference-ranking approach that determines similarity to the ideal solution (TOPSIS). Furthermore, the study reviews existing bibliometric platforms that provide scientometric and altmetric indicators. An essential feature of the proposed approach is the automation of data acquisition through APIs, ensuring real-time access to scientific repositories, bibliographic databases, and social media sources.
Student science competitions play a key role in developing young talent, stimulating innovation
and preparing future leaders in STEM (science, technology, engineering, mathematics) and other
fields. Events such as the International Science and Engineering Fair (ISEF), Google Science Fair, or
international Olympiads not only promote science, but also create a competitive environment where
the quality of assessment is crucial [
Forming a jury for such competitions is a complex task that requires ensuring competence,
impartiality, diversity and transparency. Traditional approaches to selecting experts are often
subjective, time-consuming and vulnerable to conflicts of interest, which can undermine the
credibility of the competition results [
Recent advances in information technology, in particular in big data processing and artificial
intelligence, are opening up new opportunities to automate this process. Scientometric databases
such as Scopus [
The relevance of the topic is due to several factors. The growth in the number of student
competitions, which, according to UNESCO [
The object of this study is the process of forming the jury of the competition of scientific papers.
The subject of the study is the application of multicriteria optimisation methods for selecting the jury based on the integration of scientometric and altmetric indicators of the applicant specialists.
The goal is to develop the principles of an automated technology for forming a jury for a competition, which will ensure an increase in the level of objectivity, qualification and impartiality of the evaluation of participants' works, and optimization of organizational costs.
Analysis of the current state and features of competitions has shown that the last decades are
characterized by a significant increase in the number of scientific student competitions at the global
level. This trend is due to several key factors: globalization of education and science, investments of
many countries in STEM education (science, technology, engineering, mathematics), development of
digital technologies, interest of corporations in attracting talented youth to their industries [
Student research competitions are diverse in their format, purpose and conditions of participation. They can be aimed at developing specific academic or practical skills, depending on the discipline or specialisation, and can be of different scale: from local university competitions to international competitions. Let us describe the main types of such competitions and their features.
Specialist or disciplinary competitions are aimed at students engaged in research in certain fields
of science, such as physics, biology, mathematics, engineering, computer science, etc. The purpose
of such events is to deepen students' knowledge in a particular field and develop their professional
skills. The topics of such competitions are strictly limited to the respective speciality. Participants
must demonstrate in-depth knowledge and ability to conduct research in a particular specialisation.
Entries are often evaluated by leading industry experts, allowing students to receive professional
feedback. Such competitions are held at the national or international level [
Practice-oriented competitions are aimed at creating or developing practical solutions to
realworld problems in various fields. These can be developments in engineering, information
technology, healthcare, economics, ecology, etc. The main goal of these competitions is to encourage
students to create innovative products, services or methods that can be used in real life. Practically
oriented competitions often require not only theoretical justification but also the development of
prototypes, models, software products, etc. Such competitions can involve industry partners, which
gives students the opportunity to collaborate with businesses and potential investors [
Online competitions have gained popularity in recent years as they allow students to participate regardless of their location. Such events are convenient and accessible to more participants as they do not require physical presence. Participants submit their research papers electronically, which reduces travel and organisational costs. The evaluation is carried out remotely, often using special review platforms. Online competitions provide a wide dissemination of scientific results via the Internet and can use virtual presentations and webinars to discuss the work.
According to the statistics of the State Scientific Institution "Institute for the Modernisation of
Education Content" (SSI "IMEC"), in pre-war times, more than three hundred different international
and national intellectual competitions were held annually in Ukraine. There were international
olympiads, international competitions of scientific papers, international professional creative
competitions, international tournaments, international conferences, and national olympiads, which
together are an integral part of the national system of identifying and developing young talents [
In summary, competitions are becoming more widespread and diverse in terms of disciplines, formats, and geographical coverage. All these factors create significant difficulties in addressing the task of forming a jury due to the need to scale up all organizational processes.
However, the jury selection process also involves a qualitative dimension. Conventional approaches to forming juries for student research competitions—primarily relying on the manual selection of experts by organizers—encounter several challenges that hinder the ability to guarantee objectivity, competence, transparency, and efficiency in evaluation.
Let us consider the criteria that a jury member must meet.
One of the main criteria for selecting experts in the jury is their qualifications and experience in scientific or professional activities. The jury members must have an appropriate level of education, academic degrees (e.g., PhD, Doctor of Science), and significant research experience in the field related to the competition. The deeper the expert's knowledge of the subject matter, the better he or she will be able to evaluate the work. The qualification criteria include an academic degree in a field related to the competition topic, work experience in the relevant field, and the number of publications in peer-reviewed journals that reflect the level of expert competence.
The jury members should have a reputation in the scientific community, which is confirmed by their contribution to the development of the relevant field of knowledge. This may be in the form of awards, grants, invitations to participate in scientific juries or conference committees. The criteria for recognition may include participation in editorial boards of scientific journals or conference juries, recognition in the scientific community through receiving awards or prizes for scientific achievements, experience of speaking at international or national conferences, which demonstrates involvement in active scientific work.
To ensure that the evaluation process is objective and standardised, it is important to involve experts who have previous experience in reviewing research papers or evaluating student projects. Jury members should understand the principles of academic ethics, know the criteria for evaluating research papers and be able to clearly structure their comments for participants.
It is especially important to ensure the impartiality and objectivity of the jury members. This means that the experts must be independent and not have a conflict of interest with the contestants or organisers. Objectivity of evaluation is a critical aspect that helps to maintain trust in the results of the competition and ensure fair competition.
For competitions that cover several scientific disciplines or a broad area, it is important that the jury includes experts from different fields. This allows for a comprehensive evaluation of the work, especially when the research deals with interdisciplinary topics.
Note that in practice such a diverse approach to jury selection is very difficult to implement. Manual selection of experts is extremely time-consuming. Traditional selection of jury members often depends on personal contacts, recommendations or subjective assessments of the organisers. Without clear metrics, the assessment of competence remains subjective. Usually, the selection process is not documented, which makes it difficult to verify its fairness, generates distrust in the competition, and can lead to a decrease in the motivation of participants to participate in future competitions.
The task of automated jury formation for student research competitions is to develop a system
that ensures an objective, transparent and efficient selection of experts to evaluate the participants'
work. It is necessary to create an automated approach that assesses the competence of candidates
based on scientometric indicators, such as the number of publications, h-index and citations, as well
as altmetric data, such as mentions in the media and social networks. The system should
automatically analyse the profiles of potential jury members using data from bibliometric databases
such as Scopus [
Evaluation of a researcher's scientific activity is one of the most important problems that has been
considered almost since the very beginning of science. Before the fourth information revolution, the
contribution of a scientist to scientific progress was assessed mainly on the basis of qualitative
criteria, as a relatively small number of people were engaged in scientific activities. However, as the
number of scientists and the number of research papers increased, it became more difficult to assess
their activities using traditional qualitative methods. Figure 2 shows the dynamics of publication
activity in 2018-2022 [
This has led to the need to develop new approaches to assessing scientific effectiveness, in particular methods of quantitative analysis of performance through scientometric indicators.
Today, there are two main approaches to assessing the effectiveness of scientific activity: expert and statistical. The expert approach is based on subjective assessments of the quality of work, which has two significant drawbacks: the influence of the human factor and the high cost of conducting it. 179
The expert approach is based on a qualitative assessment of scientific activities carried out by
specialists (experts) in the relevant field [
The statistical approach, also known as scientometric, uses quantitative metrics to assess the
effectiveness of research activities. It is based on the analysis of data from bibliometric databases
such as Scopus [
Scientometrics was formed as a special methodological branch of science based on the description
of various aspects of scientific activity using mathematical methods [
The Hirsch index (h-index) is one of the most popular scientometric indicators for assessing the scientific productivity of researchers.
The Hirsch index or h-index is the maximum integer h, which means that the author has
published h articles, and each has been cited at least h times [
Today, there are a number of modifications and derivatives of the Hirsch index [
The Hirsch index does not take into account the level of an author's contribution to a
multiauthored paper. One of the co-authors may make only a minor contribution, but still receive the
same "credit" for their h-index. The h-index does not take into account how long ago the research
was published. Articles published many years ago may be cited more frequently due to their
longevity in the literature, but this does not mean that they are relevant. Citations vary greatly across
different scientific fields. Researchers in highly cited fields (e.g. medicine or physics) have a higher
h-index than those in the humanities or social sciences, where citations are usually lower [
Self-citations can artificially inflate a researcher's h-index, as self-citations are also added to the total citation count. Researchers who work in large teams or on collaborative projects can receive more citations due to the large number of co-authors and publications, even if their personal contribution is minimal.
The h-index value does not increase if one or more papers receive a large number of citations.
Even if a researcher has one or more single papers with a very high impact, this will not be reflected
in the h-index, as it only cares about the number of citations above a certain threshold [
As mentioned above, the desire to avoid these shortcomings has led to many modifications of the Hirsch Hirsch index. We have structured them into subsets and presented them in Tables 1-5.
The following subsets of indicators were identified: ‘Early indices based on the h-index’, “Aggregation-based indices”, “Indices that take into account time”, “h-related indices to assess scientific production at different levels”, “Other h-index related indices”.
Gives more weight to highly cited papers by addressing a limitation of the h-index where excess citations do not affect the index.
Measures the average number of citations in the top h publications (Hirsch core).
Index gives more weight to highly cited papers, similar to the g-index, but takes the square of the citations into account.
Unlike the h-index, it reflects the impact of highly cited papers, giving more weight to them.
Unlike the h-index, it considers only the most cited papers and their average impact.
More emphasis on highly cited works than the h-index.
Combines h-index and g-index using their geometric mean.
Combines the h-index (quantitative) and m-index (qualitative), creating a more comprehensive measure. Square root of total citations in the Hirsch core, reducing sensitivity to outliers.
Balances both h and g, minimising the extreme effect of highly cited papers on the g-index.
Expands on h-index by incorporating the citation impact of top papers, considering both quantity and quality.
Takes into account total citations in the core, offering a refined alternative to the h-index.
Altmetrics are a relatively new tool for assessing the impact of research activities, focusing on social and online activities related to research. Altmetrics have emerged as a response to the limitations of traditional scientometric indicators and the implementation of the concept of "open science". They aim to take into account modern changes in scientific communication and interaction with scientific publications through digital platforms.
Unlike traditional scientometric metrics (h-index, citations), altmetrics reflect the broader impact of research papers through their presence on social media, media, politics, and other platforms.
Altmetrics measure the attention paid to research papers in non-academic environments,
covering a wide range of sources. They were introduced in 2010 as an alternative to traditional
metrics to capture the impact of science in the digital age [
Adjusts the h-index by dividing it by Normalises the h-index based on career the number of years since the first length. publication.
Assigns lower weight to older articles, Focuses on recent work, unlike h-index prioritising newer contributions. which is cumulative.
Considers when citations have been Prioritises active researchers who are made, favouring recently influential still contributing, unlike the static hworks. index.
Combines the h-index and its growth A dynamic improvement, showing how rate, rewarding ongoing citation the h-core evolves over time. accumulation.
An extension considering the Provides a more nuanced view of cognitive effort or contribution of contributions in multi-author works by each author to the papers, adding evaluating individual input and quality. another layer to the impact measurement.
Accounts for citations across multiple papers, avoiding overestimation from self-citation.
Index counts the papers fractionally according to the number of authors.
Aims to balance between perfectionist and productive researchers by combining impact and total output, achieving a higher fairness rate than h-index.
A variation that introduces a weighting scheme to address the Matthew effect, which benefits highly cited researchers.
Complements the h-index by reflecting additional impact that the h-index ignores.
Focuses on citations across multiple papers in different scientific communities.
Adjusts the h-index to account for shared authorship, unlike the traditional full count.
Higher fairness index (91% vs. 80% for the classical h-index). Accounts for both the number of publications and quality more equally.
Reduces the effect of disproportionate citation advantages by counteracting the Matthew effect in some cases...
Altmetric data is collected from a variety of sources: social media, media, academic platforms, references to research in government documents, reports or patents, forum posts or podcasts.
The Altmetric Attention Score is a key indicator that aggregates data from various sources into a
single numerical score. It reflects the level of attention to the work, taking into account the weight
of the sources [
Altmetrics have certain peculiarities. Altmetric data accumulates faster than citations, which
allows for real-time impact assessment. They reflect the impact of research or work beyond the
academic community, including society, industry and politics. They span multiple platforms,
allowing for multifaceted evaluation [
The main platforms for collecting altmetric data: Altmetric [
Altmetrics can be useful for selecting experts for a jury, as they assess the impact of research papers not only through traditional citations, but also through mentions on social media, blogs, news, and other digital platforms. Altmetrics allow us to assess the extent to which an expert's research is relevant and discussed in the scientific and public sphere.
Altmetrics are updated much faster than scientometric indicators due to the dynamism of digital
platforms and real-time data collection. While altmetric data (mentions on Twitter, media) appear
within hours or days and are updated daily, scientometric databases (Scopus, Web of Science) [
Thus, altmetrics can be a useful complement to traditional scientometric methods of jury selection, providing a more comprehensive approach to assessing the competence of experts.
The task of selecting a jury for a student research paper competition can be interpreted as a MultiCriteria Decision-Making (MCDM) task. The task is to select the optimal set of experts (jury members), taking into account several criteria that reflect various aspects of competence, impartiality, scientific reputation, etc. let – a set of candidates in the jury
The generalised mathematical formulation of the problem is as follows: where is the i-th expert candidate.
It is necessary to select K experts for the jury.
= { 1, 2 , . . . , }, = 1, ;
≪ ; − a set of criteria, that determine the competence and ability of experts to be members of the jury. The score of each i-th candidate (hybrid metric) is defined as a weighted sum of β − weight for the altmetrics score (digital/social impact); γ − weight for the Org_Links indicator.
= { 1, 2 , . . . , }, = 1, ;
The whole set of criteria consists of subsets: a subset of classical h-criteria, a subset of altmetrics (Altmetric_Score) and a set of characteristics stipulated by the terms of the competition (Org_Links).
Then and where ℎ_ based on citations); of the i-th expert- candidate; − the composite h-index of the i-th expert- candidate (measuring academic influence _
− an aggregated altmetric score (measuring social, media, and public influence)
− the composite indicator is determined by the relationships of contestants and experts-candidates, relationship of experts-candidates among themselves, etc.
To justify the values of the weighting coefficients , , , it is proposed to use the group decisionmaking method, specifically the Delphi method, which involves engaging experts to assess the importance of each criterion.This will allow setting relevant weights for the h-index, altmetrics and conflict of interest indicators (org.idicators) depending on the specifics of the competition. In addition, the developed model allows dynamically changing weighting factors depending on the type of competition or scientific field. In competitions with high demands on the jury's scientific performance, the weight of the h-index coefficient can be increased, while the weight of altmetrics coefficient can be increased to assess the social impact of works. Simulations for different types of competitions confirmed that the adaptation of weights allows to increase the accuracy and objectivity of jury selection.
Besides, each of the composite scores has a complex structure: ℎ_ _
} = { 1, 2, … , } ;
} = +1, +2, … , ; } = { +1, +2, … , } ; | | = |ℎ_ | + | _ | + |
For example, the group of quantitative indicators of academic impact contains several assessments by which candidates in the h-index group are evaluated, and can be assessed according to the following criteria 1 − h-index; 2 − number of publications; 3 − citation index; The group of altmetric indicators includes 4 − interdisciplinarity (experience in various fields of science); 5 − social activity (participation in popular science events, digital footprint on the Internet); A group of indicators stipulated by the terms of the competition (Org_Links) 6 − no conflict of interest (independence); 7 − reviewer experience (e.g. number of reviews, quality of reviews); 8 − professional recognition (awards, participation in committees).
Each candidate is evaluated for each criterion . The score of candidate for criterion is denoted as , where is a numerical score that can be obtained using expert or quantitative methods. Thus, each candidate is characterised by a set of scores. The scores form groups according to their characteristics.
= ( 1, 2, … , ) = ({ℎ_ }, { }, { _ }); where is the vector of scores for candidate ji Thus, we have a hierarchy of weighted criteria.
Criteria can have different units and ranges of values, so it is necessary to normalise the data for
further comparison. One of the most common normalisation methods is Min-Max normalisation,
which converts all values to the range [
кнорм = −
( 1, 2, … , ) ( 1, 2, … , ) − ( 1, 2, … , ) where − the expert's score for criterion , ( 1, 2, … , ), ( 1, 2, … , ) − the minimum and maximum values for this criterion among all experts.
Formally, the problem can be represented as a multi-criteria optimisation problem with Boolean variables. otherwise.
Restrictions: =1 The goal is to maximise the total weighted sum of the scores of the selected experts: ∙ ℎ_ норм + ∙ _ норм + ∙ _ норм ∙ where ∈ {0,1} is a binary variable that equals 1 if candidate ji is elected to the jury and 0 – The number of experts is limited by M – the capacity of the expert group (jury): where K − the number of jury members set for the competition.
Experts must be independent and have no conflict of interest. This restriction can be implemented by building a graph of the applicants' relationships based on the use of information from scientometric and altmetric databases. The data on interdisciplinary outlook, social activity, review experience, and professional recognition are mainly from altmetric databases.
This model presents the task of selecting a jury as a classical multi-criteria decision-making problem using mathematical programming methods. It allows to objectively take into account several important criteria, such as scientific productivity, interdisciplinarity, and reviewing experience, and at the same time ensure a balanced and impartial jury.
To select the jury based on multi-criteria indicators, the TOPSIS [
The concept of the jury selection system, grounded in scientometric and altmetric indicators is based on a comprehensive assessment of academic productivity and social impact of experts using modern digital tools and metrics. The generalised concept of the application is shown in Figure 3.
System goals Automate the process of selecting experts based on formalised criteria. Identify and eliminate conflicts of interest. Ensure transparency of the jury formation.
The core of the system is the automatic retrieval of information about candidates from scientometric databases and altmetric platforms. Several popular APIs are integrated into the system to automate the collection of scientometric and altmetric data. In particular, the Scopus API is used to obtain the h-index, number of publications and citations, which allows receiving data in real time based on ORCID or other author identifiers. Altmetric API and Dimensions API are used for altmetric indicators, which collect information about social activity and distribution of posts. Automation occurs through periodic API requests, with data automatically updated and processed through a normalization system.
The frontend of the application provides convenient tools for selecting the necessary information. It is developed on the principles of cross-platform compatibility to provide access from different devices (personal computer, phone, tablet).
The main functions of the interface: Visualisation of metrics (graphs, charts to demonstrate the dynamics of the h-index,
Altmetric Attention Score). Filtering and searching (ability to search by researcher's name, publication topic, year of publication, or number of citations). Researcher comparison (a function for comparing researchers by various metrics to quickly assess their impact).
The backend of the application processes requests from users and is responsible for interacting with the API to retrieve data from the relevant scientometric and altmetric databases. The application server processes the received indicators, structures and aggregates them, and saves them to its own database. Automating this process through the API ensures that data is obtained quickly and conveniently, and the user-friendly interface makes it easy to interpret and use this data.
The described automated jury selection system is based on the integration of various tools for
186
collecting, analyzing, and processing scientometric and altmetric data. Information is gathered via
APIs from platforms such as Scopus, Web of Science, Altmetric, Dimensions, Google Scholar [
The sequence diagram shows how the system interacts with the scientometric database (Fig. 4).
The organiser enters the search parameters through the web interface. React.js implements the user interface and sends an HTTP request to the backend (Django) at the user's choice. The backend generates a request to the appropriate external database (scientometric or altmetric) to obtain the desired indicators, for example.
The bibliometric database processes the request and returns a JSON response with the requested data. The backend processes the received data, generates complex indicators and returns the processed data to the frontend for display. The results are displayed on the frontend.
The creation of an information system for conducting student research competitions is an essential task for improving the quality and transparency of evaluations. The proposed mathematical model for selecting jury members, based on scientometric and altmetric indicators, provides an objective framework for assessing experts by considering multiple criteria, such as the h-index, altmetrics, and conflict of interest indicators.
The application of the TOPSIS method for MCDM was justified, as it evaluates how close each expert is to an ideal solution, taking into account various criteria. TOPSIS effectively balances conflicting criteria and is particularly suitable for tasks like jury selection, where multiple indicators must be considered simultaneously.
While the h-index is commonly used for measuring scientific productivity, it is essential to account for its limitations. Altmetrics serve as complementary metrics, capturing the digital and social impact of research. Combining these two types of metrics creates a more comprehensive evaluation framework, balancing long-term academic influence with immediate digital engagement.
The integration of APIs from platforms such as Scopus, Web of Science, and Altmetric allows for the automatic collection of scientometric and altmetric data. This automation ensures timely updates, transparency, and accuracy in the selection process, reducing the potential for bias or manual errors.
Future research will focus on developing a mathematical framework for determining weight distributions in a hierarchical model for expert evaluations. This will further refine the assessment of jury members, ensuring an even more balanced and precise selection process that aligns with the specific goals of different competitions.
This comprehensive approach improves the objectivity of the jury selection process, optimizes decision-making, and contributes to the efficiency of conducting research competitions.
The automated jury selection system is built on the modern technology stack that includes advanced methods for expert data collection, analysis, and evaluation. The use of APIs, machine learning, multi-criteria analysis methods, and web technologies ensures the efficiency, scalability, and reliability of this system.
During the preparation of this work, the authors used Deepl.com in order to translate text and grammar check. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.