The purpose of the study is to substantiate a model for predicting the competitive score of social projects for community development with the optimization of the architecture of a deep neural network trained on the prepared data from the implementation of previous projects. The proposed methodology for predicting the competitive score of social projects for community development is based on the use of deep neural networks and includes ten stages. These stages are reflected in the developed research algorithm. The peculiarity of the proposed methodology is that the database is formed on the basis of the collected data on the results of competitions for local initiative projects in the region. In the course of the research, 6 types of modern deep neural network architectures (FNN, RNN, LSTM, GRU, CNN, RCNN) were used. To evaluate the models of a given deep neural network architecture, such metrics as MSE, MAE, R2 Score, RMSE, Inverse RMSE, and ms/step were used. It was found that the best accuracy rates are provided by the model based on RCNN. It showed the best results among all models. It has the lowest values of MSE=6.962, MAE=2.11, and RMSE=2.638, and the highest R2Score=0.43, which indicates a better ability to explain variability in the original data. We have optimized the basic RCNN model using 4 options: 1) increasing the number of layers and neurons; 2) using another optimizer; 3) using Dropout regularization; 4) changing the loss function. It is established that the RCNN model using Dropout regularization is the best for predicting the quantitative value of the competitive score of social community development projects. Relative to the baseline RCNN model, there was a decrease in MSE by 1.22% and a decrease in MAE by 2.51%. Further research should be conducted in the direction of developing a decision support system for planning social community development projects from the proposed RCNN model using Dropout regularization to predict the competitive score of social community development projects.
In today's world, social projects for community development play an important role in improving the lives and activities of their populations [1-3]. In order to efficiently allocate resources and support the most promising initiatives, it is important to have accurate methods for predicting the success of social projects submitted for competitions. In recent decades, artificial intelligence has become a key tool in the development of society [4-6]. It can radically change a number of industries, including the social sphere and civic activities. In particular, the use of deep neural networks for forecasting is becoming increasingly common among project managers who select social projects to be submitted to a competition. Such competitions of social projects of local initiatives are held in the territory of individual communities and regions [7].
Since the threshold for a passing score fluctuates from year to year, the forecast results are the basis for a preliminary assessment by both the developers of the competition proposals and project managers of the possibilities for obtaining a passing score or improving the competition ideas. To determine a rational model for predicting the competitive score of social projects for community development, 6 types of the most common deep neural networks were considered. Subsequently, the best of the obtained models was optimized according to various criteria [8-10]. Our paper presents the results of data collection and intellectual analysis of the results of competitions for local initiative projects in the Lviv region (Ukraine). Based on the results, we conducted a study. It concerns the justification of a rational model of a deep neural network. Its architecture was also optimized, which ensures accurate prediction of the competition score of social projects submitted to competitions.
Thus, the rationale for the deep neural network model for predicting the competitive score of social projects for community development is based on the collection and use of data on previous projects, including project description, budget, community capacity, and competition results. This data was used to train a neural network that subsequently provides accurate forecasting of the competition score of new social projects submitted for the competition. The resulting model is able to demonstrate high prediction accuracy on the test data set. This indicates that it is a valuable tool for assessing the prospects of social projects for community development. 2. Analysis of published data and problem setting Today, researchers pay a lot of attention to the development of tools for project management [11-13], including for the management of social projects [14-16]. They have made a significant contribution to the development of science in this area. However, with the development of computational intelligence, research is carried out annually on the development of machine learning-based tools for various spheres of human life and activity [17-19]. At the same time, the attention of scientists is focused on the development of tools for project management based on computational intelligence technologies [20-22].
Taking into account the relevance of using both computational intelligence and the expediency of developing tools for project management, our work investigates the use of deep neural networks to predict the competitive score of social community development projects. Justification of an effective model based on deep neural networks is an urgent task for both science and practice of holding competitions for local initiative projects in individual communities and regions.
Based on the analysis, it was found that there are many scientific papers devoted to this area of research. We have analyzed the most relevant scientific works in this area. Noteworthy are the works [23-25] that deal with predicting project success using neural networks. In these works, their authors used different computational models for forecasting. Each of these papers contains the results of model development and proof that these models improve the results compared to traditional machine learning methods such as logistic regression and decision trees.
In [26-28], it is proved that recurrent neural networks are important models in the field of deep learning. This network structure is used to recursively form complex deep networks with a simple structure. It is possible to add additional weights to the network, which ensures the creation of cycles in the network graph. In addition, it is possible to use information about long-distance dependencies, which ensures high prediction accuracy with sufficient data quality. The learning rate of recurrent neural networks cannot be improved, and the gradient gradually disappears. LSTM improves the nodes of the hidden layer of the RNN into special cellular structures that can perform better in longer sequences. Paper [26] investigates the cyclic neural network algorithm and its improved model based on LSTM, and builds a model for predicting athletes' performance based on a cyclic neural network that can be used to predict athletes' performance with high prediction accuracy.
There are also studies [29-32] that have been conducted to predict the success of crowdfunding projects based on the texts of online social welfare crowdfunding projects. By calculating the amount of information and analyzing the sentimental value of the text, the authors studied how the textual information of an interconnected social security crowdfunding project affects the success of the project. It is found that imperfect R-squared indices reflect that multiple linear regression models perform poorly with respect to this prediction. Furthermore, this paper tests and analyzes the prediction performance of four machine learning models, including a multiple regression model, a decision tree regression model, a random forest regression model, and an AdaBoost regression model.
Recently, several methods have been proposed to explain the predictions of recurrent neural networks (RNNs), including LSTM [33-36]. The goal of these methods is to understand the network's decisions by assigning a relevance to each input variable, such as a word, indicating the extent to which it influenced a particular prediction.
In existing works, some of the architectures of neural network models have not been compared with each other or evaluated only qualitatively. We propose to fill this gap by quantitatively comparing different neural network model architectures for predicting the competitive score of social community development projects. Using the model that will show the best results during the research, perform optimization for it using different options. Thus, the objective of this paper is to substantiate a qualitative deep neural network model for predicting the competitive score of social projects for community development. The model will be trained on a dataset containing information about previous projects, including project description, budget, and competition results. 3. The purpose and objectives of the study The aim of the paper is to substantiate a model for predicting the competitive score of social community development projects based on an optimized deep neural network architecture trained on prepared data with the results of previous projects, including their description, budget characteristics, community capabilities, and competition results.
To achieve the presented goal, the following tasks should be solved: 1. to propose a research methodology and prepare data that will ensure the training of deep neural networks for predicting the competitive score of social projects for community development;
2. to substantiate and optimize the architecture of the deep neural network model for predicting the competitive score of social projects for community development and to evaluate its accuracy. 4. Research methodology and data preparation for training deep neural networks The main stages of the research methodology for substantiating and optimizing the architecture of the deep neural network model for predicting the competitive score of social community development projects, as well as evaluating its accuracy indicators, are shown in Fig. 1.
The proposed algorithm involves 10 main stages, which include the collection and formation of a database of implemented social projects in the territory of communities based on data with competitive proposals of social projects in the territory of communities. In particular, we have collected data on the results of competitions for local initiative projects in the Lviv region during 2018-2021. They reflect the indicators of the implementation of social community development projects in the Lviv region, which were funded from the regional budget. We received 741 copies of data on implemented social projects (Lviv region, Ukraine) (Fig. 2), which are distributed by attributes: 1) year of project implementation (Year); 2) project number (No); 3) project registration number (Registration_number); 4) project name (Project_name); 5) name of the territorial community (Territorial_community); 6) name of the settlement (Settlement); 7) total project budget (Project_budget); 8) funds from the regional budget (Regional_budget); 9) funds from the district budget (District_budget); 10) funds from the territorial community (Public_budget); 11) sponsorship funds (Sponsorship_funds); 12) financial contribution (Financial_contrib); 13) non-financial contribution (Nonfinancial_contrib); 14) percentage of non-budgetary contribution (%_non-budgetary_contrib); 15) total score (Total_score); 16) expert score 1 (score_E1); 17) expert score 2 (score_E2); 18) Expert 3 score (score_E3); 19) Taxcapacity_index; 20) Score_taxability_index; 21) Final_score; 22) Results of project selection (Results).
optimizer.
regularization.
no no 1. Collecting and forming a database of implemented social projects in
communities 2. Intelligent analysis and data preparation for training models for predicting the competitive score of social projects 3. Selection of deep neural network architectures for predicting the competitive score of social projects for community development 4. Define the architecture of the deep neural network
neural network architecture by individual metrics Have you considered all architectural options?
yes
yes
The interactive software environment Jupyter Notebook was used to intelligently
analyze data on the results of competitions for local initiative projects in the Lviv region.
This made it possible to analyze the data and quickly experiment with the code and
demonstrate the results. Based on the data analysis, it was found that some of them
(Total_score, score_E1, score_E2, score_E3) duplicate the final score (Final_score) and were
removed from the DataFrame (df). During the preliminary processing of the data in the df,
we analyzed it for gaps and anomalies. In addition, the values of individual columns of the
df were converted to floating point numbers (float64) and integers (int). To determine the
type of settlement, we used a function based on a simple method based on the prefix:
df ' Settlement ' = df ' Settlement '.apply (lambda X : map _ settlement ( X )) , (
The proposed map_settlement function (
For preliminary data processing, we used the scikit-learn library of the Python programming language to analyze the results of the local initiative project competitions. As a result, we have prepared data that will provide training for deep neural networks to predict the competitive score of community development projects (Table 1). t c e j o r P t n e m e l t t e
S … … …
To improve the efficiency of training deep neural network models, we scaled the data. To do this, we used normalization and minimum scaling of the data of individual df columns. b i r t n o c _ l a i c n a n i F
X8 0 … 15000 29833 b i r t n o c _ l a i c n a n i f n o N
X9 25.1 25.3 20.6 … 25.1 33.8 b i r t -no cy_no n r _% tae g d u b x e d n j y t i c a p a c x a
T X10
X11 0.7097 0.5 0.4156 1.5 0.4144 1.5 …
… 0.7097 0.5 0.6075 1.0 x e d n j y t i l i b a x a t _ e r o c S e r o c s _ l a n i
F
Y1
31.833
31.500
31.167
31.167
31.000
This ensured that the values Хі of the df columns were transformed so that all data in the
range from 0 to 1 were obtained:
Х і = ( Хі − min ( Хі )) / (max ( Хі ) − min ( Хі )) ,
(
We have determined the input parameters of the model for predicting the competitive
score of social community development projects based on the use of deep neural networks.
At the same time, we selected attributes that are closely related to the target feature (final
score) due to the constructed correlation matrix. For each input parameter of the model,
which are shown in Table 1, we determined their average value Хі :
(
The correlation matrix Kіj is constructed, the elements of which are determined using the formula: where cov( Хij ,Y1 ) – covariance between the input data Хij and the target feature Y1 .
The quantitative value cov( Хij ,Y1 ) is determined by the formula:
cov( Хij ,Y1 ) = 1 N( Хlі − Хі )( Хlj − Х j ),i, j = 1,m . (
N −1 l=1
As a result of the calculations, a correlation matrix between the input data and the target feature "Final_score" was built (Figure 4).
The analysis revealed strong and weak correlations between the input data and the target feature "Final_score". Based on the analysis of this correlation matrix, we have an idea of the existing links between funding sources, characteristics of social projects and their final scores. The information obtained may be valuable for further analysis or model development. 5. The results of substantiation and optimization of the architecture of the deep neural network model for predicting the competitive score of social projects for community development We begin the model justification with the choice of deep neural network architectures that will ensure the prediction of the competitive score of social community development projects. When choosing deep neural network architectures for predicting the competitive score of social community development projects, we took into account the features of the data and the task of prediction. We propose to use the following deep neural network architectures (Figure 5): 1) Feedforward Neural Network (FNN); 2) Recurrent Neural Network (RNN); 3) Long Short-Term Memory (LSTM); 4) Gated Recurrent Unit (GRU); 5) Convolutional Neural Network (CNN); 6) Recursive Neural Network (RCNN).
A linear neural network FNN is used to make predictions based on input features without depending on previous time steps or sequences. If the data has a complex nonlinear structure, FNN may be ineffective for modeling it. Recurrent neural network RNN provides processing of sequential data. However, RNNs can face the problem of vanishing gradient when training on long sequences, which leads to limitations in the model's ability to account for long-term dependencies. LSTM and GRU neural networks are advanced versions of RNNs that solve the vanishing gradient problem by providing special gates that allow the model to store and update information for a long time. They are especially effective for modeling long-term dependencies in sequential data. CNNs are commonly used for image processing, but they can also be useful for sequence analysis, as they can be transformed into a form that preserves spatial structure. Such a deep neural network is useful for analyzing relationships between attributes at different levels of abstraction. The convolutional neural network RCNN combines recursive and convolutional layers, which allows the model to analyze sequences and structural relationships in the data. This is useful for modeling sequences that reflect the relationships between social project evaluations and their characteristics.
We used the Sequential model from the Keras library, which is a sequential neural network where layers are arranged one after the other. This is the simplest type of model that is well suited for many types of machine learning tasks. It is proposed to use deep neural network architectures with 3 to 7 layers with the ReLU activation function. In the last output layer, only one neuron is used, since the task of predicting the competitive score of social community development projects is a regression task. When compiling the models, the Adam optimizer is used and the root mean square error is used as a function of loss. The models are trained on training data for 100 epochs.
To evaluate the models of a given deep neural network architecture, we used the
following metrics. An indicator for assessing the accuracy of regression models (Mean
Squared Error), which is defined as the root mean square difference between the predicted
and actual values of the competitive score of social community development projects:
1 N
MSE = (Yi − Yˆi )2 . (
N i=1 where Yi – real value of the competition score of social projects for community development, points; Yˆi – predicted value of the competitive score of social projects for community development, points; N – number of data on social projects in the dataset, units.
An indicator for assessing the accuracy of regression models MAE (Mean Absolute
Error), which is defined as the average value of the difference between the predicted and
actual values of the competitive score of social projects for community development:
(
The coefficient of determination ( R2Score ) is used to assess the quality of a regression model. It shows how well the model reflects the variation in the dependent variable. This indicator is determined by the formula: where SSres – sum of squared errors; SStot – total sum of squares.
The closer the value is R2Score to 1, the better the model explains the variation in responses.
The next indicator shows the square root of the root mean square error ( RMSE ). It is used to measure the average size of regression model errors and is calculated by the formula development, points; Yˆi – predicted value of the competitive score of social projects for community development, points; N – number of data on social projects in the dataset, units.
MAE = 1 N
Yi − Yˆ .
N i=1 R2Score = 1 − SSres .
SStot а)
The indicator RMSE is measured in units of the original variable, i.e., in our case, in the competitive scores of social community development projects. The smaller the value RMSE , the better the model's predictions.
The inverse square root of the mean ( Inverse RMS ) is worthy of note: This is simply the inverse of RMSE .
Inverse RMS = 1 . (
RMSE
If RMSE it measures the average size of the prediction errors of the competition scores of social projects for community development, the inverse indicator Inverse RMS , measures how many times the average prediction value deviates from the actual value. In other words, the higher the value of Inverse RMS , the lower the value of the model error.
Based on the above-described variants of the architecture of the studied deep neural
network architectures (Figure 5), models for predicting the competitive score of social
community development projects were created. For each of them, metrics were determined
using formulas (
They have the highest MSE, MAE, and RMSE values, as well as a very low R2Score, which may indicate a poor ability to explain variability in the data. The model based on CNN (Convolutional Neural Network) showed acceptable results, but slightly worse than FNN and RNN. The model based on RCNN (Recurrent Convolutional Neural Network) showed the best results among all models. It has the lowest values of MSE=6.962, MAE=2.11, and RMSE=2.638, and the highest R2Score=0.43, which indicates a better ability to explain variability in the original data (Figure 7).
a) b)
Thus, the best model for predicting the competitive scores of social community development projects, taking into account the numerical values of the presented metrics, is the RCNN-based model. This model demonstrates the lowest values of prediction errors and the highest level of explanation of variability in the original data.
Analyzing the rational architecture of the RCNN deep neural network model, we see that it has a sequential stack of layers. The first layer Conv1D has 32 filters and a kernel size of 3. This layer is responsible for extracting features from the input data. After that, the MaxPooling1D layer is applied, which reduces the dimensionality of the original data by half. Then there is a second layer Conv1D with 64 filters and a kernel size of 3. It is also responsible for feature extraction. After that, the MaxPooling1D layer is used again, which reduces the dimensionality of the data by half. The data is flattened with Flatten for further input into the fully connected layer. The fully connected layer has 64 neurons and uses the relu activation function. The last layer is the output layer with one neuron without activation, as we are predicting a numerical value. The total number of parameters in the model is 10,561. This includes weights and offsets for each layer. All model parameters are training parameters. The resulting RCNN deep neural network model has a fairly simple structure with only convolutional and fully connected layers. This architecture is often used to process sequential data and it makes it possible to accurately predict the quantitative value of the competitive score of social community development projects.
Subsequently, we optimized the obtained rational deep neural network model to predict
the quantitative value of the competitive score of social community development projects.
It is proposed to optimize the model by 4 options: 1) increasing the number of layers and
neurons; 2) using another optimizer; 3) using Dropout regularization; 4) changing the loss
function. For each of the variants of model training, metrics are determined using formulas
(
of layers and neurons
7.380 7.777 6.877 8.171 MAE 2.110 2.133 2.248 2.057 2.226
0.430 0.395 0.363 0.437 0.330
2.638 2.717 2.789 2.622 2.859
RMSE 0.379 0.368 0.359 0.381 0.350 ms/step 2 1 1 2 1
The basic RCNN model has good performance in terms of accuracy in predicting the quantitative value of the competitive score of community development projects. However, there is room for improvement. The proposed model using Dropout regularization shows the best results in terms of MSE, MAE, and R2 Score. It achieves the lowest mean square error and the highest coefficient of determination (R2 Score), which indicates the model's better ability to explain variability in the original data. The model with the modified loss function has the worst performance among all models. This may indicate that the chosen loss function is not suitable for the given forecasting task. Regarding the training time of the models, the models with RMSprop optimizers and the modified loss function require less time for each step, but this is not a decisive factor, since the time difference is very small. Thus, taking into account the obtained quantitative values of the accuracy indicators of the optimized RCNN models, the model using Dropout regularization is the best for predicting the quantitative value of the competitive score of social community development projects.
For the deep neural network model using Dropout regularization, a graph of the dependence of actual values on the predicted values of the competitive score of social projects for community development was constructed (Fig. 8, a), as well as the distribution of model residuals (Fig. 8, b).
а) b)
The resulting graph of the dependence of the actual values on the predicted values of the competition score of social community development projects (Fig. 8, a) shows the actual values of the competition score on the abscissa axis. These values represent the actual values of the competition score during the implementation of social community development projects in Lviv Oblast. The ordinate axis shows the predicted values of the competition scores of community development projects, which were obtained using a deep neural network model with Dropout regularization. Each point on the graph represents a pair of actual and corresponding predicted values of the competition scores of community development projects. The more the point is shifted up and to the left, the greater the error in the prediction of the deep neural network model using Dropout regularization. The diagonal line (black dashed line) represents the ideal case when the actual and predicted values coincide. The points close to this line indicate accurate predictions of the competition scores of community development projects.
The constructed distribution of residuals of the deep neural network model using Dropout regularization displays on the abscissa axis the residuals, which is the difference between the actual and predicted values of the competition score. The ordinate axis shows the number of observations falling into the corresponding residual intervals. This distribution makes it possible to determine how evenly the model residuals are distributed. The ideal case is when the residuals of the competition scores of community development projects are distributed evenly around zero without significant deviations.
Both graphs presented in Fig. 8 are important for assessing the accuracy and efficiency of the model in predicting the competitive scores of social projects. The first graph allows us to visualize how well the model predicts the values, and the second graph helps us understand the distribution of model errors. It has been found that the vast majority of error deviations are between -2 and +4 points, which meets the requirements for the accuracy of forecasting the values of the competitive scores of social development projects. 6. Conclusions 1. The proposed methodology for predicting the competitive score of social projects for community development is based on the use of deep neural networks and includes ten stages. These stages are reflected in the developed research algorithm to substantiate and optimize the architecture of the deep neural network model for predicting the competitive score of social community development projects (Figure 1). The peculiarity of the proposed methodology is that the database is formed on the basis of the collected data on the results of competitions for local initiative projects in the region. Intelligent analysis of the collected data makes it possible to assess the relationship between the input characteristics and the competitive score of social community development projects. The described methodology is the basis for performing high-quality data preparation, which is the basis for justifying an effective deep neural network model for predicting the competitive score of community development projects.
2. Based on the developed methodology, as well as on the collected data on the results of competitions for local initiative projects in the Lviv region during 2018-2021, the data were prepared for training deep neural network models. The research used 6 types of modern deep neural network architectures (FNN, RNN, LSTM, GRU, CNN, RCNN). To evaluate the models of a given deep neural network architecture, such metrics as MSE, MAE, R2Score, RMSE, Inverse RMSE, and ms/step were used. It was found that the best accuracy rates are provided by the model based on RCNN (Recurrent Convolutional Neural Network). It showed the best results among all models. It has the lowest values of MSE=6.962, MAE=2.11 and RMSE=2.638, as well as the highest R2Score=0.43, which indicates a better ability to explain variability in the original data. We have optimized the basic RCNN model using 4 options: 1) increasing the number of layers and neurons; 2) using another optimizer; 3) using Dropout regularization; 4) changing the loss function. It is established that the RCNN model using Dropout regularization is the best for predicting the quantitative value of the competitive score of social community development projects. The use of Dropout regularization in the model brought a slight improvement in the R2 Score by 1.63%, which indicates a better ability of the model to explain the variance in the original data. At the same time, relative to the baseline RCNN model, there was a decrease in MSE by 1.22% and a decrease in MAE by 2.51%. Further research should be carried out in the direction of developing a decision support system for planning social community development projects based on the proposed RCNN model using Dropout regularization to predict the competitive score of social community development projects.
Linguistics and Intelligent Systems (COLINS-2019). Volume I: Main Conference,
Kharkiv, Ukraine, April 18-19, 2019, Vol-2362: pp.96-105.
[23] H. Youneszadeh, A. Ardeshir, M.H. Sebt, Predicting project success in residential
building projects (Rbps) using artificial neural networks (ANNS), Civil Engineering
Journal (Iran), 2020, 6(
Bykovyy, E. Mikhno, O. Osolinskyi, Deep multilayer neural network for predicting the
winner of football matches. International Journal of Computing, 2020, 19(
Corn-to-Sugar Process, Processes, 2023, 11(
Artificial Intelligence in Education and Teaching Assessment, 2022, pp. 63–67.
[36] V. V. Morozov, O. V. Kalnichenko, O. M. Mezentseva, The method of interaction modeling
on basis of deep learning the neural networks in complex it-projects. International
Journal of Computing, 2020, 19(