Social unrest caused by the crisis makes it difficult to forecast the market performance for the company, and, accordingly, the decision-making process. Today's situation creates the basis for food, migration, political, and energy crises. The complexity of the representation of the situation, and the generally chaotic nature of changes necessitates the use of a more comprehensive approach to forecasting market indicators. The dimensionality reduction and parallelism principles are traditional approaches to reduce the impact of data volumes on the execution time of an algorithm. For the current study, it was decided to choose a MapReduce technology to implement a parallelization approach. Based on the results of this study, the main approaches to forecasting economic indicators were defined within the current framework. The chronological non-determinism of the target data was declared, and their co-integration was considered accordingly. The obtained data made it possible to define a family of VAR models as the basis for forecasting these indicators. A set of factors as exogenous variables and related parameters were defined. The accuracy of forecasting the specified algorithm family was checked using the e-commerce market data during recent social disasters. The obtained indicator of parallelization efficiency of these algorithms using the MapReduce technology allows us to state the expediency of a similar parallelism form for vector autoregressive forecasting models in practical application.
by the SARS-Cov-2 coronavirus [
On February 24, 2022, the situation worsened as a result of the full-scale invasion of the Russian
Federation on the territory of Ukraine. According to various analytical agencies, the war provoked
various types of crisis phenomena at the global level: energy, food, migration, etc. [
2022 Copyright for this paper by its authors.
each case and without significant reference to the event that caused it. For example, the tragedy of
September 11, according to some scientific sources, still has a social impact on the USA [
During a social disaster, forecasting market activity becomes a complex and atypical task, which
without the use of modern information technologies and/or appropriate education is an almost
insoluble task. This causes the prevalence of risk forecasting related to consumer behaviour and
means of automating these processes. One such tool that is currently gaining popularity is forecasting
indicators using autoregression algorithms [
The concept of social vulnerability is most often considered in terms of risk analysis, which has
three key stages: definition, assessment, and management. In the 20th century, to optimize this
process, decision support systems (DSS) were created [
The current paper will consider passive data-oriented systems because it was decided to
concentrate on autoregressive models. An example of such systems can be [
Not all autoregressive models will be considered either. This is explained by the fact that, in most cases, market indicators, such as the level of demand, prices, and investments, are time series and should be considered complex. The reason for this can be considered the possibility of classical paradoxical situations (Giffen’s paradox, Veblen’s paradox, etc.) and impossibility of their reflection by a single indicator. Therefore, the current study will consider only vector autoregressive models (abbreviated VAR), in particular: vector autoregression of the moving average; vector autoregression of the distributed lag; vector seasonal autoregression.
Such a choice can be explained by the general volatility of economic indicators and the aggravation of this process during social disasters. In addition, it was decided to slightly modify the input data to take into account qualitative indicators that would describe the state of the disaster.
Carry out a step-by-step review of the selected models for research, their modifications, and the technology for their parallelization. 3.1.
Before proceeding to the modification of the basic VAR family algorithms, we will consider their mathematical represantation: where .– K-dimensional time series; – matrices K×K, dimensional vector of white noise with zero mean and nondegenerate covariance matrix . and – nondegenerate matrices.
It follows from the given formula (1) that the classic models predicts only static variables. To exogenous indicators consider, it was decided to use a modification of error correction (EC). Such an adjustment is necessary if several endogenous variables have a common stochastic trend, which is typical for indicators of the market activity.
The general formula for the modified EC-VAR family of algorithms will have the following form: (2) , , (1)
– K; , . where , ,
In the selected case, an important place is occupied by exogenous variables, in fact, they will serve as the basis for taking into account the social disasters in the quantitatively presented. 3.2.
In general, it’s accepted that a social disaster affects all market subjects, from individuals to the state. The business management theory indicates that the nature of the impact on individuals is reflected in the behavior of the target audience, changes in the company’s work and market regulation e (which is a representation of the state) are combined into the microeconomic profile of the market situation. Given the fact that for each indicator it is necessary to consider the disaster itself, we will begin the modification by analyzing the possibilities of converting its description into a quantitative form. For qualitative indicators to reflect an objective view of the disaster, it was decided to conduct an expert assessment among 5 risk managers of different companies in Kharkiv and Novomoskovsk, 5 sociologists from Kharkiv and Dnipro. The following four indicators were mentioned the largest number of times: a general textual description of a social disaster; the news is related to a catalyzing phenomenon; assessment of the company’s employees regarding readiness for unforeseen circumstances; the duration of the disaster (from the moment the catalyst begins to act).
The last indicator has a quantitative representation, and the subjective assessment of employees
can easily be converted into a quantitative form if a survey is using a scale from 0 to 100, where 0 is
“the company is not prepared for unforeseen circumstances”. and 100 – "the company is fully
prepared for unforeseen circumstances." After the survey, the result will be normalized to 1. For
conversion into quantitative form, we will use the principles of content analysis. In general, the
algorithm should be as follows:
1. remove non-letter expressions from the text, in particular numbers and punctuation marks;
2. divide the cleaned text into sentences and words;
3. apply the stemming operation – shortening the word to its base;
4. to prevent stemming errors, we carry out lemmatization – bringing the word form to the
lemma;
5. unify the created dictionary;
6. remove words with minimal linguistic load, for example, conjunctions;
7. find the TF-IDF frequency response [
We can formally describe the social disaster as follows: , where SDS – content analysis result for disaster description provided by the company; SDO – content analysis result for disaster description provided by news; t – disaster duration; R – readiness for unforeseen circumstances.
Now let’s move on to the main indicators of the profile of the market situation and the target audience. For the target audience, by analogy with the social disaster indicator, it was decided to conduct an expert assessment, during which it was found that the following factors are the most important for this indicator: general text description of the target audience; audience size; volume of income divided by total income.
The last two indicators are quantitative, so it is enough to normalize them from 0 to 1. To convert
the text description, we will use the principle of cluster analysis. The first 5 steps of the algorithm are
similar to those noted in the description of the disaster. The final steps are slightly modified:
6. summarize the audience’s description using a set of features proposed by Robert Plutchik:
expectation, anger, disgust, sadness, fear, surprise, hope, and trust [
7. form the emotional colour of each word from the audience description, giving the value of each feature in the range from 0 to 100; 8. summarize the obtained values, taking into account the sign of the emotion; 9. normalize the values in the range from 0 to 1; 10. consider the general market paradoxical situations mentioned earlier (Veblen, Giffen, snob effects) by introducing the appropriate indicator, which is calculated as the module of the difference between the normalized number indicator and the income share indicator.
We can formally describe the social disaster as follows: (3) where – coefficient of market paradoxes normalization, TAD – content analysis result for target audience description provided by the company, CD – normalized number indicator, RD – income share indicator.
It is worth noting that formula (4) does not consider the impact of a social disaster as such, so we introduce the following formula for the problem under consideration: ,
Each modified algorithms of vector autoregression treat external variables as numerical series, not scalars, so the indicator obtained as a result of calculation according to formula (5) must be vectorized. To achieve this, we’ll use the obtained result as the centre of the normal distribution, which will serve as an exogenous variable for the selected algorithms of the modified EC-VAR family. Let’s move on to the next indicator that needs to consider – the profile of the market situation. According to microeconomic theory, it can include a large number of different indicators. After surveying experts, the following characteristics were identified, which will be taken into account in the current paper: volumes of innovative activity; financial stability; (5) , (4) market monopolization; the state of the target industry and the world economy; social disaster.
To measure market monopolization, we will use the Herfindahl-Hirschman Index: where N – number of companies in the selected market, s – market share owned by each company.
Considering the fact that the e-commerce market has the characteristics of monopolistic competition with some regional manifestations of oligopoly, to find the approximate value of the index, it is enough to know the value of the market shares for the 5 largest companies on it. Innovative activity expressed by the relevant indicator is as follows: ,
, where IR – share of income from innovations, MT – total monetary mass of the company.
The resulting social catastrophe is characterized by the previously mentioned SSD indicator. The
financial stability of the company is a classic economic indicator (FSI) that reflects the company’s
ability to meet its obligations in the long- and medium-term perspectives [
As in the case of the TAOI indicator, a scalar value will be obtained during the calculation according to the formula (8). To vectorize it, we will use the result as the centre of the normal distribution. The obtained numerical series does not consider the state of the target industry and the world economy. To take into account the state of the economy, we will choose several classic indicators: world GDP level; prices for energy resources level; the S&P 500 index.
The indicated data are already numerical series, so to be able to use them as exogenous variables,
it is enough to normalize them. We will determine the state of the industry based on the portfolio of
shares of the five biggest companies in the e-commerce market. According to [
As in the previous case, the data are already numerical series, but they need to be normalized for the algorithms to work correctly. Finally, the following series will be considered as external variables for the family of modified EC-VAR models: target audience characteristics; company’s activities on the market characteristics t; world economy state; industry state.
Let’s move on to consider the theoretical basis of MapReduce technology, which will be used to parallelize the selected algorithms. 3.3.
The essence of the approach, which is embedded in MapReduce technology, is to distribute the
total data set to individual nodes. According to [
As we can see, in this case, in addition to the actual mapping and reduction functions, there are distribution and combination functions, which are necessary for the response from each node to arrive in a combined form. The results are sorted before the reduction. Among the advantages of the chosen approach, it is possible to single out scalability, relative cheapness, ease of use (although the technology is complicated by the need for configuration settings), and the possibility of monitoring execution. On the other hand, among the disadvantages, it is worth highlighting the need to create a large volume of programme code and the stealth of processing. There is also a need for a lengthy configuration setup.
By the experimental environment we mean the following set of characteristics: general conditions; efficiency function; rule for comparing the efficiency of two models; errors and uncertainties.
We will gradually determine each of the specified characteristics of the environment. 4.1.
Given the peculiarities of the algorithms, it was decided to use the method of a controlled experiment. This became the basis for choosing a permanent and stable execution environment – a physical device based on Ubuntu with the following technical characteristics: CPU: Intel Core i5-1135G7; RAM: 16 Gb; SSD: 512 Gb; OS: Ubuntu 21.04.
To determine the execution speed, it was decided to use the Python 3 with libraries for natural
language processing (nltk, re, etc.) and working with data (pandas, numpy, etc.). For MapReduce
implementation, as mentioned above, it was decided to use Hadoop technology with 3 cores. Two test
samples provided by Amazon [
In Figure 3, the data for Amazon is slightly different from the previous one and contains information about fluctuations in the stock market, unlike the sales levels for Walmart:
where – speed of forecasting execution; MSE – mean square error of the forecast; – data preparation speed.
All performance metrics will be determined using the time module of the Python 3. We will calculate the MSE indicator using the following formula: where N – number of predicted values; – real value; – predicted value. 4.3.
To compare the efficiency of two models, we will use the following formula: , , where – metric value for model А; – metric value for model В.
(9) (10) (11) : model A is more efficient than В; : model А is less efficient than B;
The determined parameter C is fuzzy, so we will use the following set of rules to determine the most efficient model:
: it’s impossible to determine the most efficient model. 4.4.
Within the framework of the proposed task and the described experiment, we can outline the following uncertainties and errors: during speed testing: human factor and instrumental error; during accuracy check: data problem.
To mitigate the impact of these problems, we set the number of measurements for the performance of parallelized and sequential modified autoregressive models equal to 10. The number of measurements for determining accuracy is equal to 5, and for the speed of preparation also 5.
The first step in implementation is to programming modified autoregressive models. To avoid errors in the implementation of successive versions, it was decided to use a library of the Python 3 – statsmodel. To create a MapReduce version, develop two script files for the reducer and the mapper. To simplify the perception, we show in Figure 4 the pseudocode for the mapping function:
The next stage of implementation is the content analysis algorithm. As already mentioned, for this
it was decided to use the nltk library, which contains a large number of different corpora of words.
The Porter stemmer will be used as a base for stemming, and the WordNet lemmatizer will be used
for lemmatization. The TF-IDF frequency characteristic will be obtained using the sklearn library.
Part of the code that performs content analysis is shown in Figure 5. Data related to stock prices were
chosen to be taken from the Alpha Vantage API. They will be combined into special data frames
using pandas, a library that makes it easy to work with large amounts of data. Their aggregation and
subsequent transformation for transmission as exogenous variables will be carried out at the expense
of the already mentioned sklearn library. It was decided to take information related to the world
economy as a whole in csv format from the database of the World Bank [
We present Figure 6 as a confirmation that the EC-modification of the algorithms by adding the processing of the previously given data was necessary. Figure 6 (actual value in orange and predicted value in dashed blue) showing a modified implementation of the moving average vector autoregression algorithm indicates that the data in the basic algorithm “a” doesn’t consider the fluctuations, but the improved algorithm “b” provides fairly accurate results. Now we can proceed to the obtained results. Let's start with the execution time of the forecasting algorithms, information on 10 measurements is given in Table 1 (VARS – seasonal autoregression, VARL – distributed lag autoregression, VARMA – moving average autoregression).
Let’s find the average value for each case. We have 0.163 s for the serial VARL, and 0.064 s for the MapReduce version, we will get a speed gain of ~2.55. For VARS sequential we have 0.179 s, based on MapReduce technology – 0.062 s, we get a speed gain of ~ 2.89. For serial VARMA we have ~0.205 s, for the parallelized version – 0.078 s, we get a speed gain of ~ 2.63. So, the average speed gain is 2.69. The obtained value already allows us to state the expediency of using the technology for forecasting algorithms of the VAR family. To additional confirmation, the number of cores was increased to 4, as a result, the average speed gain for the set of models was 3.74. Now let’s consider to comparing algorithms of the VAR family with each other.
The first step of this is to check the accuracy of the forecast for the above samples. The obtained aggregated result of 5 measurements is shown in Table 2 (the result is rounded to a whole value).
As we can see from the Table, in both cases, the moving average algorithm is the most accurate, which explains its wide applicability in econometric analysis. The resulting accuracy is based on the fact that the moving average algorithm uses neighbouring values of the numerical series, this somewhat smoothest the overall forecast, making it more natural. b Figure 6: Forecasting results for Walmart’s dataset
Regarding the time of data preparation, it is about 291 s for all algorithms. To reduce the impact of this time, it is necessary to parallelize the quality indicators processing. As part of the experiment, this was done using MapReduce technology (3 cores were used). As a result, the average preparation time was 104 s, i.e., the gain in speed is ~2.8. With an increase in the number of cores, the time was 78 s, that is, the gain in speed increased to ~3.73. In the case of comparing successive algorithms with each other using formula (11), it was determined that the VARMA algorithm is the most effective when considering the e-commerce market during social disasters (in both cases, the indicator exceeded 3). However, the use of parallelization, especially with 4 cores, indicates a significant time advantage of the parallelized versions, while the prediction accuracy remains unchanged. This allows us to state the expediency of implementing this version within the framework of the proposed task.
The purpose of this work was to determine the effectiveness of using MapReduce technology to parallelize autoregressive models for e-commerce market indicators during social disasters. For this purpose, an analysis of the subject area was carried out and the main methods of forecasting market indicators were classified, the reason for choosing vector autoregression models as the basis for further research was indicated. It was found that the nature of the input data requires modification of these models, considering the non-determinism of external factors.
During the theoretical examination of these models, their features were clarified and parallelization possibilities were outlined.
The choice of MapReduce, based on Hadoop technology is justified for the next three algorithms: vector autoregression of the moving average; vector autoregression of the distributed lag; vector seasonal autoregression.
The need for additional modification was determined to take into account exogenous variable indicators characterizing the microeconomic profile of the market situation and the characteristics of the target audience during social upheavals. In accordance with the tasks set, an experimental environment was constructed to test the feasibility of using the MapReduce technology and the created modification.
Based on all the above modifications and using the MapReduce technology, a series of experiments was conducted with the data of the companies Walmart regarding the level of sales, and Amazon regarding indicators of stock market activity. In the course of the experiments, it was found that the highest accuracy (in both cases more than 93%) is guaranteed by the use of vector autoregression of the moving average. At the same time, it was found that the gain in the speed of the algorithms during parallelization can reach 3.73, and in data preparation – 3.73.
Therefore, we can state that the use of MapReduce technology in combination with additional modifications of the basic algorithms, related to taking into account the non-determinism and high volatility of the indicators of the external environment, is expedient.
The authors would like to thank the Armed Forces of Ukraine for the opportunity to write a valid work during the full-scale invasion of the Russian Federation on the territory of Ukraine.