Understanding the Relationship between Digital Inclusion and E-Participation1

Abstract

The potential impact of the digital divide on e-participation has long been a topic of discussion in research and practice. In general terms, it is expected that more access to the Internet and related technologies has a positive impact on the level of participation via online channels. This ongoing research paper takes a different approach to the subject by examining the relationship between eparticipation and the barriers to digital inclusion, expressed as the 3As - access, affordability, and ability - within the framework of the UN's Leave No One Behind (LNOB) goal. In this context, the aim of the study is to examine the relationship between digital inclusion and e-participation. To achieve this goal, multilinear regression analysis was conducted using cross-sectional data, including variables obtained from various international databases such as the UN, the World Bank, and the ITU for a sample of 192 countries. Preliminary results suggest that some of the LNOB-related variables have an impact on the level of e-participation development. Next steps will include additional analyses and a more detailed interpretation of the initial results.

1. Introduction

As a result of the rapid adoption and widespread use of information and communication technologies (ICTs) by government and society, the way citizens interact with public organizations has significantly changed [ 1 ][ 2 ]. This transformation has not only enabled citizens to benefit from online information and services but has also opened the door for them to engage more actively with government agencies through digital and mobile technologies. In this context, e-participation has the potential to have a significant impact on strengthening citizen participation [ 3 ]. Indeed, online participation can help to reduce social exclusion by enabling marginalized citizens to access the internet and its benefits [ 4 ]. However, as the world becomes increasingly interconnected, the digital divide has emerged as a critical barrier to achieving inclusive and participatory governance [ 5 ][ 6 ].

The path to a participatory system involves removing barriers to the participation of socioeconomically disadvantaged groups and encouraging them to participate [ 7 ][ 8 ]. Citizen participation is meaningful when equal participation opportunities are provided in society. In this context, e-participation requires equal access to and utilization of ICTs [ 9 ]. Some scholars claim that new developments in ICTs and e-participation channels reduce disadvantages [ 10 ][ 11 ], while other scholars argue that inequalities are deepening [ 12 ][ 13 ]. In short, the relationships between e-participation and the digital inclusion of disadvantaged groups remains a topic that needs more attention.

The digital dimension of the Leave No One Behind (LNOB) goals, including the digital divide/inequalities and participation, has been highlighted in the UN 2022 E-Government Survey report, which emphasizes that "The new face of inequality is digital." The issue of the digital divide is considered a significant obstacle in terms of e-government development and eparticipation levels [ 6 ][ 14 ]. Therefore, in the age of digital transformation, the UN' LNOB promise has emerged as a critical initiative to bridge the digital divide and ensure inclusive participation in the digital era [ 14 ]. The LNOB indicators developed in recent years have brought a somewhat different perspective to the issue in addition to traditional digital divide indicators. Within this framework, the support of digital inclusion policies, the prevention of digital exclusion, and the provision of equal digital opportunities for everyone are the main goals of the LNOB initiative.

The purpose of this study is to analyze the relationship between digital inclusion and eparticipation, using LNOB indicators. In this context, the relationship between the 3A indicators - Access, Affordability, and Ability - considered as barriers to the LNOB promise and the level of e-participation will be examined. The analysis shows the relationship between digital inclusion and e-participation in the context of the LNOB indicators through a cross-country analysis. As a result, by interpreting the relationship between changes in barriers to LNOB and changes in the level of e-participation, multiple indicators are tested and insights into the relationship between digital inclusion and e-participation are drawn.

2. Literature Review

As is known, e-participation, in its broadest sense, refers to the use of ICT to support citizen participation in public decision-making processes [ 1 ][15][16]. In the context of the 2030 Sustainable Development Agenda, citizen participation is a key necessity for sustainable development, as emphasized in SDG target 16.7, which aims to "ensure responsive, inclusive, participatory and representative decision-making at all levels." Therefore, e-participation is recognized as a significant catalyst for sustainable development and is among the objectives of the LNOB goal [17]. Furthermore, with the COVID-19 pandemic, it has become even more evident that there are inequalities in access to and usage of ICTs in every society. Digital inequality in the evolving digital society is a challenge for both developed and developing countries [ 14 ]. The importance of e-participation has been brought back into focus in addressing the shortcomings and needs observed during the COVID-19 pandemic in terms of online public information and service delivery and citizen engagement. In this context, the relationship between e-participation and LNOB promise has gained a different significance within the framework of the SDGs.

For e-participation policies to be successful, they need to encompass all segments of society and have an egalitarian structure. In this context, one of the most important concepts that needs to be examined in relation to e-participation is digital divide (or inclusion/exclusion). The relationship between e-participation and digital inclusion is valuable to both understand and evaluate the level of development and shortcomings of e-participation in a country, as well as the success of inclusion policies in the digital realm.

At the core of the digital inclusion/exclusion debate lies the issue of the digital divide. The digital divide is a concept used to describe the gap between those who have access to digital technologies and those who do not [18], and it signifies the deepening of inequality in favor of a privileged group. Predicated upon the premise that individuals with internet connectivity enjoy a more advantageous societal position compared to those without it [ 12 ], the digital divide transcends mere access to encompass levels of technological literacy and proficiency [19][20]. Consequently, the digital divide serves as a conduit for the reflection of societal disparities onto the digital landscape, creating what is commonly referred to as digital inequality, an integral facet in discussions pertaining to the demand side of e-participation [21][22].

As the prevalence of digital opportunities increases in a society, it seems clear that segments of the population with higher socioeconomic status are more integrated than those with lower status [ 13 ]. This phenomenon is attributable to the diminished likelihood of socially excluded individuals to access the internet, concomitant with their diminished digital competencies, thereby contributing to digital inequalities [17][23]. Therefore, the digital divide is a significant obstacle to e-participation being achieved equally for all, as it is a factor that restricts certain people's access to or use of digital government information and services [21][22]. As a result, digital inclusion policies arise.

Since the realization of e-participation depends on citizens' ability to access and use ICTs, it is emphasized that indicators of these factors are important in the realization of e-participation [24]. There are many studies in the literature on the relationship between e-government and the digital divide [ 7 ][ 8 ][ 12 ][25][26]. However, there are very few empirical studies designed to specifically examine the relationship between e-participation and the digital divide (or inclusion/exclusion) [27][28][29][30][ 31 ]. In these studies, the indicators used to assess both eparticipation and the digital divide also vary.

There are some studies in the literature that discuss and analyze the digital divide among the barriers to e-participation [ 3 ][ 32 ]. However, there are not many studies specifically focusing on the direct relationship between the digital divide and e-participation. In these studies, digital divide indicators typically focus on access to ICTs and ICT supply levels, as well as demographic and socioeconomic factors. Porwol et al. presented, as a preliminary result in their study, that social inclusion and digital inclusion are prerequisites for e-participation, focusing on citizens' access to ICTs and local government online information and service supply levels [27]. Similarly, Aikins & Chary focused on the information and service levels provided by local governments to increase ICT access and online participation in their research on five local government units in the US [28]. Ribeiro et al. addressed citizens' access, use, and possession of ICTs using secondary data obtained from two different national surveys in their study discussing the technological challenges and limitations of using social media in e-participation initiatives in Brazil [29]. Perez-Morote et al. used technological infrastructure and democracy indicators to measure access and use of digital opportunities in addition to socioeconomic and demographic factors in their two cross-country analyses conducted on 178 UN countries [30].

In general, studies examining the relationship between e-participation and the digital divide partially or completely utilize different approaches from the literature on the digital divide to measure the digital divide. As previously stated, the digital divide is initially a concept used to describe the gap between those who have access to ICTs and those who do not [18]. The measurement of the digital divide focused on the access variable [ 11 ]. However, later on, ICT usage and ICT skills have also been considered as important components of the assessment [ 5 ][ 7 ][ 12 ][25]. Socioeconomic and demographic factors are also crucial when measuring the digital divide. From this perspective, the digital divide should be approached as a multifaceted concept. The studies above have mostly utilized variables such as access, use, and skills of ICTs in this context. However, not only the extent to which they used these variables but also the indicators they used to measure them varied among the studies. In this context, the unique value of this study lies in its evaluation of the digital divide using the UN's LNOB approach and variables, as opposed to previous assessments that relied on different sources and variables. In light of all these explanations, the primary research question that this study aims to answer is as follows:

RQ: What is the relationship between digital inclusion and e-participation, considering the UN’s “leave no one behind” framework?

3. Research Design and Methods

This study uses cross-sectional data and a multiple linear regression model to quantitively analyze the relationship between digital inclusion and e-participation. In this regard, the UN’s E-participation Index was selected as the dependent variable, while the UN's LNOB variables, including access, affordability, and ability, are used as independent variables to measure different aspects of the digital divide. Education, location, age, gender, and income are included as demographic and socioeconomic variables. The research model created within this framework is illustrated in Figure 1. Brief explanations regarding the selection and operationalization of each of the variables are provided below.

Access Access to electricity Access to internet and mobile infrastructure Access to e-information and eservices Affordability Affordability of internet access Affordability of digital devices Affordability of e-services Ability Traditional Literacy Digital Literacy Language Literacy Demographic economic Factors Age, Gender, Urban/Rural Income, Education &

SocioEducation, Population,

E-Participation e-information e-consultation e-decisionmaking

The e-participation Index represents the dependent variable in this study. Since 2003, the UN E-Government Development Index (EGDI) has been measuring electronic government development among countries through the E-Government Survey process. The E-Participation Index (EPI), a sub-index of the UN E-Government Survey, has become a significant data source for assessing and comparing e-participation progress globally. The EPI measures a country's performance in encouraging online citizen participation [ 14 ].

As previously stated, the independent variables in this study are consistent with the UN's LNOB indicators as an alternative way to measure the digital divide. We are also using several demographic and socioeconomic indicators. The fifth chapter of the latest UN E-Government Survey report for the year 2022, titled "Leaving no one behind in the hybrid digital society," identifies access, affordability, and ability as barriers to digital inclusion [ 14 ]. To measure these three variables contributing to digital exclusion, comprehensive indicators have been proposed. In the critical area of access, criteria such as access to electricity, access to the Internet and mobile infrastructure, and access to e-information and e-services are highlighted. Similarly, in terms of affordability, criteria include the affordability of Internet access, the affordability of digital devices, and the affordability of e-services. Lastly, concerning ability, the report lists three areas of literacy relevant to e-government and e-participation: traditional (or general) literacy, digital literacy, and language literacy [ 14 ]. We identified potential indicators for each of the variables and obtained them as secondary data from international organizations. The variables, indicators, and data sources included in this study are listed in Table 1.

As shown in Table 1, this study utilizes data from several databases including the World Bank, UN, OECD, ITU, NRI, and A4AI to analyze the relationship between e-participation and digital inclusion. Due to the utilization of various data sources, the sample of the study was constructed from 192 countries based on the availability of data found in these sources. All data sources were collected based on the year 2022. Since HDI data was not available for the year 2022, HDI and GII data were obtained from the 2021 database. The selection of data for this study was influenced by the reliability of data sources, the relationships among these variables, and the theoretical connections with the level of e-participation development. Common method bias should not be a problem since the data were collected from various sources [ 33 ]. We examined suitable data for each variable and were able to access all the necessary secondary data for the study. The overall hypothesis of this study is the following and in the next few paragraphs we propose more specific hypotheses.

H1: As digital inclusion decreases, the level of e-participation development decreases.

For the operationalization of the independent variables, different data sources were utilized in accordance with the aim of the study. Particularly, multiple indicators were used to analyze each of the main concepts and their relationship with e-participation. In this context, for access to electricity, the indicator "Access to electricity (% of population)" from the World Bank database was utilized; for access to internet and mobile infrastructure, indicators "Individuals using internet (%)" and "Fixed broadband subscriptions per 100 people" from the World Bank, along with the indicator "Fixed mobile subscriptions per 100 inhabitants" from the ITU database were employed; and for access to e-information and e-services, indicators "Publication and use of open data (%)" from the NRI, and "Online Service Provision Index" from the UN EGovernment Survey database were included in the research.

H2: As access to digital technologies decreases, the level of e-participation development decreases.

H2a: As access to electricity decreases, the level of e-participation development decreases.

H2b: As access to internet and mobile infrastructure decreases, the level of e-participation development decreases.

H2c: As access to e-information and e-services decreases, the level of e-participation development decreases.

Due to the unavailability of a specific indicator compatible with the study for affordability of e-services, different proxies for affordability of Internet access & affordability of e-services variables were evaluated. In this regard, we use data obtained from the World Bank database for "GDP per capita (Purchasing Power Parity - PPP)" and data from the ITU database for "Dataonly mobile-broadband basket (PPP)" and "Fixed broadband basket (PPP)." For the variable affordability of digital devices, "Digital Device Price Index" data were obtained from the Alliance for Affordable Internet (A4AI) database.

H3: As the affordability of digital technologies decreases, the level of e-participation development decreases.

H3a: As the affordability of Internet access and e-services decreases, the level of eparticipation development decreases.

H3b: As the affordability of digital devices decreases, the level of e-participation development decreases.

Finally, for the evaluation of variables related to Ability, due to the lack of access to compatible data for measuring language literacy individually, traditional literacy & language literacy were assessed together in this research. In this context, data from the World Bank and UNDP databases for "Adult Literacy Rate (%)" and "Human Development Index" were obtained. For measuring Digital Literacy, NRI dataset were utilized to obtain "Adoption of emerging technologies" and "ICT skills in the education system" data.

H4: As the ability to use digital technologies decreases, the level of e-participation development decreases.

H4a: As traditional literacy and language literacy decrease, the level of e-participation development decreases.

H4b: As the digital literacy decrease, the level of e-participation development decreases.

Indicators were selected for the analysis of each demographic and socioeconomic variable. To represent the gender variable, the Gender Inequality Index (GII), published by the UNDP as a sub-index of the HDI, was included in the study because inequality between genders was of interest, rather than population distributions by gender. All other socioeconomic and demographic variables were obtained from the World Bank database. The proportion of elderly population considered disadvantaged in terms of access and usage of digital opportunities (% of 65+ ages of population) and life expectancy at birth indicators were employed to analyze the age variable. For measuring the education variable, enrollment data for both secondary and tertiary education were included in the study. Finally, for the income variable, GNI per capita, and for the location variable, urban population (% of total population) data were utilized. H5: As gender inequality decreases, the level of e-participation development increases. H6: As the elderly population ratio decreases, the level of e-participation development increases.

H7: As the level of education decreases, the level of e-participation development decreases. H8: As income level decreases, the level of e-participation development decreases. H9: As the urban population rate decreases, the level of e-participation development decreases.

4. Preliminary Findings

Multiple variables and indicators were used in the analysis of the relationships between digital inclusion and e-participation, along with the formulation of nine specific hypotheses to answer the research question. Before moving on to hypothesis testing, we present the descriptive statistics. We also run correlations to better understand relationships between the different variables used in the study and to what extent they are associated with e-participation. Table 2 presents the descriptive statistics (mean and standard deviation) of the variables and the correlations between the variables and the e-participation index.

Based on the correlation results, EPI has strong or moderate relationships with the majority of independent variables. Specifically, there is a very strong relationship between EPI and OSPI, while it has weak relationships with data-only mobile-broadband basket (PPP), fixed-broadband basket PPP, and the Digital Device Price Index. However, these are just preliminary observations and in order to obtain correct interpretations of these relationships along with the impacts of each independent variable on EPI we conducted multilinear regression analysis. Table 3 Preliminary multilinear regression results that include all indicators

Predictors (Intercept) Access to electricity (%) Individuals using the internet (%) Fixed broadband subs. per 100 p Mobile-broadband subs. per 100 p Open data availability OSPI GDP Fixed-broadband basket (PPP) Mobile-broadband basket (PPP) DDPI Adoption of emerging tech.

ICT skills in the educ. system Adult Literacy rate HDI GII GNI per capita Enrollment in Secondary Educ (%) Enrollment in Tertiary Educ. (%) EPI

Table 3 shows some preliminary results of a multilinear regression model, which includes all the indicators collected to measure the independent variables. As can be seen from the results, many variables do not have a significant relationship with the level of e-participation development when controlled for the other variables included in the model. However, for exploratory purposes, we also performed individual linear regression models and they showed statistically significant relationships with EPI for several variables. Therefore, these preliminary multilinear regression results indicate the need for additional tests and analyses to identify variables that significantly affect the level of e-participation development. Indeed, as shown in Figure 2, model assumptions (normality of residuals, normality of random effects, linear relationship, homogeneity of variance, multicollinearity) indicate that some indicators do not past all the tests, and we should reconsider whether and how to include them in the study.

5. Final Comments and Next Steps

The overall aim of this study is to empirically assess the effects of different indicators of access, affordability, and ability, which are listed as barriers to digital inclusion within the scope of the LNOB goal, on the level of e-participation development. As seen in the descriptive statistics in Table 2, there is an abundance of indicators for each of the independent variables, although in some cases they are proxies and do not completely represent the actual concept. Looking at the correlations, it is also clear that there are important relationships between many of the independent variables and the e-participation index. However, it seems that several independent variables are so highly correlated to each other that we obtained very few significant results when including them all in the regression model. Therefore, the assessment of the overall research question and the first hypothesis of this study should happen at the end of the process. As one of the next steps, we will also test which indicators are better representations of the three main variables: access, affordability and ability, since due to multicollinearity issues, we may need to use fewer and/or different variables for the final model.

In fact, as mentioned before, descriptive statistics and the assumptions tests show that there is a high correlation between several independent variables. This may indicate a multicollinearity issue [ 34 ]. In this case, the reliability of the estimated coefficients in the model may decrease, and the accuracy of the predictions may be affected [ 35 ]. Therefore, in the subsequent analysis, careful selections will be made among the highly correlated variables. When deciding which variables to include in the model, preference will be given to variables with fewer issues related to multicollinearity and those that better represent the concepts relevant to the study. In this context, an analysis of the indicators identified for all hypotheses will be conducted and necessary adjustments will be made. [15] Macintosh, Ann. "Characterizing e-participation in policy-making." 37th Annual Hawaii

International Conference on System Sciences, 2004. Proceedings of the. IEEE, 2004. [16] UN, E-Government for the Future We Want: E-Government Survey 2014, 2014. URL: https://publicadministration.un.org/egovkb/Portals/egovkb/Documents/un/2014Survey/E-Gov_Complete_Survey-2014.pdf [17] UN, Transforming our world: The 2030 agenda for sustainable development, 2016. URL: https://sdgs.un.org/goals [18] OECD, "Understanding the Digital Divide", OECD Digital Economy Papers, No. 49, OECD

Publishing, Paris, 2001. https://doi.org/10.1787/236405667766 [19] Helbig, Natalie, J. Ramón Gil-García, and Enrico Ferro. "Understanding the complexity of electronic government: Implications from the digital divide literature." Government information quarterly 26.1 (2009): 89-97. [20] Van Deursen, Alexander, and Jan Van Dijk. "The digital divide shifts to differences in usage." New media & society 16.3 (2014): 507-526. [21] Karabulut, Niyazi, Dijital dişlanma ve e-katilim ilişkisi: kuramsal bir inceleme ve model önerisi, in: E. Bingöl (Ed.), Kamu yönetiminde katilim üzerine güncel tartişmalar, 1st ed., Gazi Kitapevi, Ankara, 2022), pp. 255-284. [22] Yavuz, Nilay. "Digital Divide and Citizen Participation in Public Policy Making." CitizenCentered Public Policy Making in Turkey. Cham: Springer International Publishing, 2023. 115-130. [23] Yildiz, Mete. "Digital divide in Turkey: A general assessment." Handbook of research on overcoming digital divides: Constructing an equitable and competitive information society.

IGI Global, 2010. 75-89. [24] Krueger, Brian S. "Assessing the potential of Internet political participation in the United

States: A resource approach." American politics research 30.5 (2002): 476-498. [25] Bélanger, France, and Lemuria Carter. "The effects of the digital divide on e-government: An emperical evaluation." Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06). Vol. 4. IEEE, 2006. [26] Zhao, Fang, Alan Collier, and Hepu Deng. "A multidimensional and integrative approach to study global digital divide and e-government development." Information Technology & People 27.1 (2014): 38-62. [27] Porwol, Lukasz, Adegbojega Ojo, and John Breslin. "On the duality of e-participation– towards a foundation for citizen-led participation." Second Joint International Conference on Electronic Government and the Information Systems Perspective, and Electronic Democracy, EGOVIS/EDEM 2013, Prague, Czech Republic, August 26-28, 2013, Proceedings 2. Springer Berlin Heidelberg, 2013. [28] Aikins, Stephen K., and Meena Chary. "Online participation and digital divide: An empirical evaluation of US Midwestern municipalities." Digital Literacy: Concepts, Methodologies, Tools, and Applications. IGI Global, 2013. 63-85. [29] Ribeiro, Manuella Maia, Maria Alexandra Cunha, and Alexandre Fernandes Barbosa. "Eparticipation, social media and digital gap: Challenges in the Brazilian context." Proceedings of the 19th annual international conference on digital government research: Governance in the data age. 2018. [30] Pérez-Morote, Rosario, Carolina Pontones-Rosa, and Esteban Alfaro-Cortes. "Effects of the Digital Divide in E-Participation Development in the UN Countries." The 13th

References

[1] Saebø , Øystein, Jeremy Rose, and Leif Skiftenes Flak. "The shape of eParticipation: Characterizing an emerging research area." Government information quarterly 25 .3 ( 2008 ): 400 - 428 .

[2] Kim , Soonhee, and Jooho Lee . "Citizen participation, process, and transparency in local government: An exploratory study." Policy Studies Journal 47.4 ( 2019 ): 1026 - 1047 .

[3] Karkın , Naci. "E-Katılım kavramı ve süreci: Kamu siyasa oluşum sürecine vatandaş katkısının olabilirliği . " Sosyoekonomi 17 .17 ( 2012 ).

[4] Helsper , Ellen J. , and Anna Galácz . "Understanding the links between social and digital exclusion in Europe." Worldwide internet: Changing societies , economies and cultures 146 ( 2009 ).

[5] Van Dijk , Jan , and Kenneth Hacker . "The digital divide as a complex and dynamic phenomenon." The information society 19 .4 ( 2003 ): 315 - 326 .

[6] Van Dijk , Jan. " The evolution of the digital divide-the digital divide turns to inequality of skills and usage." Digital enlightenment yearbook 2012 . IOS Press, 2012 . 57 - 75 .

[7] Mossberger , Karen. "Toward digital citizenship: Addressing inequality in the information age." Routledge handbook of Internet politics . Routledge , 2008 . 173 - 185 .

[8] Dinçer , Pelin , E-Demokrasi Çerçevesinde E -Katılım ve Toplumsal Cinsiyet İlişkisi . In: A. Saylam (Ed.), Kamu Yönetiminde Elektronik Vatandaş Katılımı , 1st ed., Gazi Kitapevi , Ankara, 2021 , pp. 215 - 240 .

[9] Panopoulou , Eleni , et al. "Learning from eParticipation initiatives of regional and local level authorities in Greece and Spain." Journal of Balkan and Near Eastern Studies 13 . 01 ( 2011 ): 77 - 96 .

[10] Roman , Alexandru V. , and Hugh T. Miller . "New questions for e-government: Efficiency but not (yet?) democracy." Public Affairs and Administration: Concepts, Methodologies, Tools, and Applications . IGI Global , 2015 . 2209 - 2227 .

[11] Norris , Pippa. Digital divide: Civic engagement, information poverty, and the Internet worldwide . Cambridge university press, 2001 .

[12] J. Van Dijk , The deepening divide: Inequality in the information society , 1st ed., Sage Publications , 2005 .

[13] Polat , Rabia Karakaya . "Digital exclusion in Turkey: A policy perspective . " Government information quarterly 29 .4 ( 2012 ): 589 - 596 .

[14] UN , United Nations E-Government Survey 2022: The Future of Digital Government , 2022 . URL: https://desapublications.un.org/publications/un-e - government-survey-2022 International Conference on Digital Society and eGovernments ICDS, February 24-28 , Athens, Greece, 2019 .

[31] Perez-Morote , Rosario, Carolina Pontones Rosa, and Esteban Alfaro Cortes. "Exploring the Relation Between the Digital Divide and Government's Effort to Develop E-Participation: A Global Longitudinal Analysis." Research Anthology on Citizen Engagement and Activism for Social Change . IGI Global , 2022 . 997 - 1018 .

[32] Smith , Simon. "From inclusive spaces to inclusionary texts: How e-participation can help overcome social exclusion." Handbook of research on overcoming digital divides: Constructing an equitable and competitive information society . IGI Global , 2010 . 533 - 548 .

[33] Woszczynski , Amy B. , and Michael Whitman . "The problem of common method variance in IS research." The handbook of information systems research . IGI Global , 2004 . 66 - 78 .

[34] Hanushek , Eric A. , and John E. Jackson. Statistical methods for social scientists . Academic Press, 2013 .

[35] R Core Team , R: A language and environment for statistical computing , 2019 . URL: https://www.R-project. org.

[36] Farrar , Donald E. , and Robert R. Glauber . "Multicollinearity in regression analysis: the problem revisited." The Review of Economic and Statistics ( 1967 ): 92 - 107 .