Nowadays, the problems of retrieving, processing, and analyzing information obtained from web sources to get valuable insights, processing large data volumes, and using different data analysis techniques are extremely relevant in cross-disciplinary research, including studies on the intersection of computer science and social sciences. Various social science fields, such as political science, may benefit from the application of web data extraction techniques, when the information should be scraped from websites. Therefore, this paper proposes an approach to the use of information technology for automatic data collection from egovernment national portals, assessment of the online citizen services availability and variety, and further analysis by researchers in political science. The proposed information technology assumes detection of citizen services provided by national portals and their categorization according to the specified branches. The Python programming language is used to develop data processing components, while the Power BI analytical tool is used to visualized the obtained results on the dashboard. The performance of the proposed solution is verified, by processing national portals of several countries that have their homepages allow web scraping and offer English versions.
Today, humanity lives in the period of the information age, the determinant of the evolution of which is the rapid development of new information and communication technologies that have been developing since the second half of the 20th century. Digital technology and the new opportunities that it brings were swiftly adopted by the world society, which in turn caused the beginning of the transformation of various aspects of human life, including politics.
In the late 20th century and beginning of the 21st, new technologies significantly influenced the
course of foreign and domestic policy of different countries. Information and communication
technologies have begun to change the attitudes of states, governments, politicians, on the economy,
on methods of governance, and most importantly, on improving the communicate ways with their
citizens [
As the Internet began to spread widely around the world, quite a lot of enthusiasts interested in
improving communication with government agencies began to use the then-new technology to
create the first relevant web resources, which in turn could be supported by local governments [
Today, the digitalization of government structures and creating an e-government model is a
priority for many developed countries, and the E-Government Development Index (EGDI) shows
that some states have achieved quite tangible success in achieving this aim [
In previous research [
Political science today has a rather large arsenal of research methods. However, like all other sciences, social sciences and political science in particular are in a state of constant movement, transformation and improvement of methodology.
In terms of significant transformations, there has recently been a growing interest among
humanities researchers in computational research methods using digital technology. In this case, it
can be spotted the emergence of an interdisciplinary field, namely Computational Social Science
(CSS). The growing interest of scholars in computational social science can also be justified by the
rapid development of the internet and the growth of digital data worldwide [
It is also necessary to draw attention to the fact that in political science the use of methods related
to the application of computer technology dates back to the second half of the 20th century [
Methods related to automatic computerized text recognition have also gained some development.
of text data in a fairly short period of time [
The Internet evolution has been one of the main reasons for the widespread digitalization of
human life. Today, any user activity on the Internet can be recorded, and at the same time web pages,
social networks, online media, blogs, file exchanges, etc. can be a source of valuable information for
web pages can be used to analyze web resources. Web scraping can be understood as collecting
information from a particular web page of interest to the researcher [
Let us present the approach to automated collection and analysis of data from national e-government web portals of different countries. The main purpose of the study is to improve the assessment of the structure and content of these portals in terms of availability and variety of services provided to citizens. The analysis is aimed at identifying the key characteristics of the web portals, such as the number of available electronic services, their thematic distribution by e-government service catalog, as well as the level of richness of services in various citizen service branches.
The baseline of the data workflow is the uploading of a prepared data set containing information
about countries, their national portals and relevant e-government indices (Fig. 1).
[
To load this data (1) into the programming environment, a table read operation is used, which ensures that the spreadsheet is transformed into a data frame marked as .
Then, the data frame
is transformed into the JavaScript Object Notation (JSON) format. Let us formally describe this process as the following operation:
= { 1, 2, … , }. ( ) =
(1) (2) Were:
• • • • • • • • • • • portals analysis using web scraping techniques.
Finally, the process is finished with saving data (3) into the JSON format file. The writing operation could be formally described as following: ( ,
). is the function that stores the set of dictionaries into a JSON file;
Therefore, the data preparation process could be formally described as the set of operations that ensure the transformation (4) of the Excel spreadsheet into the JSON format file: .
Then, the obtained JSON file (Fig. 2) will be used by another software component for national data set, = 1, .
= { ′′1, ′′2, … , ′′ }. is the number of records in the data set 1, 2, … , ; ′′ is the dictionary of attributes in the key-value format, corresponding to the record in the The obtained JSON string (2) is then de-serialized into the internal Python object which could be denoted as the set of dictionaries: (3) (4) (5) is the function used to transform the set of records into the respective set of JSONbased objects
= { ′1, ′2, … , ′ };
′ is the JSON object that corresponds to a separate record in the data set, = 1, .
web portal is accessed by making Hypertext Transfer Protocol (HTTP) requests. All hyperlinks are
extracted from the resulting (Hypertext Markup Language) HTML content, which is then analyzed
for thematic keywords relevant to major government service areas and citizen services according to
Integrated Architecture Framework for E-Government (IAFEG) [
Let us formally define the set of countries and corresponding web portals as following: = { 1 = ( 1, 1), 2 = ( 2, 2), … , = ( , )}. (6) Were: • is the country data record at the EDGI website, = 1, ; • is the national web portal page corresponding to the country data record, = 1, ; • is the Uniform Resource Locator (URL), i.e. the web address of the respective national web portal, = 1, .
The initial stage of the national web portals data (6) processing flow includes the data loading from the previously created JSON file described as following: →
. is the JSON file with the prepared EGDI national web portals data; is the corresponding set of dictionaries, which contain information about countries and For each web portal page , = 1, that corresponds to the country , = 1, (7), the HTTP Let us formally define the web page request as following: =
( ), = 1, . = { 1, 2, … , }, = 1, .
= { 1, 2, … , }. is the result of the request to web portal page , = 1, ; is the URL of the corresponding national web portal page , = 1, .
Each web portal page , = 1,
library [
Where is the hyperlink extracted from the web portal page, = 1, , = 1, .
For each country web portal (9), a search is carried out by thematic categories of e-government citizen services defined by dictionaries of keywords. Let us formally describe such thematic categories using the following set: = 1, .
Where is the thematic category of citizen services (e.g. taxation, education, healthcare, immigration, employment, etc.), each is associated with the set of keywords Ω = { 1, 2, … , },
The IAFEG-based keywords for the e-government citizen services are outlined in Table 1.
Proposed thematic categories of citizen services and their keywords based on IAFEG [
health, insurance, care, sick, medical, funeral immigration, citizen, travel, visa, residence, international
employment, work, job, business, license, certification
For each hyperlink , = 1, , = 1, (9) the presence of keywords (10) in the hyperlink text is checked. If the keyword from the thematic category , = 1, is found, the hyperlink is stored and considered as corresponding to the respective e-government citizen service.
Therefore, the set of thematic categories and hyperlinks that, as we assume, provide the access to corresponding citizen services, is formulated for each country: = {( , )}, = 1, . (11)
Where: • • is the thematic category of citizen services, = 1, ; is the hyperlink extracted from the web portal page and considered as the access point to the corresponding thematic category (or its sub-category) of citizen services (e.g. taxation, education, healthcare, immigration, employment, etc.), = 1, , = 1, .
The example of United Kingdom (UK) national portal [
Where: =
⋃ ( , )∈
. =
| |. 1 | |
As can be seen (Fig. 4), for any country, a thematic structure of citizen services is built based on the collected hyperlinks.
This makes it possible to estimate the number of citizen services detected by using each thematic category (11), by introducing the following equation:
Moreover, it becomes possible to estimate the service richness [
Finally, the general process of e-government national portals data processing could be formally represented as following: (14) tools, freely accessible and easy to use for non-professionals in information technology, i.e. social or political scientists.
Hence, we propose to use Microsoft Power BI for the further analysis of the obtained web scraping
results. Power BI is a high-performance Business Intelligence (BI) tool for advanced data
visualization and data-driven decision making [
Hence, using the proposed Python-based data pipeline and the Power BI analytical tool, the
proposed solution allows to automatically collect data from national portals of different countries
mentioned in the EGDI index [
Using the proposed data analytics solution, social or political science scholars can easily compare and analyze the development of online services provided by national portals of different countries. Also, this toolkit can be used to identify the best practices in the field of online citizen services offering via e-government portals.
The general EGDI-based dataset [
However, a lot of national portals mentioned on the EGDI website [
Table 2 demonstrates the stages of EGDI dataset [
The initial EGDI [
Failed to process national web portals of 18 countries
As can be seen from Table 2, almost 55% of country records were removed from the initial dataset because of the inaccessible national portals or absence of English versions. The remaining 87 records were processed using the proposed solution. However, only 79% of the available national portals were successfully scraped.
There are such countries and corresponding national portals: • successfully processed: Mexico, Ukraine, Brazil, Maldives, Cambodia, Jamaica, Australia, Dominica, Liberia, Antigua and Barbuda, Monaco, China, Luxembourg, Micronesia (Federated States of), Republic of Moldova, Romania, Latvia, Uzbekistan, Nigeria, Somalia, Pakistan, Kazakhstan, Bahrain, Timor-Leste, United Arab Emirates, Viet Nam, Czech Republic, Croatia, Switzerland, Belgium, Austria, Germany, Canada, Cyprus, Kuwait, Japan, Fiji, Chile, Armenia, Bahamas, Barbados, Botswana, Grenada, Georgia (Country), Saint Kitts and Nevis, Montenegro, Rwanda, Singapore, Slovenia, United Kingdom of Great Britain and Northern Ireland, Vanuatu, South Africa, Sweden, Italy, Mauritius, Saint Lucia, Malawi, Netherlands, Solomon Islands, Portugal, Kiribati, Liechtenstein, Norway, Samoa, Cameroon, Finland, Trinidad and Tobago, United States of America, and Estonia; failed processing: Bulgaria, Spain, Iran (Islamic Republic of), Jordan, Lithuania, Eritrea, Ghana, Ireland, Israel, Kyrgyzstan, Malta, Namibia, Philippines, New Zealand, Morocco, Palau, Thailand, and Zimbabwe.
Fig. 7 demonstrates the created Power BI dashboard that consolidates information about
countries, Online Service Index (OSI) measures of these countries, as well as introduced measures:
•
•
number of detected citizen services (12);
richness of the citizen services (according to the IAFEG [
As can be seen from Fig. 7, country national portals are placed on the scatter chart: • •
X-axis of this scatter chart is OSI, which estimates the scope and quality of online services
provided by a web portal according to the EGDI methodology [
Sizes of each point reflect numbers of detected online citizen services (Fig. 7).
In Fig. 7 we consider the first ten countries, ordered by the OSI values Estonia, Ukraine, Singapore, United Kingdom of Great Britain and Northern Ireland, Japan, Kazakhstan, China, Germany, Australia, and Netherlands.
It is interesting, that processing of Ukraine, Australia, and Netherlands national portals has
resulted into 0 citizen services detected and, therefore, 0.00 values for the service richness measures.
However, according to the OSI measurement on the EGDI website [
The analysis of Ukrainian indicators resulted into the fact, that EGDI rating contains the URL of
the Cabinet of Ministers (CM) homepage [
This paper proposed the information technology for online citizen services detection and assessment on e-government national portals. The main purpose of this study was to improve the assessment of the structure and content of national portals in terms of availability and variety of online services provided to citizens. Such a solution can be used by political scientists to perform experiments, find best practices of online citizen services provision, compare different national portals, and get valuable insights.
The approach to data extraction from e-government national portals and further processing to assess the availability and variety of online citizen services is proposed, and the corresponding information technology is implemented using Python, third-party libraries, and Power BI. Obtained results have shown the difference between EGDI-based OSI measurements and the availability of detected citizen services. There was noticed, that some national portals provided by EGDI are not the same portals really providing online citizen services, e.g. the URL for Ukraine leads to the CM website, not Diia.
In the future, the proposed approach will be improved to traverse all national portal pages.
The authors have not employed any Generative AI tools.