The research has developed a series of steps to detect and simulate historiographic information on web forums through Web-Scraping, Data mining and Big Data analytics. The system of posting ranking is described in order to determine the relevance of the content for the needs of the historian, developed an algorithm that determines the significance of the system, its impact on the quality of the historical research. The given code snippets in the SQL query language to get the most useful aggregate numeric and text data will help simplify the research methods and develop new ways to identify knowledge and information in the content of web-forums of historical topics. Monitoring of web-forum user's activity using time series analysis has been completed. The rush hour for the generation of historiographic data in selected thematic sections of the web forum was determined. It is substantiated that processing technologies use for a large number of text data using Natural Language Processing (NLP) and Deep Learning will also allow automating the detection of valuable characteristics and features of the obtained data.
Informatization has led to an exponential increase in the amount of information, the creation of local and global systems and networks, databases and knowledge, the emergence of fundamentally new technologies that radically changed the methodology of historical research. The intellectual and informational potential has become the result of informatization, and the development of both depends on the intensity of the information society process. These two concepts can be combined in one - the information and cognitive potential that characterizes the process of informatization. An important component of information and cognitive potential is the intellectual potential, which manifests the ability of a person to solve problems using the accumulated knowledge, skills and experience. The second component is the information potential, which provides the necessary level of awareness of members of society, that is, the ability to summarize, search, store and transmit information. Accumulation of information and intellectual potential as a result of informatization is the accumulation of historiographical information and the formation of a new resource of historical knowledge in the Internet environment.
Exponential growth of electronic resources, observed at the beginning of the XXI century, opens new perspectives for the development of historical research. The global computer network of the Internet is a complete source of professional information for historians. An additional tool for cognition for historians is the study of a large array of unstructured historical information generated in web forums. To do this in Digital history, it is important to use various methods of data mining to automatically detect web documents, retrieve information from web forums, and identify its general patterns on the Internet. 2
Scholarly works include a variety of areas for applying knowledge and data
consolidation, using a wide range of resources for research on web communities. E.
Trunzer examines high-performance architecture for gathering and consolidating data
from multiple sources and resources into a single repository, their preparation for
processing, and Big Data Analytics [
The indicated methods of carrying out qualitative research of data of information
systems, ways of preparation and data collection with the help of Web scraping
frameworks with the purpose of obtaining unique scientific results by researchers in
various fields of science [
The transformation of large amounts of data into Smart Data in a wide range of digital
humanities (Digital History, Digital Sociology, etc.) is defined in the work of
M. Zeng, which describes ways of transforming "raw" data into useful for historians
and humanities information in order to identify additional knowledge [
At the same time, in the context of our study, valuable works are those that deal with
virtual communities flow. It is examined in K. Miller studies [
The purpose of our study is to apply interpreted object-oriented programming language Python for data collection, integration into the database; use of the data warehouse and the language of SQL queries for processing the historiographic information of web forums and determining the relevance of the data obtained for historians. 3
In modern conditions, changing the paradigm of historical science, the use of various research resources of the Internet is a particularly urgent task. Within the framework of Digital history, scientists will actively involve various information resources for the study of recent history; sites that cover virtual and real scientific and historical communication; sites that have sources and special research on history; sites of scientific and social funds that support historical scientific and educational projects. Among the wide variety of modern research resources of the Internet, it is worth highlighting the source-research potential of direct-communication resources, since they maximize the function of communication, and therefore generate the primary and secondary sources of historical information. The specifics of directly-communicating Internet resources (partnerships, forums, social networks, blogs, multimedia web communities, and so on) lies in their in interactivity, embodied in the very technology of the WWW.
The most popular types of web communities are forums that are designed to
communicate with network users. Features of social network historical information
are these: information content is unstructured, discussions arise spontaneously around
various information drives (photos, posts of participants, discussions) [
Data processing involves several stages that will enable the collected data to be consolidated and integrated into warehouse, make it accessible for later processing, provide historians with convenient and reliable work with the information they receive. Thus, in Fig.1, the main components of the data workflow from the web forums into historiographic information database are highlighted.
The first step involves generating a request for historical information source (web forum), parsing data with R/Python programming languages libraries and functions according to website structure (see Fig.2). Each structural element has its own tag, which is fairly easy to obtain by parsing and web scraping techniques, but one of the major drawbacks of this method is the periodic change of class names and identifiers contained within the HTML tag structure of the site. The frequent parsing problem of web forums is data encoding (Javascript tags within HTML-tags), which requires additional requests (Ajax request etc.) generation and data decoding techniques.
After that, data characteristics with applying of natural language processing/ understanding (NLP, NLU) are determined automatally, which helps to obtain the maximum amount of useful information, such as: keywords, presence of discussion in the post, negative or positive post content and other features. So, the collected data must be integrated into data warehouse or relational database (see Fig.3).
The main table - "Posts", consisting of 10 characteristics and has relations to other tables. Automated data processing and the ability to use regular expressions, NLP / NLU and Deep learning will allow us to identify such characteristics as: a keyword, a historical events that are described in the posts, the presence of discussion in the post, chronological boundaries, topics, etc.
Extracted and integrated into the database posts will be sorted by using the described below data arranging method, which allows to highlight the most relevant and interesting posts and those that contain useful for historians information. The next step is to determine the effectiveness of the ranking system by comparing the ID order by increasing of the unique post ID (Post_ID in the main table of the "Posts" database), which is the initial state; and after the post ranking process and their new sorting order definition in the main table. In order to execute this, an algorithm has been developed that identifies the difference between two sets of consolidated data.
The algorithm works as follows: two datasets are accepted at the input, which must contain the post identifier (Post_ID). The first set of data - the initial one, which was not used for post ranking, the second set - the one that "passed" the post ranking system. After that, the input table length is determined. If it is equal one to another, depending on the position of the posts and their identifiers in both sets of data, the similarity between argument 1 and argument 2 is calculated. The result is percentage of similarity. The lower the percentage of similarity is, the greater significance the post ranking system has. If the dataset length differs, the program does not perform computations. The final stage is ready-for-processing data extraction for historians, which will greatly simplify their work, and allow a qualitative summarizing and hypothesis development and building based on data analysis of posts from web forums containing historiographical information. Data querying from an existing warehouse is possible by using the SQL query language, working with relational databases. Complex queries formation for obtaining valuable information will allow getting the most necessary information in a short time spans and without high resource usage. In addition, the use of ranking system will allow us to receive up-to-date information quickly. Historiographic post ranking methods: summing up the existing data characteristics, which will allow to calculate the information content of a separate post stored in the database. For the posts ranking system, the following characteristics will be used from the "Posts" table of the historiographical web-forum information database: post date and time publication, number of keywords, presence of graphic images / illustrations in the post, appropriateness of current number of words with the optimal number of words, reference to certain resources, and presence of discussion. Therefore, most features will be sorted by the system and / or filtered by the end user (a specialist, historian). The appropriateness to the optimal length of a post calculation is possible using the following equation: (1) (2), = | − |, where – value that defines the optimal post's length. The lower this value, the higher is the appropriateness; – number of words in the ni post; – the optimal number of words for a web-forum post.
Characteristics that correspond to the logical values True or False will respectively
represent the value displayed on the equation below:
= { = 1 =
= 0 =
where – a characteristic that corresponds to the value True or False (the presence of
images in the post, calls to a certain resource, the presence of discussion in a post etc.)
User activity monitoring in a web-forum is an important feature that should be taken
into account during the stage of conducting exploratory data analysis. It will reveal
certain patterns of online activity, execute some kind of timeseries analysis and obtain
information about when most of data in the historical web forum content is
generated. Powerful libraries and tools that are part of Python's programming language allow
us to collect huge amounts of data from the Internet. For timeseries analysis and
activity monitoring, these tools were used, and the transformation & visualization of the
collected data was carried out in order to obtain certain insights on forum users
activity. Timeseries analysis is a systematic approach by which mathematical and statistical
questions are answered posed by time correlations [
Most research information products and ways to improve them occupies precisely
analysis of user behavior, in particular their activity [
The key point is to determine the weekday of deploying a discussion on web forums, especially relevant for historical topics, which allows you to determine when users themselves are able to participate in resolving disputed issues of the past as well as the present. So, as we can see in the graph, most users of historical, military-historical web forums are active on weekdays, whereas on weekends they are less and less timeconsuming. High activity is observed at the beginning of the week, but gradually it decreases. Later it rises again (Thursday), and then it finally becomes minimal (Saturday, Sunday).
It's important to watch the time of the day when new content is being published on a web forum. This allows the user, including the historian, to choose the best time to publish his own post so that other members of the web forum can familiarize with and respond to it. At the web-site under investigation, the peak of activity was at lunch (13: 00-15: 00). If we take a gap from 5:00 to 20:00, we can observe a normal distribution, while from 00:00 to 5:00 - the distribution of Poisson. Determining the activity of the users of the forum has allowed us to identify the necessary timeframes for active deployment of discussion on web forum pages, to model user behavior and analyze it. It also helped to find certain correlations between the time of day, day of the week and the number of generated posts in the content of the historical web forum. Consequently, most users are actively posting new posts on weekdays, especially during the day. Many of them join the discussion on the forum, only when it appeared in a sufficient number of posts (more than half). This allows you to find a good time to collect data, as well as participate in discussions in historical web forums.
Modern web forums generate unique sources of historical information that contain large volumes of important, valuable information from eyewitness events. Investigating such a large array of unstructured historical information through web forums is an additional tool of cognition for historians. Web forums provide wide access to information that has not been filtered. Web Mining technology provides a high-quality analysis of the content of web-forums and allows you to get new information about historical events, to conduct operational monitoring of data. Using interpreted object-oriented programming language Python helps to automate the following processes: data collection, identification of the main characteristics of the individual structural elements of the forum, data acqusition, data computing and processing. The tools used in the research provided for the rapid processing of tabular data and helped to identify the most interesting characteristics of historical information that was generated in the content of web forums.