-This paper suggests an algorithm of text data mining based on conversation analysis. Natural languages are developing dynamically nowadays. New semantic units are constantly being introduced into the spoken language. In these conditions, chains of dependency graphs of semantic units are constantly being rebuilt. This paper proposes a method for identifying synonyms based on conversation analysis. The proposed method has been tested on data collected from social networks.
The social networks are currently undergoing a turbulent
growth: every day, users send billions of messages and
submit billions of comments. Their analysis has a great
impact on many areas of business. For example, it is
impossible to overestimate the influence of internet
marketing on the promotion of goods and services. However,
in order to use these mechanisms effectively, it is necessary
to understand the demands of users. The source of such
information can be the materials published by users of social
networks, as well as the shares and reposts by users and the
entire communities [
II.
NETWORKS
The social network Vkontakte was selected as a data source for this research. The reasons for this choice are as follows: the network provides open access to its data (no restrictions on accessing the server data); Vkontakte is the most popular social network in Russia and the fifth most popular social network in the world; Vkontakte is a full-fledged social network (unlike Twitter and Instagram, which are microblogs) allowing to create thematic communities, which are particularly interesting for this study.
As part of this study, a Python software package was developed, containing an authorization module, a data collection module, and a filtration module. This software package allows to collect data and filter them to take the relevant information only. relevant information only.
Within this study, the developed software package was used to collect more than 5,000 posts and over 170,000 comments from the two most popular communities of the city of Samara (“Podslushano Samara” and “Uslyshano Samara”).
III. DETERMINATION OF THE CLOSENESS OF TEXT UNITS BASED ON CONVERSATION ANALYSIS Conversation analysis, i.e., the study of structures and formal properties of a language in its social and economic application, is related to all major areas of ethnic and methodological research.
Initially, the conversation analysis was intended for the study of verbal and everyday speech only, and more than that, only conversations between several interlocutors. H. Sacks, the creator of the method, attracted the attention of scientists to the fact that conversations are central for a social world.
A conversation shall necessarily be organized, it implies the existence of an order that does not need to be explained again and again during the exchange of phrases. The order is also needed for the spoken words to be clear to all the conversation participants. The conversation shows the social, interactive competence of people willing to explain their behavior and to interpret the behavior of interlocutors. Inside the local sequences of conversation, and only there, social institutions are finally “spoken into existence”. As a result, the smallest and seemingly insignificant details of the conversation actually become a means of actualizing the most important social institutions.
The goal of conversationalists is to describe social practices and expectations that the interlocutors rely on when constructing their own behavior and interpreting the behavior of others.
Conversion analysis focuses on particular cases as opposed to idealization that is inevitably connected with any theoretical generalization, from the point of view of Garfinkel and Sacks. In their opinion, idealization impedes scientific development, since any typology is not much connected with the content of real cases which it is supposed to be based on. Sacks sought to develop a method of analysis that would remain at the level of primary data, raw material, specific, isolated events of human behavior. In contrast to classical sociology, he argued that the details of any spontaneous human interaction are strictly organized – to the extent that provides for their formal description.
On the basis of the above prerequisites, the peculiarities of conversation analysis can be formulated as follows. First, this method follows the data, i.e. the analysis is based on empiricism without using (possibly) predetermined hypotheses. Secondly, the smallest details of the text are considered to be an analytical resource and not an obstacle to be discarded. Third, the authors of the method are convinced that the order in organizing the details of everyday speech exists not only for researchers, but – first and utmost – for the people who construct this speech [8,9].
This idea formed the basis for the study. Initially, it was when analyzing text data and use it to extract contextual suggested that on a large data set two text units have similar use distance vectors V (vector that shows how the two text units relate to each other within the data, where index i (indicates the distance between units) and Vi (the number of combinations between units, V0 - total number of uses of two text units within one sentence) serve as metrics.
The data collected from the Vkontakte social networks have been pre-processed; each text unit has been brought to its normal form (the pymorphy2 package was used for this). The data was then pre-processed to extract the necessary statistics (WordCount, maximum sentence length). The next step was to create a distance matrix.
A cosine
distance was used to calculate distances between two vectors: ∑ =1 × √∑ =1 ( )2×√∑ =1
( )2
= cos( ) =
∙ ‖ ‖‖ ‖ == (1)
of determining the closeness of text units and therefore a theory was proposed that it is possible to use this approach for the statistical definition of the author of a literary text.
The main idea of the study is to make multidimensional vectors that store distances between each pair of words in the text. It has been suggested that when comparing distances between pairs of words it is possible to estimate the degree of closeness of text fragments.
This study can be roughly divided into two tasks: 1. 2.
Data preparation.
Determining the optimal text fragment size for comparison between texts.
To check the first stage of this hypothesis it was suggested to prepare sets of text data by the sliding window method. The sliding window method is an algorithm of transformation, which allows to form a set of data from the source text, which can serve as a set for research.
In this case, the window is understood as the size of the window containing the set of texts that are used to conduct the research. During the algorithm operation the window is shifted along subchapters of the text by one measurement unit, and each position of the window forms one text. An example of the method operation is shown in Figure 2.
The next step of the second stage of the study is to compare the data obtained with different window sizes. To do this, we took the windows that include the first element of the data set. These windows have been reduced to the same size (by excluding word sets that were not included in the smaller window). Next, the Pearson correlation coefficient of matrices was calculated. The results of calculating the Pearson correlation coefficient between different sizes of the window are shown in Table 1.
The results of the second stage allow to make an assumption that it is possible not to use the whole text, but only a part of it, when analyzing text data. It can be seen from
VALUE PEARSON CORRELATION COEFFICIENT BETWEEN DIFFERENT WINDOW SIZES
The research was supported by the Ministry of Science and Higher Education of the Russian Federation (Grant # 0777-2020-0017) and partially funded by RFBR, project numbers # 19-29-01135, # 19-31-90160.
R. Deng, “Research on the Model Construction and Development of Computer Information Acquisition System”, IOP Conference Series: 5 0.79 0.8 0.8 0.82 0.83
7
A.S. Mukhin, I.A. Rytsarev, R.A. Paringer, A.V. Kupriyanov and D.V. Kirsh, “Determining the proximity of groups in social networks based on text analysis using big data,” CEUR Workshop Proceedings, vol. 2416, pp. 521-526, 2019. [7] I.A. Rytsarev, D.V. Kirsh and A.V. Kupriyanov, “Clustering of media content from social networks using BigData technology,” Computer Optics, vol. 42, no. 5, pp. 921-927, 2018. DOI: 10.18287/2412-61792018-42-5- 921-927.