Social networks play an important role in the modern world, and it is important to define the important and popular topics discussed. This article deals with data collection from the social network Twitter, and further clustering and classification of the collected data.
The extra-large volumes of data in information technology - data sets the size of
which is beyond the capabilities of typical database (DB) for collecting, storage,
management and analysis of information [
The concept of big data means working with the vast volume of information and varied composition, very frequently updated and located in different sources in order to increase efficiency and create new ones.
At the moment, social networks are at the peak of their popularity, millions of people
are already use Facebook and Twitter. Many companies need to analyze the data
collected from social networks to assess the relationship of users to their products [
Clustering (or cluster analysis) - is the task of partitioning a set of objects into groups
called clusters. Within each group must be "similar" objects, and objects of different
groups should be as different as possible, at the same time certain measure has to be
determined. The main difference between clustering and classification is that a lists of
groups are not clearly defined and determined during the process of the algorithm.
The main goal of clustering is the searching of existing structures [
k nk di ti
After determination by the TF-IDF measure, clustering can be performed by various
algorithms, such as k-means, for example [
Also, there is another approach to solve the problem of the classification - the classification of the information through the machine learning.
Machine learning - the process through which the machine (computer) is able to show the behavior which was not explicitly programmed. There are two types of learning: inductive and deductive.
In the work of researchers involved in cluster analysis of the text information in
various search engines, the inductive measure Word2vec is frequently used [
Word2vec analyzes the use of words contexts and concludes that they are or are not
close in meaning. Since word2vec making such conclusions based on large amounts
of text, the conclusions are quite adequate The algorithms on which word2vec is base
dare described in detail in [
The examples of vector distances obtained by word2vec are in Table 1.
Paris 0.978443 Spain 0.665923 Belgium 0.665923
Italy 0.633130 Portugal 0.577154
Russia 0.571507
Germany 0.563291
One type of a deductive approach can be considered is the Latent Dirichlet Allocation
(LDA). This generative model that allows to explain the results of observations with
the help of implicit groups, that allows to receive an explanation of why some of the
pars of data are similar. Typically, when using this approach, you identify a limited
number of topics and further states that each document is a mixture of a small number
of topics [
For more detailed analysis, it is best to combine different approaches and techniques depending on the amount of processed data. 3
To research the work of TF-IDF algorithm software tool that allows to collect data directly from the Twitter servers has been developed. The implementation is based on an open Twitter API 2.0 interface. As the object of the study the tweets from Samara region were taken, for this as a selection criterion of the messages geolocation was set to the Samara region (including all settlements of the region). All tweets collected in such a way need to be clustered using TF-IDF metric for short messages of 140 characters in length and the algorithm k-means.
To carry out the data collection from Twitter server a request containing a consumer key and consumersecret key was sent. In reply we have oauth.accessToken, oauth.accessTokenSecret which give us the ability to retrieve data from servers. The second step is to send the query-request, in response to which a set of tweets returns.
Next, the third step is the counting the TF-IDF metric values for each message. 4
The data was collected for 24 hours, by the query-request, which characterizes the Samara region. As the result over 6000 messages has been collected. By applying metrics TF-IDF and the k-means algorithm 22 clusters were obtained. On example, one of the obtained clusters (Figure 2) shows that the messages are similar in meaning, but among them there are messages with "foreign" theme. Apparently, such is not quite accurate result was obtained due to the fact that studied Twits have a 140 character limit. For this reason, for more accurate clustering and further classification it is necessary to modernize TF-IDF measure, introducing additional weighting coefficients corresponding to the number of symbols (words) in the message.
Moreover, high density of the clusters (Figure 3) shows the need for revision of the metric. TF-IDF metric works for short messages with relative accuracy. For this reason it is necessary to optimize this metric: adding weight coefficients, the coefficients associated with the hashtag, the introduction of the normalization coefficient associated with the length of the message and the number of words in it, etc.
Issues related to clustering and further classification of text data are relevant in rel ation to the enormous spread of social networks and online services worldwide. Approaches and techniques presented in the article are planned for testing on text data collected from Twitter social network in the Russian segment. Collecting the nece ssary data is being produced by means of the developed software system, based on time zones and geolocation. It is planned to develop the subject in the direction of the output and optimization of parallel clustering algorithms.