The intellectualization of data processing for Internet memes flow includes problems of search and identification of network memes, text and meme image recognition, analysis of relevant information, detection of the graph structure of meme flow distribution network, clustering, and visualization. The paper focuses on viral pictures consisting of an image and a text. It is a challenging problem even with such a simplified representation of memes. Memes consisting of images and text are widely spread in social networks. The modeling process involves both an immediate content of memes and information contained in the environment and comments. The intellectualization of processing such specific information is accompanied by expert assessments. Tool design for processing and analysis of the Internet meme flow is based on a group of methods to solve the following problems: extracting and analyzing text from an image; text classification; implementation of different network metrics to adjust the classification, collecting different metrics to create a predictive system. The research paper introduces a novel approach to the development of methodology and software designed for automatic and intelligent detection of political Internet meme's influence on Russian-speaking Internet users. The social network Vkontakte was used as a meme dissemination medium.
Social network research is getting increasingly popular nowadays in the world [
According to the Merriam-Webster dictionary, the meme is defined as "an idea, behavior, style, or usage that spreads from person to person within a culture". Internet memes are represented in the form of viral images visualized as a picture typically accompanied by a piece of text.
When studying the processes of meme dissemination and visualization, it is
necessary to take into account particular semantic environments, the text itself, word
combinations, words, and concepts as a whole, all of them being the elements of the world’s
language picture [
Geo-linking of memes to a certain distribution agent can be achieved in geosocial
(location-based) networks. Relevant tools use GPS data in a mobile device and require
access to its geolocation information. The network proximity parameter is used to find
friends in a social network, make recommendations, and collect behavioral patterns for
specific users. An overview of approaches to network proximity and the application
that uses network proximity to distribute social content is provided in [
The current work aimed to develop an intelligent system designed to extract text fragments from images presented as Internet memes, to classify them, to use various network metrics to make corrections to its distribution forecast model, and to define particular classes where specific Internet memes might belong.
Internet memes analyzed within the present work can be classified as viral images consisting of some text combined with a picture. The social network Vkontakte was selected as an Internet medium for meme dissemination. 2
Let there be a certain environment (in particular, a social network) for the Internet memes dissemination. The objects of this environment are Internet memes characterized by properties inherent to the distribution environment. The goal is to develop and implement specific tools intended to classify objects belonging to a given Internet meme dissemination environment and predict its distribution.
Due to the open nature of viral memes, there is no basic list that describes (or can describe) each meme, and there is no authorized Internet body to index new memes according to some conventions. Hence, a trained model cannot be effectively trained based on every class that existed before or may appear in the future. Thus, adaptive models and algorithms are needed for effective analysis and prediction.
The authors selected the social network Vkontakte as a dissemination medium. Records that contain text and an application in the form of an Internet meme will be referred to as objects of this environment. For example, we will classify each selected Internet meme as having political or non-political content.
The expected input to the system is a link to a meme in Vkontakte or a direct image with text. The ultimate goal of the project lies in creating a system that can correctly label Internet memes and predict their distribution.
Given the specifics of Internet memes, this problem cannot be solved by conventional algorithms. To solve the problem, it is necessary to design special methods and algorithms for detecting, recognizing, and classifying text, using network metrics, as well as clustering algorithms.
To design special tools for such a system, the following intermediate tasks must be solved: 1. extracting text from a meme image; 2. classification of a text piece associated with an image; 3. use of different network metrics to adjust the classification method for Internet memes and to create for them a distribution prediction system; 4. analysis of data obtained using the system; 5. clustering and visualization of the Internet meme database.
Extracting Text From the Image Since Internet memes are generally images containing a picture and a text, it is necessary to design a recognition system (OCR) capable of detecting a text fragment on a noisy image and extracting it. To extract text from Internet memes, OCR is supposed to: 1) find text fragment; 2) pre-process image containing text; 3) recognize text.
Initially, for text recognition, we used the freely distributed Tesseract text
recognition software developed by Hewlett Packard. Tesseract allows recognizing text based
on the LSTM neural network [
However, in the course of the development process, it became obvious that Tesseract algorithms were not always sufficient to effectively select text fragments superimposed on top of an image and achieve satisfactory results when classifying small phrases or phrases characteristic of Internet memes. Hence, it became necessary to develop a more specialized OCR.
To solve the first problem of finding text within the image, it was selected the EAST
algorithm (An Efficient and Accurate Scene Text Detector) [
At the stage of preprocessing images, containing mostly text, one needs to select exclusively text and get rid of the noise. Problematic is that the text color is unknown while it may contain several shades of the same color. Preprocessing includes image clustering, creating a mask separating text from the background, and determining background color.
To implement the symbol classification algorithm, the convolutional neural network
was used. Such a network (CNN or ConvNet) belongs to the class of deep neural
networks and is most often used for the analysis of visual images. CNN networks are
regularized versions of multilayer perceptrons. Multilayer perceptrons usually belong to
fully connected networks, i.e. each neuron in one layer is connected to all neurons in
the next layer. The "full connectivity" makes these networks predisposed to data
overload. Typical regularization methods include adding some form of weight measurement
to the loss function. However, CNN takes a different approach to regularization: they
take advantage of hierarchical templates applied to data and collect more complex
templates using smaller and simpler templates. Thus, on the scale of connectivity and
complexity, CNN can be placed at a lower level. The ConvNet network can successfully
capture spatial and temporal dependencies in an image using appropriate filters. The
convolutional neural network architecture provides better alignment with the image
data set by reducing the number of parameters involved and allowing re-use of weights.
In other words, the network can be taught to better understand complex images [
The input data is an image in the RGB color model, separated by three color planes – red, green, and blue. The ConvNet is used to transform images into a form that can be more easily processed without a loss of properties critical for making a good prediction. The data used as a training sample included 59567 objects divided into 37 classes: 33 letters of the Russian alphabet of upper case and 4 letters of the lower case – "а", "б", "е", "ё". 1814 fonts were used to create the sample. The sample was divided into a training sample containing 47,653 objects and a test sample containing 11,914 objects. The neural network was implemented using Python and the Keras library. The convolutional neural network model for letter classification was trained on 30 epochs and showed 98.8% quality on training data.
Activation layer LeakyRelu, alpha=0.1
Activation layer LeakyRelu, alpha=0.1
Activation layer LeakyRelu, alpha=0.1
Smoothing layer Layer with dimension 128 and linear activator
Activation layer softmax with dimension n=number of classes
At the stage of line recognition and combining recognized letters into words for selected paths, centers of mass were found, and then basic strings making up the text were constructed. If the top and bottom points lie above and below the text line, respectively, then this path also lies on it. If the distance between paths is greater than the average distance multiplied by a certain coefficient, it means that there is a gap between paths. This not only allows to determine the correct sequence of character processing but also eliminates the noise that could remain after image processing. Then, based on the intervals between letters, paths are combined into words.
As a result, the pseudocode of the text recognition program looks like this Fig. 2: Algorithm Input: Sample Xm={x1,…,xm} – number of clusters; ℎ – line spacing threashold; Output: _ – image text 1. Image clustering 2. Masque building 3. Letter paths selection c lines = [] for cnt in contours: if moment of cnt not in lines then add cnt on line for line in lines: mean_dist = mean distance between paths for <i:=0 to len(line-1)> do: if dist(cnt[i], cnt[i+1) > mean_dist + threshold then separate cnt[i] and cnt[i+1] with space result = “” for <line in lines> do: for <i:=0 to len(line)> do: result += Classification cnt[i]
The developed OCR effectively highlights text and classifies small words. However, during the development process, there were problems when classifying large volumes of text. Therefore, the final version of the program for extracting text from images includes two stages. At the first stage, the EAST algorithm selects text blocks. If their amount is big and they form a large group, then Tesseract is used to recognize them. If the number of text blocks is not large, the formerly described approach for text extraction is used.
Pseudocode for the final text extraction and recognition algorithm "Internet meme": img; ◃<S – input image > n = minimal number of words; textblocks = EAST(img); ◃<extracting text from image> combining all intersecting text blocks; if len(textblocks) > n then txt = Tesseract(textblocks); else txt = TEXTEXTRACTOR(textblocks); show txt; The social network Vkontakte was chosen as a medium for Internet memes dissemination. The objects of classification are records containing text, an application in the form of an Internet meme, and comments. Such objects can be classified as political or nonpolitical. An object is classified according to the class of the record text, the Internet meme text, and comments text. So, it is necessary to build a classifier detecting whether the text is political or not.
To solve this problem, we used a sample consisting of 63 political memes and 44 non-political ones. Some additional objects were taken from the social network VKontakte and were represented by comments having both political and non-political content. In total, 168 phrases were used for training. A validation sample contained 11 political and 7 non-political memes. The data preprocessing stage included data normalization with the stemming algorithm and converting data into a vector representation.
The support vector machine [
Word clouds were built for both classes (Fig. 4, 5). Using these clouds it’s possible to note that the "political memes" class includes images containing the names of countries, regions, and political leaders. The "non-political memes" class does not have a clearly defined group of words, because sentences related to this class have different topics.
The classifier demonstrated an accuracy of 81.25 % on the test sample. With the growth of the training data, the accuracy of the classifier will be improved.
Five groups from the social network Vkontakte were selected for analysis. We managed to extract 43 political memes using the developed tools. A list of extracted political memes allowed to obtain lists of participants who "rated" the selected entry.
A total of 66,184 participants were collected. A separate list was created for cities where the selected participants live. This list was used to construct a pie chart displaying the most politically active regions of Russia. The most active regions are Moscow, St. Petersburg, Yekaterinburg, Novosibirsk, Krasnodar. Cities with a frequency less than 0.4% were assigned to a separate category "other".
To identify groups of cities with the same political activity in the obtained data we
used the K-means algorithm [
Algorithms and software to select text from images, classify text, and cluster obtained data are elaborated. Distinct algorithms of text and image classification are subjected to comparative analysis. The algorithms to extract text from an image and to render images are implemented: ─ finding text using the algorithm EAST; ─ grouping images containing text information; ─ preprocessing images containing text information; ─ detecting text lines on images; ─ extraction of letter paths; ─ classification of letters; ─ combining letters to get a text.
A system to process and analyze the Internet memes flow is designed.
The drawbacks of the obtained model refer to the accuracy of meme classification.
So, it demonstrated only 85% accuracy in determining whether a record belongs to
political memes or not. It is mainly caused by an insufficient set of training data, lack of
image classification, errors when extracting text from images. To improve the quality
of classification, it is necessary to increase the amount of training data, improve the
image extraction algorithm, and implement an image recognition algorithm. Some of
the obtained results are represented in [