Analysis of public Internet data is a popular topic for research. Study prospects of using personalized data led the scientists to the problem of constructing a social profile. It is structured set of information, which is able to uniquely characterize a particular person. The social profile building is carried out through the analysis of the filtered Internet open source data. Raw social profile data are subdivided on static and dynamic parts. Dynamic unstructured data includes text and multimedia information and cannot be handled by classical analytic means. The analytical task of social profile data is achieved through mathematical tools of the set theory, Big Data software, NoSQL data stores and analytic tools for social media. Also, mod-ern methods for analysis of the multimedia data are helpful. The techniques review for the analysis of multimedia content (graphics, sound) is offered. Analysis of multimedia resources is complicated by the variety of information processed types. The current work is devoted to combine existing experience by automating non-textual information processing in the task of a social profile building. We consider such areas as Big Data analysis, visual analysis, Optical Character Recognition, speech recognition, neural networks and specialized algorithms for specific objects recognition and linking them with a social profile. Automated processing of multimedia information will improve the completeness and accuracy of the final social profile. In addition, the results of this work can be used to study the social phenomenon of viral media.
Processing of heterogeneous Internet open data covers a very wide range of
applicability in various fields of human activity. Application analysis results of such
information starts with contextual sampling recommendations and end large-scale
scientific society research for the purpose of counteraction to criminality. A social profile
building is one of these generally applicable tasks. Social profile [
Next step is the information obtained analysis after the person identification on the
Internet, collecting and filtering of primary social profile data [
Issues of social media analysis profile of a man using a variety of means (Big Data, OCR, image analysis, neural networks, etc.) raise in this paper. In particular, the processing of graphical and audio content, that may be associated with a particular personality, is described. Also, there is attempts to automate the unstructured data analysis. It's necessary to extract the statistical and semantic components from them and to bind them with the social profile data. 2
Studies on analysis of multimedia information are not as common as works related to
processing text data. The reasons for this are simple:
• The diversity of multimedia information types;
• Semantic variability of content, depending on the recipient subject;
• A large amount of data, compared with the text;
• The existence of distortion;
• Difficulties of machining processing and structuring the results.
Methodology of speech and visual objects recognition fairly well researched at the
current time. But differentiation issues of recognized entities not yet resolved. Also,
determining the meaning and symbolism of the certain media image in the particular
situation poorly studied. This theme is dealt with in article named "Unveiling the
multimedia unconscious: implicit cognitive processes and multimedia content
analysis"[
Authors of "Triangulating Social Multimedia Content for Event Localization using
Flickr and Twitter" [
Jinglan Zhang, Kai Huang, Mark Cottman-Fields and others present an overview
of techniques for collecting, storing and analyzing large volumes of acoustic data
efficiently, accurately, and cost-effectively in their work called "Managing and
Analyzing Big Audio Data for Environmental Monitoring" [
A paper named "Fusing audio, visual and textual clues for sentiment analysis from
multimodal content" [
Research work called "Mining Melodic Patterns in Large Audio Collections of
Indian Art Music" [
Experience of works discussed above can be useful for handling multimedia social profile data.
It is need to analyze the existing text data before proceeding the multimedia processing.
Filling information card produced during collection and filtering stage of source data.
Dynamic text data processed by subsystem of social entities and relationships analysis.
It uses specialized tools of NLP (natural language processing), a tonality definition of
the text and the predictive text mining methods. IBM ContentAnalytics software [
Next step is to create a mathematical model for the construction of iBase social profile database. Model based on the results of the textual data analysis using BigInsights and Content Analytics. Social graph is built on this basis. It specifies the possible relationships between the considered person and social profile entities (mentioned persons information about persons, associated to the considered person in any context; organizations - information about various facilities related to personalities from the social profile; events - information about events, bringing together a group of people by some common features; contact details of the person; activities, specialization, attainments and hobbies of person). A such graph schematic example is shown in Figure 1. The main work on the identification of relationships and dependencies is performed in IBM i2 TextChart program. Data analysis is performed using the following algorithm: 1. Raw social profile data shall be input in the TextChart project via import CSV-file with source information. 2. The text is allocated first read important information, then search is carried for repetition and synonyms throughout the text by using the Find tool. 3. The results are added to the project as social profile entities (considered person, activities, etc.). 4. Words are highlighted in the processed text, expressing the relationships between the generated objects. Then corresponding objects are selected in the dependency graph window. 5. The desired connection type is selected by clicking the Insert Link button, then the expressions are added as the connections. 6. Similarly, new objects and attributes are added to existing ones by means of navigation between entity search results. 7. Entries calculated for each option when finding conflicting information. Then it concludes about their truth: false information is removed from the social profile or additional specifying search is performed.
The results of visual analysis are the social graph and iBase social profile database. It is possible to find implicit relationships built on the graph in addition to the hidden information directly identified during the analysis. Analyze the social profile text in such way will form the basis for relations with the social objects. These objects obtained in the analysis of the multimedia information. 4
In contrast to textual data, multimedia content is difficult to analyze using traditional
means. Therefore, it is necessary to resort the implementation of big data solutions,
machine learning and neural networks. Examples of such systems can be considered
Google Analytics, MS Azure, Multimedia Mining Marvel, Quaero. The most
common multimedia information are the images and sound. There are a number of
multimedia data analysis strategies [
Multimedia content can appear in two different versions as part of social profile building: 1. Multimedia data, viewed and created by a considered person. That information can speak about the activities and preferences of the person. Also it applies to the author's content. Analysis issue consists in comparing the multimedia objects with the existing samples in the Internet and a social profile (e.g. definition of personal musical preferences across multiple recordings). 2. Content that contains the information about the considered person itself. The analysis purpose is to identify the essential information from the direct media object (e.g. emotion recognition in the photo, the semantic expressions selection of audio).
Content-based analysis method appropriate to use for multimedia information
processing of the first category [
User content may contain different identification labels in the service data (information about the author, the developer device, spatial data) and within the multimedia entity. Examples of these labels can be: the author's signature or watermark, typical author's style of the object, author's mention in comments, etc. Some of this information is implicit, so the possibility of automatic processing for user-generated content is very limited. The use of visual analysis is desirably. Processing algorithm will be different from one presented in paragraph 3 of this article that, you need to consider abstractions in addition to text information extracted from the multimedia objects. These abstractions are mined manually and have a fuzzy interpretation.
Another object of the user-generated content analysis is to define options for using multimedia entities. In this case, the analyst should link a particular media object to the text definition of the event meaning in which the object is mentioned. Big Data means searches for all possible situations in Internet of the multimedia entity use. Further, statistical analysis and tone selection are performed for the accompanying text. Also, it is recommended to apply visual analysis means as in the previous case.
Let's take a second data category. Content-interpretative method is used for its
analysis [
A speech and intonation recognition are the main focus of audio analysis in the construction task of social profile. The results of the audio analysis are: the sound characteristics of the considered person voice, attached to the social profile; recognized text linked to the original record. The characteristics of the other people voices also may be taken from the analyzed audio file for comparison with other social profiles. It should be noted that the sound voice characteristics can`t be regarded as an information card elements. They can vary considerably depending on the age and condition of the per-son, environment and recording quality.
Currently, there are a sufficient number of freely distributed speech recognition systems with open source: CMU Sphinx, Julius, HTK, Praat, SHoUt, VoxForge and others. Many of them are based on the use of hidden Markov models and neural networks. Sound voice characteristics include spectral-temporal, cepstral, amplitudefrequency and signs of nonlinear dynamics.
It should be mentioned that there are seven types of intonation construction in
Russian language (3 interrogative, 2 exclamatory, 2 narrative). Intonation recognition of
human speech can take place in three steps [
Accuracy of this algorithm depends on the quality and duration records, characteristics of speech, etc. 6
Analysis of the graphical information includes: image, text recognition and compare the results with the creation date of the file under consideration. The approaches of the graphic information recognition are divided into three categories: iteration methods, artificial neural networks, and object search paths to the further study of their properties. OCR technology (Optical Character Recognition) allows you to find the printing and handwriting (rarely) text. The original image quality has a strong influence on the final result of the recognition algorithms. As in the case of the audio analysis, the image-recognized text should be attached to the original object. Further it will be treated by means of text analytics. Findings data are reduced in the resulting table after completion of the image processing. That table contains the following key parameters: detected faces, their emotions, the list of labels, environmental data (the recognized objects), service image information (size, creation date, name, etc).
Pattern recognition in the task of a social profile building is divided into: search people in the images, the determination of their sentiments and the environmental elements allocation. Face Detection Services provide by projects such as ASID, FaceID, FindFace, Vissage Gallery and others. Emotion recognition is a more complicated procedure, major problems which are to determine the face position and color, quality of illumination, foreign objects in the image foreground. However, despite this, there are ready-made solutions: FaceReader, FaceSecurity, etc. Development of systems for determining environmental elements is not sufficiently well-researched now. There-fore, it is advisable to use specially trained neural network to solve this problem.
The degree of analytical subsystem maturity for social profile building affects the information content and correctness of the final results. This is especially true for multimedia processing, because these data are diverse and difficult to automate processing. A review of existing approaches to the multimedia content analysis is provided in this paper. Also, possibility of their application in the task of a social profile building is considered. It was revealed that is possible to use existing algorithms for processing audio data, while the analysis of graphic information requires improving recognition technology.