The speed of full-text search directly affects the process of text analysis. Search engine creates a text index, which is used for fast full-text search. Solr and ElasticSearch are two popular search engines. A text analysis system requires fast implementing searching and indexing at the same time. This paper describes preprocessing workflow of the analysis system called Formal Concept Analysis Research Toolbox (FCART) and experiment of searching and indexing social networking service data at the same time. Results of the experiment show which search engine is better as the core of FCART search subsystem.
Formal Concept Analysis Research Toolbox (FCART)[1] is a data analysis system,
which supports texts knowledge discovery techniques, including those based on
Formal Concept Analysis[2], clustering [
In this paper we describe key features of search engines mentioned above, describe
preprocessing workflow of FCART, requirements to the full-text search engine as a
part of an analytic system and then describe the experiment of searching and indexing
at the same time of data that was gathered from social network service LiveJournal
[
Formal Concept Analysis (FCA) has many applications in different fields [
From an analyst point of view, basic FCA workflow in FCART has five stages (see Fig. 1). 1. Filling Intermediate Data Storage of FCART from various external SQL, XML or JSON-like data sources (querying external source is described by External Data Query Description (EDQD); 2. Data indexing and preprocessing; 3. Loading a data snapshot from a local storage to the current analytic session (snapshot described by Snapshot Profile). Data snapshot is a data table with annotated structures and text attributes, loaded in the system by accessing external data sources; 4. Transforming the snapshot to a binary context (transformation described by
Scaling Query); 5. Building and visualizing concept lattices and other artifacts arising from formal contexts (binary data tables) within an analytic session. External Data Source
External Data Set
FCART IDS Import/Export
Tools FCART Client Import/Export
Tools
FCART Intermediate Data Storage Local Session
Data Base Analytic Artifacts Pattern Structures
Clusters Other artifacts
JSON Collection
Data Snapshot (Multivalued Context)
Binary Context Concept Lattice
External Data Query
Description Snapshot Profile Scaling Query +
Graph Generators
All steps of data analysis are accomplished with the use of Client program. In the background, Client interacts with Server using Server REST interface commands. In the previous articles, we described Server REST interface in detail.
In the current version of FCART the preprocessing workflow consists of two steps: 1. Normalizing incoming documents 2. Creating index At normalizing step, FCART creates a document in JSON format, selects one text field as identifier and calculates document metrics (size, word count, etc.). At index creating step, FCART creates index on one or multiple text fields. This index is used for fast data search. 3
Full-text indexing library adds content to a full-text index. It then allows performing queries on this index, returning results either ranked by the relevance to the query or sorted by an arbitrary field such as document's last modified date. The library gets fast search responses because -- instead of searching the text directly -- it searches the index. The library cannot be used independently, it can be used as a database plugin or as a part of another software system.
There are several key features of the full-text search library: 1. Search, initial indexing and reindexing speed; 2. Support of languages and dialects; 3. List of supported programming languages; 4. Format of supported documents;
Stemming algorithms;
List of metrics and ranking methods;
The full-text indexing system is a standalone, independent product. It is based on
fulltext indexing library and provides more features. For example, Solr and ElasticSearch
are based on same library called Lucene[
LiveJournal is a popular in Russia social blogging system. Using LiveJournal Web API we have extracted 1000,100 000, 1000000 person’s profiles. Then we carried out some classical SNA experiments and tested improved subsystems of FCART. Initially, IDS takes five initial profiles and initiates process of downloading the neighbourhood of these persons by relation “has a friend”. Each profile consists of several blocks of fields : 3. Nickname (“nick” – unique identifier). 4. List of friends (“friends” – array of nicknames). 5. List of users who checked this profile as a friend (“friendsOf” – array of nicknames). 6. List of interests (“interest” – array of tags). 7. List of watching communities (“watching” – array of community names). 8. List of memberships in communities (“memberOf” – array of community names). 9. List of posts (not used in next stages). 10. Other personal information. threads
Profiles
100 18 2 0.2 195 21 3 3671 291 32 42710 4483 471
Solr 5.4 and ElasticSearch 2.0 have been deployed on a computer with
installed INTEL Xeon Processor E5-2670 (2,60 GHz, 4 cores) 8Gb of RAM
[
Experiment shows advantage ElasticSearch over Solr from 3 to 7 times.
ElasticSearch 1000 profiles 3000 5 10000 profiles 5700 16 100000 profiles 6200 29 1000000 profiles 6900 42
In this paper, we have described key features of two popular full-text search engines,
architecture and preprocessing workflow of FCART software. The case of searching
data and indexing at the same time was considered. Our experiment have shown
significant advantage ElasticSearch over Solr. In future work we will use
ElasticSearch as the core of the search subsystem. Next FCART release will be
integrated with ConceptCloud[
This work was carried out by the authors within the project “Data mining based on applied ontologies and lattices of closed descriptions” supported by the Basic Research Program of the National Research University Higher School of Economics.