In this paper we present main challenges related to scaling-out the NCBO Resource Index further. We look into several recent developments that can relate to those challenges. Finally, we propose a solution that we plan to implement together with a description of an intended evaluation.
The NCBO Resource Index is a system for ontology based annotation and indexing of
biomedical data. The key functionality of this system is to enable users to locate
biomedical data resources related to particular concepts. That functionality is based
on semantically-enhanced search1. The Resource Index currently includes 22 different
data resources comprising over 3.5 million data elements resulting in 16.4 billion
annotations stored in a 1.5 terabyte MySQL database. The semantic expansion
consumes considerable amount of resources in terms of storage and processing power.
Based on research in density and evolution metrics it became possible to significantly
reduce processing requirements so the Resource Index can be computed in an
acceptable time on a single machine [
In recent years data-intensive, non-relational processing has seen significant
development starting with Google’s MapReduce paper [
development of Hadoop2 and related technologies. It seems natural to investigate those technologies in the context of our problem.
In Section 2 of this paper we name particular challenges that we face while scaling the NCBO Resource Index processing and maintenance. We analyze related work to search for possible approaches to those challenges in Section 3. Finally, in Section 4, we propose architecture of a solution that we plan to implement and evaluate. We conclude the main points in Section 5.
Storing source data. The data that are indexed can be divided in two major types.
The first type is raw data harvested directly from each resource before any processing
is done. The second type is data after concept recognition that includes term
expansion based on semantics. Storing and maintaining the semantically expanded
data is what causes the system to exceed 1.5 terabytes at the moment. As the number
and kinds of resources grows, we expect to see a more than 10-fold increase in the
storage space requirement, far exceeding the capabilities of typical systems. For
example, to include the entire PubMed resource (20M articles) in the Resource Index,
it was projected to consume nearly 20TB of disk space. Protection against data lost
due to mechanical or program failure will add additional complexity–keeping up to
three replicas of the data. Fig 1 (A) illustrates the expected increase in data storage
requirements. These estimations do not include possible compression.
Processing source data into the Resource Index. There are three main processing
steps required to perform the semantic expansion. The first step is concept
recognition, which can be classified as embarrassingly parallel—it can easily be
divided among processing nodes. The second step is term expansion based on
semantics, where the most computationally expensive process relies mostly on join
operations. During that stage, information on ontological distance between concepts is
also utilized. For every recognized term, approximately 14 additional terms are also
associated. Finally, an inverted index is applied on the entire set of associated terms
for efficient search. Unlike concept recognition, the expansion step, i.e. join
operations, is not easy to parallelize; however, based on the earlier metrics [
In general, processing time is expected to increase linearly with the amount of data. The linear increase is in itself not a problem. However, the expected increase in data amount is significant. Processing capabilities of a single machine are already limiting frequency of Resource Index update process and that will of course deteriorate even further with more resources. Therefore, it becomes clear that some form of dataintensive distributed processing is necessary.
Maintaining the Resource Index. The Resource Index that is used for search purposes is a product of several complex processes. However, that does not provide a full picture of complexity of updating Resource Index. At the same time, Resource Index has to be available for search and it would be beneficial to introduce the update as soon as possible. We need special architecture that will allow for concurrency of updates and searches. That will allow for continuous updates to Resource Index without taking the search offline. Moreover, such architecture should support many concurrent requests without noticeable decrease in performance.
In this section we present related work that has the potential to help with the aforementioned challenges. The biggest focus is not on scientific publications, but on several open-source projects that could be directly or indirectly applied to our scenario. However, some scientific publications are also mentioned where relevant.
We focus on solutions in so called NOSQL or non-relational domain. An alternative could be a distributed relational database. It would require fewer changes in the current architecture. However, less structured nature of the non-relational solution fits our data better. It also seems that non-relational approach should offer greater flexibility in future.
Our approach in this work focuses on materialization of semantic expansion. That is, data is expanded on disk and queries are directly performed on the data. Query should be understood here as a search query, not as a SQL nor as a SPARQL query. The alternative approach would be to expand the query and perform it on nonexpanded data. It would save disk space, but might negatively impact the performance at run time as some query expansions might have significant size. The queryexpansion approach is the topic of separate research work.
Storing source data. One of the prominent developments in storage is the Hadoop
Distributed File System (HDFS)3 designed based on Google File System [
HyperTable6 is scalable database also based on BigTable. It can run on top of HDFS and several other file systems. It offers relatively more limited integration with 3 http://hadoop.apache.org/hdfs/ 4 http://hbase.apache.org/ 5 http://hadoop.apache.org/mapreduce/ 6 http://www.hypertable.org/ other Hadoop subprojects than HBase. Cassandra7 is a more recent development also inspired by BigTable; however, to a lesser extent. It provides limited support in terms of further data processing.
Processing source data into the Resource Index. Processing mechanisms are
generally connected to a particular storage system. HDFS and HBase offer probably
the biggest variety of choices. First and foremost, they support Hadoop MapReduce
which is a generic processing framework based on Google MapReduce [
HyperTable has its own HyperTable Query Language with a SQL-like syntax. Casandra does not offer support of any query language at the moment. Maintaining the Resource Index. Maintaining the Resource Index includes two main tasks: updating and making it available for rapid search. In the best scenario both tasks should work in parallel. Cassandra and Voldemort11 can both be considered here as they focus on fast data serving in contrast with bulk processing in, for example, HBase. They can also handle parallel read and writes well. However, they do not offer any dedicated mechanism for handling the connection with the bulk processing backend. Though, it can be constructed. The most recent project that emerged is ElephantDB12. It consists of two integrated components where one is dedicated to creating the Resource Index and the other to serving it. 7 http://wiki.apache.org/cassandra/ 8 http://hive.apache.org/ 9 http://pig.apache.org/ 10 https://github.com/nathanmarz/cascalog 11 http://sna-projects.com/voldemort/ 12 https://github.com/nathanmarz/elephantdb It this section we outline a solution we intend to implement and a related evaluation plan. In Fig. 1 (B) we present a sketch of the architecture to be implemented. The main processing will be evaluated on a 10 node cluster, and the query processing part will be implemented on a smaller cluster.
Based on the available solutions we plan to implement data storage both directly in HDFS and in HBase. We will evaluate relative performance differences. In particular, potential benefits of standard and custom HBase index (not to be confused with the Resource Index) and its multidimensional structure. The multidimensional structure of HBase will be also investigated as a way to reduce storage requirements for semantic expansion. In later phases, we will investigate possible applications of time stamping in HBase for analyzing the Resource Index evolution with time. The subject of evaluation will be mostly performance (processing and storage); however, additional elements like ease of maintenance or support for processing approaches will be also assessed. Independently of other tests, compression will be applied to data to examine its impact on both storage requirements and processing time.
The processing stage will be initially implemented in Pig. In later phases, we will compare it with pure MapReduce jobs to potentially leverage a custom HBase index. Finally, we plan to implement the Resource Index maintenance using ElephantDB. To our knowledge this would be first independent evaluation of that system.
In this paper we presented challenges related with scaling-out the NCBO Resource Index. A parallel could be found between our challenges and recent developments in data-intensive processing. By looking into those developments we proposed a solution that we plan to implement and evaluate. This should allow scaling-out the NCBO Resource Index to cover all the required resources. In addition, the evaluation will fill gaps in knowledge about relative (processing and storage) performance of the aforementioned technologies.