Motivation: High-throughput technologies, like gene expression arrays and next-generation sequencing, provide enormous data sets, which are too large to transfer or download quickly. The study of such data, for our application this means explaining the measurements with a molecular interpretation of disease etiology, requires continuous updates and refinements as novel interpretations are pursued. The complexity of the problem requires a diverse range of expertise. And thus a shared view is crucial for a successful collaboration - within and between institutions. Web services and traditional web pages provide centralized data storage and synchronized presentation. Relying on a single central server, though, comes with its own flavor of reliability and performance issues. Every time the server is busy solving a request, the user is forced to wait. It is therefore very beneficial to combine the integrity of web services and the share-ability of web pages with the fluency of a desktop application. Increasing the interactivity of data presentation to each individual user allows for a more interactive knowledge exchange on the group scale.
Here, we present a combination of Open Source technologies for distributed, synchronized and failure-resistant storage of huge data sets as the technological basement for globally fast access to research data. Accordingly, this work explores the derived possibilities for interactive presentation to a group of locally distributed researchers, as enabled by a problem-tailored web application. To aid in the investigative work, the user interface shows minimal latencies. These goals are achieved by capitalizing on related developments in distributed data storage and asynchronous web technologies, most notably the non-relational database Apache Cassandra and the Google Web Toolkit. This combines efficient pre-processing with parallelisation.
The developed web application looks akin to a typical desktop application and is highly responsive, since it downloads needed data in parallel, while the user is happily working. The researcher can prepare different data set views for different aspects of his analysis, which are immediately available for colleagues and collaborators. By *To whom correspondence should be addressed. underpinning every decision and conference call with a synchronized shared data set, group communication is greatly improved. This work demonstrates that the interactivity with the user to work on large data sets is strengthened with remote applications and typical “show next page” delays are overcome by employing the latest web technologies. This way, the strong server-user interaction allows for the seamless extension for serving additional users and thus allow for collaborations.
Availability: Source code for the web application and the data storage back-end was released under GNU Lesser General Public License and is freely available for download from http://github.com/fxtentacle/eQTL-GWT-Cassandra 1
The problem is as old as Bioinformatics itself: biological research yields more data than a human can handle manually.
Technological advancements in biochemistry and better insights in computational biology together have accelerated and broadened the avalanche of information. We are already losing this fight, as much data is generated in labs, which is never rendered available for analyses in other contexts, since the information is not publicly available. Given the inter-individual differences of patients and controls, many new model organisms, many more tissues investigated at an ever more detailed level and additional test conditions, the avalanche of usable knowledge will not stop in any foreseeable future.
For the analysis of large data, one needs to find humandigestible aggregations of it. It shall be a goal-oriented presentation where the essential information to form hypotheses is brought together. Further statistical evidence in the data will guide the downstream analysis, as will additional external information. For our application, the molecular interpretation of disease phenotypes, this is a process of continuous refinements and updates, accompanied by fruitful discussions with fellow researchers and collaborators. Every researcher needs software to look at and evaluate all possible measurements and explanations, without being forced to manually download or update the huge underlying data set. The researcher is doing investigative work and, therefore, it should be possible to browse all of the data interactively. 1.1
The most common approach found in the Bioinformatics community for presenting and browsing data sets is to store research data inside a relational database and write custom-made software or web pages to present the data. Some approaches also include the ability to produce diagrams, but most are limited to text and tabular data. While generated web pages are usually static, this form of presentation is to be considered interactive if the user can issue filtering requests to the web server and retrieve a new web page containing the response within an acceptable response time.
Inside a relational database, data are stored as sorted by the primary key and separate look-up tables are generated for column range and equals queries. Since these look-up tables need to be identical for every database server in the system, common database software does not allow write requests to be distributed. There is, however, a range of solutions for replicating databases to multiple servers such that read requests can be distributed. This means that a relational database scales well with huge data sets when it comes to read requests and does not scale at all with write requests.
For very basic applications where research data are imported only once and then never modified, this works well. However, as soon as data are being processed or annotated on the user’s behalf, the write rate of a single server is sustained. With traditional databases, only reads can be executed on a replica of the data and, therefore, the system as a whole cannot scale for writes.
We would like to see a word-processor like working environment: every data editable and inspectable with local views on the full data, helped by search tools and situation-dependent statistics like word counts. We want to avoid data transfer (takes too long), web-forms based “paged” interactions, any non-scalable components and want to directly support inter-user communication. 2
The system evolved in Java from the PHP implemented TiQS interacting QTL System (tiqs.it). It does not require any local installation work apart from unzipping and executing a shell script. Web service requests are handled by a custom-written Java servlet inside a Jetty 6 (codehaus Foundation) servlet container. Jetty was chosen for its capability to run with the same configuration file on Windows, Mac OS X and Linux. Its direct competitor, Apache Tomcat, instead needs to be set up individually for each machine.
Data describing the topology of the system are stored in a PostgreSQL
database (www.postgresql.org) with Hibernate 3 (www.hibernate.org). The
valuable scientific data is stored inside a distributed Apache Cassandra
From the previous PHP implementation, no code could be saved and a separate execution environment was designed from scratch. It was taken special care to ensure the application to remain compatible with common IT constraints in research institutions: the application needs HTTP access on a random port for each worker node, as well as two configurable ports on which the peer-to-peer communication of the distributed database will take place. The system can work with almost any memory and hard disk configuration and every Windows, Mac OS X and Linux computer can be turned into a worker node simply by copying a folder and running a shell script. The worker node software could also be deployed remotely.
The used database is data center aware in its distribution of redundancy. Accordingly, a complete self-contained copy of the data is kept at each physical facility, if the researcher has appointed one or more machines to use for data storage. The researcher can immediately start working, even while a local copy of the data set is being synchronized with coworkers and collaborators round the globe automatically. There is no initial waiting time for downloading the complete data set or manually deploying updates. Current Linux distributions like e.g. Debian (Möller et al., 2010) have all packages readily available or downloadable at the developers’ websites. 3.2
Most current research software does not include any sort of failure tolerance and data replication, which comes as a surprise given the price of good research data. With growing data sets, storing and retrieving the correct subset is not a simple task anymore. A chain is only as strong as its weakest limb and therefore a bad database and schema choice will completely ruin a data exploration software.
Apache Cassandra was chosen because it is stable, fast and replicates data. This database is used in production at Facebook with billions of rows and therefore can be assumed to be reliable. Moving away from conventional relational databases towards a novel distributed system out of a key-value store and manually managed indexes payed off well.
Database query times average at 25ms and the system has shown a maximum in throughput of 10,000 expression QTL entries written per second on a single machine. Evaluating the performance on three machines showed that scalability was achieved and is simple to set up. When the database system is using the same configuration file on every machine, different worker nodes will automatically find each other and relocate the distributed data accordingly. Fault tolerance was evaluated by randomly disconnecting one of the three machines from the network. While the processing time for background tasks did go up when disconnecting an active worker node, the presentation front-end still responded as fast as before.
It was also verified that the data set stays complete and consistent as long as not more than half of the worker nodes are disconnected A B C
D E at the same time. So from a scalability and reliability point of view, this novel approach is superior to any conventional system relying on a single centralized database. 3.3
Above described technologies where applied for interactively
presenting high-throughput data in statistical genetics. A web
application provides a front-end to expression QTL data in a genomic
context provided by DAS
On the lower half of the screen, one can see the chromosome browser with DAS tracks and annotations for the provided research data. One feature of the novel approach that was received especially well was the ability to change the viewing area in the chromosome browser without reloading the page. The user can click on and drag the chromosome to scroll. While the user is moving the display area using his mouse, the web application downloads the needed data in parallel, so it can update the view while the user is still scrolling. The table view below is also dynamically updated to always show the most relevant data rows, in this case the 25 most probable expression QTL in the specific area of the chromosome, which is currently visible in the chromosome browser. The chromosomal map view shown in figure 2 was found to work great for getting a quick overview of which gene is interacting with which loci and later on investigating those locations. To ease the process of following up on these locations, transcripts and known genes retrieved from Ensembl DAS tracks are displayed alongside the expression QTL data when the user zooms in. The columns used for positioning along the X and Y axis can be freely chosen to accommodate different flavors of two-dimensional data. The web application approach is highly reactive in comparison to normal web pages. That is presumably because most calculations are done when data are written. Hence, the data are stored already prepared, sorted and preformatted, which makes their presentation cheap in terms of network and CPU usage.
By working with a synchronized local copy of the database, read latency is as low as for traditional desktop applications. Each separate user benefits from this increased interactivity, thereby accelerating overall team communication. 3.4
3.4.1.
With exceptions for mobile computing, a local copy is always faster to access than relying on a remote service. However, users are not willing to wait hours or even days for a slow initial download of the whole data set. Therefore, our software system replicates commonly-accessed data automatically while falling back to remote access while replication or synchronization is in progress. This allows new user to immediately start working and by dynamically creating a local replica of the data set, we ensure that no row needs to be sent twice.
A research collaboration includes a number of computers distributed over several networks. It is safe to assume that on such a scale, at least one machine or network connection will fail. When scaling to work with huge data sets, even more computational power is needed and with more machines, component failures will get more and more often. In fact, we planned for and accepted them as part of the normal operation of a distributed software system. For the system to stay operational and interactive under such conditions, the data need to be replicated. One simply cannot afford to lose data. Also, no worker node in the system should pose a single point of failure. Therefore, all nodes are running the same software and communicate with each other as equal peers. This approach is a stark contrast to the commonly used pattern of master-slave database replication. In cloud environments, this homogeneous configuration enables us to provide demand driven load balancing. 3.4.2.
Not duplicating read-only data has been common sense in operating system design for decades, however with the growing amount of data stored in research databases, it is becoming increasingly important for maintaining a high read throughput. While creating a newly aggregated data presentation should preferably be fast, it is a rare event when compared to inspecting the data through already existing presentation views.
A research database should therefore rather create a modified and filtered copy of the data rather than using complex WHERE clauses and JOINs. If JOINs are unavoidable, most database systems provide a VIEW capability which gives the developer a warm fuzzy feeling of having thought ahead. Sadly, most VIEWs are not materialized by default. Therefore, performance of a standard database VIEW is as bad as calling the underlying JOINs and WHERE clauses on every access to any row of the VIEW. This might seem trivial to state, but if one knows beforehand, that a certain VIEW will only be modified sparingly, that VIEW should be manually materialized. (CREATE TABLE … SELECT …) This trades a one-time creation overhead in return for significantly increased read throughput on following queries.
Using these two optimization techniques, the computational work can be moved from the presentation towards storage of the data, which allows for parallelisation and distribution on a compute grid. Keeping data stored the way it’s supposed to be presented also ensures that all researches in the collaboration, independent of their available computing power, can immediately access and work with all presentations of the data. 3.4.3.
Research data sets may easily contain thousands of rows. While enriching, annotating or filtering the data set, these rows can be processed independently. By distributing one-time computational tasks, such as the creation of a new presentation of the data, to all machines in the collaboration, everyone can see the result data set faster.
In our case, using a distributed database gives every worker node low-latency access on the whole data set and so one can actually do workload distribution on a row scale. If you’re forced to stick with a conventional relational database system, work distribution Relevancy scores are stored alongside in the index table, to allow for dynamic merging of blocks. If the user requests the range 0-500 Resolution1: Resolution2: Resolution3: of low-level tasks might even be counter-productive or should at least be done with reasonably-sized blocks of data.
Tables are the dominant form of presentation for research data. In Bioinformatics, gene locations and interactions play an important role and, therefore, different flavors of genome scales, chromosome browsers and interaction maps have been invented. Especially graphical presentations provide the researcher with a quick overview of his data. The possibility to zoom in and move around in a map of his data closes the gap between an overview of the complete data and a detailed close-up of specific features. Interactive bars and maps constantly require new data to be shown, as the user is moving around and inspecting different aspects of the experiment. This puts an enormous strain on the back-end database, since it means a constant flow of search queries for aggregating the data to show and distance-comparisons are usually O(N2). It is also important, that only the most relevant information for a given zoom level is displayed, to make the resulting graphic not only sufficient, but also succinct.
While updating the data or creating a new presentation view, the location where each item will be visible on the bar or map is usually known beforehand. Similarly, when applying our suggestion of copying the data on write time, the relevance of every item for every zoom level can also be calculated offline.
We therefore propose to divide the possible view area into a hierarchical set of equally-sized blocks, as seen in figure 4. Assuming that display position and relevancy score have already been calculated, the data should look akin to table 1.
We now compare these positions to figure 3. On zoom resolution 1, all items are in the first block. The same applies to zoom resolution 2. On resolution 3, the item with ID=1 is in the second block, while the other two items are in the first block. Now we can create a look-up table for every block at every resolution, which is easy given the schema-less nature of our chosen distributed key-value store. Example look-up tables are shown in table 2. Please note that the rows of each table have been sorted by their relevancy scores. shall compare other contemporary eQTL infrastructures with what they could achieve, when they adopted those technologies. 4.1
A related project is the eXtensible Genotype And Phenotype
platform (XGAP)
Another related project is Gene Network
Obvious shortcomings of the Gene Network are that data transfer is not being encrypted using the industry standard HTTPS and that there exists no version which the researchers could deploy on-site inside their own firewall. Apart from security issues, WebQTL allows for a very convenient analysis of small data sets. It provides a plentiful selection of visualization methods, such as box plots, correlation diagrams and even directed graphs.
From a technical point of view, the Gene Network is considered to be inferior to both the novel approach presented as well as XGAP. Data set presentation in Gene Network is implemented as downloading ready-made images from their web servers. This leaves the user with no further possibility for interaction than changing parameters and waiting for the next image to be downloaded. Since the researcher has no possibility of running Gene Network on own computational resources, the web servers provided by Gene Network are essentially shared by all users.
In its choice of presentation methods, Gene Network is very similar to the R language for scientific computing. Due to Gene Networks focus on reasonably small data sets, using such a scripting language as computational back-end seems a wise choice. Gene Network is there wholeheartedly recommended for analytical and investigative work on classical QTL. 4.3
By going new ways in terms of data storage, we combined the low latency of local data storage with the benefits and integrity of a centralized storage server. This technological design decision allowed us to greatly increase interactivity of our data presentation, without forcing the user to download the complete data set beforehand. While during development, there was a strong focus on expression QTL, that is their positioning on the chromosome and their associated genes, the system was designed to be plug-able for a multitude of data processors and data visualisation applications. The chromosome browser allows for arbitrary chromosomes to be displayed along with arbitrary annotation information, as long as the DAS file format is being used. Similarly, the map view allows for arbitrary positioning measures to be used on the X and Y axis, as long as there is a data processor available to calculate said positions. While theoretically any user could develop such data processors using a simple Java API, it might be beneficial to broaden our showcase of example processors to support additional forms of high-throughput data.
Data replication allows a whole team a consistent shared view of their experiment. A new presentation created by one collaborator is immediately available to the entire team. Driven by the high interactivity between every user and the web application, overall team communication is speeding up, and there is a general demand for a deeper integration of social aspects into the data presentation. We envision a future version of our system where researchers can discuss current and past measurements in real-time using a special comment and annotation system adapted to work directly on the data presentation.
With computing nodes gradually getting cheaper and more readily available, dynamic grid brokering will replace static worker queues and present us with unprecedented peak amounts of compute power. While grid technologies traditionally suffer from their own transiency, the distributed and homogeneous nature of our proposed system can easily compensate for node failures, while still retaining near-perfect performance.
The authors thank Thomas Martinetz and Saleh Ibrahim for comments and a nice working atmosphere. Lydia Lutter is thanked for her critical reading of the manuscript. Jay Shendure, Gregory J. Porreca, Nikos B. Reppas, Xiaoxia Lin, John P. McCutchJonathan Brown.