The UniProt Knowledgebase offers both manually curated and automatically generated information on proteins, and is one of the leading biological databases. While it is one of the largest free data sets that is available in RDF, our infrastructure and website are not based on RDF. We present numbers about the volume and growth of UniProt and show why this volume of data prevents using RDF triple stores and SPARQL with currently available tools.
UniProt is released every three weeks. In each such period, there is a time window of only six days to prepare all source data for publication. When the curators freeze their work, we need to retrieve all related data from our global partners. We then convert all source data (42 GB as of release 15.8) to RDF/XML
Total 8,612,297 36,461 195,249
7,641 84,308,128 4,537 8,661 3,493,485
6551 640,191,073 1,572,097,256
487,209,989 2,796,164,777 8,616,640 36,461 195,249
7,649 82,605,680 4,532 8,673 3,520,871
6551 691,432,967 1,610,723,778
499,947,698 2,897,100,198 a Automatically generated, not public. (180 GB, 18 GB compressed), while validating it. This conversion took around 16 hours for release 15.8 of September 2009.
The UniProt.org [
UniProt.org runs on three mirrors, each of which needs to be provisioned with all data in time for a release. Data size is a major factor in our deployment as we are limited by upload speed. UniProt 15.8 as needed for the website consumes 39 GB when gzip compressed, and takes four hours to upload. Larger data sizes increase the risk of transfer failure.
We also have an internal website for our curators and internal software tools. Because it needs to reflect current work, we rebuild the manually curated UniProtKB/Swiss-Prot every night with a time window of 3.5 hours for converting, loading and validating about 160 million triples.
UniProtKB/TrEMBL with supporting data is converted, loaded, and validated every Sunday. At this occasion we generate about 1.4 billion triples while checking the data against our validation rules, making us indeed weekend triple
billionaires. Current generic RDF stores are unable to handle this amount of data on the limited hardware budget available. a www.franz.com/agraph/allegrograph/ b www.ontotext.com/owlim/OWLIMPres.pdf c www.oracle.com/technology/tech/semantic technologies/htdocs/performance.html d Storage cost of Virtuoso was not multiplied as they used a dataset with a higher number of triples with large literals in comparison to LUBM8000. e virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#StorageCostPerTriple f Our current custom solution, described in Section 2, not a SPARQL engine 3
We have to deal with ever growing data volumes (Fig. 3). When evaluating tools, we need to take into account not just performance on today’s data, but also on the expected data in five years. The yearly growth rate of the core UniProtKB data is currently 51%. This is a doubling time of 17 months, faster than Moore’s Law5 that predicts a doubling time of 24 months.
Assuming that the current growth rate of UniParc does not accelerate from the current 270% a year (doubling every 7 months), we will see at least 10 billion sequences in UniParc in five years. This estimate translates to 3 × 1011 triples, around 470 times the current size. 4
We have met the requirements for data processing speed and query performance for the near future, unfortunately on a non-RDF technology stack. Currently the available tools do not meet our performance needs for UniProt.org, especially since any RDF solution must preserve the current user interface and full text search. We do however aim to deploy a public SPARQL endpoint once it becomes feasible.
. 5
UniProt is mainly supported by the National Institutes of Health (NIH) grant 2 U01 HG02712-04. Additional support for the EBI’s involvement in UniProt comes from the European Commission (EC)’s FELICS grant (021902RII3) and from the NIH grant 1R01HGO2273-01. Swiss-Prot activities at the SIB are supported by the Swiss Federal Government through the Federal Office of Education and Science and the European Commission contracts FELICS (021902RII3) and SLING (226073). PIR activities are also supported by the NIH grants and contracts HHSN266200400061C, NCI-caBIG, and 5R01GM080646-04, and the Department of Defense grant W81XWH0720112. This work has benefited from Oracle funding support.