The growth of the Life Science Semantic Web is illustrated by the increasing number of resources available in the Linked Open Data Cloud. Our SWIT tool supports the generation of semantic repositories, and it has been successfully applied in the eld of orthology resources, helping to achieve objectives of the Quest for Orthologs consortium. However, our experience with SWIT reveals that the time required for the generation of datasets is longer than desired. In this work we present the application of High Performance Computing techniques, mainly memory optimization and parallelization, to speed up SWIT.
The application of SWIT to the transformation of large datasets has revealed
that despite the computational complexity increases linearly with the number
of entities to be transformed, it is slower than expected, and we have identi ed
some limitations to the performance: (
2. We have optimized the way SWIT manages the ontology individuals during the transformation process. We use now two hash maps composed of pointers to individuals and not copies, so keeping the coherence and reducing memory consumption. One map grants that no failure happens when searching for an individual, if this exists, while the other map could miss in some searches since it acts as a greedy algorithm. The speed up of the execution of searches with the two maps is up to 2x.
3. The optimization of memory management also a ects the process of identifying equivalent individuals using identity rules. Identity rules are de ned using AND and OR conditions, and the new method uses one hash map of vectors for each type, where pointers to hashed individuals are stored. The hashing is carried out di erently depending on the map. For AND conditions, a hash is performed along all the properties of the entity. Contrariwise, for OR conditions, the entity properties are hashed separately, having several pointers to the same individual along the hash map.
4. The SWIT parallelization is done by using gnu parallel tool4. The parallel design consists in setting one input le and one SWIT instance per core, so the parallelization only works when multiple les are established. When transforming a single large le, we need to split it in several smaller les to enable parallelization. It provides the largest speed-up. 2
Results Our tests show a speed-up of 1000x, 4100x and 7800x in the InParanoid [2] datasets E.coli, H.arabidopsidis and H.sapiens respectively. The executions were tested on a high performance server that provides 2 chips of Intel R Xeon R E52698 v4 with 20 cores each (2 hyper-threading), making a total of 40 physical cores or 80 virtual cores, running at 2,2 GHz and 128 GB RAM DDR4.