-The lessons learned during six years of experience in design, development, and operation of four Science Gateway (SG) generations motivated us to develop yet another generation of platforms coined “Rosemary”. At the core of Rosemary the three fundamental SG functions, namely related to data, computing, and collaboration management, are integrated together. Our earlier studies showed that complete integration between these functions is a feature that is usually overlooked in the existing SG platforms. Rosemary provides a generic data model, RESTful API, and responsive UI that can be customized through programming to build customized SGs. Moreover, Rosemary is designed and implemented to be flexible to changes in e-Infrastructures and user community requirements. The software frameworks, tools and libraries employed in the realization of Rosemary streamline the development, deployment and operation of customized SGs for the users needs. So far the platform has been used to implement prototypes of three SGs for highthroughput analysis and management of neuroimaging data, sharing of data in in-vitro fertilization research, and provenance tracking of DNA sequencing data. This paper presents the design considerations, data model, and system architecture of Rosemary and highlights some of the features that are intrinsic to its design and implementation with examples from the three prototypes.
Between 2010
The third generation was based on a custom platform built on the Spring framework and the MOTEUR workflow management system [6]. It focused on providing computing power to biomedical scientists for neuroimaging, genomics and proteomics data analysis [7]. The fourth generation enriched SGs utility by integrating data management, in addition to facilitating access to the Dutch grid infrastructure. This generation was based on the WS-PGRADE/gUSE SG framework [8], and used to build customized SGs for neuroimaging data analysis [9] and molecular docking simulations [10]. Both the third and
I. INTRODUCTION fourth generations were deployed and used daily in biomedical Science Gateways (SGs) are web-based enterprise informa- research projects, and helped researchers to handle large tion systems that provide scientists with customized and easy computations, for example refer to [11]. access to community-specific data collections, computational These SG generations were developed and studied in tools, and collaborative services on e-Infrastructures [1], [2]. partnership with the computational neuroscience, bioinforThe construction of SGs is challenging because they include matics, and medical chemistry research communities from a large number of heterogeneous, distributed, and evolving AMC and Amsterdam region. The lessons learned in this software components and services. Moreover, there is a plethora process motivated the development of yet another generation of evolving and alternative technologies and platforms to choose of SG coined Rosemary, which are described in the following from, as well illustrated by the rich information contained in sections. the EGI Science Gateway Primer [3] and the XSEDE Science Gateway cookbook [4]. III. REQUIREMENTS AND SYSTEM DESIGN
In this paper we present yet another SG framework, Rose- To better introduce Rosemary SG platform, the user commary, that was designed in response to lessons learned during 6 munity requirements and the functions to address them are years of SG development and operation at the Academic Med- explained in Section III-A. Additionally, the considerations ical Center (AMC) of the University of Amsterdam. Rosemary during the design phase are explained in Section III-B to addresses the need of flexible and light platform to address support its design, implementation, and technology choices. modern user interface requirements while integrating data, processing and collaboration functions as first-class concepts. A. Three Pillars of Requirements and Functions The paper details the motivation and presents the requirements During the life cycle of a research project, several researchers and system design of Rosemary in Sections II and III. The collaborate with a wide spectrum of complementary background and expertise, such as domain-specific science, statistics, data functions related to these three functional requirement groups processing, and information technology. These collaborators are shown by columns that are rooted on data, computation and take various roles along the project life cycle, such as data collaboration. The rows represent the e-Infrastructure resources collector, data analyst, support, and principal investigator. and the integration and enrichment layer implemented by the Moreover, researchers perform several tasks in different phases SG. Integration functions cross-cut and manage the complex of the research life cycle based on their roles. Finally, on top relations and interplay among data, computing, and community of all that, some characteristics may vary in each discipline, management functions. These functions for example capture of depending on local culture and research project setup. All these data and process provenance, coordination of computation and characteristics translate to an overwhelming and complex set data-related tasks, security, monitoring, notification, discussion of requirements for a science gateway that can efficiently and among users, analytics of domain data and SG activity, and effectively support research activities. billing management. The detailed explanation of these functions
In spite of all this diversity, in our research we identified and a comparison of some of the well-known SGs based on three foundational groups of functional requirements at the core their provided functions can be found in [12]. of any SG, namely related to data, computing, and collaboration. It is also worth noting that SG design should be flexThese are extensively described in [12] and summarized below: ible enough to accommodate potential changes in the eInfrastructure resources and in the three functional requirement groups due to expected evolution of research practices.
plex, distributed, and heterogeneous datasets on eInfrastructures. Examples of data-related requirements include storing, annotating, searching, retrieving, replicating, and archiving datasets, as well as capturing metadata and provenance.
Computing-related requirements concern management of complex, computationally demanding, distributed and coordinated data processing on e-Infrastructures. Examples of such requirements include negotiating resources, instantiating and setting up computer programs from application repositories, scheduling as well as managing the execution order and data flow between computer programs and handling failures.
Collaboration-related requirements concern management of communications and interactions among scientists involved in a research project. Such collaborations entail, among others, definition of team membership; sharing or reusing data, computer programs, and resources; and Science Gateway [Cyber/eInfrastructure] Resources
Data
Computation
Collaboration cFun Iteng $ Provenance ito tra sn ion
' Coordination | Analytics Data Management & Monitoring + Billing # Computing
Management % Security ( Notification * Discussion "Community
Management
Data Infrastructure
Computing Infrastructure
Community
the complex set of requirements of the research communities in the context of SGs when organizing them explicitly around data, computing, and collaboration requirements. For example, we started in the first generation motivated by the need for highthroughput computing. It was only when we deployed the third SG generation, which addressed the most prominent computingrelated requirements, that the scientists and ourselves understood that there was a second dimension to consider, namely the data-related requirements. Likewise, when the data-related requirements were better addressed in the fourth generation, the need for more sophisticated collaboration mechanisms became evident, motivating for a fifth SG generation.
Furthermore, we found that all of the three functional requirement groups should be considered in the design. The SG should integrate data, computing, and collaboration resources seamlessly. Failing to integrate any of these resources will result in a SG that is not effective and will find limited use. A highlevel illustration of SGs is depicted in Figure 1, where the SG The design of Rosemary considers all the three functional requirement groups. It provides a metadata-rich abstract integration layer in which data, computing, and collaboration management functions are integrated together at its core. The rich metadata and integration level provided by Rosemary foster its capabilities to supply data provenance, increase automation, and unlock advanced features such as full search, user guidance, and data mash-up. Moreover, its implementation aims to ensure flexibility to accommodate changing requirements and e-Infrastructures.
Gateways: a) to customize existing SGs or b) to develop from scratch, for example based on web application frameworks or Content Management Systems (CMS) [3], [13]. Customizing the existing SGs enables utilization of already provided SG functions such as integration and management of computing resources, which reduces development time. However, this approach also confines the developers to a certain software Although we opted for developing Rosemary SG framework stack and SG functions, which implies understanding the from scratch, we didn’t (still don’t) intend to repeat ourselves architecture and details of the existing SG before being able to and others [15]. It was our goal to maximize reusing code customize it or add new functions. For example, take the and software components developed for the previous genercase of the Distributed Research Infrastructure for Hydro- ation of our SGs and by others. Therefore we used a SOA Meteorology project (DRIHM) science gateway, where they through microservices and wrapped and packaged (existing) report that building a SG based on a sophisticated modular software components into independent services. More details and extensible SG framework was not smooth because of are presented in Section V. the amount of necessary modifications to address the user In our previous SG generations a big portion of development requirements, being confined in a certain software stack, and time was spent on integration tests. Moreover, operation lack of integration between main entities [14]. In contrast to this of those SGs turned out to be challenging because of the approach, developing SGs from scratch gives the developers complexity of the system. We aimed at mitigating these full control over the software stack and SG functions, but challenges by utilizing modern web application frameworks it increases the development time. However, even then it is (Play Framework for the back-end and AngularJS for the frontpossible to integrate with existing software components, for end) and compartmentalizing system components (using Docker example from other SG frameworks, when these are decoupled and Ansible). Rapid development is facilitated by utilization of and provide a clear abstraction layer. It is therefore important high-level programming languages (Scala, CoffeeScript) and to choose the approach that suits best the requirements based the availability and richness of tools and libraries (Bootstrap, on available resources and expertise. SBT, Gulp, Bower, Apache Lucene, Akka). These modern
Our third SG generation was developed from scratch based software platforms and libraries also enabled us to provide a on the Spring framework [7], while in the fourth generation Rich Internet Application (RIA) and responsive UI that can we customized the WS-PGRADE/gUSE SG framework [8] be used from various devices with various dimensions. More to develop our specific SGs [9], [10]. In the fifth generation, details are presented in Section V.
Rosemary, we opted for developing it from scratch based on the Play framework to be able to have full control over the IV. BASIC CONCEPTS: DATA MODEL implementation of the new SG data model (see Section IV) The basic concepts of the data model are explained through and its software stack. Note that we only implemented an the three pillars (see Section III-A) and the integration layer “integration layer” from scratch that integrates data, computing, between them. The simplified Entity Relationship (ER) diagram and collaboration functions seamlessly. Base on our previous is depicted in Figure 2, where the most relevant attributes that analysis in [9], this level of integration is not offered by existing are explained in this paper are highlighted. Note that all entities SG frameworks, as explained in Section VII. have a set of <key, value, unit> attribute attached to them to
In summary, the following items were the key features store metadata. considered in the design and implementation of Rosemary:
Fully integrate the three pillars of SG and provide func- A. Data Management tions for data, computing, and collaboration management. Datum entity holds the domain data and metadata. Datum Maximize utilization of existing services and software objects can form a hierarchical structure by declaring other components. objects as their children. One Datum object can be declared as Maximize flexibility for accommodating changes in e-In- child of multiple Datum objects. This relationship is used to frastructures and user community requirements. capture the notion of association and derivatives, for example, Minimize changes when customizing the SG for various multiple samples that belong to a subject or a 3D model that disciplines. is reconstructed from multiple brain scans. Streamline software development, deployment, and operation. B. Computing Management
Provide an easy-to-use and ubiquitous user interface (UI). Recipe entity captures what can be done with data and is These features have various consequences on the design meant to store data processing functions. Examples of Recipe and implementation of Rosemary. The most prominent conse- objects are program (or workflow) descriptions or simply a quences are the effect on the data model and its implementation, document that describes how one sort of Datum can be derived utilization of a Service Oriented Architecture (SOA) and from another one. Recipe objects may also have a set of input microservices, and use of modern software platforms and tools. abstract ports that describe the inputs expected from users. A
The new data model was essential to take into account all Recipe object may have a program description attached to it, the three functional requirements and to integrate them all into and in that case, it also includes a) the abstract input and output the SG. When designing the new data model we also aimed ports as expected and generated by that program; and b) a at maximizing flexibility and generality so that it can cope transformer that specifies how user inputs could be validated with the changes and can be customized to address various and transformed into program inputs, and how program outputs requirements of different research disciplines. More details are should be interpreted and transformed by the system to Datum presented in Section IV. objects.
on data. A Processing object has one or multiple Datum as inputs and outputs, a set of status, and one or multiple Recipes to specify which data processing functions have been used. Examples of Processing objects are program (or workflow) executions or a user action such as manual file edit. Processing objects can form a hierarchical structure to group them together and capture multiple levels of abstraction. For example, the Datum selected by the user in the SG as the input of one Processing can be (automatically) transformed based on the domain information into multiple files that will be used in multiple processings organized into a group.
email address, and preferences.
Thread entity stores an array of messages (see Tags in Section IV-D) to capture the communication between users around a specific topic involving SG activity. For example, users can discuss the outputs of a certain data processing or receive a domain or system expert opinion for troubleshooting a data processing.
Notification entity holds the information about a system event or a user action, for example, when a Processing has been completed, or a message has arrived on a Thread.
entities, Tag and Resource, and the relationships between these entities and the rest of the entities.
Tag entity is located at the heart of the data model and is one of the key factors that make the data model flexible and customizable. Tags are used to annotate objects to create groups and deliver functions such as filtering, access control, and communication. Access to Tags is controlled by defining their owner and members. Examples of the usage of Tags include:
other. The access to Datum, Recipe, Processing, Resource, Thread, and Notification objects is controlled through the access control of the workspace tags. This means that when an object is annotated with a workspace Tag it would become accessible to anybody that has access to that Tag. Workspace-specific information, such as community credentials for resources, can also be stored in these Tags. Annotate Datum or Processing objects with a category name that can be used for filtering during search. Tag labels are used to customize the SGs for specific domains. Group together or filter some Datum or Processing objects based on user-defined Tags.
Annotate Datum and Processing objects with messages and form a discussion around them. A message in a communication Thread is indeed a Tag.
resources or computing platforms. Datum entities are originated from, and have replicas on, data Resources. Recipes can also have replicas on data Resources, i.e., to specify the location of program binaries or Recipe descriptions. Additionally, Recipes can be eligible to be executed on a certain computing platform. Regarding credentials to access Resources, users could also have their own personal credential in addition to community credentials.
The front-end uses the back-end RESTful API to provide data, computing, and collaboration management functions to users. The UI is made responsive to the screen dimension through the Twitter Bootstrap1 front-end framework. The frontend has been implemented based on the Google AngularJS2 web application framework to deliver a single-page RIA. The front-end controllers are implemented using the CoffeeScript3 programming language that enhances JavaScript’s readability and conciseness. All these frameworks and high-level programming languages improved the development and test cycles and therefore enhanced front-end stability.
The back-end provides a secure RESTful API to integrate, enrich, and manage data, computation and collaboration functions provided by the e-Infrastructure resources. It is implemented based on the Play web application framework4 and coded in the Scala5 functional and object-oriented programming language. The SG information is stored in MongoDB6, a document-oriented database that offers the required flexibility and scalability. The information about Datum and Processing objects, including their metadata and annotating Tags, is indexed and searched upon using Apache Lucene7 search engine. The back-end also includes a Data Source Manager (DSM) that provides an abstraction layer over management 1http://getbootstrap.com/ 2https://www.angularjs.org/ 3http://coffeescript.org/ 4https://www.playframework.com/ 5http://www.scala-lang.org/ 6https://www.mongodb.com/ 7https://lucene.apache.org/ of data and metadata that is stored on various data sources such as iRODS8, XNAT9, and WebDAV10. The DSM utilizes the API of the e-Infrastructure data sources to provide the functionality to import data and metadata into the SG. Similarly, it provides export and replication functions to those data sources. Another important component of the back-end is the Processing Manager (PM) [16], which provides a RESTful API and an abstraction layer over the computation management functions on computing platforms. The PM potentially handles simple jobs and workflows depending on the underlying middleware or service used through its plug-ins. Currently, the PM uses Distributed Infrastructure with Remote Agent Control (DIRAC [17]) Interware to interface with various computing platforms such as Grid, Cloud, and Cluster. DIARC utilizes the API of the e-Infrastructure computing platforms to run, manage, and monitor computations on them. The PM also stages input and output data and application to/from the computing platforms using a simple data plug-in. Compared to the version described in [16], the PM included in Rosemary has been reimplemented in Scala based on the Akka toolkit11 and RabbitMQ12 message broker to enhance its scalability and transparency for troubleshooting.
The development of Rosemary is streamlined by the ecosystem of tools including SBT13, Gulp14, and Bower15 to manage library dependencies and automate system builds. The Play framework in particular makes it easy to hot redeploy the SG and test its functionality during development. Deployment is streamlined by encapsulating various parts of the system, i.e., MongoDB, Processing Manager, Back-end and Front-end, in separate Docker16 containers and configuring those containers using Ansible17 configuration management platform.
include:
Create and manage workspaces in which users collaborate in data analysis tasks. Workspaces bring a collection of data, recipes (applications), processings, communication threads, notifications, users, and community resources and their credentials together.
Authenticate to various external data and computing resources using the available community or user credentials. Import data and metadata from external data sources into a workspace so that it can be managed or processed through the SG.
Configure available applications and submit and manage data processings to computing platforms. Necessary steps 8https://irods.org/ 9http://www.xnat.org/ 10https://en.wikipedia.org/wiki/WebDAV 11http://akka.io/ 12https://www.rabbitmq.com/ 13http://www.scala-sbt.org/ 14http://gulpjs.com/ 15http://bower.io/ 16http://www.docker.com/ 17https://www.ansible.com/ for data processing, such as application and data staging and cleanup, are performed automatically by the SG. Search and filter data and processings based on metadata and annotated tags such as category or user-defined tags. Communicate with other users with reference to data and processings.
Receive notifications of system events and user actions. Investigate provenance information to establish trust and troubleshoot problems.
Export data and metadata to an external data source, for example for data preservation or archival.
implemented based on the Rosemary platform:
site neuroscience data management, computing management on Grid computing resources, and collaboration management among scientists. See example of UI on Figure 4.
The In Vitro Fertilization (IVF) gateway provides complex multi-site IVF data sharing management, specifically to follow a workflow that involves authorization from multiple centers for each requested dataset.
The Genomics gateway provides wet-lab and dry-lab sequencing data management with detailed provenance tracking, as well as collaboration management among scientists.
has been presented in our recent work [9]. From an analysis
of liter
Here we build upon this previous work to compare Rosemary to existing systems. Only the information available in the scientific publications was considered, therefore some degree of inaccuracy is possible in the summary presented below.
Delivery functions for humans and programs are implemented in Rosemary as the responsive front-end and the RESTful API provided by the back-end. These are also found in all of the frameworks, with the exception of Apache Airavata and NEWT, which do not implement user interfaces.
Coordination functions are also present in all of the frameworks, with the exception of ICAT and NEWT. Some frameworks integrate advanced workflow management systems such as Pegasus [27] and WS-PGRADE/gUSE [8], while others are based on pilot job abstractions (e.g., DARE and inSilicoLab). In Rosemary coordination functions are realized through the Processing Manager, which includes data and DIRAC plug-ins for data staging, program instantiation, pilot job abstractions, and simple workflows.
Security functions are implemented by all of the frameworks at various levels of sophistication. For example, the Catania SG and iPlant use Shibboleth18 for single sign-on, whereas DARE uses standard security mechanisms from SAGA19. In Rosemary security is still primitive at the moment, being implemented through internal credential management components and tokenbased authentication.
Monitoring functions are also present in all of the frameworks to some extent. The Catania Portal and NEWT platform, for example, provide system, resource and job monitoring and accounting functions. Rosemary implements monitoring functions not only for computation, but also for data transfers and collaboration, which are all delivered to the users through a notification system.
Provenance functions are present in only a few frameworks. The best example is Pegasus, which keeps track of executed software, data locations and execution parameters. In Rosemary special attention has been given to provenance functions. PM in particular captures and stores the provenance information about data processing in the PROV standard20. Additional information about the sources of data and their replicas is captured in the Rosemary Data Model.
Functions for Data and Computing management are found in all the studied frameworks, with the exception of Globus Online (no computation) and DARE (no data). Typically the frameworks exploit existing middleware and standardized abstractions, such as SAGA (e.g., Catania SG, DARE) and BES (e.g., WS-PGRADE/gUSE) for data and computing management. In Rosemary, the Processing Manager and the Data Source Manager play an important role as abstraction layer to the resources, where plug-ins can be developed to interface with nearly any required resource. It however does not adopt any standard.
Finally, Community management functions are present in most frameworks, normally in simple form such as group and role management. We believe that this is the aspect that most differentiates Rosemary from other frameworks, since it has integrated the community management functions with the data and processing management functions at its core, which is demonstrated by features such as workspaces and communication threads.
Rosemary is a SG framework that needs to be customized for a particular user community by programming and possibly by adding new functions to it. Therefore it requires understanding different parts of the system, as well as being able to 18http://shibboleth.net/ 19http://saga-project.org 20https://www.w3.org/TR/prov-overview/ program in Scala or Java. Our experience with the three SG implementations based on Rosemary showed that the generic data model and platform implementation, the highlevel programming languages, and the employed software development ecosystem facilitate adding or customizing functions quickly. The Neuroscience gateway was the initial usecase by which Rosemary platform was implemented. Based on that, the IVF gateway was implemented in one month by a programmer that was not familiar with the platform. Finally, the Genomics gateway was implemented in one month by two programmers that were familiar with the platform. However, with this approach it is also necessary to re-factor the generic parts of new functions and make them available for existing and future gateway instances. To streamline such a code re-factoring process it seems necessary to implement more SGs and reevaluate and re-organize the code based on that experience.
The three SG prototypes have attracted significant attention so far, mainly because of their flexibility, lightness, and being able to customize them quickly for various disciplines. In addition, based on the initial feedback, its responsive UI, richness of metadata management, and search functionality are the most attractive Rosemary features. Although the initial feedback and enthusiasm from users are promising, usage by a larger group is required to understand shortcomings of the system and further improve it.
and metadata-rich data model and a software framework that can be customized by programming for rapid implementation of new SGs that fulfill user requirements and provide rich user experience. It also utilizes high-level programming languages, modern software framework and libraries, and a software ecosystem that supports continuous testing and rapid deployment, all of which streamline software development, deployment, and operation. Rosemary framework will be published as an open source project to increase its uptake and invite other developers to improve it by their contributions.
Mahdi Jaghoori, Juan Luis Font-Calvo, Sara Ramezani, Jalmar Teeuw, and Matthan Caan for their various contributions to the Rosemary platform. We are grateful to our collaborators at various departments of the AMC (KEBB; Radiology; Immunology; Medical Chemistry; ADICT) and the University of Amsterdam (ICTS and FNWI), as well as and our former colleagues at the AMC e-Science group for various contributions, inspiration and support. We also thank the anonymous reviewers for their feedback and suggestions. This work was carried out on the Dutch national e-Infrastructure with the support of SURF Cooperative. This publication was supported by the Dutch national program COMMIT/ and the High Performance Computing and Networking (HPCN) Fund of the University of Amsterdam.