Motivation: We our efforts to work with HUBzero in a significant collaboration-oriented project, as well as our impressions of HUBzero after nearly a year of interaction with the platform. Our assessment is a mixed one: while the HUBzero platform does a good job of bringing users and modeling tools together, we found some noteworthy limitations of this recently open-sourced technology. For example: hubs are generally configured to deploy simulation tools of modest means with input and output data not readily shared across user accounts; by default, the security configuration of a HUBzero server is very open; hubs are inflexible in terms of their deployment requirements, and there is a need for more detailed documentation. In answer to the first (and most serious) of these issues, we have extended HUBzero to enable the use of more sophisticated modeling tools, such as openModeller Desktop, and to provide seamless access to external Network File System drive space.
The open source HUBzero platform offers an effective way for
scientific and educational communities to share information,
interact, and run simulations
We are currently employing HUBzero as part of an effort to
build a so-called collaboratory: a virtual space in which users may
share information and innovate. The goal of our work is to support
a multidisciplinary community of scientists with a focus on
adaptation strategies for biological systems. In addition to
publishing and discussing content related to adaptation, users of
the HUBzero portal (or “hub”) need to leverage relevant
simulation tools, such as openModeller and openModeller Desktop
Such tools have requirements that may not be satisfied by a
typical hub deployment. For example, there may be a need for
software libraries that do not normally exist in HUBzero’s
virtualized Linux environment
Tangentially related to these requirements is the need to restrict access to a hub’s OpenVZ-based Workspace tool. Through Workspace a user is given a private, virtual Linux workstation that may be employed for nearly any purpose. Generally, this is *To whom correspondence should be addressed. intended to offer an environment for users to create and work with simulation tools that are made available in a hub. However, there is potential for accidental and malicious abuse of such a powerful virtual environment; for example, by gratuitous resource use or hacking.
To provide hub access to openModeller and accommodate the
aforementioned needs, we have extended HUBzero in three ways.
First, we have significantly expanded the OpenVZ environment to
support the execution of a tool as complex as openModeller. Next,
we have integrated NFS (Network File System)
To permit long-running, computationally intense openModeller simulations, we have embarked on a separate project to kick off and manage such jobs outside of the HUBzero environment. The results of these simulations can be made available to our hub through NFS.
The remainder of this paper is organized as follows. Section 2 presents the state-of-the art in collaborative, virtual organization tools. Section 3 discusses the Collaboratory Project and simulations tools that we integrate with the HUBzero infrastructure. In section 4 we present the HUBzero architecture. Section 5 reviews the OpenVZ configuration and what we have done to enhance this configuration. Section 6 addresses our integration of NFS with HUBzero, including our use of public and private directories for each Workspace/openModeller user. Section 7 describes our changes to HUBzero to restrict access to the Workspace and openModeller tools. Section 8 briefly looks at a separate project to handle more intense openModeller jobs outside of our hub. Section 9 outlines our overall impressions of working with HUBzero. Finally, in Section 10 we offer some concluding remarks about this project. 2
HUBzero offers many features that researchers need for effective collaboration. Major components include: various interactive simulation tools that can be accessed through almost any Web browser, a repository for online course materials and other publications, mechanisms for uploading new resources, a tool development area, ratings and citations abilities, content tagging mechanisms, wikis and blogs, private and public collaboration areas, usage statistics, news and events, and more. While relatively few platforms are as feature-rich, there are, nevertheless, some worth mentioning.
MyExperiment
HUBzero is often compared to the highly successful Open Courseware Initiative from MIT (web.mit.edu/ocw). However, HUBzero is more than just a repository for course materials. It is a place where researchers and educators can meet and accomplish real work. The HUBzero platform offers groups for private collaboration, software development projects in its forge area, event calendars, and many other services designed to connect researchers and build a community. But most importantly, it connects users to the simulation tools they need for research and education. Simulation jobs can be dispatched on national grid resources, including the NSF TeraGrid, the Open Science Grid, and others. HUBzero’s middleware hides much of the complexity of distributed and grid computing, handling authentication, authorization, file transfer, and visualization, and letting the researcher focus on research.
Aside from its core features, HUBzero was chosen for our collaboratory project due to its wide adoption by the US research community as well as our easy access to support services that are provided by Purdue University. We believe that MyExperiment could be adapted to our needs, but its focus on workflows wasn’t as good a fit as HUBzero. Our project heavily depends on climate and adaptation simulation tools that need to be run within the Collaboratory’s system.
Climate change has compelled researchers to look not only for causality, but also for adaptation strategies. Aimed at building a virtual environment for comprehensive and informed decisionmaking, the Collaboratory Project is comprised of a team of researchers in ecology, computer science, law, and social science. We anticipate that our work will help foster an understanding of complexity in natural systems and charter a path from data to knowledge to insight and, finally, to action. Integrated with ongoing and proposed survey research on expert opinion, we also plan to carry out social network analysis (collaboration networks, in particular) and study the impact of a virtual organization on the decision-making process. We anticipate that our efforts will result in various cyber tools that can be adapted by any domain requiring collaborative decision-making.
The role played by our hub (see Figure 1) is to operate as a shared space where researchers from numerous fields related to climate adaptation can run projection and simulation tools, devise new hub tools, and share results with colleagues and the public. Our initial work focused on making computational simulation and projection tools publicly available. However, we soon extended our hub architecture to better enable easy data sharing with added privacy control. As more tools become available, we plan on our hub becoming a central point of contact for experts in climate change and adaptation strategies—a space where a suite of computational tools can be leveraged to simulate the effects of climate change on species and project future distribution patterns. As its community of users grows in both number and diversity, our hub will become more than a repository of publicly available tools and data: it will evolve into a virtual community for climate adaptation research where members contribute back by sharing new tools, publications, use cases, and data.
Source Project, and middleware that enables the execution of hub tools inside a user’s Web browser. To provide an easy-to-use portal system for the scientific community, HUBzero also incorporates features for publishing articles and presentations, a Q&A area, a wish list feature, and user group management. All hub tools run in restricted virtual containers that control access to the underlying file system, networking, and other processes. Each user is granted access to their own home directory with quota
OpenModeller Desktop is an open source, fundamental niche
modeling library with embedded Geographic Information System
(GIS) using the Quantum GIS (QGIS) libraries. It provides a
uniform method for modeling distribution patterns using a variety
of widely used algorithms
Projected occupancy maps (see Figure 3) as well as other output can be shared, but with an option to keep user-specified files private. Ecologists, biologists, policy makers, and experts in other fields can use scientific data and simulation results to make informed decisions about climate change and its effect on species population. 4 4.1
The HUBzero platform was developed at Purdue University to
support nanoHUB.org, an online community for the Network for
Computational Nanotechnology
Our hub users can access various resources about climate adaptation strategy ranging from legal documents to simulation tools. Users can also become contributors by uploading articles, documents, and event information; creating new tools; and participating in forum discussions. As new tools are added to the hub, they contribute to the overall capacity for analysis and lead to more data; in turn, these data become input for steps in the adaptation decision-making process.
Adding a new tool to the hub begins with a registration process on the site, which alerts the hub administrator to create a code repository for the tool. The owner of the tool then iterates though a software engineering process until the tool is ready for publication to the hub’s community. There is, of course, extensive documentation available on the hub to assist newcomers with all HUBzero features, including tool creation.
User groups may also be created to facilitate discussions and the exchange of ideas through the hub. This mechanism offers different levels of privacy, ranging from public to “by invitation only,” and permits users to more finely manage their discussions and the sharing of data. 5 5.1
HUBzero’s middleware relies on OpenVZ to operate multiple, isolated containers (or virtual environments [VEs]). Because
User's desktop
VE NFS Client
Web Server
VE NFS
Client NFS Client NFS Server
VE NFS
Client Middleware The standard open source installation of HUBzero includes a basic OpenVZ configuration that we chose to enrich with additional software packages. For example, Subversion can be added to provide a more functional development environment for hub tools (although, in the most recent release of HUBzero, Subversion is already included). We found significant value in adding a graphical editor (gedit), Web browser (kazehakase), a compression package, and other essential utilities.
The installation of gedit was not limited to just package
deployment. Additional changes to HUBzero’s File Alteration
Monitor (FAM) scripts are required before gedit can be installed
without errors. FAM notifies applications when specific files or
directories are changed
In order to successfully execute a hub tool inside a VE, libraries on which the tool relies must also be added to the OpenVZ environment. For our hub, we needed to provide various C/C++ libraries (starting with the installation of the Debian build-essential package) and an extensive list of additional packages such as qgis, gdal, sqllite, expat, grass and qt. In addition, through a locally mounted directory accessible within our hub’s VEs, we provided OpenVZ utilizes Linux, any computational tools deployed within a hub must also be able to run under Linux. When a hub user launches a tool, the corresponding application is executed within a lightweight VE.
In addition to OpenVZ, HUBzero employs VNC
Hub Extended Architecture Database 5.3
access to precompiled libraries for openModeller (in general, the libraries found inside our VEs differ from those of the underlying HUBzero host system).
We found it beneficial to test new tools with the hub Workspace tool. Through Workspace, error messages are displayed and provide hints about any missing libraries or problems with invocation scripts. For example, the binary package we manually created for openModeller contained several libraries that were not carried along internally during HUBzero’s process of promoting the openModeller tool from initial registration to installation. In this case, we had to manually copy the missing libraries to their destination. This issue may have been the result of the HUBzero project not foreseeing the use of tools as extensive and complex as openModeller.
For other hub tools developed in Java, we configured a more complete Java environment for use by OpenVZ. In addition to the installation of the sun-java6-jdk package, we had to make changes to x86_64-linux.gnu.conf file for Java libraries.
Beyond providing access to all the necessary utilities and libraries for our openModeller hub tool, we also added an X Window window manager, IceWM, to properly display openModeller Desktop within the HUBzero TightVNC Java applet. With IceWM in place, openModeller Desktop appeared and behaved as it normally would on a graphical workstation. We employed the same window manager configuration used by the HUBzero Workspace tool, although we changed themes and preference files to better handle the openModeller icons and window behavior.
It is noteworthy that a special script is needed to invoke a hub tool; for openModeller certain requisites had to be in place within this script. In particular, we had to ensure that various openModeller libraries were pointed to and the IceWM window manager was started.
As a result of our efforts, users familiar with openModeller Desktop who visit our hub do not have to learn a new interface. We have also embarked on an effort to construct new openModeller plugins for our hub tool. These extensions will be made available for hub users as they become ready.
In a hub, files (as opposed to Web content) are typically accessible through the Workspace tool, which provides a Linux workstation within a VE. Our extensions to HUBzero employ NFS thin clients that operate within our hub’s VEs and on our hub server. 6.1
NFS allows sharing the resource of an external file system with many thin clients. In our case, these clients execute inside hub VEs and permit users to access an external file system as though it were local to the hub’s server. As a result we can 1) use less disk space on our HUBzero server and provide access to external, scalable data storage; and 2) leverage the external NFS storage to permit public and private directories for hub users.
Initially, we made an effort to avoid running the NFS client on our server, but found that this made collecting and processing user data difficult and did not easily integrate with HUBzero’s existing configurations. Also, it became cumbersome to automate the creation of public and private user account directories under NFS (HUBzero’s middleware already has access to the user information we need for the dynamic creation of public/private directories under NFS file system).
With our present architecture, an NFS client executes on both our hub server and inside the VEs. Under this configuration, our NFS space is mounted when the OpenVZ service starts. We made our NFS mount persistent for all VEs such that each VE becomes an NFS client that is accessible by the user via the Workspace tool. This permits us to access user data on the host machine while simultaneously providing access to the NFS space from inside the hub VEs. With this, NFS can be used for storing and sharing openModeller input and results. 6.2
While data sharing is an important task, the ability to control data access is also important. To accommodate our researchers’ need to manage how their data are shared, we automated our hub to create a public and a private directory for each user who is permitted to use the Workspace tool. When a given user launches the Workspace tool, HUBzero automatically checks to see if this is a new user, and, if they are new, creates their public and private directories in NFS space. If the user is later deleted, a maintenance routine is run to reclaim the defunct user’s storage. A user’s public NFS directory is accessible by all Workspace tool users as a readonly folder, while their private NFS directory is accessible only to them.
By default, the HUBzero platform makes the Workspace tool available to any registered user. With such a configuration, our system’s resources (including NFS) would be open to use and, possibly, abuse by anyone who registered with our hub. Given the CPU cycles and large data sets involved in simulations, the misuse of hub resources is a significant risk.
To better manage resource usage, we modified HUBzero so that new users are granted access to the Workspace tool only upon request and after being vetted. When an unauthorized (but registered) user tries to launch Workspace, they are given an option to request access through a trouble ticket. This request is dispatched to the hub administrator who may then determine whether or not to approve the desired access. If approved, the user will be able to employ Workspace with access to both their home directory and NFS public/private space.
Our extensions to restrict the Workspace tool required a number of changes to the tool’s configuration. First, a new group was created to manage the use of Workspace. Next, we created a new resource through the HUBzero administrative interface, associating the aforementioned group name with this resource. Finally, we published the new resource and manually updated the hub database so that Workspace access became limited to the newly created group.1 In a related enhancement, we also extended the HUBzero ticketing system logic to include a ticket status for Workspace access approval (as touched upon above). 1 This method was provided to us through communications with the HUBzero project team.
Access control for tools similar to openModeller can be handled during the tool publishing process and restricted to a previously created group.
It is worth noting that testing HUBzero extensions can become somewhat cumbersome due to a requirement that each registered user have only one unique email address. For example, with our added restrictions to the Workspace tool, we needed to test multiple users who are members of various groups. Each user account, of course, requires a unique email address in order to be registered. However, this creates significant limitations for fabricating test accounts, since a legitimate email address must be devised for each account. Thus, we altered our hub so that this requirement, which we find useful in general, is relaxed for specific accounts that we set aside for testing purposes only.
The expectations of our hub’s user community are that simulation results are forthcoming with little delay. Even for small, simple jobs, however, openModeller’s memory and CPU overhead is not insignificant. Moreover, the openModeller Desktop user interface can contribute a fair amount to this overhead, too. In general, such resource consumption can have a deleterious affect on any system if enough users simultaneously execute large openModeller jobs.
To better handle this type of workload, we have begun implementing a stand-alone system to execute multiple numbers of simultaneous openModeller jobs. Although job processing is handled outside of HUBzero’s environment, job submission will take place through a webpage hosted on our hub, while simulation results will be accessible through the same NFS space that is made available to users of our hub’s Workspace tool.
After having been under development for a number of years, HUBzero was released to open source in 2010. This happened to coincide with the start of our collaboratory project and, for the reasons already noted, we decided to try HUBzero and deploy our own hub rather than pay for a hosting plan through Purdue University.
Because we have only recently made our project’s hub available to the Internet, we have yet to accumulate enough traffic and benchmarking to offer a worthwhile opinion of HUBzero’s capabilities while in production use and under load. Nevertheless, we can comment on deploying and securing HUBzero, as well as the support offered by the HUBzero project. 9.1
We have deployed the open source HUBzero platform onto three
different servers (one test and two production systems) multiple
times. Each server operates Debian Lenny as a virtual machine
(VM) running under the Kernel-based Virtual Machine (KVM)
hypervisor
The HUBzero installation process at the time of this writing consists of 77 steps spread over 25 major areas. This complexity often results in errors that are challenging to debug. One option the HUBzero project might consider implementing is a checkpoint system for their install process. With this, at key intervals it would be possible to know that a given installation is correct. To this end, we have compiled our own internal list of checkpoints, especially with regard to Workspace configuration. In a related matter, although we have not yet tested it, the HUBzero project has recently devised a beta version of a simplified installation process.
Over time our familiarity and expertise improved regarding HUBzero installations. Our team now maintains an internal wiki to help maintain institutional knowledge on this front and keep solutions to common problems close at hand. It now takes us roughly eight hours to setup a new HUBzero server (without adding our extensions). 9.2
An informal benchmarking of simultaneous openModeller Desktop tool executions offered some encouraging results. In our test (see Figure 5), each openModeller Desktop tool processed through a 149-megabyte data set, resulting in no perceptible delay for 18 simultaneous sessions. The maximum CPU utilization was 56% and the total running time was 76 seconds. As the number of sessions was decreased, we observed that the total maximum running time changed only slightly (e.g., 70 seconds for six simultaneous sessions); we believe this was due to the abundance of available resources for the tested number of sessions. For comparison, we found the maximum CPU utilization was 37% for 12 simultaneous sessions and 19% for six sessions. Memory usage was not significant given the amount of data processed (it remained below 6% of our hub’s total memory throughout the test). On a previous HUBzero VM (four CPU cores, four gigabytes of RAM, and 500 gigabytes of hard disk) we had carried out a similar test of the openModeller Desktop tool, but with fewer simultaneous sessions. There, for 15 simultaneous sessions we noted the memory usage peaked at 31%, while CPU utilization reached 94%; running time was considerably greater at about three minutes. 9.3
The “out of the box” security posture of HUBzero is best described as open. To begin with, a newly installed hub will start life with several critical services exposed to its network. These include: SSH, SMTP, MySQL, and LDAP. Each of these should be appropriately restricted—in the case of SMTP, MySQL, and LDAP, these can be safely restricted to the hub itself. Next, other than “root,” it is not possible to install a user account on a hub. This makes it very difficult to know who is using the root account (since anyone logging into a hub server must do so as root) and significantly reduces the utility of OS activity audit trails. Related to this, the root account is exposed to password guessing through SSH login attempts. Finally, efforts to install file space monitoring tools such as Tripwire appear to cause malfunctions in HUBzero. (Although, we cannot offer much detail about Tripwire’s erroneous interactions with our hub deployments, as we elected not to spend time diagnosing this issue.) The support available through the HUBzero project has improved significantly over time. We typically see responses to our email queries within 24 hours and, in some cases, even on weekends. Often, the issues we raise are already known to the HUBzero project team and do not require much troubleshooting on their part. Both a knowledge base and a “Questions & Answers” resource are available through the HUBzero project website. Although anyone can post a question, only a handful of institutions appear to be deploying their own hubs (as opposed to purchasing a hosting plan through Purdue), so there is relatively little information to be found in existing support resources. 10 CONCLUSION The abilities to execute simulation/modeling tools and share information are paramount to our collaboratory project. As such, we found the HUBzero platform to be a proper fit for sharing computational tools and data. While the HUBzero platform does a good job of bringing users and modeling tools together, we extended its functionality to enable the use of sophisticated, computationally intensive modeling tools. Also, we discovered that a default hub requires additional security measures to protect its network services (e.g., SMTP, MySQL, and LDAP) and restrict access to its OpenVZ environment. In terms of resource consumption, we tested relatively modest sets of data with the openModeller Desktop tool and will continue evaluating resource consumption for larger data sets as they become available. Finally, we provided seamless access to external Network File System drive space to store and share input and output data.
Science Foundation under Grant No. 1029584.