=Paper=
{{Paper
|id=Vol-2357/paper2
|storemode=property
|title=Multidisciplinary Collaboration Through Online Virtual Research Environments (VREs): What Do VRE Users Need?
|pdfUrl=https://ceur-ws.org/Vol-2357/paper2.pdf
|volume=Vol-2357
|authors=Yi Yin,Anneke Zuiderwijk,Marijn Janssen,Xander de Ronde,Keith Jeffery
|dblpUrl=https://dblp.org/rec/conf/iwsg/YinZJRJ18
}}
==Multidisciplinary Collaboration Through Online Virtual Research Environments (VREs): What Do VRE Users Need?==
https://ceur-ws.org/Vol-2357/paper2.pdf
10th International Workshop on Science Gateways (IWSG 2018), 13-15 June 2018
Multidisciplinary Collaboration Through Online Virtual
Research Environments (VREs): what do VRE users
need?
Yi Yin Anneke Zuiderwijk Marijn Janssen
Delft University of Technology Delft University of Technology Delft University of Technology
Delft, The Netherlands Delft, The Netherlands Delft, Netherlands
Y.Yin@tudelft.nl a.m.g.zuiderwijk-vaneijk@tudelft.nl M.F.W.H.A.Janssen@tudelft.nl
Xander de Ronde Keith Jeffery
Delft University of Technology ERCIM
Delft, The Netherlands Faringdon, United Kingdom
X.E.J.deRonde@student.tudelft.nl keith.jeffery@keithgjefferyconsultants.c
o.uk
Abstract—Making new data combinations and collaborating climate change research, or research on the Internet-of-Things.
with researchers from different disciplines are becoming In a study conducted by Zuiderwijk, Jeffery [3], they state that
essential ingredients of scientific research. These activities are VREs consist of three major components or layers, namely:1)
increasingly contributing to solutions for multidisciplinary global e-Infrastructures(e-Is) providing Information and
problems, such as climate change and energy transition. Virtual Communication Technology (ICT) facilities; 2) e-Research
Research Environments (VREs) can potentially support making Infrastructures (e-RIs) providing access to data, software and
data combinations and researcher collaborations by providing a computing resources; 3) the VRE with its users, who can
multiplicity of data and services. Many VREs have been cooperatively work through the VRE to conduct various
developed already and are used in specific research domains.
research activities [3-5].
However, there is a lack of insight into what is needed to develop
a multidisciplinary VRE in comparison with monodisciplinary Many VREs have been developed and used for specific
VREs. This is currently blocking the development of innovative research domains. For example, the EVER-EST [6] and the
multidisciplinary VREs. This study aims to investigate the EPOS [7] VRE in earth sciences, the VI-SEEM [8] VRE in life
requirements for building a multidisciplinary VRE and to study sciences, climatology and digital cultural heritage, and the
the key differences between monodisciplinary VREs and GenePattern [9] VRE in biological sciences. Requirements for
multidisciplinary VREs. Our study shows that comprehensive VREs in monodisciplinary research have already been
requirements in nine categories need to be fulfilled when
investigated. They include easy-to-use interfaces, adequate
designing a multidisciplinary VRE. Lack of considering many
data storage, available analysis tools, high performance
requirements and limit focus in monodisciplinary VREs hinder
the wide use of current VREs in multidisciplinary research.
computing resources [10, 11], secure access mechanisms via
the same credentials [11], metadata management [12], help and
Keywords—VRE; research data sharing; requirements; training support for VRE users [10].
multidisciplinary Virtual Research Environment; science gateway Multidisciplinary research, intending to solve many
I. INTRODUCTION problems, such as climate change, environmental pollution,
and earthquakes monitoring and prediction, needs to combine
Virtual Research Environments (VREs) have become data from several disciplines and requires collaboration.
critical to modern research processes [1]. VREs or Science However, there is a lack of insight into what is needed to
Gateways, aim to support researchers from multiple disciplines develop a multidisciplinary VRE in comparison with
to collaborate [2]. They do so by managing the increasingly monodisciplinary VREs. This is currently blocking the
complex range of tasks involved in carrying out research on development of innovative multidisciplinary VREs. The
both small and large scale, such as tracking the change of data objective of this study is twofold: 1) to investigate the
using information from seafloor scans for undersea requirements for building a multidisciplinary VRE and 2) to
archaeology, using data on greenhouse gas concentrations for study the key differences of current practices of
monodisciplinary VREs in comparison with multidisciplinary that this process consists of three steps, namely: 1)
VREs. To attain this objective, we have investigated VRE information gathering, 2) representation and 3) verification
requirements using multiple methods, including a literature [15]. Maguire et. al (2002) mentioned that the requirement
review, interviews with potential VRE users and developers, analysis process encompasses 4 steps, namely 1) information
and the characterisation of existing research infrastructures. gathering, 2) user needs identification, 3) envisioning &
This paper is organized as follows. Section 2 describes the evaluation and 4) requirement specifications. Parviaien et. al
requirements engineering approach applied in this study. (2003) stated three phases in the requirement engineering
Section 3 and 4 describe the requirements for monodisciplinary processes, including 1) requirements elicitation, 2)
VREs and multi-disciplinary VREs accordingly. In Section 5, requirements analysis & negotiation and 3) requirements
we compare the requirements between monodisciplinary VREs validation [20]. We state that in reality the requirements
and multidisciplinary VREs. Section 6 concludes this paper. collection is a continuous and iterative process which needs to
accommodate changes from the involved organizations,
II. REQUIREMENT ENGINEERING APPROACH environment and stakeholders. From these engineering
VREs support research by interconverting between the processes we derived four common elements as shown in
multiple underlying e-RIs supported by e-Is, while the VRE Figure 1, namely 1) elicitation, 2) analysis and negotiation, 3)
users neither know nor care about the underlying e-Is [3]. evaluation and 4) evolution management. Below we explain
Depending on e-RIs, VREs are on a higher level of hierarchy how we identified and elicited requirements in this research
than e-RIs and underlying e-Is, and provide more advanced through each of the steps shown in Figure 1.
functionalities for their end-users which are mainly researchers.
The perspective of the user, i.e. the researcher, is essential for Step 1: Elicitation
developing the VREs. Understanding user requirements is Requirements elicitation helps to discover and
generally recognized as the most crucial and the most difficult conceptualize system requirements through information
stage for the successful development, deployment and gathering and user needs identification. We collected
evolution of information systems [13-15]. This is also the case background information from interviews and publicly
for the VREs. Aybuke and Claes [16] state that requirements available documentations of existing VRE research projects.
should include both user needs and needs arising from other After the user information is collected, analysis can start to
stakeholders like organizations, governmental bodies and identify the real user needs and expectations. In this research
industry standards. What is common among requirement we use existing VRE projects and the characterisation of e-RI
definitions is that they refer to describing what the proposed projects to identify user needs. Interview protocols were
information system is supposed to do and how it should do this created to collect information from the end-users and VRE
[13, 16, 17]. However, the understanding regarding “what” and developers [21]. Ten interviews have been conducted (see
“how” differs per stakeholder, and it is not easy to identify the Table 1).
differences between various requirement classifications in
practise [18]. In this study, we adopted the definition of TABLE 1 Overview of interviews for requirements information collection
requirement from Aybuke and Claes [16], namely descriptions
Interviewee Role of interviewee Experience Research
of how a software products should perform. # with VREs domain
Figure 1 The requirement engineering process used in the research 1 Potential VRE end user No Civil engineering
2 Developer Yes Earth science
3 Potential VRE end user Yes Earth science
4 Potential VRE end user Yes Physics
5 Potential VRE end user No Physics
6 Developer and Yes Health
potential VRE end user
7 Developer Yes Computer
science
8 Developer Yes Information
science
9 Potential VRE end user Yes Library
10 Developer Yes Environmental
sciences
A VRE allows for connecting existing VREs and e-RIs,
we have analyzed the functionalities in the existing VRE
projects as a starting point to understand the VRE
requirements. Eight ESFRI landmark projects have been
selected for analysis. These projects are relatively mature
VREs or e-RIs which have been developed or are already in
The requirements engineering process concerns the operation now. These VREs focus on a single discipline such
investigation and learning about the problem domain in terms as earth science, social science, or life science. A protocol
of understanding the actual goals, needs and expectations of guiding and structuring the characterisation of e-RIs was also
the users regarding a system [19]. Browne et. al (2001) stated created on the basis of six key types of functional elements in
an e-RI defined by the ENVRIPlus project [22]. The
questions of these three protocols were created using the In this study, we focused on the research results from the
Reference Model of Open Distributed Processing (ODP) [23]. step 1 and step 2 while the results from step 3 and step 4 are
The questions covered each of the five ODP viewpoints: beyond the scope of this paper.
enterprise (science), information, computation, engineering
and technology. The VRE should account for the needs of III. ELICITATION OF MONO-DISCIPLINARY VRE
heterogeneous user groups. In addition, the questions REQUIREMENTS
concerning activities of VRE users addressed user activities in In order to tackle the global challenges and solve complex
line with those mentioned in the literature [24, 25]. scientific problems, scientists need to use VREs as shorthand
Step 2: Analysis and negotiation for the tools and technologies. They can conveniently make
use of resources and technical infrastructures available both
Once an initial set of user requirements has been locally and remotely to conduct their research and to interact
formulated, requirements can be detailed, discussed and with other researchers who might be from different countries.
agreed by stakeholders in the analysis and negotiation phase,
including two main steps: Therefore, VREs need to provide tools and computing
resources related to data acquisition, data storage, data
1) Analyse and envision. When analysing and describing processing, and data analysis. According to the e-infrastructure
the requirements, it is essential to fully document “the design research project ENVRIplus, VREs should meet requirements
element or its interfaces in terms of requirements (functional, and provide six types of functionalities, including data
performance, constraints and design characteristics)” [26]. identification and citation, i.e. assigning global unique
After describing the requirements, it is also necessary to identifiers to data; data curation, i.e. data quality check; data
develop a conceptual prototype to illustrate the requirements cataloguing, i.e. adding metadata to datasets; data processing,
and get feedback from the stakeholders. On the basis of the i.e. converting data format and data visualization; data
feedback, the requirements are evaluated and may be optimization, i.e. data compartmentalization; and data
modified. provenance, i.e. tracking the changes of data. We also add
2) Specification and negotiation. In the analysis step of collaboration, training and support as an additional category in
user requirements, the following should be discussed with all this table, since researchers are in need of support related to
stakeholders and documented within the specification: finding collaboration for research projects, i.e. finding the
identification of the range of relevant users, clear design goals, collaborators with specific expertise, writing grant proposal
the requirements with prioritized levels and evaluation criteria and research project management tools, according to
to test the requirements whether they will be fulfilled and interviewees #1, #8, and #9.
evidence of acceptance of the requirements by stakeholders. According to the interviewees #1, #3, #4, #5, and #9 (als
The following methods are used for specification and researchers that are potential VRE end-users), a quickly-
negotiation: function mapping, requirements categorisation accessible, reliable, easy-to-use, low-cost VRE is expected.
[14]. Therefore, when designing the VRE, it is very important to
also consider the non-functional performance-related
Step 3: Evaluation
requirements defined by commonly used software engineering
The evaluation of requirements is to check the consistency standards in FURPS+ and ISO 25010:2011 such as efficiency,
and completeness of the requirements [20]. This phase is usability, reliability, maintainability, sustainability,
concerned with the examination of the requirement description compatibility and portability.
to ensure that it defines the system in an accurate and
comprehensive way. In this research, we have used use case In addition, interviewee #6 indicated that all VREs have to
analysis, online questionnaire and several workshops with carefully deal with data containing privacy sensitive
VRE experts to evaluate the collected requirements. After these information. Therefore, privacy, security, trust and legal
activities, we have also designed a VRE system architecture to requirements are necessary to be considered in term of
accommodate all collected requirements. regulatory compliance. Privacy and security requirements
specify how the use of the VRE should be robust against cyber-
Step 4: Evolution management attacks in term of enhanced privacy and security. Trust
requirements specify the acceptable behaviours of the
Requirements are the starting point for the system design
stakeholders in the VRE, such as users, system developers and
phase [20]. However, we cannot wait for complete
service providers. Legal requirements specify that the whole
requirements as the content and the priority of the initial development of VRE should comply with all legislation,
requirements may evolve and change during the development especially the new General Data Protection Regulation issued
process. Therefore, we also keep track of changes in or new by the European Parliament, the Council and the Commission
requirements. These initial requirements have been used to in 2015.
define the system functionalities when designing the VRE
architecture. During the development of the VRE architecture Research into mono-disciplinary VREs already shows that
and analysis of use cases, some additional requirements have many challenges exist for the use of VREs [27], including: data
been identified. These requirements are not new but support context issues (understanding the creation context of research
requirements identified in step 2. data), data heterogeneity issues (large amount of data
generating from various sources), data quality issues (it is not
easy to control data quality), privacy issues (datasets
containing privacy information need to shared and reused), processing towards desired effects from the viewpoint
user experience issues (the expectation of users on the system of data object creator or users;
varies), availability of technology issues. Previous research
also shows that for multidisciplinary VREs, even more 6) Data provenance requirements which define the needs
challenges should be added to this list, since multidisciplinary of making logs on the transformation and
VREs need to interoperate between a large variety of computational process on data objects;
standards, ontologies and terminologies used by different 7) Collaboration, training and support requirements
research disciplines in different countries. According to the which define the needs of providing research
information from interviews and the e-RI characterisations, collaboration tools and manuals for using VREs
additional challenges concern the availability of data from systems;
different sources, data use licensing for different organizations,
scalability in terms of connecting High Performance 8) None-functional requirements which define the
Computing (HPC) facilities, system management responsibility software performance related objectives such usability,
and financial support. Another challenge concerns the access stability;
policy. VRE system administrators prefer one certificate per 9) Security, privacy trust and legal requirements which
research community in order to lower the effect on user define the needs of system design in compliance with
credential management[11], while many organizations cannot all regulation and improvement measures in terms of
easily make agreements on sharing certificates to grant access improving users’ overall trust on the VREs.
to VREs.
TABLE 2 Overview of requirements for a multidisciplinary VRE
From the requirements collection work, we have analyzed
eight monodisciplinary VREs or e-RIs based on seven Category Requirement example
functional requirement categories. These VREs provide Data - Ability to assign (global) unique identifiers (e.g. DOIs,
integrated services and datasets and cross-country access to identification ePIC, URIs) to data contents
various resources for research. Researchers from the same and citation - Ability to assign an accurate, consistent and
research domain can use these resources. Some VREs only standardized reference to a data object, which can be
cited in scientific publications
provide these services to authorized researchers. Researchers
Data Curation - Ability to detect and correct (or remove wrong data
from other disciplines or general public cannot easily access - Ability to support manual quality checking
some of these VREs. Some VREs are still in the development Data - Ability to associate a data object with one or more
phase, although some functionalities in the seven categories are Cataloguing metadata objects which contain data descriptions
being designed or already implemented. The usability of these - Ability to select a subset of individuals from within a
functionalities significantly varies. These VREs mainly provide statistical population to estimate characteristics of the
whole population
dataset download and limited data analysis tools. The
Data - Ability to convert data from one format to another
collaboration, training and support functionalities are largely processing format
missing in these VREs and e-RIs. - Ability to inspect, clean, transform data, and to provide
data models with the goal of highlighting useful
IV. ANALYSIS OF REQUIREMENTS FOR MULTIDISCIPLINARY information, suggesting conclusions, and supporting
VRES decision making
Data - Large datasets processing
In this section we describe the requirements for optimization - Data compartmentalization
multidisciplinary VRE collaboration, compared to mono-
disciplinary VRE use. Table 2 presents a list of requirements Data - Data Provenance: Ability to provide “pathways of data”
provenance or the history of data information (provenance data)
for the development of a multidisciplinary VRE, containing - Data publication information: Ability to provide data
nine categories of requirements, namely: publication information (e.g. which data was accessed,
which data is not accessible, which query was carried out
1) Data identification and citation requirements which and when)
define the approaches to provide everlasting and Collaboration, - Notifications: Sending notification when certain
unique references to each research data object; training and information becomes available to the users
support - Finding collaborators: Ability to locate previous
2) Data curation requirements which define the needs of collaborators and potential collaborators
processes to assure the availability and quality of data Non- - Usability
object over the long term; functional - Performance efficiency
(System - Reliability
3) Data cataloguing requirements which define the Performance) - Maintainability
needs of easy and quick access to data objects by - Sustainability
queries over catalogues; Privacy, - Specified service authorization contract
security, trust - Specific definition regarding software service
4) Data processing requirements which define the needs and legal authorization in compliance with legislation
of providing computational transformation software on requirements - Secure storage and use of data, especially data
containing privacy information
data objects;
5) Data optimization requirements which define the This list contains 148 requirements in 9 categories which
needs of providing computational transformation and have been reported in a project deliverable of the VRE4EIC
project [21]. Our requirements provide a comprehensive
overview of many perspectives that need to be considered When a new VRE becomes available, researchers do not
during the development of VREs. The requirement categories just move from one e-RI or VRE which they are already
mentioned in Table 2 are also important to monodisciplinary familiar with to another. The process of transferring from one
VREs, however, the examples of the requirements themselves VRE to another creates challenges for researchers. Since the
are specific to multidisciplinary VREs. usability of existing mono-disciplinary VREs developed based
on the requirements in Table 3 significantly varies, the
V. DISCUSSION interoperability of those VREs cannot meet the researchers’
We have analyzed eight e-RIs and VREs to understand the demands in multi-disciplinary research. In addition,
current practices in the development of VREs. Table 3 showed researchers do not want to spend much time on learning how
the implementation of functional requirements in these to use new software or work in a new online environment.
projects. From the table we can see that many projects have Therefore, easy access to data from multiple disciplines and to
considered data-related requirements while collaboration computing resources are crucial. A portal or gateway
requirements are largely ignored. Although the categories of connected with those resources might be suitable to fulfill this
the requirements have been covered by many existing VRE task.
projects, but the range and details of specific requirements in In multidisciplinary research, researchers desire to use a
each categories are not comprehensive enough for developing single tool with an easily understandable Graphical User
a multi-disciplinary VRE. Privacy and trust related Interfase (GUI) and plug-and-play features provided by
requirements have not been identified in these projects. different VREs or e-RIs to submit their experiment tasks, data
TABLE 3 Characterisation of the e-Research Infrastructures and Virtual analysis assignments and to monitor the status. They do not
Research Environments. want to know the complexity behind simultaneously running
e- these tasks. A powerful workflow engine with an intuitively
Requirements
Research usable GUI needs to be designed in a multidisciplinary VRE
Infrastruc Cat Collabor to integrate several mono-disciplinary VREs.
ture / Identifi
Cu alo Data Data Data ation,
Virtual cation
rati gui Proce Optimi Proven training
Different research communities use different standards and
Research and data models to process research data. Researchers from the
on ng ssing zation ance &
Environm citation
support same research domain can use their own standards and
ent practices for data processing. In multidisciplinary research,
EURO- researchers need to combine data from different research
● ● ● * * ● ○
ARGO domains with interoperable data processing tools. Our
ICOS ● ● ● ● ● ● ○ research showed that they do not want to encounter errors
when they put the data in the VRE system. In the interviews,
researchers also expressed their concerns related to the control
EPOS ● ● ● ● ● ● *
of their data if shared with other researchers. They want their
ELIXIR ● ● ● ● ● ● ○ work and data to be acknowledged and properly referred to
Lifewatch ● ● ● ● ● ● ○
when used by others. In a multidisciplinary VRE stored data
and data use are more complex compared to a mono-
CESSDA ● ● ● ● ● ● * disciplinary VRE, thus better security mechanisms are
required without hindering the ease-to-use of the system.
ENVRIPL
● ● ● ● ● ● * In general, we found that there remains a big gap in the
US
completeness of requirements fulfilled by the existing mono-
CLARIN ● ● ● ● ● ● * disciplinary VREs towards multidisciplinary VREs. This is
shown in Table 3. A comprehensive multidisciplinary VRE
● Requirements covered should fulfill the requirements described by in the
○ Requirements not covered aforementioned nine requirement categories (see section IV).
* Requirements suggested by the project vision or limitedly covered
Multidisciplinary VREs need to be developed in terms of
A multidisciplinary VRE is ideally open to any researcher. interoperability, a single gateway with an intuitive GUI to
In our study, we found that many researchers are already using easily access data and computing resources, and complying
some domain-specific resources like research data, software with data protection regulation.
tools or e-infrastructures to support their research activities.
However, these resources are only known or open to a small VI. CONCLUSIONS
research community. Some of the mono-disciplinary VREs we This research aims to 1) study the requirements for developing
studied claim to be openly accessible, but they are de facto a multidisciplinary VRE, and 2) investigate the requirement
only open to some researchers due to bureaucratic user differences between the current practices of monodisciplinary
registration and approval processes. It is very difficult for VRE and the requirements of developing a multidisciplinary
researchers from other domains to find these existing VREs VRE. A comprehensive set of requirements needs to be
since they are not aware of the VRE development in the considered when developing a multidisciplinary VRE.
research domain other than their own science community. Building on the ENVRIplus categorization for e-Research
Infrastructure requirements, we categorized functional
requirements in the following categories: data identification [7] EPOS. European Plate Observing System. 2018; Available from:
https://www.epos-ip.org/.
and citation, data curation, data cataloguing, data processing,
[8] Vi-SEEM, Virtual Research Environment (VRE) in Southeast Europe
data optimization, data provenance, collaboration, training and and the Eastern Mediterranean (SEEM). 2018.
support. Non-functional requirements were added to this [9] GenePattern. GenePattern: A Patform for Reproducible Bioinformatics.
categorization and collected in the categories of performance- 2018; Available from:
related requirements, security, privacy, trust and legal http://software.broadinstitute.org/cancer/software/genepattern#.
requirements. [10] McGrath, A., et al., The Essential Components of a Successful Galaxy
Researchers’ concerns on losing control of their own research Service. Journal of Grid Computing, 2016. 14(4): p. 533-543.
data, lack of interoperability between different VREs and e- [11] Gesing, S., et al., Using Science Gateways for Bridging the Differences
between Research Infrastructures. Journal of Grid Computing, 2016.
RIs and e-Is, as well as lack of comprehensive consideration 14(4): p. 545-557.
of various requirements in monodisciplinary VREs limit their [12] Grunzke, R., et al., Metadata management in the MoSGrid science
adoption for multidisciplinary research. When developing gateway-evaluation and the expansion of quantum chemistry support.
multidisciplinary VREs, we have to take all these Journal of Grid Computing, 2017. 15(1): p. 41-53.
requirements into consideration and choose suitable [13] Yu, E.S.K. Towards modelling and reasoning support for early-phase
technologies to meet them. However, we have to admit that requirements engineering. in Proceedings of the 1997 3rd International
Symposium on Requirements Engineering. 1997. Los Alamitos, CA,
developing VREs is a rather complex engineering process. United States, Annapolis, MD, USA: IEEE.
Requirements identified in this study may not be implemented [14] Maguire, M. and N. Bevan, User requirements analysis, in Usability.
at once, but fulfilled stage by stage during the development of 2002, Springer. p. 133-148.
mature multidisciplinary VREs. [15] Browne, G.J. and M.B. Rogich, An empirical investigation of user
requirements elicitation: Comparing the effectiveness of prompting
ACKNOWLEDGMENT techniques. Journal of Management Information Systems, 2001. 17(4):
p. 223-249.
This work was carried out within the VRE4EIC project [16] Aybuke, A. and W. Claes, Engineering and Managing Software
and received funding from the European Union’s Horizon Requirements. 2005, Springer-Verlag Berlin.
2020 research and innovation programme under grant [17] Robinson, W.N., S.D. Pawlowski, and V. Volkov, Requirements
agreement No. 676247. The authors should like to thank their Interaction Management. ACM Computing Surveys, 2003. 35(2): p.
colleagues in this project for their input for this paper 132-190.
(particularly Daniele Bailo, Zhiming Zhao, Valerie Brasse, [18] Curtis, B., H. Krasner, and N. Iscoe, A field study of the software design
Theodore Patkos, and Jacco van Ossenbruggen for process for large systems. Communications of the ACM, 1988. 31(11):
p. 1268-1287.
coordinating the interviews). The views expressed in this
paper are the views of the author and not necessarily of the [19] Koukias, A., et al. Approach on analysis of heterogeneous requirements
in software engineering. in 11th IFAC Workshop on Intelligent
VRE4EIC project. Manufacturing Systems, IMS 2013. 2013. Sao Paulo.
[20] Parviainen, P., et al., Requirements engineering inventory of
technologies. VTT PUBLICATIONS,
REFERENCES [21] Yin, Y. and A. Zuiderwijk, State-of-the-art and user requirement
analysis. 2016.
[22] ENVRIplus. 2018 [cited 2018; Available from:
[1] Buddenbohm, S., et al., Success criteria for the development and http://www.envriplus.eu/.
sustainable operation of virtual research environments. D-Lib Magazine,
2015. 21(9/10). [23] Linington, P.F., et al., Building Enterprise Systems with ODP. An
Introduction to Open Distributed Processing. 2011, Washington:
[2] Susha, I., M. Janssen, and S. Verhulst. Data collaboratives as a new Chapman & Hall/CRC Press.
frontier of cross-sector partnerships in the age of open data: Taxonomy
development. in Proceedings of the 50th Hawaii International [24] Buddenbohm, S., et al., Success Criteria for the Development and
Conference on System Sciences. 2017. Sustainable Operation of Virtual Research Environments. D‐Lib
Magazine, 2015. 21(9/10).
[3] Zuiderwijk, A., et al., Using Open Research Data for Public Policy
Making: Opportunities of Virtual Research Environments, in Conference [25] De Roure, D., C. Goble, and R. Stevens, The design and realisation of
for E-Democracy and Open Government. 2016: Krems an der Donau, the Virtual Research Environment for social sharing of workflows.
Austria. Future Generation Computer Systems, 2009. 25(5): p. 561-567.
[4] Crosas, M., The dataverse network®: an open-source application for [26] IEEE-STD, ISO/IEC Standard for Systems Engineering - Application
sharing, discovering and preserving data. D-lib Magazine, 2011. 17(1): and Management of the Systems Engineering Process. ISO/IEC 26702
p. 2. IEEE Std 1220-2005 First edition 2007-07-15, 2007: p. c1-88.
[5] Edwards, P., et al., Lessons learnt from the deployment of a semantic [27] Zuiderwijk, A., et al. Using Open Research Data for Public Policy
virtual research environment. Web Semantics: Science, Services and Making: Opportunities of Virtual Research Environments. in 2016
Agents on the World Wide Web, 2014. 27-28: p. 70-77. Conference for E-Democracy and Open Government (CeDEM). 2016.
[6] EVER-EST. EVER-EST. 2018; Available from: www.ever-est.eu.