10th International Workshop on Science Gateways (IWSG 2018), 13-15 June 2018 Multidisciplinary Collaboration Through Online Virtual Research Environments (VREs): what do VRE users need? Yi Yin Anneke Zuiderwijk Marijn Janssen Delft University of Technology Delft University of Technology Delft University of Technology Delft, The Netherlands Delft, The Netherlands Delft, Netherlands Y.Yin@tudelft.nl a.m.g.zuiderwijk-vaneijk@tudelft.nl M.F.W.H.A.Janssen@tudelft.nl Xander de Ronde Keith Jeffery Delft University of Technology ERCIM Delft, The Netherlands Faringdon, United Kingdom X.E.J.deRonde@student.tudelft.nl keith.jeffery@keithgjefferyconsultants.c o.uk Abstract—Making new data combinations and collaborating climate change research, or research on the Internet-of-Things. with researchers from different disciplines are becoming In a study conducted by Zuiderwijk, Jeffery [3], they state that essential ingredients of scientific research. These activities are VREs consist of three major components or layers, namely:1) increasingly contributing to solutions for multidisciplinary global e-Infrastructures(e-Is) providing Information and problems, such as climate change and energy transition. Virtual Communication Technology (ICT) facilities; 2) e-Research Research Environments (VREs) can potentially support making Infrastructures (e-RIs) providing access to data, software and data combinations and researcher collaborations by providing a computing resources; 3) the VRE with its users, who can multiplicity of data and services. Many VREs have been cooperatively work through the VRE to conduct various developed already and are used in specific research domains. research activities [3-5]. However, there is a lack of insight into what is needed to develop a multidisciplinary VRE in comparison with monodisciplinary Many VREs have been developed and used for specific VREs. This is currently blocking the development of innovative research domains. For example, the EVER-EST [6] and the multidisciplinary VREs. This study aims to investigate the EPOS [7] VRE in earth sciences, the VI-SEEM [8] VRE in life requirements for building a multidisciplinary VRE and to study sciences, climatology and digital cultural heritage, and the the key differences between monodisciplinary VREs and GenePattern [9] VRE in biological sciences. Requirements for multidisciplinary VREs. Our study shows that comprehensive VREs in monodisciplinary research have already been requirements in nine categories need to be fulfilled when investigated. They include easy-to-use interfaces, adequate designing a multidisciplinary VRE. Lack of considering many data storage, available analysis tools, high performance requirements and limit focus in monodisciplinary VREs hinder the wide use of current VREs in multidisciplinary research. computing resources [10, 11], secure access mechanisms via the same credentials [11], metadata management [12], help and Keywords—VRE; research data sharing; requirements; training support for VRE users [10]. multidisciplinary Virtual Research Environment; science gateway Multidisciplinary research, intending to solve many I. INTRODUCTION problems, such as climate change, environmental pollution, and earthquakes monitoring and prediction, needs to combine Virtual Research Environments (VREs) have become data from several disciplines and requires collaboration. critical to modern research processes [1]. VREs or Science However, there is a lack of insight into what is needed to Gateways, aim to support researchers from multiple disciplines develop a multidisciplinary VRE in comparison with to collaborate [2]. They do so by managing the increasingly monodisciplinary VREs. This is currently blocking the complex range of tasks involved in carrying out research on development of innovative multidisciplinary VREs. The both small and large scale, such as tracking the change of data objective of this study is twofold: 1) to investigate the using information from seafloor scans for undersea requirements for building a multidisciplinary VRE and 2) to archaeology, using data on greenhouse gas concentrations for study the key differences of current practices of monodisciplinary VREs in comparison with multidisciplinary that this process consists of three steps, namely: 1) VREs. To attain this objective, we have investigated VRE information gathering, 2) representation and 3) verification requirements using multiple methods, including a literature [15]. Maguire et. al (2002) mentioned that the requirement review, interviews with potential VRE users and developers, analysis process encompasses 4 steps, namely 1) information and the characterisation of existing research infrastructures. gathering, 2) user needs identification, 3) envisioning & This paper is organized as follows. Section 2 describes the evaluation and 4) requirement specifications. Parviaien et. al requirements engineering approach applied in this study. (2003) stated three phases in the requirement engineering Section 3 and 4 describe the requirements for monodisciplinary processes, including 1) requirements elicitation, 2) VREs and multi-disciplinary VREs accordingly. In Section 5, requirements analysis & negotiation and 3) requirements we compare the requirements between monodisciplinary VREs validation [20]. We state that in reality the requirements and multidisciplinary VREs. Section 6 concludes this paper. collection is a continuous and iterative process which needs to accommodate changes from the involved organizations, II. REQUIREMENT ENGINEERING APPROACH environment and stakeholders. From these engineering VREs support research by interconverting between the processes we derived four common elements as shown in multiple underlying e-RIs supported by e-Is, while the VRE Figure 1, namely 1) elicitation, 2) analysis and negotiation, 3) users neither know nor care about the underlying e-Is [3]. evaluation and 4) evolution management. Below we explain Depending on e-RIs, VREs are on a higher level of hierarchy how we identified and elicited requirements in this research than e-RIs and underlying e-Is, and provide more advanced through each of the steps shown in Figure 1. functionalities for their end-users which are mainly researchers. The perspective of the user, i.e. the researcher, is essential for Step 1: Elicitation developing the VREs. Understanding user requirements is Requirements elicitation helps to discover and generally recognized as the most crucial and the most difficult conceptualize system requirements through information stage for the successful development, deployment and gathering and user needs identification. We collected evolution of information systems [13-15]. This is also the case background information from interviews and publicly for the VREs. Aybuke and Claes [16] state that requirements available documentations of existing VRE research projects. should include both user needs and needs arising from other After the user information is collected, analysis can start to stakeholders like organizations, governmental bodies and identify the real user needs and expectations. In this research industry standards. What is common among requirement we use existing VRE projects and the characterisation of e-RI definitions is that they refer to describing what the proposed projects to identify user needs. Interview protocols were information system is supposed to do and how it should do this created to collect information from the end-users and VRE [13, 16, 17]. However, the understanding regarding “what” and developers [21]. Ten interviews have been conducted (see “how” differs per stakeholder, and it is not easy to identify the Table 1). differences between various requirement classifications in practise [18]. In this study, we adopted the definition of TABLE 1 Overview of interviews for requirements information collection requirement from Aybuke and Claes [16], namely descriptions Interviewee Role of interviewee Experience Research of how a software products should perform. # with VREs domain Figure 1 The requirement engineering process used in the research 1 Potential VRE end user No Civil engineering 2 Developer Yes Earth science 3 Potential VRE end user Yes Earth science 4 Potential VRE end user Yes Physics 5 Potential VRE end user No Physics 6 Developer and Yes Health potential VRE end user 7 Developer Yes Computer science 8 Developer Yes Information science 9 Potential VRE end user Yes Library 10 Developer Yes Environmental sciences A VRE allows for connecting existing VREs and e-RIs, we have analyzed the functionalities in the existing VRE projects as a starting point to understand the VRE requirements. Eight ESFRI landmark projects have been selected for analysis. These projects are relatively mature VREs or e-RIs which have been developed or are already in The requirements engineering process concerns the operation now. These VREs focus on a single discipline such investigation and learning about the problem domain in terms as earth science, social science, or life science. A protocol of understanding the actual goals, needs and expectations of guiding and structuring the characterisation of e-RIs was also the users regarding a system [19]. Browne et. al (2001) stated created on the basis of six key types of functional elements in an e-RI defined by the ENVRIPlus project [22]. The questions of these three protocols were created using the In this study, we focused on the research results from the Reference Model of Open Distributed Processing (ODP) [23]. step 1 and step 2 while the results from step 3 and step 4 are The questions covered each of the five ODP viewpoints: beyond the scope of this paper. enterprise (science), information, computation, engineering and technology. The VRE should account for the needs of III. ELICITATION OF MONO-DISCIPLINARY VRE heterogeneous user groups. In addition, the questions REQUIREMENTS concerning activities of VRE users addressed user activities in In order to tackle the global challenges and solve complex line with those mentioned in the literature [24, 25]. scientific problems, scientists need to use VREs as shorthand Step 2: Analysis and negotiation for the tools and technologies. They can conveniently make use of resources and technical infrastructures available both Once an initial set of user requirements has been locally and remotely to conduct their research and to interact formulated, requirements can be detailed, discussed and with other researchers who might be from different countries. agreed by stakeholders in the analysis and negotiation phase, including two main steps: Therefore, VREs need to provide tools and computing resources related to data acquisition, data storage, data 1) Analyse and envision. When analysing and describing processing, and data analysis. According to the e-infrastructure the requirements, it is essential to fully document “the design research project ENVRIplus, VREs should meet requirements element or its interfaces in terms of requirements (functional, and provide six types of functionalities, including data performance, constraints and design characteristics)” [26]. identification and citation, i.e. assigning global unique After describing the requirements, it is also necessary to identifiers to data; data curation, i.e. data quality check; data develop a conceptual prototype to illustrate the requirements cataloguing, i.e. adding metadata to datasets; data processing, and get feedback from the stakeholders. On the basis of the i.e. converting data format and data visualization; data feedback, the requirements are evaluated and may be optimization, i.e. data compartmentalization; and data modified. provenance, i.e. tracking the changes of data. We also add 2) Specification and negotiation. In the analysis step of collaboration, training and support as an additional category in user requirements, the following should be discussed with all this table, since researchers are in need of support related to stakeholders and documented within the specification: finding collaboration for research projects, i.e. finding the identification of the range of relevant users, clear design goals, collaborators with specific expertise, writing grant proposal the requirements with prioritized levels and evaluation criteria and research project management tools, according to to test the requirements whether they will be fulfilled and interviewees #1, #8, and #9. evidence of acceptance of the requirements by stakeholders. According to the interviewees #1, #3, #4, #5, and #9 (als The following methods are used for specification and researchers that are potential VRE end-users), a quickly- negotiation: function mapping, requirements categorisation accessible, reliable, easy-to-use, low-cost VRE is expected. [14]. Therefore, when designing the VRE, it is very important to also consider the non-functional performance-related Step 3: Evaluation requirements defined by commonly used software engineering The evaluation of requirements is to check the consistency standards in FURPS+ and ISO 25010:2011 such as efficiency, and completeness of the requirements [20]. This phase is usability, reliability, maintainability, sustainability, concerned with the examination of the requirement description compatibility and portability. to ensure that it defines the system in an accurate and comprehensive way. In this research, we have used use case In addition, interviewee #6 indicated that all VREs have to analysis, online questionnaire and several workshops with carefully deal with data containing privacy sensitive VRE experts to evaluate the collected requirements. After these information. Therefore, privacy, security, trust and legal activities, we have also designed a VRE system architecture to requirements are necessary to be considered in term of accommodate all collected requirements. regulatory compliance. Privacy and security requirements specify how the use of the VRE should be robust against cyber- Step 4: Evolution management attacks in term of enhanced privacy and security. Trust requirements specify the acceptable behaviours of the Requirements are the starting point for the system design stakeholders in the VRE, such as users, system developers and phase [20]. However, we cannot wait for complete service providers. Legal requirements specify that the whole requirements as the content and the priority of the initial development of VRE should comply with all legislation, requirements may evolve and change during the development especially the new General Data Protection Regulation issued process. Therefore, we also keep track of changes in or new by the European Parliament, the Council and the Commission requirements. These initial requirements have been used to in 2015. define the system functionalities when designing the VRE architecture. During the development of the VRE architecture Research into mono-disciplinary VREs already shows that and analysis of use cases, some additional requirements have many challenges exist for the use of VREs [27], including: data been identified. These requirements are not new but support context issues (understanding the creation context of research requirements identified in step 2. data), data heterogeneity issues (large amount of data generating from various sources), data quality issues (it is not easy to control data quality), privacy issues (datasets containing privacy information need to shared and reused), processing towards desired effects from the viewpoint user experience issues (the expectation of users on the system of data object creator or users; varies), availability of technology issues. Previous research also shows that for multidisciplinary VREs, even more 6) Data provenance requirements which define the needs challenges should be added to this list, since multidisciplinary of making logs on the transformation and VREs need to interoperate between a large variety of computational process on data objects; standards, ontologies and terminologies used by different 7) Collaboration, training and support requirements research disciplines in different countries. According to the which define the needs of providing research information from interviews and the e-RI characterisations, collaboration tools and manuals for using VREs additional challenges concern the availability of data from systems; different sources, data use licensing for different organizations, scalability in terms of connecting High Performance 8) None-functional requirements which define the Computing (HPC) facilities, system management responsibility software performance related objectives such usability, and financial support. Another challenge concerns the access stability; policy. VRE system administrators prefer one certificate per 9) Security, privacy trust and legal requirements which research community in order to lower the effect on user define the needs of system design in compliance with credential management[11], while many organizations cannot all regulation and improvement measures in terms of easily make agreements on sharing certificates to grant access improving users’ overall trust on the VREs. to VREs. TABLE 2 Overview of requirements for a multidisciplinary VRE From the requirements collection work, we have analyzed eight monodisciplinary VREs or e-RIs based on seven Category Requirement example functional requirement categories. These VREs provide Data - Ability to assign (global) unique identifiers (e.g. DOIs, integrated services and datasets and cross-country access to identification ePIC, URIs) to data contents various resources for research. Researchers from the same and citation - Ability to assign an accurate, consistent and research domain can use these resources. Some VREs only standardized reference to a data object, which can be cited in scientific publications provide these services to authorized researchers. Researchers Data Curation - Ability to detect and correct (or remove wrong data from other disciplines or general public cannot easily access - Ability to support manual quality checking some of these VREs. Some VREs are still in the development Data - Ability to associate a data object with one or more phase, although some functionalities in the seven categories are Cataloguing metadata objects which contain data descriptions being designed or already implemented. The usability of these - Ability to select a subset of individuals from within a functionalities significantly varies. These VREs mainly provide statistical population to estimate characteristics of the whole population dataset download and limited data analysis tools. The Data - Ability to convert data from one format to another collaboration, training and support functionalities are largely processing format missing in these VREs and e-RIs. - Ability to inspect, clean, transform data, and to provide data models with the goal of highlighting useful IV. ANALYSIS OF REQUIREMENTS FOR MULTIDISCIPLINARY information, suggesting conclusions, and supporting VRES decision making Data - Large datasets processing In this section we describe the requirements for optimization - Data compartmentalization multidisciplinary VRE collaboration, compared to mono- disciplinary VRE use. Table 2 presents a list of requirements Data - Data Provenance: Ability to provide “pathways of data” provenance or the history of data information (provenance data) for the development of a multidisciplinary VRE, containing - Data publication information: Ability to provide data nine categories of requirements, namely: publication information (e.g. which data was accessed, which data is not accessible, which query was carried out 1) Data identification and citation requirements which and when) define the approaches to provide everlasting and Collaboration, - Notifications: Sending notification when certain unique references to each research data object; training and information becomes available to the users support - Finding collaborators: Ability to locate previous 2) Data curation requirements which define the needs of collaborators and potential collaborators processes to assure the availability and quality of data Non- - Usability object over the long term; functional - Performance efficiency (System - Reliability 3) Data cataloguing requirements which define the Performance) - Maintainability needs of easy and quick access to data objects by - Sustainability queries over catalogues; Privacy, - Specified service authorization contract security, trust - Specific definition regarding software service 4) Data processing requirements which define the needs and legal authorization in compliance with legislation of providing computational transformation software on requirements - Secure storage and use of data, especially data containing privacy information data objects; 5) Data optimization requirements which define the This list contains 148 requirements in 9 categories which needs of providing computational transformation and have been reported in a project deliverable of the VRE4EIC project [21]. Our requirements provide a comprehensive overview of many perspectives that need to be considered When a new VRE becomes available, researchers do not during the development of VREs. The requirement categories just move from one e-RI or VRE which they are already mentioned in Table 2 are also important to monodisciplinary familiar with to another. The process of transferring from one VREs, however, the examples of the requirements themselves VRE to another creates challenges for researchers. Since the are specific to multidisciplinary VREs. usability of existing mono-disciplinary VREs developed based on the requirements in Table 3 significantly varies, the V. DISCUSSION interoperability of those VREs cannot meet the researchers’ We have analyzed eight e-RIs and VREs to understand the demands in multi-disciplinary research. In addition, current practices in the development of VREs. Table 3 showed researchers do not want to spend much time on learning how the implementation of functional requirements in these to use new software or work in a new online environment. projects. From the table we can see that many projects have Therefore, easy access to data from multiple disciplines and to considered data-related requirements while collaboration computing resources are crucial. A portal or gateway requirements are largely ignored. Although the categories of connected with those resources might be suitable to fulfill this the requirements have been covered by many existing VRE task. projects, but the range and details of specific requirements in In multidisciplinary research, researchers desire to use a each categories are not comprehensive enough for developing single tool with an easily understandable Graphical User a multi-disciplinary VRE. Privacy and trust related Interfase (GUI) and plug-and-play features provided by requirements have not been identified in these projects. different VREs or e-RIs to submit their experiment tasks, data TABLE 3 Characterisation of the e-Research Infrastructures and Virtual analysis assignments and to monitor the status. They do not Research Environments. want to know the complexity behind simultaneously running e- these tasks. A powerful workflow engine with an intuitively Requirements Research usable GUI needs to be designed in a multidisciplinary VRE Infrastruc Cat Collabor to integrate several mono-disciplinary VREs. ture / Identifi Cu alo Data Data Data ation, Virtual cation rati gui Proce Optimi Proven training Different research communities use different standards and Research and data models to process research data. Researchers from the on ng ssing zation ance & Environm citation support same research domain can use their own standards and ent practices for data processing. In multidisciplinary research, EURO- researchers need to combine data from different research ● ● ● * * ● ○ ARGO domains with interoperable data processing tools. Our ICOS ● ● ● ● ● ● ○ research showed that they do not want to encounter errors when they put the data in the VRE system. In the interviews, researchers also expressed their concerns related to the control EPOS ● ● ● ● ● ● * of their data if shared with other researchers. They want their ELIXIR ● ● ● ● ● ● ○ work and data to be acknowledged and properly referred to Lifewatch ● ● ● ● ● ● ○ when used by others. In a multidisciplinary VRE stored data and data use are more complex compared to a mono- CESSDA ● ● ● ● ● ● * disciplinary VRE, thus better security mechanisms are required without hindering the ease-to-use of the system. ENVRIPL ● ● ● ● ● ● * In general, we found that there remains a big gap in the US completeness of requirements fulfilled by the existing mono- CLARIN ● ● ● ● ● ● * disciplinary VREs towards multidisciplinary VREs. This is shown in Table 3. A comprehensive multidisciplinary VRE ● Requirements covered should fulfill the requirements described by in the ○ Requirements not covered aforementioned nine requirement categories (see section IV). * Requirements suggested by the project vision or limitedly covered Multidisciplinary VREs need to be developed in terms of A multidisciplinary VRE is ideally open to any researcher. interoperability, a single gateway with an intuitive GUI to In our study, we found that many researchers are already using easily access data and computing resources, and complying some domain-specific resources like research data, software with data protection regulation. tools or e-infrastructures to support their research activities. However, these resources are only known or open to a small VI. CONCLUSIONS research community. Some of the mono-disciplinary VREs we This research aims to 1) study the requirements for developing studied claim to be openly accessible, but they are de facto a multidisciplinary VRE, and 2) investigate the requirement only open to some researchers due to bureaucratic user differences between the current practices of monodisciplinary registration and approval processes. It is very difficult for VRE and the requirements of developing a multidisciplinary researchers from other domains to find these existing VREs VRE. A comprehensive set of requirements needs to be since they are not aware of the VRE development in the considered when developing a multidisciplinary VRE. research domain other than their own science community. Building on the ENVRIplus categorization for e-Research Infrastructure requirements, we categorized functional requirements in the following categories: data identification [7] EPOS. European Plate Observing System. 2018; Available from: https://www.epos-ip.org/. and citation, data curation, data cataloguing, data processing, [8] Vi-SEEM, Virtual Research Environment (VRE) in Southeast Europe data optimization, data provenance, collaboration, training and and the Eastern Mediterranean (SEEM). 2018. support. Non-functional requirements were added to this [9] GenePattern. GenePattern: A Patform for Reproducible Bioinformatics. categorization and collected in the categories of performance- 2018; Available from: related requirements, security, privacy, trust and legal http://software.broadinstitute.org/cancer/software/genepattern#. requirements. [10] McGrath, A., et al., The Essential Components of a Successful Galaxy Researchers’ concerns on losing control of their own research Service. Journal of Grid Computing, 2016. 14(4): p. 533-543. data, lack of interoperability between different VREs and e- [11] Gesing, S., et al., Using Science Gateways for Bridging the Differences between Research Infrastructures. Journal of Grid Computing, 2016. RIs and e-Is, as well as lack of comprehensive consideration 14(4): p. 545-557. of various requirements in monodisciplinary VREs limit their [12] Grunzke, R., et al., Metadata management in the MoSGrid science adoption for multidisciplinary research. When developing gateway-evaluation and the expansion of quantum chemistry support. multidisciplinary VREs, we have to take all these Journal of Grid Computing, 2017. 15(1): p. 41-53. requirements into consideration and choose suitable [13] Yu, E.S.K. Towards modelling and reasoning support for early-phase technologies to meet them. However, we have to admit that requirements engineering. in Proceedings of the 1997 3rd International Symposium on Requirements Engineering. 1997. Los Alamitos, CA, developing VREs is a rather complex engineering process. United States, Annapolis, MD, USA: IEEE. Requirements identified in this study may not be implemented [14] Maguire, M. and N. Bevan, User requirements analysis, in Usability. at once, but fulfilled stage by stage during the development of 2002, Springer. p. 133-148. mature multidisciplinary VREs. [15] Browne, G.J. and M.B. Rogich, An empirical investigation of user requirements elicitation: Comparing the effectiveness of prompting ACKNOWLEDGMENT techniques. Journal of Management Information Systems, 2001. 17(4): p. 223-249. This work was carried out within the VRE4EIC project [16] Aybuke, A. and W. Claes, Engineering and Managing Software and received funding from the European Union’s Horizon Requirements. 2005, Springer-Verlag Berlin. 2020 research and innovation programme under grant [17] Robinson, W.N., S.D. Pawlowski, and V. Volkov, Requirements agreement No. 676247. The authors should like to thank their Interaction Management. ACM Computing Surveys, 2003. 35(2): p. colleagues in this project for their input for this paper 132-190. (particularly Daniele Bailo, Zhiming Zhao, Valerie Brasse, [18] Curtis, B., H. Krasner, and N. Iscoe, A field study of the software design Theodore Patkos, and Jacco van Ossenbruggen for process for large systems. Communications of the ACM, 1988. 31(11): p. 1268-1287. coordinating the interviews). The views expressed in this paper are the views of the author and not necessarily of the [19] Koukias, A., et al. Approach on analysis of heterogeneous requirements in software engineering. in 11th IFAC Workshop on Intelligent VRE4EIC project. Manufacturing Systems, IMS 2013. 2013. Sao Paulo. [20] Parviainen, P., et al., Requirements engineering inventory of technologies. VTT PUBLICATIONS, REFERENCES [21] Yin, Y. and A. Zuiderwijk, State-of-the-art and user requirement analysis. 2016. [22] ENVRIplus. 2018 [cited 2018; Available from: [1] Buddenbohm, S., et al., Success criteria for the development and http://www.envriplus.eu/. sustainable operation of virtual research environments. D-Lib Magazine, 2015. 21(9/10). [23] Linington, P.F., et al., Building Enterprise Systems with ODP. An Introduction to Open Distributed Processing. 2011, Washington: [2] Susha, I., M. Janssen, and S. Verhulst. Data collaboratives as a new Chapman & Hall/CRC Press. frontier of cross-sector partnerships in the age of open data: Taxonomy development. in Proceedings of the 50th Hawaii International [24] Buddenbohm, S., et al., Success Criteria for the Development and Conference on System Sciences. 2017. Sustainable Operation of Virtual Research Environments. D‐Lib Magazine, 2015. 21(9/10). [3] Zuiderwijk, A., et al., Using Open Research Data for Public Policy Making: Opportunities of Virtual Research Environments, in Conference [25] De Roure, D., C. Goble, and R. Stevens, The design and realisation of for E-Democracy and Open Government. 2016: Krems an der Donau, the Virtual Research Environment for social sharing of workflows. Austria. Future Generation Computer Systems, 2009. 25(5): p. 561-567. [4] Crosas, M., The dataverse network®: an open-source application for [26] IEEE-STD, ISO/IEC Standard for Systems Engineering - Application sharing, discovering and preserving data. D-lib Magazine, 2011. 17(1): and Management of the Systems Engineering Process. ISO/IEC 26702 p. 2. IEEE Std 1220-2005 First edition 2007-07-15, 2007: p. c1-88. [5] Edwards, P., et al., Lessons learnt from the deployment of a semantic [27] Zuiderwijk, A., et al. Using Open Research Data for Public Policy virtual research environment. Web Semantics: Science, Services and Making: Opportunities of Virtual Research Environments. in 2016 Agents on the World Wide Web, 2014. 27-28: p. 70-77. Conference for E-Democracy and Open Government (CeDEM). 2016. [6] EVER-EST. EVER-EST. 2018; Available from: www.ever-est.eu.