This vision paper introduces a pioneering data lake architecture designed to meet Life & Earth sciences' burgeoning data management needs. As the data landscape evolves, the imperative to navigate and maximise scientific opportunities has never been greater. Our vision paper outlines a strategic approach to unify and integrate diverse datasets, aiming to cultivate a collaborative space conducive to scientific discovery. The core of the design and construction of a data lake is the development of formal and semi-automatic tools, enabling the meticulous curation of quantitative and qualitative data from experiments. Our unique "research-in-the-loop" methodology ensures that scientists across various disciplines are integrally involved in the curation process, combining automated, mathematical, and manual tasks to address complex problems, from seismic detection to biodiversity studies. By fostering reproducibility and applicability of research, our approach enhances the integrity and impact of scientific experiments. This initiative is set to improve data management practices, strengthening the capacity of Life & Earth sciences to solve some of our time's most critical environmental and biological challenges.
quire massive amount of data, even in continuous mode.
This has been no diferent for experimental and observational sciences like Life & Earth sciences. Accessibility to data about the Earth and its biodiversity, with varying levels of provenance, quality and reliability, opens up the possibility of constructing diferent perspectives on the phenomena observed, leading to scientific conclusions with diferent depths that target a wide range of knowledge consumers (civilians, decision-makers, scientists).
Traditional schema-on-write approaches, such as the Extraction, Transformation and Loading (ETL) process, are inefective for the data management requirements of these experimental sciences. Data lakes are becoming increasingly common for the management and analysis of massive data. Data lakes are repositories that store raw data in its original format. They can be well adapted for storing data harvested from digital sources (observation stations), social media, Web and in situ collectors.
The extraction of value through data-driven experiments in the Life & Earth sciences is determined by two main elements: • The maintenance of metadata gathering the conditions under which experiments are performed (quantitative perspective) to preserve the memory of the experimental process of knowledge production process, and to enable understanding and reproducibility. • An open science perspective that can go beyond data sharing and must consider the sharing of know-how, decision-making, expertise, project with the ability to collect data based on specific character management, and people within the projects that sequences. 80legs2 ofers sequential data extraction from define the research must be considered. websites. Octoparse3 simplifies the data extraction process by enabling users to create a scraping workflow with
This vision paper introduces our approach to designing clicks. It includes features like URL and string lists for and building a data lake for collecting and integrating targeted scraping and ready-to-use templates for popular data and meta data of Life & Earth sciences’ data-driven sites like Amazon and Google. FactExtract [3] is tailored experiments. for aggregating content from specific Senegalese news
The remainder of the paper is organised as follows. sources, boasting automatic language detection for ten Section 2 gives a general overview of approaches that ad- languages, data cleaning, and analysis, all whilst avoiddress curating and managing knowledge in Life & Earth ing data duplication. This tool, which utilises Python’s sciences. Section 3 describes the challenges associated Newspaper library, also features automated daily updates with curating data and data-driven experiments in Life & for the news content it monitors. ENoW - News Data ExEarth sciences often guided by researchers. In particular, tractor from the Web4 is a news scrapping system that exthe section gives the general challenges for building data plores online newspapers. ENoW receives search strings lakes containing curated data and producing knowledge as input and stores in a relational database data extracted derived from data-driven experiments. Section 4 intro- from the news and their full content. duces the general principle for building, maintaining and exploiting a data lake. The data lake allows the creation 2.2. Data curation of "dataverses" that can export the history of the development of experimental processes that lead to knowledge According to Garcov et al., [4], research data curation in Life & Earth sciences. Finally, Section 5 concludes the is “preparing research data and artefacts for sharing and paper and discusses future work. long-term preservation”. Research repositories are the standard for publishing data collections to the research communities. Datasets at an early collection stage are 2. Related work generally not ready for analysis or preservation. Thus, We introduce the main topics and approaches that un- extensive preprocessing, cleaning, transformation, and derline the vision of maintaining and sharing data to documentation actions are required to support usability, perform data-driven experiments: data harvesting tools, sharing, and preservation over time [5]. Curated data data curation techniques, data labs, data lakes, science collections have the potential to drive scientific progress lakes and dataverses. [6], are relevant for reproducibility and improve the reliability of sciences [7]. However, data curation introduces challenges for supporting data-driven applications [8] 2.1. Data harvesting adopting quanti-qualitative methods. For example, reData available on the Web play a determining role in search challenges curating material across time, space decision-making in personal and corporate life. Collect- and collaborators [7]. Quantitative and qualitative reing and storing this data in a structured model helps inte- search methodologies apply ad-hoc data curation strategrate them with other sources and use the dataset in var- gies that keep track of the data that describe the tools, ious applications, such as event detection and sentiment techniques, hypothesis, and data harvesting criteria demonitoring. Online newspapers are essential sources of ifned a priori by a scientific team. information, accessed daily by thousands of people. Several software tools that apply statistical techniques
Various works in the literature report manual eforts to and machine learning algorithms are available for qualiextract data from pages on the Web [1, 2]. However, these tative researchers. Woods et al. [9] argue that Computereforts have been eased by applying Web scraping tech- Assisted Qualitative Data Analysis Software (CAQDAS) niques. Some work complements automated extraction is a well-known tool for qualitative research. These tools processes to obtain clean and analysed data by imple- support qualitative techniques and methods for applymenting curation procedures [3]. Among the various ing Qualitative Data Analysis (QDA). ATLAS.ti [10], Deexisting tools available on the Web for data extraction, doose [11], MAXQDA [12], NVivo [13] implement the we can highlight ParseHub1 is a web scraping tool that REFI-QDA standard, an interoperability exchange forfacilitates data extraction from websites through an interactive click-based interface, saving the data directly to the cloud in JSON and CSV formats. It navigates through continuation pages and captures complete news articles,
3https://www.octoparse.com/ 4L Reips, M Musicante, G Vargas-Solar, ATR Pozo, C.S Hara, ENoWExtrator de Dados de Notícias da Web, Demonstration Anais Estendidos do XXXVIII Simpósio Brasileiro de Bancos de Dados, 2023, 78-83
mat. CAQDAS [14] researchers and practitioners can thereby facilitating the discovery of high-quality data perform annotation, labelling, querying, audio and video across diferent scientific fields. transcription, pattern discovery, and report generation.
Furthermore, CAQDAS tools allow the creation of field 2.4. Data lake, science lake and dataverse notes, thematic coding, search for connections, memos (thoughtful comments), contextual analysis, frequency Data lakes are expansive storage repositories that hold analysis, word location and data analysis presentation vast raw data in their native format until needed. Stein in diferent reporting formats [ 15]. The REFI-QDA (Rot- and Morrison [20] emphasised their potential for scalaterdam Exchange Format Initiative)5 the standard allows bility and flexibility in handling big data from various the exchange of qualitative data to enable reuse in QDAS sources. In recent studies, Dixon in 201010 defined the [16]. QDA software such as ATLAS.ti [10], Dedoose [11], term and its initial application in big data analytics. Quix MAXQDA [12], NVivo [13], QDAMiner [17], Quirkos et al. (2016) [21] delved into the architectural consid[18] and Transana [19] adopt REFI-QDA standard. erations and challenges such as data governance and
We assume that data curation consists of identifying, metadata management. systematizing, managing, and versioning research data, Science lakes, an ofshoot of data lakes, are tailored considering versioning artefacts an essential component specifically for the scientific community to address the of tracking changes along the research project. need for interdisciplinary research, data management and complex analytics. Russom (2016) [22] suggested 2.3. Data labs that science lakes provide a more discipline-specific approach to data handling, enabling better metadata curaData science environments provide data labs like Kag- tion and domain-specific data models, which are crucial gle6 and Dryad7 with stacks of services for (externalised) for reproducibility in scientific research. data storage, tagging and exploring tools. These environ- A data lake is a vast storage system that houses extenments allow a collective sharing space of highly curated sive volumes of raw data in its original format. This verdata collection maintenance tools. There are specialised satile system accommodates a range of data types, includrepositories like DataOne8 and data repositories re3data9. ing structured, semi-structured, and unstructured forms.
DataONE (Data Observation Network for Earth) is a Data lakes are essential in environments focused on big community-driven project that provides access to various data analytics and are designed to manage data characenvironmental and ecological data across multiple mem- terised by large volume, high velocity, and diverse variety ber repositories. It is designed as an innovative frame- from multiple sources. They are commonly utilised for adwork aimed at facilitating research and enabling scien- vanced data processing activities such as machine learntists and researchers to preserve, access, use, and increase ing and predictive analytics. Unlike traditional databases the impact of their data. The platform provides robust following the schema-on-write approach, data lakes foldata management tools, ensuring datasets’ preservation low the schema-on-read approach, providing flexibility and integrity. DataONE underscores data stewardship in how data is formatted and used. as a federated resource and supports scientific collaboration and reproducibility. It is invaluable for researchers Dataverse. The concept of dataverse takes the noseeking to address complex environmental challenges tion of data lakes further by creating a networked space through shared data and knowledge. where data is stored, actively managed, and shared within
Re3data is a global registry of research data reposito- the scientific community. A dataverse is a data reposries that ofers a comprehensive directory for researchers itory platform for publishing, citing, and discovering seeking to access, store, share, and manage their datasets. datasets. It enables researchers to publish, cite, and disIt represents a variety of academic disciplines and pro- cover datasets while providing metadata and tools to vides detailed information about each repository, such ensure others can understand and use data. Dataverses as access policies, standards, and contact details. re3data are often domain-specific and support the principles of promotes data sharing, visibility, and reuse as a critical open science, providing features such as data version reference point for finding suitable repositories for data control, digital object identifiers (DOIs) for citation, and deposition. The platform enhances transparency in re- tools for data analysis within the platform. They are search data management. It supports open science by community-driven and emphasize the accessibility and guiding users to trustworthy and reliable repositories, reusability of research data.
The most prominent example is the open-source
Dataverse project developed by the Institute for
Quantitative Social Science at Harvard University. The Dataverse
5https://www.qdasoftware.org
6kaggle.com
7https://datadryad.org/stash
8https://www.dataone.org/about/
9https://www.re3data.org
Project, initiated by King [
Diferent academic institutions have built their data- to and draw from a collective data pool. For instance, a verses for sharing and disseminating experimental sci- dataverse in these fields might include a combination of entific results, including the data collections they curate: high-throughput experimental data, field observations, University of Arizona11, the Diferent universities and and simulation outputs. The combination of openness academic institutions have promoted their dataverses and rigorous data management positions dataverses as like the University of Hamburg12, the University of Michi- critical resources in pursuing scientific discovery in Life gan13 and the Grenoble Dataverse14. & Earth sciences.
In life and earth sciences, data lakes are pivotal for conSummary. Together, these systems represent a shift solidating scientific data collected from various biodivertoward more open, integrated, and eficient ecosystems sity studies and geological events like earthquakes. Once for data management, ofering novel solutions to the curated, processed, and analysed, this data contributes challenges posed by the vast amounts of data generated significantly to data-driven experiments underpinned by in modern research. They move away from traditional well-established protocols. The harvested data enriches databases and toward more fluid, dynamic systems that the data lake and supports the creation of detailed, cucan accommodate the ever-changing landscape of big rated views for dissemination through dataverses. data and scientific research. Our vision emphasises the importance of developing
A dataverse and a data lake are concepts related to data and maintaining data lakes with partially curated constorage and management but serve diferent purposes and tent in life and earth sciences, facilitating the continuous are designed with varying cases of use in mind. While cycle of experimental data feeding back into the lake and a dataverse is a scholarly platform aimed at curating, subsequently sharing via dataverses. sharing, and preserving research data with rich metadata and community collaboration features, a data lake is a more generalised and scalable storage solution for raw data to support diverse data analytics and processing workflows.
2.5. Data lakes and data verses in Life & Various data on life and earth sciences have been
acEarth sciences quired from diferent sources [
In Life Sciences, dataverses often focus on genomics, experiments carried out on the data, their results and the
proteomics, clinical trials, and other biological data, in- conditions under which they were carried out.
Maintaintegrating various sources of information to aid in com- ing a catalogue of data-related questions and experiments
plex analyses like phenotype-genotype correlations. For can promote open science, share data and knowledge, and
Earth Sciences, dataverses might concentrate on geospa- share the data and knowledge the scientific community
tial data, climate models, seismic activity records, and has gained from it [
Then, algorithms (statistical and Artificial Intelligence) and researchers can process, filter and classify data. This ifltering process produces and stores meta-data in the data lake. Data exploration and integration (cleaning and engineering) processes can be performed on data samples from the data lake. They can be used for experimental purposes to produce content associated with the data stored in the data lake. Clean and curated data associated with meta-data representing the quantitative and qualitative perspective of the experiments can then be shared in a data verse (right-hand side of the figure).
Challenge 2: How to integrate data in the data lake?
Since the experiments require several data collections,
integrating the data into the data lake must be part of
a pipeline that includes data discovery, exploration,
selection and integration. This process should be designed
based on the requirements of life and earth science
experiments [
Challenge 3: How to integrate data in the data lake
considering scientists’ needs? The researcher’s
intervention, defined as a researcher-in-the-loop (RITL)
[
Scientific content should be extracted and computed,
including data, analytics tasks (manual and AI models),
and associated metadata. This curated content allows the
produced knowledge to be reusable and analytics results
to be reproducible [
Curation, maintenance, and exploration of data collections for bringing data value from in situ observations and experiments. Since data acts as a backbone in modelling phenomena for understanding their behaviour, it is critical to developing good collection and maintenance: which are available data collections? Are they complete? Which is their provenance? In which conditions were they collected? Have they been processed? In which cases have they been used, and what are the associated results? We propose techniques to explore data collections using graphs that can be explored and enriched while new data and analytics results are produced. Data curation also means keeping track of the type of experiments run on data, their results, and the 15https://hai.stanford.edu/news/humans-loop-design-interactive-ai-syscteomnsditions in which they were performed. Maintaining a catalogue of data-related questions and experiments can promote open science and share data and the knowledge that the scientific community has derived from it.
Modelling and simulating experiments to answer questions in life and earth sciences. Answering research questions through data-driven experiments implies: • Designing ad hoc experiment artefact models and programming languages for enabling friendly, context-aware, and declarative construction of experiments in life and earth sciences. • Collecting execution of experiments data (raw input data, prepared datasets, experiments’ tasks calibration and associated results).
Pilot experiments. The data lake will be tested in real scenarios through collaboration with domain experts in seismology and biodiversity studies in Brazil. The entry point will be two pilot experiments, namely: 1. the classification process of seismic signals collected by stations through diferent observations to detect "natural" and human-made earthquakes in the northern human-made earthquakes in the northern region of Brazil; 2. the classification of in situ observations of the "carabela portuguesa"16 and modelling its behaviour on the Brazilian coast. 16The Portuguese caravel (Physalia physalis) is a monotypic colonial species of siphonophore hydrozoan of the family Physaliidae. It is commonly found in the open ocean in all warm waters of the
In both cases, it is necessary to (i) apply statistical methods to investigate and unveil new patterns in seisms and biodiversity data, answering open problems or leading to new research questions; (ii) build predictive models to better describe or approximate phenomena, increasing the knowledge about our planet. The conditions in which statistics and prediction are performed, results, observations, interpretation and validation of the results are data to be integrated into the data lake.
Discussion. The originality of the work is to address the construction of a data lake that includes: 1. Raw collected data representing life and earth sciences phenomena (streams, batch, multimedia, proprietary). 2. Data produced along data-driven experiments adopting data science techniques including artificial intelligence algorithms (ML-driven data lakes). 3. Contextual data describing the conditions in which data are collected, and experiments are designed and enacted. The data lake will provide data curation modules for extracting metadata according to a well-adapted model and modules exploring data and using them for designing new experimentations, thereby adopting an open science perspective. world, especially in the tropical and subtropical regions of the Pacific and Indian Oceans, as well as in the Atlantic Gulf Stream.
Its sting is dangerous and very painful https://es.wikipedia.org/ wiki/Physalia_physalis.
the LETITIA17 project, funded by the Fédération Informatique de Lyon18.
The contribution is the design and exploration techniques of a data lake with a well-adapted model for metadata about life and earth sciences experiments consuming and producing quantitative and qualitative data. An important work will be to define exploration operators and pipelines to exploit the content for further maintaining and developing new life and earth sciences experiments.