SoBigData RI has the ambition to support the rising demand for cross-disciplinary research and innovation on the multiple aspects of social complexity from combined data and model-driven perspectives and the increasing importance of ethics and data scientists' responsibility as pillars of trustworthy use of Big Data and analytical technology. Digital traces of human activities ofer a considerable opportunity to scrutinize the ground truth of individual and collective behaviour at an unprecedented detail and on a global scale. This increasing wealth of data is a chance to understand social complexity, provided we can rely on social mining, i.e., adequate means for accessing big social data and models for extracting knowledge from them. SoBigData RI, with its tools and services, empowers researchers and innovators through a platform for the design and execution of large-scale social mining experiments, open to users with diverse backgrounds, accessible on the cloud (aligned with EOSC), and also exploiting supercomputing facilities. Pushing the FAIR (Findable, Accessible, Interoperable) and FACT (Fair, Accountable, Confidential, and Transparent) principles will render social mining experiments more eficiently designed, adjusted, and repeatable by domain experts that are not data scientists. SoBigData RI moves forward from the simple awareness of ethical and legal challenges in social mining to the development of concrete tools that operationalize ethics with value-sensitive design, incorporating values and norms for privacy protection, fairness, transparency, and pluralism. SoBigData RI is the result of two H2020 grants (g.a. n.654024 and 871042), and it is part of the ESFRI 2021 Roadmap.
In particular, the currently active project is SoBigData++ with the objective of building
efective national research systems within a federated European platform by planning, designing,
and integrating services in a research infrastructure called SoBigData RI. The aim is to allow users
to access methods and algorithms provided by RI as services in the cloud of e-infrastructure to be
executed using the SoBigData RI computational resources. This aspect will follow the concepts
of meta-modelling as well as FAIR and FACT principles. Optimal transnational cooperation
and competition: this aspect will be granted in 13 European countries, ofering world-leading
research expertise from multiple disciplines, as well as Big Data computing platforms, big social
data resources, and cutting-edge computational methods [
In the following sections, we will describe the diferent pillars of the SoBigData RI, starting from the Exploratories, which are thematic virtual laboratories where the researchers study the complexity of social phenomena and collect data to produce new methods and algorithms which are published in the RI Catalogue to be shared with the community or executed in the cloud computational resources ofered by the platform.
SoBigData RI has made a relevant design choice for creating its e-infrastructure and community
by introducing the concept ’exploratory’, i.e., a vertical social mining research environment
focused on specific societal challenges. The aim was to exemplify the cutting-edge,
multidisciplinary research supported by the RI and drive the integration into the e-infrastructure of
the resources available within the national laboratories [
Demography, Economy and Finance 2.0 is devoted to the study of traditional, complex socio-economic and financial systems as well as emerging ones (e.g., block-chain and cryptocurrency markets). The exploratory aims to extend the existing approaches grounded on statistical physics for the analysis of real-world (economic and financial) networks by considering innovative models to analyze the dynamics of and on networks and infer their structural details from partial information extending the rich formalism of entropy-based null models towards the development of non-linear Exponential Random Graphs and employ renormalization methods.
Migration Studies tries to estimate flows and stocks from available, real-time data, building models that track indicators extracted from unconventional and oficial data sources. This exploratory evaluates migrants’ integration in new communities through social network and retail data analysis since migration can generate cultural changes in the local and incoming population.
Social Impact of AI and explainable ML studies how the AI can replicate/simulate/infer people’s behaviours in the field of computational social choice. This interdisciplinary research area deals with the aggregation of agents’ preferences to reach a consensus decision that achieves some social objectives by finding "simpler" ways to explain this to humans.
Societal Debates and Misinformation uses the state-of-the-art algorithms for supervised prevalence estimation that can track collective sentiment even when expressed on fine-grained ordinal scales. We analyze discussions using data from online social networks to investigate topics relevant to society and phenomena such as opinion polarization and echo chambers.
Sports Data Science analyses the way scientists, fans, and practitioners conceive sports performance is changing rapidly due to the proliferation of new sensing technologies that provide reliable data streams extracted from every game. Combining this flow of (big) data with robust data science and AI tools, we now have the chance to unveil the great complexity underlying sports performances and work towards many challenging goals, from automatic tactical analysis to data-driven performance ranking, from game outcome prediction to injury forecasting.
Sustainable Cities for Citizens develops innovative methodologies to help Italian municipalities and other decision-makers in producing policies and strategies for urban sustainability and improve the environmental performance concerning digitization, energy, water, and waste management and pollution. The methodologies are also directed to citizens to be used as a tool to increase their sustainability awareness. This exploratory also focuses on analyzing human mobility using mobile phone traces, vehicular GPS, and social media data are all Big Data sources and proxies of individual and collective behaviours.
Network Medicine aims to create a new approach to biomedical science, combining principles and approaches from systems biology and network science to both understand the causes of human diseases and find and develop new personalized treatments. This exploratory also studies and develops new compressed indexes and algorithms for storing, indexing, and mining knowledge graphs and key-value stores that are the backbone of modern Computational Biology platforms.
Disaster response and recovery focusing on the research of methods and tools to analyze, monitor, and improve post-disaster reconstruction processes in socio-economic areas, spatial planning, and environmental health in cooperation with national and international institutions. ICT-enhanced solutions to the disaster management cycle’s response and recovery phases can improve the eficacy of "search & rescue" activities, emergency relief, reconstruction, and rehabilitation. This exploratory was explicitly created to tackle the problems with a strong connection with the local institution in the area of L’Aquila; for this reason, it has a dedicated access point1).
Exploratories had and still have a significant impact on the use of the platform, both in terms of the number of users and experiments carried out; they can be seen as think tank producing research, dataset, and algorithm which are refined and ofered by the RI.
SoBigData RI adheres to the EU vision on Responsible Research and Innovation and
operationalizes values driving the ongoing reform of the EU Data Protection and Fundamental Rights
legislation [
This can be demanding in an environment where regulations tend to change. The fact that the potential of technology is not always captured in time by the legislative system can result in gaps that need to be anticipated and, if possible, prevented. We are facing a continuous evolution of the ethical-legal framework. After GDPR, many legislative initiatives are developing new ethical-legal dimensions impacting data processing for statistics and scientific purposes and innovation.
There are numerous challenges to be faced. Since many private companies and industries have dificulties including and exploiting ELSEC principles in their project, we can act as facilitators showing how to include them in practice (as already done in SoBigData). Another fundamental challenge is the potential conflict of interest that can arise. In some cases, there may be a tension between the requirement to operate ethically and the need to develop and market as many services as possible, to as many users and as many diferent companies as possible.
In this challenging scenario several actions are put in place: 1) the design of a pipeline for data science experiments and services that adheres to the legal-ethical principles but is also efective; 2) an Ethical and legal board (with experts in IT law, IT ethics and data protection) which accommodate within-infrastructure ethical issues as well as will aim at answering new questions on the scope, interpretation, and application of the GDPR along with the expanding role of AI, machine learning and data mining; 3) white papers on ethical issues from actual cases and practical solutions for social science; 4) a variety of privacy-enhancing and discrimination discovery algorithms; 5) guidelines for legal, ethical, methodological and infrastructural issues arising from working with social data, to help scientists to focus on their research; 6) clear policies and agreements for the sharing of resources according to the Privacy and Ethic EU regulations; 7) tools that enable technologies and ethical values transfer to industry and PA, which are not always aware of the responsible data science pipeline, and show how to exploit its full potential in a sustainable way for a long-term added value.
In this section, we describe the services ofered by the RI, which are designed for diferent
stakeholders and attract and foster a solid and active user community[
The leading virtual access service is the RI catalogue which contains all the metadata of the resources, and it is accessible through the web interface. To access existing resources of the e-infrastructure, the user must log in to perform a free registration or use any academic/social credentials supported by the EOSC. The access to the SoBigData RI also grants access to the Catalogue, SoBigData Lab, and the e-learning area. The SoBigData VA services have the main entry point at the site: www.sobigdata.eu. The gateway of the SoBigData RI publishes all the services related to e-infrastructure organized into six main areas: 1) the catalogue enables the user to search an item given a set of keywords; 2) SoBigData Lab where users can execute methods on the e-infrastructure; 3) the description ant the link to of all the applications available; 4) the e-learning area that enables the user to access all the training material related to the SoBigData community; 5) the portal to access to our HPC facilities, and 6) on the left the work space, an online environment to support secure and controlled data storage and sharing. The catalogue is the primary tool for discovering and searching for an item inside the SoBigData RI. All the elements inside the SoBigData RI are discoverable through this service. The user can insert a set of keywords, and the list of the results will be visualized. The search result lists items included in the catalogue and their classification (e.g., Method, Training Material, Dataset). The complete description is provided on the dedicated page, accessible by clicking on the item. These features can be added to the search filter, which will be recalculated in real-time. The search result can be sorted alphabetically concerning the insertion date or popularity. The SoBigData Lab integrates diferent methods that can be invoked under the same environment through SoBigData e-infrastructure. A method is the implementation of an algorithm/procedure or is an algorithm that requires an engine to be executed. Diferent kinds of integration are available based on the programming language in which a method is implemented. Once a method is integrated into the platform, the final user has a homogeneous web form for inserting parameters and invoking it independently from the programming language employed. In particular, JupyterHub is easily accessible by clicking on the link on the top of SoBigData Lab VRE. After starting the server by selecting one of the default profiles available, the user can start to use Jupyter Notebook as the local version. Several services are ofered in the form of SoBigData libraries, consisting of a set of thematic methodologies available on the cloud and usable both in a JupyterHub environment or in the SoBigData engine and applications.
TA visit is a powerful tool to drive to share knowledge and expertise throughout Data Science. By benefitting from a SoBigData TA visit, the researcher is also guided and supported to become a responsible and ethical Data Scientist; this, in turn, will promote one of our primary objectives of creating and maintaining a trusted Research Infrastructure with a highly respected status in the field. TNA, by definition, shares and disseminates expertise from many of the partners of our community. The goal is to provide researchers and professionals with access to big data computing platforms, big social data resources, and cutting-edge computational methods. A TNA visitor will interact with the local experts, discuss research questions, run experiments on non-public big social datasets and algorithms, and present results at workshops/seminars.
The RI expertise ofer covers multiple and wide-ranging skills and knowledge. The exploratories described in Section 2 provide areas that pull together the strands of the RI across lateral and vertical thematic environments for supporting TNA visits. The RI can now ofer a visit to one of 17 European SoBigData nodes providing wide-ranging and varied expertise. The visit is a short-term scientific mission (STSM) between 2 weeks and 2 months, covering the travel and living expenses costs2.
2http://www.sobigdata.eu/content/call-2022-23-sobigdata-transnational-access
The technological model of SoBigData RI has the form of a system of systems considering
the principles of autonomy of constituents (independence and evolution), openness (join
and leave; dynamic reconfiguration), and distribution (interdependence and interoperability).
The cited principles represent the building blocks of the policies governing the rules of
participation of the national nodes. From the technical point of view, the RI is a hyper-converged
infrastructure, including both the storage area network and the underlying storage abstractions
are implemented virtually in software rather than physically in hardware [
The paper presented the main impacts and services that SoBigData RI has in the research community, mainly related to social mining and AI. Our ambition is to update our repertoire of social mining continuously and AI methods (from knowledge extraction, multilingual text classification, and enhanced privacy technology to federated machine learning) with cuttingedge techniques spurring from partners’ research activities or wrapping useful new openlyavailable methods. In this perspective, Figure 2 completes the SoBigData RI description providing some numbers related to our activities from 2015 to now. At the moment (March 2022), we have more than 9000 registered users, we fully support more than 80 TNA visits, and we trained more than 750 in our courses. We organized events disseminating and organizing workshops on thematic areas, e.g., Special Action on Gender, and with the presence at European Parliament.
In the next future, we think that SoBigData RI will contribute to the creation and maintenance of local ecosystems that go in the direction of dynamic and open data spaces aligned with the ambition of the European strategy for data.
This work is supported by the European Union – Horizon 2020 Program under the scheme “INFRAIA-01-2018-2019 – Integrating Activities for Advanced Communities”, Grant Agreement n.871042, “SoBigData++: European Integrated Infrastructure for Social Mining and Big Data Analytics” (http://www.sobigdata.eu).