Building a monolithic data marketplace is challenging due to complex inter dependencies, leading to cumbersome and error-prone development where a single failure can disrupt the entire system. To address this, we propose a modular approach using dynamic plugins in the UPCAST project. Our flexible framework allows components to be activated or deactivated as needed, enhancing scalability and resilience. By decoupling functionalities into interchangeable modules, we mitigate the risk of single points of failure, simplify maintenance, and facilitate customization for more robust marketplace solutions.
Data sources are annotated as a dcat:Dataset, with the data model designed as a knowledge graph using both the DCAT2 and UPCAST vocabularies. Users initiate this plugin to specify the details required to create a new resource. The creation of a new resource involves the following sub-procedures: • Import UPCAST vocabulary and domain-specific vocabulary in machine-readable format; • Define metadata of the resource; • Define access and usage policies of the resource; • Assign energy profile to the resource that will be used to optimise the environmental impact; • Associate price to the resource for further negotiations; • Create resource profile/summary. CEUR
ceur-ws.org
The data profiling service generates a profile for a dataset using a specified profiler given dataset metadata, sample data or the whole dataset, and other supplementary materials. A number of profilers can be connected to provide the “plug-and-play” profiling service according to the needs and requirements of the user, for example, profilers that give statistics on the dataset or provide semantic information about the data. In UPCAST, the main purpose of the profiling service is to enhance the representation of data to improve data discoverability, in particular, through semantic profiling.
The Resource Discovery Plugin acts as an intermediary, facilitating the retrieval of resource specifications.
Resource consumers can request and retrieve information from the available resources provided by
various providers. While searching the knowledge base, users may also find similar sources through
semantic similarity search[
After the resource specification, the resource provider defines constraints on the resources using Open Digital Rights Language (ODRL)3 rules, leveraging both the UPCAST and domain-specific vocabularies. On the other hand, the resource consumer specifies the intentions via a Data Processing Workflow (DPW) specification and outlines any organisation-specific access and usage control rules, as well as rules prescribed by applicable regulations (e.g., GDPR). Subsequently, conflict identification occurs between the provider’s constraints, the consumer’s intentions, and internal rules, making the derivation of authorisation decisions possible. The functionalities of the plugin can be summarised as below: • Transform the resource provider constraints to privacy and usage control rules; • Define rules for the resource consumer; • Manage rules; • Identify conflicts between the provider’s constraints and the consumer’s intentions; • Access and usage decision making.
Often, the processing intentions of a data consumer for a dataset of their interest difer from what the
data provider is willing to allow. These diferences may include the purpose of the processing, the
time interval for which the provider is willing to allow access, or the price to pay. Nevertheless, these
diferences are not necessarily irreconcilable, and both parties can often reach an agreement through
negotiation [
Upon the initiation of a negotiation process, the plugin provides a centralised platform for discussing terms, pricing, and specifications, allowing users to track, and finalise negotiations seamlessly. Moreover, the plugin incorporates robust contract management features, allowing users to create, review, and execute contracts with ease. By automating routine tasks and ofering customisable Data Processing Workflows (DPWs), it enhances data sharing while ensuring compliance with regulatory requirements.
The provider will ultimately decide the negotiation’s outcome by agreeing, rejecting, or sending
another counterofer. The result of a successful negotiation process is a data sharing contract [
The contract generation process within the UPCAST plugin is significantly enhanced by the integration of Large Language Models (LLMs). These advanced AI models facilitate the automatic generation of comprehensive and precise contracts based on the negotiation outcomes. By analyzing the details of the negotiation, including usage policies, pricing structures, and specific data processing requirements, the LLM can draft contracts that accurately reflect the agreed terms. This automation not only speeds up the contract creation process but also reduces the risk of human error and ensures that all legal and regulatory aspects are meticulously addressed. The LLM’s ability to understand and generate natural language makes it an invaluable tool for creating clear and enforceable contracts, thereby streamlining the entire negotiation and contracting workflow within the UPCAST platform. 4https://internationaldataspaces.org/ 5https://w3c.github.io/dpv/dpv/
In conclusion, this paper has introduced a modular approach to building data marketplaces, addressing the challenges posed by traditional monolithic systems. By utilising dynamic plugins within the UPCAST project, our solution provides a flexible framework that enhances scalability, resilience, and ease of maintenance. The decoupling of functionalities into discrete modules mitigates the risk of single points of failure and allows for tailored customisation to meet specific marketplace needs. This approach not only simplifies system upgrades and maintenance but also ensures robust and adaptable data marketplace solutions, demonstrating significant advantages over conventional monolithic designs.
This work was funded by the UKRI Horizon Europe guarantee funding scheme for the Horizon Europe projects UPCAST (101093216101093216) and RAISE (101093216101058479).