This paper presents an index-time merging approach to federated search designed for a European Connected Factory Platform for Agile Manufacturing (EFPF) project and its federated digital ecosystem. The presented approach supports search and recommendations for products, services as well as suitable business partners for the creation of networks within the business ecosystem. We discuss the overall architecture of the search and recommendation system, major challenges and the current state of development. Federated search, recommendation, digital platforms, platform search Federated search systems emerged in the late 1990s in order to support library users to simultaneously search multiple library sources and catalogues [1]. Early federated search systems were designed to collect search criteria from users, submit search queries to sources to be retrieved, wait for a response to arrive, and filter and organize the query results before sending the results back to the users. For example, Windows 7 services differentiate between two types of search, Windows Search and Federated Search [2]. Windows Search enables searching of the entire local system and a limited amount of external resources such as shared drives. Likewise, the selection-based search in MacOS Spotlight requires an index of all items and files that exist in the system to be created.
2020 Copyright for this paper by its authors.
The authors in [3] differentiate between federated search and metasearch. Metasearch engines crawl open websites and return results of all kinds, such as apps, webpages, contacts and documents that are drawn from different sources. In contrast to metasearch, federated search systems focus on subscribed databases that are processed through an indexing mechanism that enables structured, in-depth, contentoriented retrieval [4]. A number of attempts to implement federated search arose from the 1990s onwards, e.g. MetaLib and SFX implemented at Boston College [5], MetaLib and SFX with Ex Libris at the Five Colleges Libraries in Massachusetts [6], Georgia Library Learning Online (GALILEO) with its hybrid approach to federated search products [7], and more. Many interviews showed that the users were dissatisfied with federated search performance of that time, due to limitations in database availability, search speed, retrieval precision, and result comprehensiveness [8]. Users preferred simple interfaces over federated search and had little interest in advanced searching techniques. In recent years, several attempts to improve federated search were conducted. In 2004, the University of Nevada at Reno experimented with the Google Search Appliance to be used as a federated search utility. This required partnership between Google, the library and a test vendor, and it still experienced technical and interoperability challenges [9].
Federated search is a technique for simultaneous search of multiple data sources using a single query and search interface. The examples of federated search can be found in many sectors, facilitating search of thousands of products, each sorted into different categories, e.g. search of numerous websites serving various purposes and stakeholders [11]. The majority of the research on federated search is focused on system performance and technical development, system usability testing, user interaction, interface customization, authentication, design, database communication protocol, system platform [10].
The two fundamental components of federated search systems comprise of an index that is a reference to the data to be searched and a search algorithm. These two components interact using either search-time, index-time or a federated search interface [11]: Search-time merging runs separate search algorithms on each data source that is searched. It uses multiple indices and aggregates the returned search results into a final list, which is then presented to the user. Obtaining the results can be slow, if the central search engine needs to wait until all of the local search engines have responded. Index-time merging requires a central index to be built for all of the data that need to be included in the search results. It requires one search engine and one index, and is faster than search-time merging. To aggregate data from multiple sources and formats into a single index, interoperability issues need to be addressed. Federated search indexing is an extension of the search-time merging method that requires a robust search solution in order to index different types of content in different indices and create the unified federated search interface.
Our solution to a federated search and recommendation system, designed in EFPF, includes the following tree steps: firstly, in order to achieve a technical federation of digital platforms, we design a common federated search capability. Secondly, we collect the results of federated search to support federated recommendations. Finally, both federated search and recommendations support advanced matchmaking and agile network creation mechanisms for business transactions across the digital platform ecosystem. Federated search in EFPF supports queries for products, services and business partners. The search criteria and the results are collected to support the recommendation processes, based on different techniques of information pattern matching, e.g. information retrieval and similarity matching techniques, which are both based on Machine Learning (ML) and data analysis. After the recommendation algorithm has identified the most suitable products, services and/or partners, the user evaluates the results based on selected indicators, e.g. cost, reliability, quality, etc. In the final step, the user makes a decision and initiates a business transaction.
In EFPF, federated search is based on the index-time merge architectural approach (Figure 1). This approach requires content from all base platforms participating in the federation, to be channeled into a central index of the EFPF platform. The four base platforms in EFPS are NIMBLE (https://www.nimble-project.org/), COMPOSITION (https://www.composition-project.eu/), vf-OS (https://www.vf-os.eu/) and DIGICOR (https://www.digicor-project.eu/). The EFPF Data Spine is designed as an integration flow engine, enabling toolsets and services from the base platforms to connect with each other and offers their capabilities to the users of the different platforms. For the implementation of the Federated Search Index, we use Apache Solr (https://lucene.apache.org/solr/) that enables distributed indexing, replication and load-balanced querying, automated failover and recovery, configuration, etc.
One major disadvantage of the index-time merge architecture concerns the data connectors that need to be implemented to integrate different types of data sources. In addition, acquiring the content from different repositories and data sources requires considerable efforts; e.g., it needs to be done using scheduled read-only processes that need to be designed and implemented at the data integration layer. This also requires a decision about the frequency of the data ingestion into the central index. Data ingestion frequency needs to be configured e.g. hourly, daily or weekly, depending on the data velocity of the base platforms.
The user interaction data in EFPF is stored using the User Activity Log Service, which listens to all user interaction events generated from the EFPF portal (https://efpf-portal.ascora.eu/), e.g. purchased items, etc. The user interaction data are also stored using Solr Index, in order to support ML model creation. The outcomes of the matchmaking are presented to the EFPF users through an intuitive UI that is integrated in the portal.
Federated search, ML and recommendation techniques can be combined in an innovative way to deliver better results for businesses, improving user experience, increasing user engagement through incentives and rewards mechanisms, boosting conversion rates and fully operationalizing businesses using digital platform solutions. However, there are still many questions to be explored in the intersection of these fields, e.g.: Which recommendation algorithms are more suitable for federated platform ecosystems? What is the minimum amount of data to run an effective recommendation system? How symbolic AI methods can compensate for the missing data and better support recommendation? This paper has presented the main challenges and the initial design of the federated search architecture. The above questions will be further answered over the course of the project implementation. In 2021, an open call for third-party use of the platform infrastructure will offer the opportunity for large-scale validation of the search and recommendation system.