=Paper=
{{Paper
|id=Vol-2900/WS7Paper4
|storemode=property
|title=Federated Search and Recommendation
|pdfUrl=https://ceur-ws.org/Vol-2900/WS7Paper4.pdf
|volume=Vol-2900
|authors=Dileepa Jayakody,Nirojan Selvanathan,Violeta Damjanovic-Behrendt
|dblpUrl=https://dblp.org/rec/conf/iesa/JayakodySD20
}}
==Federated Search and Recommendation==
https://ceur-ws.org/Vol-2900/WS7Paper4.pdf
Federated Search and Recommendation
Dileepa Jayakodya, Nirojan Selvanathana and Violeta Damjanovic-Behrendta
a
Salzburg Research, Jakob Haringer Strasse 5/II, 5020 Salzburg, Austria
Abstract
This paper presents an index-time merging approach to federated search designed for a
European Connected Factory Platform for Agile Manufacturing (EFPF) project and its
federated digital ecosystem. The presented approach supports search and recommendations for
products, services as well as suitable business partners for the creation of networks within the
business ecosystem. We discuss the overall architecture of the search and recommendation
system, major challenges and the current state of development.
Keywords 1
Federated search, recommendation, digital platforms, platform search
1. Introduction
Federated search systems emerged in the late 1990s in order to support library users to
simultaneously search multiple library sources and catalogues [1]. Early federated search systems were
designed to collect search criteria from users, submit search queries to sources to be retrieved, wait for
a response to arrive, and filter and organize the query results before sending the results back to the users.
For example, Windows 7 services differentiate between two types of search, Windows Search and
Federated Search [2]. Windows Search enables searching of the entire local system and a limited
amount of external resources such as shared drives. Likewise, the selection-based search in MacOS
Spotlight requires an index of all items and files that exist in the system to be created.
By contrast, Federated Search enables the simultaneous search of multiple sources, including
document repositories, databases, remote repositories and external Web sites. To enable federated
features for websites to be effectively searched by the user, search connectors need to be implemented
and configured for each website [2].
With the maturing of Cloud Computing, Distributed Machine Learning (DML), big data, fog and
edge computing technologies, new opportunities for exploiting computing resources on demand
emerged, creating huge possible impacts for various sectors. However, implementing such federated
systems is still a challenging task due to cross-cutting requirements related to search and distributed
data management, data governance and policies that need to be simultaneously implemented and
administered.
The core issue related to federated search is about resource accessibility, which is required to create
one large index from those multiple sources that constitute the federation. Federated search allows for
each source to be indexed locally and searched across many sources. Another issue of federated search
is about the relevance of results that are obtained from a variety of sources. The relevance of results
needs to be synchronized and compared across sources, in order to merge the results in a meaningful
way. In practice, even the results from the same source are not comparable when different search
engines are used to retrieve these sources.
In this paper, we present the design of federated search and recommendation systems for digital
platforms that is currently ongoing work in the EFPF (European Connected Factory Platform for Agile
Manufacturing) project. The aim of EFPF is to create a federation of several digital platforms, while
overcoming the interoperability challenges between different platforms and their services.
Proceedings of the Workshops of I-ESA 2020, 17-11-2020, Tarbes, France
EMAIL: dileepa.jayakody@salzburgresearch.at
©️ 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
2. Literature review
The authors in [3] differentiate between federated search and metasearch. Metasearch engines crawl
open websites and return results of all kinds, such as apps, webpages, contacts and documents that are
drawn from different sources. In contrast to metasearch, federated search systems focus on subscribed
databases that are processed through an indexing mechanism that enables structured, in-depth, content-
oriented retrieval [4]. A number of attempts to implement federated search arose from the 1990s
onwards, e.g. MetaLib and SFX implemented at Boston College [5], MetaLib and SFX with Ex Libris
at the Five Colleges Libraries in Massachusetts [6], Georgia Library Learning Online (GALILEO) with
its hybrid approach to federated search products [7], and more. Many interviews showed that the users
were dissatisfied with federated search performance of that time, due to limitations in database
availability, search speed, retrieval precision, and result comprehensiveness [8]. Users preferred simple
interfaces over federated search and had little interest in advanced searching techniques. In recent years,
several attempts to improve federated search were conducted. In 2004, the University of Nevada at
Reno experimented with the Google Search Appliance to be used as a federated search utility. This
required partnership between Google, the library and a test vendor, and it still experienced technical
and interoperability challenges [9].
3. Problem statement
Federated search is a technique for simultaneous search of multiple data sources using a single query
and search interface. The examples of federated search can be found in many sectors, facilitating search
of thousands of products, each sorted into different categories, e.g. search of numerous websites serving
various purposes and stakeholders [11]. The majority of the research on federated search is focused on
system performance and technical development, system usability testing, user interaction, interface
customization, authentication, design, database communication protocol, system platform [10].
The two fundamental components of federated search systems comprise of an index that is a
reference to the data to be searched and a search algorithm. These two components interact using either
search-time, index-time or a federated search interface [11]:
Search-time merging runs separate search algorithms on each data source that is searched. It
uses multiple indices and aggregates the returned search results into a final list, which is then
presented to the user. Obtaining the results can be slow, if the central search engine needs to wait
until all of the local search engines have responded.
Index-time merging requires a central index to be built for all of the data that need to be included
in the search results. It requires one search engine and one index, and is faster than search-time
merging. To aggregate data from multiple sources and formats into a single index, interoperability
issues need to be addressed.
Federated search indexing is an extension of the search-time merging method that requires a
robust search solution in order to index different types of content in different indices and create the
unified federated search interface.
4. Federated search and recommendation in EFPF
Our solution to a federated search and recommendation system, designed in EFPF, includes the
following tree steps: firstly, in order to achieve a technical federation of digital platforms, we design a
common federated search capability. Secondly, we collect the results of federated search to support
federated recommendations. Finally, both federated search and recommendations support advanced
matchmaking and agile network creation mechanisms for business transactions across the digital
platform ecosystem. Federated search in EFPF supports queries for products, services and business
partners. The search criteria and the results are collected to support the recommendation processes,
based on different techniques of information pattern matching, e.g. information retrieval and similarity
matching techniques, which are both based on Machine Learning (ML) and data analysis. After the
recommendation algorithm has identified the most suitable products, services and/or partners, the user
evaluates the results based on selected indicators, e.g. cost, reliability, quality, etc. In the final step, the
user makes a decision and initiates a business transaction.
5. Design of a federated search and recommendation system
In EFPF, federated search is based on the index-time merge architectural approach (Figure 1). This
approach requires content from all base platforms participating in the federation, to be channeled into
a central index of the EFPF platform. The four base platforms in EFPS are NIMBLE
(https://www.nimble-project.org/), COMPOSITION (https://www.composition-project.eu/), vf-OS
(https://www.vf-os.eu/) and DIGICOR (https://www.digicor-project.eu/). The EFPF Data Spine is
designed as an integration flow engine, enabling toolsets and services from the base platforms to
connect with each other and offers their capabilities to the users of the different platforms. For the
implementation of the Federated Search Index, we use Apache Solr (https://lucene.apache.org/solr/)
that enables distributed indexing, replication and load-balanced querying, automated failover and
recovery, configuration, etc.
One major disadvantage of the index-time merge architecture concerns the data connectors that need
to be implemented to integrate different types of data sources. In addition, acquiring the content from
different repositories and data sources requires considerable efforts; e.g., it needs to be done using
scheduled read-only processes that need to be designed and implemented at the data integration layer.
This also requires a decision about the frequency of the data ingestion into the central index. Data
ingestion frequency needs to be configured e.g. hourly, daily or weekly, depending on the data velocity
of the base platforms.
Figure 1: The index-time merge architecture in EFPF
The user interaction data in EFPF is stored using the User Activity Log Service, which listens to all
user interaction events generated from the EFPF portal (https://efpf-portal.ascora.eu/), e.g. purchased
items, etc. The user interaction data are also stored using Solr Index, in order to support ML model
creation. The outcomes of the matchmaking are presented to the EFPF users through an intuitive UI
that is integrated in the portal.
6. Conclusion
Federated search, ML and recommendation techniques can be combined in an innovative way to
deliver better results for businesses, improving user experience, increasing user engagement through
incentives and rewards mechanisms, boosting conversion rates and fully operationalizing businesses
using digital platform solutions. However, there are still many questions to be explored in the
intersection of these fields, e.g.: Which recommendation algorithms are more suitable for federated
platform ecosystems? What is the minimum amount of data to run an effective recommendation
system? How symbolic AI methods can compensate for the missing data and better support
recommendation? This paper has presented the main challenges and the initial design of the federated
search architecture. The above questions will be further answered over the course of the project
implementation. In 2021, an open call for third-party use of the platform infrastructure will offer the
opportunity for large-scale validation of the search and recommendation system.
7. Acknowledgements
This research has been co-funded by the European Commission within the H2020 project NIMBLE,
No. 723810 and the H2020 DT-ICT-07-2018-2019 project EFPF, No. 825075.
8. References
[1] A. C. Elguindi, K. Schmidt, “Discovery systems, layers and tools, and the role of the electronic
resources librarian”. Electronic Resource Management, 2012, p. 109-139.
[2] J. Orchilles, “Introduction to Windows 7”, Microsoft Windows 7 Administrator’s Reference, p. 1–
91, doi:10.1016/b978-1-59749-561-5.00001-2, 2010.
[3] X. Fei, “Implementation of a Federated Search System: Resource Accessibility Issues“, Serials
Review – SER REV. 35, 2009, p. 235-241.
[4] R. Tang, I., Hsieh-Yee, S. Zhang, “User Percep. of MetaLib Comb. Search: An Inv. of How Users
Make Sense of Federated Search,” Intern. Ref. Serv. Q. 12, 2007, p. 211–236.
[5] B. Geritty, T. Lyman, E. Tallent, “Blurring Services and Resources: Boston College’s
Implementation of MetaLib and SFX”, Ref. Serv. Rev.30, no.3, 2002, p. 229–241.
[6] L.S. Mestre C. Turner, B. Lang, B. Morgan, “Do We Step Together, in the Same Direction, at the
Same Time? How a Consortium Approached a Federated Search Implementation”, Internet
Reference Services Quarterly 12, no. 1/2, 2007, p. 111–132.
[7] L. Fancher, “Wanted, Dead or Alive: Federated Searching for a Statewide Virtual Library”,
Internet Reference Services Quarterly 12, no. 1/2, 2007, p. 133–158.
[8] K. Calhoun, “An Integrated Framework for Discovering Digital Library Collections”, Journal of
Zhejiang University Science 6A, no. 11, 2005, p. 1318–1326.
[9] M. Taylor, “Using the Google Search Appliance for federated searching”, Internet Reference
Services Quarterly, 10(3), 2006, p. 45–55.
[10] D.G. Dorner, A.M. Curtis, “A Comparative Review of Common User Interface Products,” Library
Hi Tech 22, no. 2, 2004, p. 182–197.
[11] Algolia, “What is Federated Search?“, URL: https://blog.algolia.com/what-is-federated-search/,
2019.