Similarity search in large database systems is a crucial feature in many applications and often requires a manual adjustment of parameters to suit various search scenarios [ 17 ]. Such parameters are hard to optimize by randomly probing the search space, but they significantly influence the retrieved results’ quality [ 7 ]. In many cases, even experts with prior domain knowledge struggle to understand the inner workings of the used mining models and the influence of abstract model parameters, which prevents them from reaching the desired analysis goal. Systematic steering and exploration of diferent parameter settings can help to obtain the proper combination more efectively. Thus, domain experts need concurrent access to models, parameters, and results, enabling them to understand how parameters influence the results and how they can be refined to match the analysis goal.

Visual analytics enables users to explore and analyze data and models by providing integrated visual representations for data, models, and parameters. Such visual techniques enable interactive parameter adjustment during exploration and analysis [ 6 ]. Visual analytics bridges the gap between heuristics to find suitable parameters and domain experts with the knowledge to steer results in a human-centered direction. For instance, a visual interactive what-if analysis facilitates experts to understand black-box model decisions by enabling direct data and parameter manipulation [ 13 ]. The comprehensive understanding of the relationship between model parameter choices and outcome is a fundamental requirement for well-informed decision making [ 20 ]. By applying standard visual analytics techniques, such as aggregation, filtering, or speculative execution [ 18 ], the vast results- and parameter spaces can be interactively explored, despite the algorithms being time- and resource-consuming. Thus, visual analytics supports the comprehension of parameter choices in similarity search applications for users and domain experts. Visual analytics enables informed reasoning about a query’s results, allows the understanding and diagnosis of parameters, and supports the user in refining those parameters to get the best possible results.

We propose a visual analytics workspace to support users in result understanding and model refinement on a ranking-based similarity search algorithm in the context of large data foundations. Our system consists of a user-centered visualization of parameters and results to facilitate the users’ exploration and understanding of the parameter choices. We enable users to interactively update model parameters based on their domain knowledge and findings during the analysis process. Our visual analytics system further facilitates real-time analysis using speculative execution on a time-intensive similarity search algorithm, enabling online exploration and execution of the ofline algorithm.

Summarizing, we present a visual analytics system for similarity search, providing the following main contributions: (1) our system supports the understanding of results and parameters, emphasizing the most critical data characteristics by mapping similarity to spatial distance and highlighting communities of similar attribute combinations. (2) Our system enables the diagnosis of results and parameters by allowing the real-time interactive exploration of the parameter space to investigate the influence of parameter choices, enabled by the speculative execution. (3) Our system supports the refinement of the involved parameters, supporting the iterative guided optimization of the model to solve a given analysis task. 2

To cover the various involved research domains and applications, we structure our related work into sub-topics, summarizing the most relevant works regarding one aspect of our approach. Visual Analytics Foundations — Similarity searches in large database systems are often automatically executed using predefined similarity functions and distance measures. However, user-adaptable similarity search applications increase in importance, and user integration rises [ 17 ]. Visual analytics combines automated analysis techniques with interactive visualizations to enable users to understand and reason about large datasets [ 6 ]. Sacha et al. [ 14 ] have presented a knowledge generation model that describes how knowledge is generated during the analysis process, building upon prior methodologies in visual analytics [ 2, 12 ]. Besides the computer system that visualizes and models data, they describe the human as a core element whose creativity, interaction abilities, and perception help find and comprehend patterns hidden in the data.

Weight Space Exploration — As visual analytics is concerned with integrating human knowledge with automated machine learning, it is frequently used for model exploration and optimization. Sedlmair et al. [ 16 ] provide a conceptual framework of visual parameter space analysis, structuring the design space. Pajer et al. [ 10 ] present a tool for the visual analysis and exploration of weight spaces, tackling the problem of setting abstract weight parameters. Their tool supports the understanding of sensitivity and helps identify weight regions of interest for a desired output. Mühlbacher et al. [ 9 ] present TreePOD, a sensitivity-aware approach to selecting Pareto-optimal decision trees. In contrast to most existing work, we tackle the exploratory analysis of similarity queries and rely on the analyst’s intuition rather than on quality metrics.

Parameter Optimization for Mining Models — Parameter optimization for data mining systems or hyper-parameter optimization in machine learning is an open problem that frequently occurs in scientific or industrial use-cases. Analytic optimization or exhaustive search for parameter optimization is often impossible in these models due to black-box methods or highdimensional parameter spaces. Torsney et al. [ 22 ] apply a guided semi-automatic method to this problem by first sampling from the parameter space and then guiding the user by estimating the efects of parameter changes on the result. Schall et al. [ 15 ] propose a heat-map method to superimpose the prediction of a deep neural network over its input image. This allows the model engineer to identify problems in the prediction and tune the hyper-parameters accordingly. The resulting workflow is iterative and guided by the provided visualization. This method is applied to ofline handwriting recognition, where spatial information is essential but not available in ground-truth data.

While search is an essential tool to locate entities of interest in large data foundations, it has significant limitations when the data distribution is unknown and, hence, explorative access to the data is required. Specifically, the exact attribute combination of the results might not be known beforehand, or multiple entities in a particular region might be of interest. The used similarity search (SimSearch) algorithm [ 11 ] fulfills these requirements by considering entities that feature attribute combinations close to the desired search parameters. By specifying the number of ranked closest matches, the analyst can explore the region of interest and refine the search parameters according to the analysis goal. The high-dimensional search space poses particular challenges for the visual representation of the results: pair-wise distances between entities and the root search have to be considered, as well as the influence of each single search attribute.

The variety of data types and -domains that might occur in the data attributes requires the concurrent use of diferent distance SimSearch VA Backend b SimSearch Workspace a e h c a C

To avoid computationally-, time-, and storage-expensive operations in the frontend, our implementation splits the SimSearch system into frontend (2a) and backend (2b). The backend interfaces with the similarity search model, being exposed via REST API. The result of a request to the SimSearch API consists of (1) a ranked list of the top- similar results together with (2) a × cross-similarity matrix, denoting the pair-wise similarities between every two entities. The raw results are cached by the backend application for later search queries with similar parameters. The results are then transformed from the -dimensional attribute space down to the two-dimensional screen space and converted into a graph representation using a specified projection algorithm. We include diferent projection methods to achieve good results for varying search attributes and input parameters: for low-dimensional searches ( ≤ 4), the system supports PCA and MDS, based directly on the attribute values or the crosssimilarity matrix, respectively. For higher-dimensional searches, UMAP [ 8 ] can provide fast and stable projections highlighting connections in the data while preserving its global topology. The decisive criterion for choosing the provided projection methods was their ability to derive a stable transformation under a changing set of input vectors. The cross-similarity matrix is filtered for its top values and converted into a list representation to reduce network load and computational complexity in the frontend. Both results, the projected graph, and the cross-similarity list are then cached and returned to the frontend application. Figure 2 shows the architectural details of the system, including data paths, caching, and the applied data transformations. Caching Strategy and Requirements — The cache has a crucial impact on the system’s responsiveness, requiring the caching strategy to obtain the best possible balance between data topicality and system performance. Since this choice is strongly dependent on the frequency with which data evolution events occur in the data foundation, we tackle this challenge by occasionally querying the similarity search engine despite the results already being present in the cache. Since this strategy triggers a request of multiple similar parameter combinations, the results in the local search space are updated, maximizing the probability of future cache hits with the most recent data entities. The cache’s required storage space is neglectable since, in a typical scenario, = 50 can be taken as a reasonable upper-bound for the top- results of interest. The storage consumption for a query grows linearly with , except for the × cross similarity matrix, which grows quadratically. Taking the upper bound of = 50, we can estimate its storage consumption as 50 · 50 · 64 bit = 160 000 bit ≈ 20 kB. 3.2

To allow the interactive analysis of the SimSearch algorithms’ results and enable informed decision making during the parameter tuning process, our proposed similarity search workspace combines multiple components in a comprehensive user interface, shown in Figure 1.

We show the applicability and advantages of the proposed SimSearch workspace based on two exemplary use-cases. The first use-case (subsection 4.1) is hands-on and describes in detail how our proposed system can be used to reach the analysis goal, while the second use-case (subsection 4.2) demonstrates how our system can be applied to varying tasks and domains. 4.1

This use-case is based on a real-world, large-scale (≈ 120 GB) dataset containing information about companies in Italy.

In the use-case, a small company with ≈ 50 employees plans to expand, for which several potential new locations are considered. Since the company is dependant on the local infrastructure and other supplying companies, geographical proximity to those companies is an essential requirement. Simultaneously, the company wants to avoid direct local competition through other companies working in the same sector and having a similar corporate structure. Our proposed SimSearch workspace supports the search and interactive exploration of the potential company locations to fulfill the company’s requirements.

By specifying the attribute combinations in the parameter designer according to the desired or declined company profiles together with the considered company location, the local search space can be explored. The projection view reveals the most similar companies and indicates their pair-wise relationships, revealing communities and enabling the analyst to assess the most influential search attributes. In doing so, it becomes clear that the geolocation only has marginal influence on the search results, and the shown companies are too far for a business relationship.

Since the numerical search attributes, such as the number of employees, can not be objectively compared to the geospatial n o it a c o l o e g ∆ s(c0,c1)

s(c1,c2) s(c0,c2) ∆ num_employees (a) n o it a c o l o e g ∆ · w0 s(c0,c1)

s(c1,c2) s(c0,c2) w1 · ∆ num_employees (b) company location, the weights in the parameter designer have to be iteratively refined to match the analyst’s understanding of each attribute’s desired influence on the results. Figure 4 shows how the weight adjustment helps to balance the diferent distance functions. By indicating the changes in the result set for a possible weight adjustment, the analyst can exploit the systems speculative execution feature to observe changes in real-time and assess the most purposeful operation before the actual, timeconsuming execution. Using the tabular results view, the analyst can verify the possible changes in detail by observing how each attribute’s ranking would change under the operation or if the company would be excluded from the result set. By iteratively refining the search parameters, the analyst can explore the search space ideally for each potential location, leading to well-informed decision making for the new location. 4.2

This use-case is based on internal mail forwarding within a large company. Incoming postal mail is automatically opened and digitized, using an OCR system, on arrival at the company headquarters. The digitized mail item is then used as a search query on a structured database of the company’s customers, contracts, products, or projects to electronically forward the scanned document to the staf responsible for working this task needing the document. This use-case requires both a robust search engine for retrieving database entries (e.g., contracts) containing keywords, names, or numerical values similar to the query document and semantic understanding of the content to weigh these attributes.

The structured data in the database consists of categorical attributes, person or item names, as well as spatial, temporal, numerical, or general ontological values. These may occur within the scanned document with diferent individual similarities as well as in many diferent combinations. Thus the need arises to weigh these database attributes against each other to model the overall semantic similarity. This configuration of the search query likely is done by a human engineer with expert domain knowledge. One approach here is to use a set of example documents for evaluation and repeatedly querying for them and modifying the attribute weights until relevant database entries are found with high overall similarity to the query document, with less relevant entries being significantly dissimilar.

We propose to use SimSearch in this process to both see the overall similarity of the diferent database entries using the current configuration and identify clusters in the embedding space. The embedding method will be chosen to reflect the expert’s domain knowledge of semantically diferent and similar documents. Cross-similarities will show potential miss-classifications. This allows adjusting the weights of the similarity search to increase the similarity to semantically relevant documents and separate them from semantically distinct ones. 5

While the presented similarity search workspace implements a variety of features and techniques to make the data search space and the model parameter space accessible by the analyst, possible extensions could further strengthen the system’s usefulness. Such extensions could include improvements to the search functionality and the explanation of results or the implementation of advanced guiding techniques. Furthermore, the presented approach could be generalized to other domains and tasks with a similar problem setting, i.e., where high-dimensional result entities of complex mining models have to be visualized, and the model must be refined to match a particular analysis task. Extending the Search Functionality — Additional views could augment the existing visualizations with an abstract overview of possible actions and the resulting changes, enabling the analyst to identify possible changes at first glance before descending into detailed views. For example, an additional view visualizing all possible weight combinations probed by the speculative execution component and their likely outcomes could provide first hints where the region of interest might be located. Additional interestigness measures could augment the parameter designer’s weight sliders with information on the intervals corresponding with the most significant changes in the result set. Extending the interestingness feature, decision boundaries could be estimated by probing the search space in regions with a high gradient, providing a sensitivity analysis for each parameter. Extending Guidance — The system currently provides orienting guidance to users alerting them to similar weight configurations that produce significantly diferent search results. In addition to highlighting diferent possible weight settings, the system could actively propose user actions like moving weight sliders or switching to diferent projection methods. By analyzing and learning from user interactions, the system could identify the users’ preferences and provide suggestions adapted to their understanding of the domain and analysis task. By giving the system more initiative in the exploration process, the system should become both more efective and eficient to use. Generalization as Visual Analytics Technique — There are several other problems in automated data mining pipelines with the same or a similar structure as the similarity search application addressed in the presented system, such as clustering, classification, or graph merging. Specifically, our approach can be generalized to understand, diagnose, and refine models where (1) the result is a number of -dimensional entities with arbitrary distance functions associated, and (2) the outcome depends on a set of parameters whose influence on individual results is opaque. Scalability — The system’s scalability is directly dependent on the underlying similarity search algorithm. Despite implementing various techniques (caching, speculative execution) to enable interactive visual analytics on the ofline search algorithm, the similarity search model’s response time is the limiting factor for the approach. While response times of 1 − 30 s can be bridged by applying the implemented techniques, longer response times render an online analysis increasingly dificult since (1) nonideal sampling points might have been chosen for speculative execution or (2) the analyst might change the search space context more rapidly than results can be preemptively queried and cached. The response times of the similarity search algorithm could be reduced by parallelizing the main stages of the algorithm, namely (1) generating a ranked list of results for each queried attribute and (2) compiling the ranked lists into a list of top- results [ 11 ]. Limitations and Future Work — Currently, views of higher abstraction giving the analyst reference points on promising analysis directions are missing. We will tackle this issue by adding a third view to the similarity search workspace, showing all possible weight combinations in a matrix view and indicating the regions of the highest expected result change. Currently, the analyst has to evaluate the speculative changes in results manually by observing the predicted outcomes and comparing them across the diferent parameter combinations. In future versions, we will automatically highlight regions of interest using the number of changes in the result set for each combination as an interestingness measure. This functionality will be strengthened by implementing interactive, adaptive guidance. If one operation has significantly higher interestingness than others, it will be actively proposed as a possibly rewarding action. Furthermore, by tracking recent interactions of the user with the system, we will estimate the likelihood of future interactions based on the history, adapting the guidance to user preferences. Despite the presented use-cases proving our approach’s applicability in different real-world scenarios and data domains, a future user study will further validate the system’s usefulness and provide insights on both benefits and open challenges. Besides measuring quantitative criteria, such as task completion time and comparing the analysis results to ground-truth data, an additional qualitative evaluation will expose additional user requirements and future points for improvements of the system. 6

Applying complex data mining models to large data foundations introduces particular challenges to the analysis process. Both the parameter space and the search space might be opaque, requiring manual probing to approach the regions of interest and, hence, rendering an interactive exploration impossible. Applying visual analytics, models, parameters, and results can be made accessible through interconnected visualizations, revealing hidden connections between components and providing advanced mechanisms, such as speculative execution, to enable the real-time exploration of otherwise time-consuming data processing pipelines.

The presented system implements views and techniques to make the parameters and results of a novel similarity search algorithm accessible to the analyst. Specifically, we provide a projected view of the search results, highlighting the similarity to the root query, the pair-wise similarity between the result entities, the stability of the results, as well as communities of close entities. The projected view is complemented with and linked to a tabular view of the results, indicating their rank and providing sorting functions on distinct attributes or their corresponding similarity. Supporting parameter refinement and search space exploration, the system implements speculative execution on the time-consuming similarity search operation, presenting the user with possible outcomes of parameter changes on-demand before actually performing an action. The projection and tabular views are coupled with the parameter refinement functionality, integrating the speculative results into their visual representation.

We show our proposed similarity search workspace’s applicability and usefulness based on two use-cases, both anchored in real-world application examples and datasets.

This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agree