Efective retrieval is essential to strong case-based reasoning performance, and retrieval quality is critically dependent on case indexing. Such indices are not always feasible to generate manually, and so a thorough exploration of how features and weights may be generated automatically (especially using deep learning) is necessary. To that end, this summary outlines a research plan for investigating structural influences on feature quality, how learned features may be used in concert with knowledge-engineered features, and how weights may be generated in feature-dense spaces created by feature learning. It also proposes a methodology for modular exploration of various models, training set sizes, numbers of features generated, etc., to provide a comprehensive foundation of index generation using deep learning. Finally, it points to already-published research that works towards some of these goals and illustrates how results from these existing projects inform future research plans.
Case-based reasoning (CBR) performance relies significantly on retrieving useful cases from the
case base. In turn, retrieval quality depends on indices used to characterize/discriminate between
cases. High-quality indices can be derived through manual knowledge engineering (e.g., [
These problems may be addressed using feature and weight learning. Initially, this was
achieved using symbolic methods (e.g., [
This research summary outlines strategies for exploring feature and weight learning using DL in greater depth, presenting both a blueprint detailing potential research objectives and methodologies as well as an overview of steps taken so far and the resulting publications.
This research broadly seeks to deeply explore methods for leveraging DL models to generate indexing information to supplement or replace information gathered through knowledge engineering for computer vision-related tasks. This overarching goal can be subdivided into the four primary investigation regions described below, which are followed by a proposed research plan outline for exploring them.
Under the umbrella of network-based feature generation, there exist multiple potential variables
that can influence feature quality. For one, diferent model structures provide unique pathways
for feature generation, exemplified by diferent computer vision approaches (e.g., comparing
AlexNet, Inception, DenseNet, transformers, and MLP-based models); additionally, architectures
may be leveraged in diferent ways (e.g., using ensembles of networks for localized feature
generation in the multi-net approach explored in [
While feature learning can be useful in domains for which generating features through knowledge-engineering is not feasible, more research is required to evaluate feature learning augmenting incomplete knowledge-engineered indices. This includes exploring methods for efectively using both feature sets in concert, for which it may be necessary to mitigate harmful efects of a “curse of dimensionality" as a result of extracted feature spaces being generally denser than knowledge-engineered ones. Integrating knowledge-engineered and network-generated features also requires investigation into potential discretization of continuous network-generated features as well as into the independence of generated features and/or their correlation with knowledge-engineered features.
As mentioned above, using learned features can result in similarity assessment being performed
in feature-dense spaces, for which conventional methods of weight learning (e.g., [
Investigation of hybrid systems leveraging interpretable CBR structures alongside more opaque DL systems demands contextualization relative to explainability. Innately, DL-CBR hybrid systems imply an overall architecture that is more interpretable than an out-of-the-box DL model but less so than a CBR system using only knowledge-engineered information. Thus, in addition to optimizing index quality to maximize retrieval accuracy, it is important to assess where on an explainability spectrum that this work sits and to take measures where possible to maximize interpretability.
The four research objectives described above encompass a diverse range of research avenues, and they present specific sub-goals that align with an overarching three-step process that guides the proposed research methodology. Specifically, this process begins with deep exploration of network-based index generation methods, including diferent ways in which generated indices may be integrated into a CBR system. The second step involves post-processing of these indices, particularly for optimizing CBR system performance/accuracy, but also potentially including discretization for better combination with knowledge-engineered index information where applicable. Finally, the resulting DL-CBR hybrid model is analyzed/contextualized with respect to explainability, especially in comparison to CBR systems using only knowledge-engineered indexing information.
To this end, initial experiments imitating established index generation methods (e.g., [
Beyond this point, future experiments could continue in any of several directions. For one, feature weighting strategies may be more deeply investigated and/or revised in the context of potentially denser feature spaces created using network-generated features. For another, potential relationships between knowledge-engineered and network-generated features may be explored for explainability purposes. Such investigations would include dependency correlations between knowledge-engineered and network-generated features and/or methods by which continuous network-generated features may be discretized, along with the resulting impact on feature quality.
As of the writing of this summary, most progress to date has focused on providing a conceptual foundation and initial empirical tests for DL-CBR hybridization for retrieval, in line with the objectives presented above. Specifically, such explorations include using network-generated features in concert with knowledge-engineered features for greater classification accuracy, as well as investigating structural influences (e.g., network architecture/structure and feature extraction location) on feature quality, both using retrieval accuracy as proxy. The following subsections summarize the associated publications.
This research [
Results supporting this hypothesis are obtained using a zero-shot learning dataset for computer vision. Each image is associated with a unique case, and per-class feature information from the dataset is perturbed based on a random coeficient and combined with values extracted from the image using a convolutional neural network (CNN) to form the case’s feature set. The two combined sets of values represent knowledge-engineered and network-generated features, respectively. Retrieval accuracy values for the aggregated feature set are compared against corresponding accuracy values using either component set exclusively (Figure 1).
Based on the outcomes from these initial tests, combining feature sets does improve retrieval accuracy. However, additional variables such as the reliability of knowledge-engineered features and the number of features extracted from the CNN may significantly influence the magnitude of accuracy improvement. In addition, preliminary tests regarding weight extraction in parallel with feature extraction suggest that more feature-dense spaces created by extracting features from CNNs seem to accommodate established weighting strategies poorly.
In contrast to the previous description, this work [
The experiments investigate the hypothesis that extracting features from later in the network results in higher-quality features from the perspective of the CBR system. Retrieval accuracy values are used as proxy for feature quality among the three proposed methods and a CNN baseline (Figure 2). Additionally, keeping in mind the consequences of feature-dense spaces discovered in the previous work, varying numbers of features are extracted from each model.
The results support the hypothesis and suggest that localized feature sets may be especially accurate for feature-sparse scenarios preferred by CBR systems (if at the cost of increased training time). The number of features does significantly impact retrieval performance as well, both as a “curse of dimensionality" for large numbers of features and as a minimum requirement for DL model convergence for smaller numbers of features.
Future work will build on the current publications’ findings while moving forward to address other objectives. In the short term, research will focus on exploring feature generation using both diferent DL models across multiple datasets and diferent experimental parameters (e.g., number of training examples). Later experiments will investigate potential weight generation methods, as well as how weight generation methods are afected by the number of features.