This paper explores the efective combination of embedding models through a similarity analysis. Embedding models are characterized by heterogeneous architectures, training objectives, and training corpora, resulting in distinct semantic representations and domain-specific proficiencies. By integrating such diverse models, we aim to harness their complementary strengths. However, the potential for performance improvement is contingent on the specific models selected for combination. To address this, we propose a strategy that utilizes similarity scores between model pairs as strong predictors of their combined performance. First, we provide a similarity analysis of embedding models as an unsupervised evaluation framework from a retrieval perspective. Second, we empirically demonstrate that combining embeddings from diferent models can significantly enhance retrieval performance, but only in the case of certain model pairs. Third, we propose an adjusted dissimilarity measure that accounts for both the similarity and performance gap between the models. Our experiments reveal that the adjusted dissimilarity scores between models are strongly correlated to their combined performance. This finding paves the way for organizations with resource constraints to use such a similarity framework to select combinations of smaller models whose performance closes the gap to large, inaccessible models.
academia [
In this paper, we focus specifically on the efective combination of embedding models through
a similarity analysis. Diferent works in the literature have discussed the benefits of a similarity
analysis of LLMs. Caspari et al. [
We start from the observation that models exhibit diverse architectures, training objectives, and fine-tuning strategies, allowing them to capture distinct semantic, syntactic, and contextual aspects of text. Additionally, training on diferent corpora results in domain-specific capabilities. By combining these models, we aim to leverage their complementary strengths and mitigate individual biases and errors. However, the existence of improvement and its magnitude depends on the specific models involved. The embeddings might also contain conflicting semantic information – for example, architectural/training improvements in successive iterations within a model family might specifically target a particular kind of error or bias in these models. Combining such models (the predecessor and successor in a particular model family) might obscure these improvements and result in minimally improved or even impaired combined performance.
Given the possibility of beneficial or detrimental combinations and a huge number of possible combinations, a strategy to guide the selection of model combinations is essential. We posit that the similarity scores between pairs of models can serve as strong predictors of the potential performance of their combination. We do note, however, that similarity scores reflect both performance gaps along with complementarity of text representations. To predict improvements in combined performance, it is necessary to consider both of these factors that contribute to model similarity. In particular, we make the following contributions: (i) We provide a similarity analysis of several models from the MTEB benchmark using both representational, as well as functional similarity measures tailored for retrieval. (ii) We evaluate combinations of embedding models based on retrieval performance, demonstrating that performance gains are limited to specific model pairs. (iii) We formulate an adjusted dissimilarity measure that accounts for the performance gaps between models and demonstrate a strong correlation between this measure and combination performance. The code for this paper is publicly available1.
We review the literature from two perspectives. First, works that cover a similarity analysis of neural networks or, more specifically, language models. Second, we explore works related to ensembling or combining embeddings.
Measuring the similarity of neural networks has received increasing attention, resulting
in numerous approaches or metrics being proposed [
Our work on combining the embeddings of multiple embedding models draws inspiration
from the extensive work in model ensembling [
None of the works in the literature extensively evaluate combinations of several recent embedding models. To the best of our knowledge, there are also no papers that connect the two directions of this work, namely a similarity analysis and embedding combinations.
Our first step for finding suitable model combinations is to determine the similarity between
two embedding models, as discussed in [
Representational Similarity has been estimated using Centered Kernel Alignment [
Functional Similarity has been estimated using Jaccard and rank similarity [
The resulting score is bounded in [
Rank similarity measures how similar the order of common elements in two sets is, where elements that are closer together are weighted higher. Common text chunks ∈ | ∩ ′| receive a rank according to their position in the retrieval step, i.e. () = if chunk was the top- retrieved result in . With this, rank similarity is computed as follows: (, ′) = 1 ∑︁
2
(| ∩ ′|) ∈|∩′| (1 + | () − ′ ()|)( () + ′ ())
where (| ∩ ′|) = ∑︀=|∩′| 1 denotes the K-th harmonic number and normalizes the
=1
score. Like the previous measures, rank similarity is bounded in [
To adhere to computational constraints, for this study, we use the 4 smallest datasets from the
MTEB/BEIR [
Fig. 1 displays two heatmaps, one for Jaccard and another for CKA, of the pairwise similarities of all our models averaged across our four datasets. Since the rank similarity scores closely resemble the Jaccard results, we have not included a third heatmap here. However, these scores are incorporated into our analysis in Section 5. Examining the Jaccard heatmap, the most striking feature is the relatively small overlap in retrieval results for the majority of model pairs. This indicates that the top retrieval results for these models contain several unique documents. We see that for each of the metrics, the similarities within families are higher than across families at least when accounting for other factors such as size. The CKA scores are fairly high across the board but are less intuitively interpretable. Notably, the similarity of the two larger arctic models with all other models is significantly lower than the average similarity according to the CKA metric. This pattern is not particularly evident with the other similarity measures.
In the next step, we look at the impact of combining embeddings of pairs of embedding models. There are several methodological options for combining embeddings. These lie on the spectrum of aggregation-based methods such as averaging or concatenating over multiple embeddings, or methods that involve some form of learned combination where the combined embedding is optimized on task-specific objectives.
We take the simple approach of concatenating the embeddings of pairs of models for both queries and documents. We then evaluate these combined embeddings on diferent retrieval tasks from the MTEB benchmark. We use the same set of embedding models and datasets as outlined in 3.
For a dataset comprising a set of queries = { }=1 and a set of chunks = {}=1, the embeddings from two models ℰ1 and ℰ2 are concatenated to form a unified or combined representation:
Equery( ) = ℰ1( )‖ℰ2( ) Echunk() = ℰ1()‖ℰ2() ∀ ∈ ∀ ∈
For the retrieval step, we use an exact nearest neighbour search. We compute the Normalized Discounted Cumulative Gain at 10 documents (NDCG10) to evaluate retrieval performance. As each chunk in our retrieval list stems from a document in our corpus, we can compute the performance for a list of documents instead of a list of chunks as shown by [27]. This is necessary as the relevance judgments for these datasets are at a document and not chunk level.
Of course, while the NDCG10 directly informs us about how well a particular combination performs, the score is only of use relative to the scores of the two individual models that form the combination. From a practical perspective, there is little justification for utilizing a combination of models if its performance does not surpass that of both individual models. Consequently, in our analysis, we adopt a metric for measuring performance improvement (ℐndcg). Specifically, we define performance improvement as the diference between the NDCG10 score of the combined embedding model and the maximum NDCG10 score of the individual models within the pair. ℐndcg = NDCG10(ℰ1‖ℰ2) −
max (NDCG10(ℰ1), NDCG10(ℰ2))
In the remainder of this paper, we refer to the maximum of the two individual models’ scores as Max. We note that the scores are not perfectly comparable to those on the MTEB leaderboard due to diferences in our pipeline. The first stems from our data ingestion step, where we split the documents into 256 token-sized chunks for each of our models. This results in relevant information contained in the documents possibly being fragmented across several chunks. Secondly, as we test on small corpora and want to include as many queries as possible, we report retrieval scores on all queries in the datasets and not just those included in the test splits. This being said, the scores we report largely preserve the ranking of the models as reported on MTEB.
soed l M (c) Nfcorpus (d) Scidocs Figure 2: Improvement in performance through model combination. The diference in NDCG10 refers to comparing the performance of two sets of embeddings for the retriever: one where the embeddings of two models are combined and another where only the embeddings of better-performing model of the two is used.
In Figure 2 we report the ℐndcg values for all combinations of our selected models on four datasets. Across each dataset, ℐndcg is predominantly positive, indicating that combining a randomly selected pair of models generally leads to improved retrieval performance compared to Max . This re-afirms our argument that these models, in many instances, generate heterogeneous representations that complement one another. However, it is also apparent that the magnitude of improvement or even impairment, is dependent on a particular combination and dataset. To begin, we assess the influence of diferent model pair selections on the ℐndcg scores. For example, the e5-large-v2 model tends to result in positive ℐndcg values for nearly all of its combinations across all four datasets. Similarly, combinations involving the model bgesmall-en-v1.5 mostly result in positive ℐndcg. Such performance gains can provide significant benefits. For example, for the FIQA dataset the combination of the e5-small-v2 and snowflakearctic-embed-s models, both of whom are very small models (0.12 GB in memory usage each), result in a NDCG10 higher than that of the e5-large-v2 model. This would entail a climb on the public leaderboard corresponding to around 10 positions. We also observe large positive ℐndcg values in the combinations of bge-small-en-v1.5 and snowflake-arctic-embed-m across all our datasets.
Certain combinations, on the other hand, do not fare as well. The Arctic model family, particularly the small version, is also often present in combinations with a negative ℐndcg. These combinations are of no practical benefit as one could simply use the better performing model in the combination. To a lesser extent, for the Nfcorpus and Scidocs datasets, we see a similar efect for the e5-small model. We note that these models are also the ones with the lowest retrieval performance out of our set of models. Assuming that the ‘general’ quality of the representations generated by these models corresponds to their retrieval scores, we can reasonably expect significant deficiencies of certain lower performing models to have a negative impact on the overall retrieval performance when combined with stronger counterparts. These findings indicate that both the complementarity of the model representations and the disparities in their quality significantly influence the efectiveness of their combined retrieval performance. Across the analyzed datasets, we observe that certain datasets not only exhibit a higher proportion of model combinations yielding positive ℐndcg values but also demonstrate larger magnitudes of these improvements. For the Scidocs dataset, 71 combinations out of 105 improve over their Max, whereas for FIQA the number is 98. The magnitude of improvements is also on average higher on FIQA and Scifact compared to the other two datasets.
We hypothesize that enhancements in retrieval performance resulting from model combinations are attributable to their complementary diferences. Similarity scores provide a quantitative means to compare model characteristics — the diferences in architecture, training, data, etc. should reflect in the orientations of the embeddings in vector space (captured by representational similarity measures) and/or in the outcome on downstream tasks (captured by functional similarity measures). Therefore, we ask the question: Can we use these similarity scores as predictors of the compatibility of models to be combined? Assuming that models need to be suficiently distinct to be complementary, similarity metrics that accurately capture these diferences would allow us to identify such distinct model pairs. The following parts describe our framework to test this hypothesis.
We use our similarity scores and ℐndcg values from sections 3 and 4. Our central hypothesis
posits that the performance gains from combining two models are positively correlated with
their dissimilarity; i.e., more dissimilar models are expected to yield greater improvements when
combined. Therefore, we transform our existing similarity metrics into dissimilarity scores.
Given that each of our three similarity metrics is normalized to lie within the interval [
However, a critical observation arises: the raw dissimilarity scores are intrinsically coupled with the performance gap between model pairs. Specifically, a model exhibiting superior performance () is inherently more dissimilar to models with significantly lower performance ( ) compared to models with similar performance levels. Our similarity analysis in Section 3 confirms this. This coupling poses a confounding challenge, as the observed dissimilarity may partially reflect performance disparities rather than purely complementary representations of the text. To predict improvements in combined performance, we would need to account for both these factors that contribute to dissimilarity.
To address this, we introduce an Adjusted Dissimilarity metric. This metric modifies the dissimilarity score between models by accounting for the performance gap between them. We define the normalized performance gap between models ℰ and ℰ as follows: = | − | max where max is defined as the maximum absolute performance diference observed across all possible pairs of models: The Adjusted Dissimilarity AD is then defined as:
AD = × + (1 − ) × (1 − ) where is a parameter that balances the influence of the raw dissimilarity and the normalized performance gap on the adjusted dissimilarity score. Intuitively, the term (1 − ) inversely relates to the performance gap. When the performance gap is large, (1 − ) becomes smaller, reducing the contribution of this term to overall adjusted dissimilarity.
Correlation Analysis: For each model ℰ, we compute the adjusted dissimilarity scores AD with every other model ℰ , as defined in Equation (6). Similarly, we compute the ℐndcg values, resulting from combining ℰ with ℰ . Given these values, for each model ℰ, we compute Pearson’s correlation coeficient between the set of its Adjusted Dissimilarity scores {AD } and the corresponding ℐndcg values {ℐndcg} across all model pairs (ℰ, ℰ ) for a particular similarity metric . This measures the linear relationship between dissimilarity and performance improvement. After computing for each model, we aggregate these coeficients by calculating the mean correlation coeficient () across all models. We perform this analysis per model to ofer more granular insight into model-specific behaviour. We also report an overall correlation coeficient (ove)rall per metric for all model pairs for a dataset. (5) (6) Model
For each of our datasets, we provide a detailed overview of our correlation analysis in Table 2. As per Eq. 6, we need to set a value for the
parameter to compute the adjusted dissimilarity scores. For this study we use
values of 0.7 for Jaccard, 0.5 for CKA, and 0.8 for rank similarity.
These values were tuned on Nfcorpus and then kept constant for each of the other datasets. This would indicate, contingent on the results across datasets, that these alpha values could serve as general recommendations for each metric. model’s ℐndcg values and adjusted dissimilarity scores for a particular metric. Also displayed is the overall coeficient for a particular metric
() ( overall), computed for all model pairs on a dataset.
Nfcorpus
SciFact Baseline
Jaccard
Rank Sim.
Baseline
Jaccard
Rank Sim.
Scidocs
FIQA Baseline
Jaccard
Rank Sim.
Baseline
Jaccard
Rank Sim.
Does using the similarity scores improve over simply using the performance gap as a measure? As the objective is to compare the metrics, we focus on the values across columns for each dataset. For the first column, we define the baseline as the correlation of how close the models are in terms of performance (by setting = 0 in Eq. 6) to the ℐndcg values. This would be the case of simply picking the two models that are closest in terms of retrieval performance to combine. We argue that this is a fair baseline given that we measure the improvement in retrieval performance over the better model. Using this strategy, we see a moderate positive association between our baseline scores and ℐndcg values, with the highest correlation reported for the Nfcorpus dataset. For the adjusted dissimilarity scores to be of practical value, the correlation scores for a particular metric would have to be higher than the baseline. We see that this is not the case when using CKA as a metric, with nearly identical () and (ove)rall values as the baseline for all our datasets (with the exception of FIQA where it is better). When it comes to the two functional similarity measures, Jaccard and rank similarity, we see an improvement over the baseline in terms of a stronger positive association (both () and (ove)rall) on almost all datasets. The only exception being Scidocs for rank similarity. For Scifact and FIQA, both these metrics show a significantly higher association than the baseline.
In addition to the results presented in the table, we also computed (ove)rall for each of FIQA, Scifact and Scidocs while using the similarity scores computed on NFcorpus. The resulting coeficients were 0.784, 0.758 and 0.759 respectively. This re-afirms our findings in Section 3 that the similarity scores are indeed consistent and need not be necessarily re-computed for each dataset.
In terms of the more granular model-specific view, the majority of models show a strong association between their adjusted dissimilarity scores (excluding CKA) and the ℐndcg values. However, there is a trend of lower correlation values the larger the model is. Why this is the case must be further investigated and can be considered as a current limitation of our analysis.
In this paper, we present several key findings. First, our similarity analysis reveals signicfiant diferences in both the embedding spaces learned by various models and the retrieved lists they generate. Importantly, these similarities between model pairs remain consistent across diferent datasets, allowing practitioners to avoid repeating similarity evaluations for every new task. Second, our extensive evaluation of over a hundred model pairs demonstrates that, in certain cases, concatenating embeddings leads to substantial improvements in retrieval performance. These enhancements can propel models several positions higher on the MTEB leaderboard. However, we also identify model pairs that show no improvement or even a decline in performance, underscoring the necessity for a strategic approach in selecting model combinations. Third, to address this, we leverage our similarity analysis by transforming similarity scores to an adjusted dissimilarity measure that accounts for the performance gap between models. Our results indicate a strong correlation between performance improvements and this measure, particularly when using Jaccard and rank similarity metrics. Consequently, practitioners can utilize model similarities to inform their decisions on combining embeddings, thereby reducing the need to evaluate every possible combination manually.
Looking towards the future, we aim to expand our similarity evaluation and provide a public benchmark for the same. Additionally, we plan to use our evaluation framework to deploy promising combinations of smaller models publicly.
This work is part of OpenWebSearch.eu, funded by the EU under GA 101070014, and part of CAROLL, funded by the German Federal Ministry of Education and Research (BMBF) under 01|S20049.
During the preparation of this work, the author(s) used ChatGPT in order to: Improve writing style, Paraphrase and reword. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. benchmark for zero-shot evaluation of information retrieval models, in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL: https://openreview.net/forum?id=wCu6T5xFjeJ. [27] M. Günther, I. Mohr, B. Wang, H. Xiao, Late chunking: Contextual chunk embeddings using long-context embedding models, 2024. URL: https://arxiv.org/abs/2409.04701. arXiv:2409.04701.