Semantic search is a key task in today s world, where the amount of data is growing rapidly. In this work, the main role is devoted to cases when long answers must be found to a short query (known as asymmetric search). The teacher model with 30,522 tokens of vocabulary and the student model with 119,547 tokens of vocabulary are basic for training a multilingual asymmetric semantic search model using multilingual knowledge distillation. The authors used the reciprocal rank (RR), the mean average precision (MAP), and the normalized discounted cumulative gain (NDCG) to evaluate the obtained model.
Semantic search is a set of search algorithms that work based on understanding the meaning of text.
This approach effectively handles synonyms, abbreviations, and spelling errors, unlike keyword
search engines that rely on exact lexical matches to find documents. It is useful for grading and
assessing academic work [
Ontologies and knowledge graphs are classical approaches to the design and implementation of
semantic search systems, but recently these approaches have been enriched or replaced by statistical
algorithms [
This work aims to study the possibility of using the bi-encoder architecture to implement asymmetric semantic search when storage contains indexed data in English with the necessary to search information in Ukrainian.
The intention behind semantic search is to transform passages into a multidimensional vector space. During the search, the query is similarly embedded in this vector space, which allows to identify the desired quantity of most relevant matches. This approach ensures that even if the wording is different, the meaning remains the same and the system can provide accurate retrievals (Figure 1).
There are two main types of semantic search that work with different types of data. For symmetric search, queries and passages in the corpus have approximately the same length and content. In turn, asymmetric search typically uses short queries (e.g., a question or a few keywords) and longer passages answering those queries.
Nowadays, a lot of trained asymmetric semantic search models based on both bi-encoder and
cross-encoder architectures [
To use asymmetric semantic search for less common languages or even a mix of them, we need
to train new models, which will require a lot of data and computational power. Fortunately, there is
a way to facilitate this using multilingual knowledge distillation [
This approach requires a teacher model for the source language and a set of pairs (each pair includes a sentence in the source language and its translation). A new student model attempts to approximate the output of a teacher model for both source and target sentences using the mean squared error (MSE) loss. The student model could have the structure and weights of the teacher model, or it could be a different network architecture since the student model learns representation of the teacher model. This allows the student model to achieve robust generalization across languages.
It is critical that identical sentences in different languages have similar vector representations. That is why the vector space properties of the source language, obtained from the teacher model, must be applied to other languages.
From the information above, it is evident that the input parameters of semantic search models are queries and passages. In the field of multilingual search, there are three main paradigms: • Multilingual-to-monolingual: this approach allows to accept queries in multiple languages and compare them with passages in a single language. • Monolingual-to-multilingual: conversely, the idea behind this paradigm is to accept queries in a single language and return passages in multiple languages. • Multilingual-to-multilingual: provides the highest search adaptability and allows users to formulate queries and peruse passages across multiple languages. However, this flexibility may reduce the average accuracy due to the large number of possible language pairs.
This paper explores the scenario when a storage contains indexed data in English and it becomes necessary to search information in other languages, particularly Ukrainian. We will take the original model (data in the index must be produced by this model) trained for asymmetric semantic search, using the bi-encoder architecture, and add capabilities to handle multilingual queries (in our case, bilingual, i.e., English and Ukrainian).
Using knowledge distillation will significantly reduce the training time since we will mimic the teacher model on translated queries. To avoid unnecessary training on passages and to ensure consistency between previously indexed and new documents, we will continue to use the teacher model to create embeddings for new passages. The output of both models can be evaluated using a similarity metric to find the most relevant passages (Figure 3).
To determine how similar two vectors are, we can use different metrics (cosine similarity, Euclidean distance, and dot product). Measuring Euclidean distance for high dimensional vectors becomes impractical, as they will be very far apart simply because of the vastness of the space they inhabit. Using cosine similarity, which measures the angle between two vectors by paying attention to their direction and ignoring magnitude, or dot product, which measures the overall congruence .
In practice, we can generate sentence embeddings using specially designed modifications of the Bidirectional Encoder Representations from Transformers (BERT) model, known as Sentence Transformers (SBERT). It provides two types of state-of-the-art asymmetric semantic search models, one tuned for cosine similarity, the other for dot product. Cosine similarity tuned models prefer to retrieve shorter passages, while dot product tuned models prefer to retrieve longer passages.
3.1. MS MARCO
Microsoft Machine Reading Comprehension (MS MARCO) [
The dataset comprises over a million anonymous questions offering a collection of short, real-world, natural language queries split into the train (808,731 queries), development (101,093 queries), and evaluation (101,092 queries) subsets. We will use the train and development subsets to train and evaluate the student model respectively, so it makes sense to look at their length and word patterns (Figure 4).
Furthermore, the dataset contains 8,841,823 passages that are required to provide natural language answers (a question may have multiple answers or no answers at all). Although passages are not involved in the knowledge distillation for multilingual-to-monolingual models, it still seems reasonable to analyze them to figure out what the teacher model was trained on (Figure 5).
The length distribution of the combination of queries from the train and development subsets is markedly skewed, with the majority being quite brief, typically under 100 characters, predominantly concentrated between 15-25 characters. The frequency drops significantly for longer queries, with very few extending beyond 100 characters. The most common bigrams (two-word sequence of words) indicate that the majority of queries are fact-oriented. These patterns reveal that a significant number of queries are short questions asking , , ,
The length distribution of the passages has a right-skewed plot with the noticeable peak around 200-300 characters. The character type distribution shows that letters are mostly used, with a median of about 250 characters per passage and a wide interquartile range reflecting considerable variation. Digits and punctuation marks are used less frequently, but there are outliers indicating some passages with a large number of such characters.
The scale and real-world nature of the dataset makes it attractive for training and evaluating
machine learning models, but the original MS MARCO contains only English search queries. To train
and evaluate models for other languages, the dataset must be translated in one of the available ways.
For Ukrainian, the OPUS-MT English to East Slavic neural machine translation model [
Although the deliberate selection of the queries at an intuitive level implies high-quality data and a lack of outliers, it still seems reasonable to check directly the length and word patterns (Figure 6).
The length distribution peaks at 20-25 characters, indicating that most queries are concise and right-skewed, with fewer longer queries beyond 60 characters. The most frequent bigrams begin with common phrases such as "what," "definition," and "how," suggesting that queries are structured as questions, often seeking definitions or factual information.
Looking at the queries from both datasets, it is clear that TREC 19 queries are generally shorter and more focused on concrete evidence, while MS MARCO queries might be significantly longer and more complex. It is also important to note that TREC 19 does not have outliers, unlike MS MARCO, where some queries can exceed 400 characters.
These days, the TREC 19 has become a recognized benchmark for various information retrieval and deep learning tasks such as document and passage retrieval. In the task of full search, we can evaluate up to 1,000 passages for each query based on their estimated likelihood of containing the answer.
To evaluate the trained model, all 200 queries were manually translated into Ukrainian by a native speaker, which guarantees the veracity of the results.
We will use the DistilBERT base multilingual (cased) [
The MS MARCO DistilBERT base v4 [
We trained the student model for 5 epochs with a batch size of 24, 10,000 warm-up steps, and a learning rate of 2e-5. The entire training process on the train and development (used for intermediate evaluation) subsets took about 8 hours using the Apple M3 Max chip (16- ore CPU, 40- ore GPU). To measure the difference between computed and target query embeddings we used the MSE loss (Figure 7).
The teacher model has a vocabulary of 30,522 tokens, the same as the original BERT base. This vocabulary includes common words, sub-words, and special tokens to deal with English. The student model, on the other hand, is a distilled version of the BERT multilingual. The extended vocabulary of 119,547 tokens covers a variety of multiple symbols from different languages, allowing to efficiently process and understand text in different linguistic contexts.
Any dataset for semantic search or information retrieval systems has selection bias. It can be related to the data source, the date of publication, and the personal preferences of the publisher. Achieving high performance on cross-lingual tasks depends on the ability to seamlessly map sentences from different languages into a single vector space. The similarity between alphabets also significantly affects accuracy. Languages with similar alphabets, such as English and German, typically produce more accurate results than languages with dissimilar alphabets, such as English and Ukrainian.
Using the original English and machine-translated Ukrainian search queries, we achieved a reasonable level of accuracy (about 93%) for English to Ukrainian and Ukrainian to English translation tasks, indicating a high level of proficiency in both directions (Figure 8).
It is very important to monitor performance on the subset that did not participate in training to avoid overfitting. MSE on the evaluation subset steadily decreases, indicating that the student model learns to more accurately mimic the results of the teacher model (Figure 9).
Evaluating the performance of information retrieval systems is a critical step in improving their efficiency. NIST assessors labeled the TREC 19 using multi-graded judgments, making it easy to measure all the necessary and widely used state-of-the-art metrics.
Reciprocal rank (RR) calculates the score for the first relevant passage in the ranked list. This metric is very important when we need to evaluate the occurrence of the most relevant passage: where rel the rank of the first relevant passage in the list.
Mean average precision (MAP) evaluates both the relevance of the suggested passages and the position of the most relevant passages at the top. For each query, the average precision (AP) is determined by calculating the arithmetic mean of the precision scores for every position in which a relevant passage was found. This metric focuses on the ability to distinguish relevant and irrelevant items: where Q
the total number of queries.
Normalized discounted cumulative gain (NDCG) measures the ability of machine learning algorithms to sort passages by relevance. NDCG is determined by dividing the discounted cumulative gain (DCG) by the ideal DCG, representing the best version of the rating. This is useful in many scenarios where we expect passages to be sorted by relevance: =
, 1 ∑ , (1) (2) (3) where k the number of items considered in the calculation.
Both MAP and NDCG reflect ranking quality, but account for rank reduction in different ways. MAP gives more weight to relevant passages at the top of the list because this metric is based on precision, making it more sensitive to changes in early positions. DCG assigns decreasing weights to passages proceeding down the hierarchical order, but they are logarithmic and reduce the contribution of passages not very quickly.
In the case of models based on the bi-encoder architecture, it makes sense to evaluate up to 100 passages using the above metrics (Table 1). If the obtained performance is not sufficient, a reranking model based on the cross-encoder architecture can be used to improve results. Performance of the original and trained model on TREC 19
msmarco-distilbert-base
v4 msmarco-distilbertmultilingual-en-uk
RR 0.96
The bilingual model trained using knowledge distillation inherited the capabilities of the monolingual teacher
model and shows reasonable results for both languages. Obtained 768-dimensional vectors are optimized to work with cosine similarity as expected.
We can notice a slight decrease in performance for English, which is not critical since we got the ability to search passages using multiple languages. A higher score does not necessarily mean higher performance in production. At some point, models can become too specialized on the MS MARCO and its selection bias. and passages.
It is impossible to create a perfect dataset for semantic search. Manual creation of a subset even for evaluation is quite expensive and unfortunately always has some selection bias. This is a longrecognized problem, but there is no good solution for it, especially at the scale of millions of queries
The number of languages may not be limited to two, but it is better to reasonable add only justified languages and keep a balance between performance and multilingual search capabilities. Otherwise, new languages to the model degrades performance because the capacity of the model remains the same.
The performance of the trained model was also affected by the quality of the data, which in our case is a combination of original and machine-translated queries. Cloud solutions such as Google Translate or DeepL could marginally improve the results but would not compare to manual translation by native speakers. The time taken to create pairs using the neural machine translation model was about 50% longer than the time required to train the student model. Intuitively, it makes the knowledge distillation less of a training and more of a translation task.
For multilingual processing of asymmetric semantic search queries, the DistilBERT base multilingual (cased) as a student model and the MS MARCO DistilBERT base v4 as a teacher model can be used as components in the scenario when storage contains indexed data in English with the necessity to search for information in Ukrainian up to the reciprocal rank (RR), the mean average precision (MAP), and the normalized discounted cumulative gain (NDCG) as its evaluating.
The authors achieved a reasonable level of accuracy (about 93%) for English to Ukrainian and Ukrainian to English translation tasks, indicating a high level of proficiency in both directions. In the future, this result will be improved by the quality of the data, which can be a combination of original queries and manual translations by native speakers. The resulting model is useful in industries such as finance, healthcare, and e-commerce, where huge data sets are prevalent and asymmetric semantic search plays a key role to quickly retrieve relevant information.