In this work, we present the submission of TU Dresden to WOWS 2024. Four student teams assembled different approaches for Genre Classification, Text Snippet Extraction, Query Expansion, and Text Features. Each implemented component integrates seamlessly into the open web search ecosystem. We present each approach alongside a short evaluation of possible use cases, and hope that our submission will contain viable building blocks for future research to be build on top.
This report describes the submission of the team at TU Dresden for the Workshop on Open Web Search WOWS 2024 [1]. The work was conducted during a university-organized hackathon targeted at students. Details about the setup are included in the Appendix in Sec. A. Four teams, consisting of two to three students contributed four components for the open web search ecosystem. We hope that with our submitted components future research on Information Retrieval (IR) can be facilitated.
In summary, this paper is discusses the following four components: Sec. 2 reports the work of the group Genre Classification, which categorizes web pages based on the intent of the page, such as Discussion or Shopping. In Sec. 3 we detail our the submission for the extraction of text snippets. Here, the goal is to divide long documents into shorter ones and return a list of the best snippets. Sec. 4 provides details on the work of the group Query Expansion, which employed Large Language Modelss (LLMs) to generate more related information or variants for a given query. The results for the extraction of text features is highlighted in Sec. 5. The goal of this component was to quantify syntactic or semantic features of natural language such as the readability of a web page. Finally, Sec. 6 draws the conclusions of all our submissions.
The rule-based classifier makes use of a vocabulary list of relevant terms per genre. Comparing the intersection between terms in the genre-specific vocabulary lists, and the terms in the document, the most probable category is the one with the highest intersection. We first remove stop words and subsequently extract the 75 most frequent terms that we compare to the vocabulary lists to classify the genre. We use Snorkel AI [4] for implementation.
The rule-based classifier can be adapted to a precision-oriented method, where the most probable genre needs to be better than a threshold compared to the second most probable genre, otherwise the classification result is abstain.
As a typical Machine Learning based method a neural network was used for classification. As features the web pages were converted into a tf-idf vectorspace. We use the Python library scikitlearn [5] for the implementation of the Multi-Layer Perceptron classifier. After an empirical hyperparameter search a neural network using a single hidden layer of 50 neurons, ReLU activation function, stochastic gradient descent in the Adam variant using momentum for optimizations, and a constant learning rate of 0.001 was used.
For evaluation we used the Genre-KI-04 dataset [2]. This includes vocabulary lists, and the following classification categories: articles, discussion, download, help, link lists, portrait (non private), portrait (private), and shop. Details about the genres can be found in the original paper.
Since sophisticated neural ranking models such as cross-encoders generally require a lot of computational effort, a customary retrieval pipeline first retrieves a number of (e.g., 1000) documents using a fast but imprecise retrieval method and then re-ranks those documents using a more precise weighting model [6]. Cross-encoder as introduced by Nogueira and Cho [6] are an example for the latter, which are used to calculate scores for query-document pairs. Apart from their comparatively high computational cost, cross-encoders have another disadvantagetheir limited input size. This weakness is typically mitigated by truncating the document once the maximum number of input tokens is reached. The problem of this procedure is that content which is not in the beginning of a document is not taken into account by cross-encoders. As a result, the ranking of documents may be biased towards those that address the query early on.
In this part, we therefore present a simple method of extracting a number of snippets, i.e., smaller chunks of the document which fit in the cross-encoder as an additional component in a larger retrieval pipeline. Instead of simply truncation documents after a fixed number of tokens, we search for the most relevant passages (ranked snippets) in the document. These ranked snippets are used for the cross-encoder with the goal of a more precise ranking. We show the benefits of this method on two exemplary datasets which contain long documents.
The re-ranking process with ranked snippets consists of five steps. An example of those steps for the re-ranking of n = 3 documents (d 1 , d 2 , and d 3 ) is shown alongside the explanation.
First, we subdivide all n documents into snippets. The maximum length of those snippets may be chosen arbitrarily-we defaulted to 250 tokens which is the passage size used by Dalton et al. [7]. The actual length of the snippets may vary since the division process aims to retain context by not separating sentences. For example, we may start with three documents d 1 , d 2 , d 3 . After the first step, each of these documents is divided into several snippets:
, s 1 3 s 2 3 . . . s l 3 3 where s j i denotes the j-th snippet of document d i for j ∈ { 1, . . . , l i } and l i is the number of snippets of d i .
In Step 2, we pre-rank all extracted snippets in relation to the query. To accomplish this, we view the set of all snippets of a document as a corpus. From this corpus, we can create a ranking for the query using one of the following weighting models: Term frequency (TF), BM25 or PL2. We do not use cross-encoder for the pre-ranking of documents, because there may be a multitude of snippets per document depending on document length and therefore ranking all snippets using a cross-encoder can drastically slow down the re-ranking process. After this pre-ranking step, our example snippets might be ranked in the following way:
Step 3, we can obtain the top k relevant snippets of each document from the pre-ranking, which are later ranked using a cross-encoder. This step ensures that the cross-encoder only needs to rank n • k snippets for n documents instead of all snippets. In order to reduce computational cost, we defaulted to k = 3. In our example, this step results in the following selection:
Here, s 4 3 is not selected as one of the top snippets of d 3 since it is the (k + 1)-th snippet of d 3 in the ranking despite being ranked relatively high.
In Step 4, the top k snippets of all documents are ranked using a cross-encoder (CE). That way, similar to Step 2, we can more accurately deduce which snippets best match the query-but now the ranking is more precise since we used a CE instead of the simple weighting models used in Step 2. An examplary ranking for our snippets might be: s 4 2 > s 3 3 > s 2 2 > s 5 1 > . . . . The final document ranking ensues from this snippet ranking in Step 5, i.e., the document that provided the best snippet is ranked first. Our example documents are therefore ranked in the following way:
It should be noted that the goal of this section is to rank documents with regard to a query, and not only passages. Therefore, the result is a ranking of documents. Details on our implementation can be found in Appendix B.1.
In this section, we conduct tests to study the possible improvements of our cross-encoder re-ranking of top k snippets. As baselines, we use BM25 and the dense retriever MonoT5. All further ranking is performed on the top 20 documents retrieved by these two systems. We evaluate the re-ranking with our TF-ranked snippets. For this, we load the previously saved top 3 snippets for each document. To re-rank the documents, we follow the "weakest link" principle, selecting the minimum TF score among the top 3 snippets. This results in the methods BM25+TF-SP and MonoT5+TF-SP. We denote by +CE that the 3 snippets are further re-ranked by a cross-encoder. In addition, we compare the performance of these systems to the cross-encoder's performance when only evaluating the first snippet of each document (which resembles the naïve application of a cross-encoder). These results are denoted by BM25+CE and MonoT5+CE.
To measure the performance of the approaches, we utilize normalized discounted cumulative gain at 10 (NDCG@10) and mean reciprocal rank (MRR). We conduct our tests on the ClueWeb12 [8] and ClueWeb09 [9] datasets, which differ in document size: ClueWeb12 has an average document size of 5641.7 tokens, and ClueWeb09 has an average document size of 1132.6 tokens. The results for the two datasets are plotted in Fig. 2. Our approach of cross-encoder re-ranking with TF-pre-ranked snippets achieves the best performance in both metrics across all our tested datasets (see Appendix B.2 for diagrams of other evaluated datasets). The impact of our TF-ranked snippet pre-selection is relatively high on ClueWeb12 with long documents, while it is more marginal on ClueWeb09. This highlights the importance of snippet pre-selection for longer documents. ClueWeb09 consists of approximately 6 snippets, and ClueWeb12 consists of approximately 23 snippets per document. We assume for our naïve snippet generation approach that information is equally spread throughout a document. A cross-encoder taking the first snippet as input is assumed to capture more relevant information of a document with a size that is closer to the cross-encoder input size. This also explains why MonoT5 scores better on the shorter dataset, especially in comparison to BM25, since MonoT5 also suffers from a limited input size. This proves that there is a need to address the problem of limited input size, especially in large documents like those in ClueWeb12. That information is not always equally spread over a document, like we assumed for our snippet generation, can be concluded when comparing Figs. 4b and 4c. This raises the need of a more advanced approach for snippet generation.
Overall, our results show that selecting top-k pre-ranked snippets is a viable approach to tackle the problem of input size restrictions on Transformer-based retrieval systems. Especially, crossencoders can benefit from this approach since they are inefficient on large documents. Further testing to edge out efficiency and reduce context loss with snippets will be required. Also, it would be beneficial to test multiple pre-ranking systems and values of k for top-k snippet selection. The code for this part can be found in the accompanying repository2 .
Query Expansion and User Query Variants are two common methods to increase the recall of an IR system [10,11,12]. Both methods are based on modifying the query to include more related keywords, thereby causing the IR system to score relevant documents higher. In addition to conventional techniques such as the Kullback-Leibler Divergence (KL) [13,14] or Relevance Model 3 (RM3) [15], recent approaches have embraced the utilization of Large Language Models (LLMs). In this part, we employ various prompts to generate improved and expanded queries using LLMs [16,17], in particular, GPT-3.53 , Llama2 [18] and FLAN-UL2 [19].
LLMs have previously been in use for the task of query expansion and studies have been conducted using various methods and language models [16,20,21,22]. Wang et al. [21] employ query2doc, a method where the LLM generates a document for a given query, which is then used as Pseudo-Relevance Feedback (PRF). Model parameters, for GPT-3.5 the parameter min. tokens is unavailable in the API previous work demonstrates improvements across different datasets. In order to weigh the original query more heavily, multiple concatenations of the original query q with a single instance of the LLM's output may be used [16,21]. The resulting expanded query is of the form q ′ = concat({q} * n, LLM out ), where n is the number of times q is concatenated with itself, and LLM out is the LLM-generated version of q. We adopt this approach in our work with n = 5 and employ modified versions of the prompt types suggested by Jagerman et al. [16] While GPT-3.5 and FLAN were prompted to generate five similar queries, Llama was asked to answer the query. Apart from this difference, the prompts for in all experiments are similar and comparable.
Initially, the query, along with the prompt, is fed into the LLM, and its response is concatenated with the original query (n = 5). For evaluation, the Recall@1000 metric of the original and modified queries is compared on the given dataset using BM25. The specific LLMs in use are GPT-3.5 45 , Llama 2 [18] and FLAN-UL2 [19]. Llama 2 and FLAN-UL2 were run locally. Table 1 shows the model configurations that we used in our experiments. The temperature values were chosen empirically in a way such that model outputs are roughly similar. The lower and upper token limitations prevent generation edge cases such as empty responses or endless output, while still allowing for expressive responses. Local models had to fit GPU memory constraints. Hence, we had to employ the quantized versions of the models. We conducted experiments for the prompt types presented above: CoT, Q2E/FS, and Q2E/ZS. While FLAN-UL2 and GPT-3.5 can be prompted without further changes, Llama 2 requires the chat-prompt to follow a pre-defined format, our version of which can be found in the project's repository 6 . We utilize BM25 as the retrieval system in the default configuration of the Tira-framework [23]. The query expansion baselines consist of an unmodified BM25, BM25 with Kullback-Leibler Divergence (KL) and BM25 with RM3.
We measure the recall, which is aggregated over 18 datasets per model, and per prompt type. The datasets cover a range of diverse topics and were provided as part of TIRA [24] / TIREx [25]. The aggregated results can be observed in Table 4.2. Avg easy excludes evidently (cord19, longeval) and presumably (medline) difficult datasets. This highlights the difficulties LLMs experience on specific datasets, especially domain-specific ones: Excluding those, CoT+OpenAI GPT-3.5 Turbo (GPT) now performs 0.03 points better than baseline models. Note that the two Avg rows cannot be compared to one another, as baseline scores have also shifted due to the exclusion of generally low-performing datasets. Detailed results for each dataset can be found in Appendix C.1. For our query expansion approaches, it is evident that the choice of prompt has a large impact on recall performance. The combination of CoT and GPT consistently yields the highest recall in absolute numbers. However, with other prompt types such as Q2E/ZS and Q2E/FS, GPT also frequently achieves the highest recall per dataset, albeit less frequently compared to CoT. In this regard, our results are consistent with those reported in [16]. Although CoT generally performs the best, it exhibits poorer results than the baselines in datasets such as cord19 or the longeval datasets. In these cases, Q2E/ZS and Q2E/FS emerge as better choices, but are still commonly outperformed by the baseline models. Q2E/FS exhibits less convincing effectiveness, presumably because it mimics the relatively short responses of example queries through the Q2E/FS method, resulting in short queries with few new keywords. Q2E/ZS behaves similarly. Although the responses of the LLMs are longer compared to Q2E/FS, as the LLMs do not conform to the rather short examples, the generated responses are overall less extensive than those of CoT, likely resulting in inferior effectiveness. Considering the longeval datasets and cord19, it is evident that they contain either very general or highly specific queries. In the case of nonspecific queries, there is a risk that they may be muddled by the consequently more general, and in the case of CoT, extensive responses from the LLMs. This effect might potentially be reversed by conveying the user intent to the LLM, indicating whether, for instance, in the case of the query "car," the user intends to buy one or have it repaired. With domain-specific queries, it is plausible that models were trained with insufficient knowledge on the subject, resulting in subpar effectiveness.
While our main evaluation is conducted using recall@1000, we also evaluated nDCG@10. RIX [34] Long Word Count Sentence Count
Implemented Text Features with the respective formulas. Syllable count and word count were implemented using the provided tools by the Text Feature Libraries Textstat and spaCy.
The results for this metric are detailed in Table 4 in the appendix. Overall, our conclusions for nDCG are similar to those for recall. The generations for each model and each prompt are publicly available in our repository 7 .
In this part, we generated different versions of query expansions using three LLMs and three prompt templates. We were able to demonstrate that LLMs are capable of improving the recall of user queries. The combination of the prompt CoT alongside GPT proves to be the most promising, improving recall scores by up to 15%. Future research could focus on further templates for using the generated expansions since we only evaluated the qqqqq, response-format.
Text Features are quantified metrics describing syntactic or semantic features of natural language. An example is the readability of a text, useful for returning user-dependent search results. A search engine targeted to school children should return results with a high readability score, whereas a search engine with domain experts as target audience will also include texts with low readability scores. Additionally, this could be used to filter out noisy websites. This Open Web Search component8 incorporates two tools for computing text features, namely Textstat [26] and textdescriptives [27] from spaCy [28]. SpaCy's Text Feature analysis is more comprehensive than the one in Textstat, but is less efficient. Per design of the pipeline approach of SpaCy many things are computed in the background, from which only a few are required for the calculation of the text features. This overhead results in a longer runtime which should be considered. Table 3 displays the implemented text features.
Additional contributions besides the integration of the text features components include examining a potential correlation between text features and documents evaluated as relevant by ranked retrieval models. For easier exploration of the document corpus we provide an interactive Jupyter Notebook showing correlation graphs between Ranked Retrieval and Text Features, applicable to arbitrary datasets, as well as the analysis of correlations between Ranked Retrieval and Text Measures.
To verify the capability to differentiate between levels of Readability, unit tests were used. The test data consists of multi-sentence snippets from web pages. These were categorized by difficulty in the following categories (including the amount of test documents): children (3), teenagers (3), academic (3), and simple language (2), depending on what demographic the source was directed to. Initial tests involved a project member assessing the reading level of excerpts and comparing their assessments to the classifications provided by the automated measures, thus proving correct usage of the used text feature libraries at least for the readability scores.
Compared to human assessments, the automated Text Measures often overestimated the reading level, possibly failing to capture the complexities of human reading abilities within their respective indexes. Large-scale dataset computations further highlighted the discrepancies between predicted and human-classified reading levels, corroborating these findings. Despite the observed differences in assessment, the data suggested an inverse proportional relationship between comprehension levels and readability measures.
The experiments were run on the "antique/test" [35] dataset from the ir_datasets collection [36]. Based on TIRA [37] ranked retrieval models were used to create top-10 results.
A primary objective of our project was to investigate whether the ranking of relevant documents by ranked retrieval models correlates to document Readability.
First we looked at the top 10 retrieved documents for all queries across multiple retrieval models, the resulting distributions are displayed in Figure 3. The majority of results, assessed using the Flesch Reading Ease, indicates comprehension levels at or below an eighth-grade level, implying a high degree of readability. The high degree of readability was consistently observed across multiple retrieval models. Compared to the overall readability across all documents in a collection, we found that some retrieval models like SBERT or MonoT5 indeed result in a higher readability in the retrieved documents compared to the rest of the corpora, suggesting a potential relationship between relevancy and readability, whereas other retrieval models such as BM25 do not share this characteristic.
The paper's work was carried out by students from TU Dresden as part of a one-week hackathon.
The workshop was open to students in the Computer Science program and related fields, and they could earn ECTS credit points for lab work. The hackathon was advertised on the mailing lists of the beginner Information Retrieval courses from the past three years. Interested students could fill a survey indicating their preferred timeframe for the hackathon.
After a date was decided, 10 students signed up for the hackathon, three from the Bachelor's program and seven from the Master's program. The university supervisors prepared four topics, which were advertised beforehand, and the students signed up for their preferred topics. The text features topic was designated to the 3 Bachelor students. The Master students were provided with a peer-reviewed research paper as additional material, which they were required to read and understand before the hackathon.
On the first day of the hackathon, an invited member of the Open Web Search project provided a brief introduction to the Open Web Search ecosystem and TIRA/TIREx. Following this, the teams worked on their components, with supervisors providing guidance through daily checkins. On the fifth and final day of the hackathon, a short presentation from each team was held. Following the hackathon, the students were requested to prepare a report on their work, which served as the basis for this paper.
In retrospective, the short amount of time, one week, motivated the students to work diligently on their project. However, at the end of the week, the students had several open ideas for future work which they could not finish in time. Therefore, more time, even a few days more, might be beneficial for the next iteration of the hackathon. The size of the group ranged between two and three members. The small group size facilitated the organization within each group and kept the management overhead small. The topics of the hackathon were aligned with the basics gained during the Information Retrieval course, but required also reading additional literature and research.
To implement the described re-ranking steps, we utilized several Python libraries, detailed below to facilitate reproducibility. For snippet extraction in Step 1, we adapted the Spacy-PassageChunker class from the corpus_processing package, as provided by Dalton et al. [7], to allow for variable snippet sizes. The class requires spaCy [38]; we used version 3.3.0 for our implementation. The snippet pre-ranking in Step 2 was implemented using PyTerrier [39], version 0.10.0. For Step 4 we utilized ms-marco-MiniLM-L-6-v2 which has been published on HuggingFace.co [40]. To embed the model into our project, we used the transformers library [40], version 4.38.2, and the PyTorch library [41], version 2.2.0. The results of the preparation steps are accessible via TIRA [24] / TIREx [25].
We would like to express our gratitude to the Open Search Foundation for organizing the WOWS 2024 and especially Maik Fröbe, who supported us and our student teams in organizing and conducting our Hackathon which made this submission possible.
In addition, the authors gratefully acknowledge the computing time made available to them on the high-performance computer at the NHR Center of TU Dresden. This center is jointly supported by the Federal Ministry of Education and Research and the state governments participating in the NHR (www.nhr-verein.de/unsere-partner).
Prompt formats used for Meta Llama 2 7B Chat (Llama). Note the necessity for a system prompt and the additional formatting sequences due to the instruction fine-tuning of Llama-Chat. Prompts were modified to fit Llama's behaviour.