Enhancing Fusion-in-Decoder for Multi-Granularity Ranking

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance across various natural language tasks, leveraging extensive knowledge from massive datasets. However, their reliance solely on parametric knowledge often leads to the generation of inaccurate or outdated content, particularly in domain-specific tasks. Retrieval Augmented Generation (RAG) has emerged as a promising approach to address this limitation by incorporating external knowledge without necessitating re-training. While RAG enhances the accuracy of LLM-generated content, efectively retrieving external knowledge remains a challenge due to potential noise and computational costs. To address this, traditional information retrieval systems adopt two-stage approaches, utilizing eficient retrievers followed by reranking mechanisms. Recently, transformer-based architectures, including BERT and T5 models, have shown promise as efective rerankers. However, such models have limited context size and only perform single-granularity ranking at a time, hindering their efectiveness and eficiency. In this paper, we first explore the existing rerankers such as RankT5 and RFiD, highlighting challenges in multi-granularity ranking. Subsequently, we introduce PFiD (Passage Fusion-in-Decoder), a simple yet eficient approach aimed at efectively ranking both document and passage simultaneously. Through empirical evaluation, we demonstrate the eficacy of PFiD in improving efectiveness and eficiency, ofering a promising direction for further research in this domain.

Keywords

• eol>Information Systems
• Retrieval Augmented Generation
• Large Language Model

1. Introduction

Despite their remarkable capabilities and growth, Large Language Models (LLMs) [ 1, 2, 3, 4 ] still tend to generate factually incorrect or outdated content as their knowledge solely relies on their parametric knowledge, especially in domainspecific or knowledge-intensive tasks [ 5, 6, 7 ]. Retrieval Augmented Generation (RAG) approaches [ 8, 9, 10, 11 ] have gained significant attention, which improve the quality of LLM-generated output by grounding on external knowledge to supplement the LLMs’ parametric knowledge, without having to re-train the LLMs. RAG leverages a powerful information retrieval model, which is designed to search large datasets or knowledge bases. The retrieved information is then incorporated into LLMs, enabling it to generate more accurate and contextually relevant content. By incorporating external knowledge, RAG can efectively reduce the problem of generating factually incorrect or outdated content in LLMs [ 12, 13 ].

However, current RAG frameworks have major challenges when it comes to the efectiveness and eficiency of information retrieval systems: First, LLMs tend to generate inaccurate responses on distracting (or noisy) contexts, thus the performance of retrieval models has a significant impact on the quality of RAG’s responses [ 14, 15, 11 ]. Second, the retrieval component of RAG requires searching through large-scale knowledge bases or the web, which can be computationally expensive and slow [ 11 ]. Due to the above challenges, existing retrieval systems adopt two-stage approaches, an eficient first-stage retriever such as BM25 [ 16 ] and DPR [ 17 ] retrieves a set of documents from a larger dataset, and then a second-stage reranker is used to rerank retrieved documents for precise ranking. Recently, with the advent of transformer-based models such as BERT [ 18 ] and T5 [ 19 ], more architectures including bi-encoder [ 17 ], crossencoder [ 20 ], encoder-decoder [ 21, 22 ], and decoder-only models [ 23 ], have gradually shown their efectiveness as a reranker. However, these models have limited context size and only perform single-granularity ranking during inference, which hinders their efectiveness and eficiency in real-world RAG scenarios.

To this end, in this paper, we focus on the multigranularity ranking task, which ranks both document and passage simultaneously. Specifically, we first investigate the single-passage cross-encoder models such as MonoT5 [ 22 ] and RankT5 [ 21 ]. It achieves superior performance across various ranking tasks, but due to the constraint of input tokens, its eficiency is limited in real-world RAG scenarios. Next, we present the use of multi-passage cross-encoder, such as FiD [ 9 ] and RFiD [ 24 ]. These models alleviate the input tokens limit by leveraging multi-passage, but they directly use the cross-attention score of the decoder as a passage relevance, which is implicitly learned, and encounter dificulty with distinguishing relative diferences between passages. Thereafter, we propose a simple and efective PFiD (Passage Fusion-in-Decoder) for multi-granularity ranking. PFiD extends the FiD model by generating a document-level relevance token, enabling both document retrieval and passage ranking. Furthermore, PFiD adopts the inter-passage attention mechanism to learn relative passage relevance explicitly, using the special tokens at the beginning of the input text to represent the entire context.

Experiments on MIRACL passage ranking dataset [ 25 ] demonstrate that PFiD improves efectiveness and eficiency compared to existing approaches, especially in RAG scenarios.

2. Preliminaries

2.1. Task definition

Given a user query and a document (or passage) corpus = {1, 2, ..., }, the goal of document retrieval is to ifnd the documents that are most relevant to the query . In our multi-granularity ranking setting, which consists of document retrieval and passage ranking tasks, the document retrieval task is to perform reranking on BM25 retrieved top- documents. While traditional passage ranking tasks typically involve ranking entire passages, in this paper, the passage ranking task focuses solely on ranking passages within the retrieved document itself, which aligns more closely with real-world RAG scenarios and is thus more feasible.

2.2. Ranking models

Pre-trained Language Models (PLMs) are currently the most efective ranking models, which can be categorized into: biencoders and cross-encoders. Bi-encoders encode a query and a passage separately to obtain semantic representations [ 17 ], emerging as powerful first-stage retrievers by pre-computing the passage representations ofline. Instead, cross-encoders take the concatenation of the query and a passage, and perform query-passage interactions [ 20 ], which have been conceived as second-stage rerankers, designed to explicitly refine the results provided by the firststage retrieval. In this paper, for brevity, we also refer other PLMs, such as encoder-only [ 17, 20 ], decoder-only [ 23 ], and encoder-decoder [ 21, 22 ] models that perform querypassage interactions simultaneously, as cross-encoders.

There are several PLM-based cross-encoders, including sequence-to-sequence language models such as MonoT5 [ 22 ] and RankT5 [ 21 ] for ranking task, as well as multi-passage reader models like FiD [ 9 ] and RFiD [ 24 ] for RAG tasks, which have been demonstrated superior effectiveness.

MonoT5. MonoT5 [ 22 ] is the first work to define a ranking task as a text generation task by leveraging T5 [ 19 ] encoderdecoder model. A query-document pair is concatenated into an input sequence Query : Document : Relevant:, and utilizes true and false as target tokens to represent their relevance. Then, the model is fine-tuned for text generation task. After training, the ranking scores are derived from the logits of true token, based on the softmax applied only on the logits of the true and false tokens. RankT5. Following MonoT5 [ 22 ], the input sequence is similar except that RankT5 do not include the Relevant: postfix. Then, the model use the <extra_id_10> as target token to learn unnormalized ranking score. The model is trained with list-wise ranking loss directly, instead of using text generation loss as in MonoT5 [ 22 ]. However, these models cannot be directly used for long document retrieval due to the maximum input length constraint as in most PLMs, which hinders their efectiveness in the document retrieval task.

FiD. The FiD model further extends T5 [ 19 ] encoder-decoder model, taking multiple passages as input, encoding separately, and then feeds the concatenated encoder hidden states into a T5 decoder to generate the answer. Relevance scores for passages are computed using cross-attention scores, which entail averaging the attention score across all tokens within the passage and all layers and heads within the decoder [ 26 ].

RFiD. While FiD [ 9 ] treats all passages equally within its encoders, solely depending on the cross-attention mechanism to establish correlations between the decoder and encoders, which may identify the incorrect answer by referring to spurious passages. Instead, RFiD [ 24 ] improves FiD by identifying potential answer-containing passages (or rationale) among the candidates and guiding the decoder with the identified rationales. Afterward, cross-attention scores are directly regarded as passage relevance scores the same as in [ 9 ]. However, even with the rationale, the cross-attention mechanism still lacks for distinguishing relative diferences between passages, as it is implicitly guided by a rationale classifier solely trained on point-wise binary classification loss.

3. Method

In this section, we briefly discuss a simple but efective Passage Fusion-in-Decoder (PFiD) for multi-granularity ranking. PFiD adopts the FiD [ 9 ] architecture as a base model, further extends FiD by utilizing true and false token as a target token to model document-level relevance, enabling multi-granularity ranking simultaneously. Additionally, PFiD integrates inter-passage attention to learn relative passage relevances explicitly, which is similar to the listwise training objective of RankT5 [ 21 ].

Fusion-in-Decoder for Document Retrieval. Formally, Given a question and a set of passages within the document = {1, 2, ..., }, the FiD encoder outputs the -th passage embeddings Hk ∈ R× , where denotes the maximum token length, and denotes the dimension of hidden states, which are then concatenated as the input of the fusion decoder [H1, H2, ..., Hk].

Hk = FiD-Encoder( + ) (1)

The FiD decoder utilizes [H1, H2, ..., Hk] to generate the target token = true false. Therefore, the loss function can be defined as follows: ℒ = − ∑︁ log (|1, 2, ...., − 1, [H1, H2, ..., Hk]) =1 (2) Inter-passage Attention. Previous work [ 24 ] tackled the issue of spurious passages by employing a binary classifier on the first token’s encoder hidden states Hk,1, to determine whether the passage is a rationale passage to the query. Then, guide the decoder by appending the additional embeddings toward the end of the encoder’s hidden states [H1, H2, ..., Hk, Hk+1], where Hk+1 ∈ R2× is trainable rationale embedding. However, as Table 2 shows, it drastically underperforms in passage ranking tasks by a large margin, as it does not explicitly model relative passage relevance.

Instead, to mitigate this, we utilize inter-passage attention to model interactions between passages explicitly. PFiD builds a set of input sequences by appending the first token hidden states of each pair as B = [H1,1, H2,1, ..., Hk,1], where H, denotes the -th token embeddings of -th passage. In a standard cross-encoder, the first token of the encoder aggregate query-passage information to compute a relevance score. We further use this token to depict the relative semantics via self-attention mechanism. Inspried by [ 27 ], we consider single-layer transformer model to depict relative passage relevance as follows: B̃︀ = softmax ︂( QK⊺ )︂ √ (3) in which matrices WQ, WK, WV ∈ R× are learnable parameters. The information from diferent passages is fused V, where Q = BWQ, K = BWK, V = BWV and exchanged via the self-attention mechanism. The training loss used for inter-passage attention can be defined as follows:

= softmax(B̃︀W ) ∈ R2, ℒ = − ( log() + (1 − ) log(1 − )) where is the passage relevance label, and the overall training objective of PFiD is:

ℒ = ℒ + ℒ, where is a hyperparameter to balance two losses. (4) (5)

4. Experimental setup

4.1. Datasets

We use MIRACL [ 25 ] passage ranking dataset for our experiments. The MIRACL [ 25 ] dataset is a large-scale, opendomain, human-generated multi-document ranking dataset which is similar to MS MARCO [ 28 ], but MIRACL owns its advantage by providing segmented document collection, enabling both document retrieval and passage ranking.1 For the document retrieval task, we construct the document retrieval dataset by regarding a document with at least one positive passage, as a positive document. Table 1 shows the statistics of the datasets.

Task Document Retrieval Passage Ranking # train

4.2. Baselines

We compare PFiD against the following three types of ranking baselines. The first is Single-Passage Cross-encoder (SPC) baselines, including MonoT5 [ 22 ], and RankT5 [ 21 ]. Due to the constraint of input tokens, we only take the ifrst- tokens in the document retrieval task. An alternative approach is to score each passage independently, and then take the passage with the highest score as the representative for ranking the document, or directly perform retrieval over the segmented passages. However, we will omit these approaches as the former lacks eficiency, and the latter is not scalable for real-world RAG scenarios. Then, the model is trained list-wisely with randomly sampled negatives from the entire passage sets; The second is MultiPassage Cross-encoder (MPC) baselines, including FiD [ 9 ] and RFiD [ 24 ]. For comparison in our experimental setting, both FiD and RFiD models are trained with the target token of true false, enabling both document retrieval and passage ranking. All SPC and MPC baselines used in this experiment are initialized with T5-base model; The third is the most frequently employed lexical ranker BM25 [ 16 ]. We use the Elasticsearch engine with default parameters 1 = 1.2, and = 0.75. 1MS MARCO also provide segmented document collection, but the segmented corpus do not align with passages in passage ranking tasks.

# of Documents = 2 2 3 4 # of Passages # of Documents = 5 2 3 4 # of Passages # of Documents = 10 0.7 We adopt T5-base [ 19 ] as our base model, using Adam [ 29 ] with a learning rate of 10− 4 and a dropout rate of 0.1. For both training and inference, we use the top-100 passages and truncate them to 200 of the maximum token length. The hyperparameter is set to 0.5. For the document retrieval task, we perform ranking on BM25 top-100 retrieved documents, whereas passage ranking ranks the passages within the given positive document. We also conduct experiments on real-world RAG scenarios, considering both document retrieval and passage ranking simultaneously. We use the evaluation metric of the nDCG [ 30 ], Recall, and MRR scores to evaluate the efectiveness. All experiments are conducted on a single NVIDIA A100 GPU (40GB). In this work, we do not consider other training approaches including data augmentation, knowledge distillation, or negative sampling strategies as delving into their efects falls outside the scope of our objectives.

5. Results and Analysis

Retrieval and Ranking. Table 2 presents our evaluation results on document retrieval and passage ranking tasks. The key observations are as follows: (i) MPC significantly outperforms SPC in document retrieval task by aggregating multiple passages, alleviating the problem of limited context size in SPC. In particular, one can see that PFiD outperforms RFiD by a large margin on both document ranking and passage ranking task. This indicates that by leveraging passage-wise context to guide the decoder, we can better identify relative passage relevance. Note that compared with the existing SPC baselines, our method achieves ranking eficiency by explicitly removing the need for each granularity ranking. PFiD directly consumes the entire document, and scores the relevance of the entire passages and document simultaneously. (ii) RFiD, implicitly guiding the decoder with rationale embedding shows improvement over FiD by a large margin, however, it is still even worse than BM25. It suggests that implicitly guiding indeed benefits the model’s ranking ability to some extent. However, when ranking 0.8774 0.8916† 0.3725 0.6036 0.8726 0.7856 various passages from multi-documents, traditional MPC is completely indistinguishable, suggesting cross-attention score from the decoder is not suited for the passage ranking task. (iii) SPC achieves superior performance over MPC in passage ranking task, as it is trained with rich negative samples from other documents, while MPC is only trained with in-document negatives. Additionally, even with indocument negatives, when trained with inter-passage attention, PFiD can achieve ranking efectiveness that rivals that of SPC, suggesting that incorporating an additional module to identify relevant passages is more efective than relying solely on the cross-attention mechanism of the decoder. Results on real-world RAG scenario. Next, we investigate the efectiveness of PFiD in real-world RAG scenarios. We first retrieve # documents from the candidates, and rerank # passages within the retrieved documents. Figure 1 represents the result of our evaluation. Notably, from Table 2 we observed that MPC outperforms SPC in document retrieval tasks, however, the performance drastically drops in this setting, as cross-attention scores from the decoder are indistinguishable across passages from multi-documents. Additionally, despite RankT5 reaching the best efectiveness on the passage ranking task, it did not exhibit any improvements over our method in real-world RAG scenarios, suggesting the importance of the multi-granularity ranking. Instead, PFiD consistently outperforms all baselines, by leveraging the complementary nature of SPC and MPC. PFiD is capable of more eficiently retrieving documents and ranking passages, and capturing the relative semantic correlation between diferent passages, leading to superior performance.

Cross-attention vs PFiD. As discussed above, PFiD has the advantage of identifying relevant passages compared to previous models like RFiD since it explicitly models relative passage relevance. We investigate the efects of the crossattention scores of the decoder and our passage ranking scores for the passage ranking task. Figure 2 illustrates the distribution of the rank of positive passages. As depicted in Figure 2, the PFiD is more strongly correlated with passage relevances than cross-attention scores, suggesting the PFiD focuses more on positive passages by explicitly learning relative passage relevance. Our experimental results show that the enhanced ability to identify relevant passages contributes to overall performance improvement.

References

[1] OpenAI, Gpt-4 technical report , 2024 . arXiv: 2303 . 08774 .

[2] H. Touvron , L. Martin , K. Stone , P. Albert , A. Almahairi , Y. Babaei , N. Bashlykov , S. Batra , P. Bhargava , S. Bhosale , D. Bikel , L. B. et al., Llama 2 : Open foundation and fine-tuned chat models , 2023 . arXiv: 2307 . 09288 .

[3] R. Anil , A. M. Dai , O. Firat , M. Johnson , D. Lepikhin , A. Passos , S. Shakeri , E. Taropa, P. Bailey , Z. C. et al., Palm 2 technical report , 2023 . arXiv: 2305 . 10403 .

[4] T. B. Brown , B. Mann , N. Ryder , M. Subbiah , J. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry , A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger, T. Henighan , R. Child , A. Ramesh , D. M. Ziegler , J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler, M. Litwin , S. Gray , B. Chess , J. Clark , C. Berner , S. McCandlish , A. Radford , I. Sutskever , D. Amodei , Language models are few-shot learners , 2020 . arXiv: 2005 .14165.

[5] A. Mallen , A. Asai , V. Zhong , R. Das , D. Khashabi , H. Hajishirzi , When not to trust language models: Investigating efectiveness of parametric and nonparametric memories , in: A. Rogers , J. Boyd-Graber , N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers) , Association for Computational Linguistics , Toronto, Canada, 2023 , pp. 9802 - 9822 . URL: https://aclanthology.org/ 2023 . acl-long . 546 . doi: 10 .18653/v1/ 2023 . acl-long . 546 .

[6] J. Maynez , S. Narayan , B. Bohnet , R. McDonald , On faithfulness and factuality in abstractive summarization , in: D. Jurafsky , J. Chai , N. Schluter , J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Online, 2020 , pp. 1906 - 1919 . URL: https:// aclanthology.org/ 2020 .acl-main. 173 . doi: 10 .18653/ v1/ 2020 .acl-main. 173 .

[7] L. Huang , W. Yu , W. Ma, W. Zhong , Z. Feng , H. Wang , Q. Chen , W. Peng , X. Feng , B. Qin , T. Liu, A survey on hallucination in large language models: Principles, taxonomy , challenges, and open questions, 2023 . arXiv: 2311 . 05232 .

[8] P. Lewis , E. Perez , A. Piktus , F. Petroni , V. Karpukhin , N. Goyal , H. Küttler , M. Lewis , W. tau Yih, T. Rocktäschel, S. Riedel , D. Kiela , Retrieval-augmented generation for knowledge-intensive nlp tasks , 2021 . arXiv: 2005 .11401.

[9] G. Izacard , E. Grave, Leveraging passage retrieval with generative models for open domain question answering , in: P. Merlo, J. Tiedemann , R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics , Online, 2021 , pp. 874 - 880 . URL: https://aclanthology.org/ 2021 .eacl-main. 74 . doi: 10 .18653/v1/ 2021 .eacl-main. 74 .

[10] G. Izacard , P. Lewis , M. Lomeli , L. Hosseini , F. Petroni , T. Schick , J. Dwivedi-Yu , A. Joulin , S. Riedel , E. Grave, Atlas: Few-shot learning with retrieval augmented language models , Journal of Machine Learning Research 24 ( 2023 ) 1 - 43 . URL: http://jmlr.org/papers/v24/ 23 - 0037 .html.

[11] Y. Gao , Y. Xiong , X. Gao , K. Jia , J. Pan , Y. Bi , Y. Dai , J. Sun , M. Wang , H. Wang , Retrieval-augmented generation for large language models: A survey , 2024 . arXiv: 2312 . 10997 .

[12] H. He , H. Zhang , D. Roth, Rethinking with retrieval: Faithful large language model inference , 2022 . arXiv: 2301 . 00303 .

[13] N. Thakur , L. Bonifacio , X. Zhang , O. Ogundepo , E. Kamalloo , D. Alfonso-Hermelo , X. Li , Q. Liu , B. Chen , M. Rezagholizadeh , J. Lin , Nomiracl: Knowing when you don't know for robust multilingual retrievalaugmented generation , 2024 . arXiv: 2312 . 11361 .

[14] F. Shi , X. Chen , K. Misra , N. Scales , D. Dohan , E. H. Chi , N. Schärli , D. Zhou , Large language models can be easily distracted by irrelevant context , in: A. Krause , E. Brunskill , K. Cho , B. Engelhardt , S. Sabato , J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research, PMLR , 2023 , pp. 31210 - 31227 . URL: https://proceedings.mlr.press/v202/shi23a. html.

[15] A. Asai , Z. Wu , Y. Wang , A. Sil , H. Hajishirzi , Self-rag: Learning to retrieve, generate, and critique through self-reflection , 2023 . arXiv: 2310 . 11511 .

[16] S. Robertson , H. Zaragoza , The probabilistic relevance framework: Bm25 and beyond , Foundations and Trends in Information Retrieval 3 ( 2009 ) 333 - 389 . doi: 10 .1561/1500000019.

[17] V. Karpukhin , B. Oğuz , S. Min , P. Lewis , L. Wu , S. Edunov , D. Chen , W. tau Yih, Dense passage retrieval for open-domain question answering , 2020 . arXiv: 2004 .04906.

[18] J. Devlin , M.- W. Chang , K. Lee , K. Toutanova , Bert: Pre-training of deep bidirectional transformers for language understanding , 2019 . arXiv: 1810 .04805.

[19] C. Rafel , N. Shazeer , A. Roberts , K. Lee , S. Narang , M. Matena , Y. Zhou , W. Li , P. J. Liu , Exploring the limits of transfer learning with a unified text-to-text transformer , 2023 . arXiv: 1910 .10683.

[20] R. Nogueira , K. Cho , Passage re-ranking with bert , 2020 . arXiv: 1901 .04085.

[21] H. Zhuang , Z. Qin , R. Jagerman , K. Hui , J. Ma, J. Lu, J. Ni , X. Wang , M. Bendersky, Rankt5: Finetuning t5 for text ranking with ranking losses , 2022 . arXiv: 2210 . 10634 .

[22] R. Nogueira , Z. Jiang , J. Lin , Document ranking with a pretrained sequence-to-sequence model , 2020 . arXiv: 2003 .06713.

[23] X. Ma , L. Wang , N. Yang , F. Wei , J. Lin , Finetuning llama for multi-stage text retrieval , 2023 . arXiv: 2310 . 08319 .

[24] C. Wang , H. Yu , Y. Zhang , Rfid: Towards rational fusion-in-decoder for open-domain question answering , 2023 . arXiv: 2305 . 17041 .

[25] X. Zhang , N. Thakur , O. Ogundepo , E. Kamalloo , D. Alfonso-Hermelo , X. Li , Q. Liu , M. Rezagholizadeh , J. Lin , MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages, Transactions of the Association for Computational Linguistics 11 ( 2023 ) 1114 - 1131 .

[26] G. Izacard , E. Grave, Distilling knowledge from reader to retriever for question answering , 2022 . arXiv: 2012 .04584.

[27] J. Yang , Z. Liu , C. Li , G. Sun , X. Xie , Longtriever: a pre-trained long text encoder for dense document retrieval , in: H. Bouamor , J. Pino , K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Singapore, 2023 , pp. 3655 - 3665 . URL: https://aclanthology.org/ 2023 .emnlp-main. 223 . doi: 10 .18653/v1/ 2023 .emnlp-main. 223 .

[28] P. Bajaj , D. Campos , N. Craswell , L. Deng , J. Gao , X. Liu , R. Majumder , A. McNamara , B. Mitra , T. Nguyen , M. Rosenberg , X. Song , A. Stoica , S. Tiwary , T. Wang , Ms marco: A human generated machine reading comprehension dataset , 2018 . arXiv: 1611 . 09268 .

[29] D. P. Kingma , J. Ba , Adam: A method for stochastic optimization , 2017 . arXiv: 1412 . 6980 .

[30] K. Järvelin , J. Kekäläinen , Cumulated gain-based evaluation of ir techniques , ACM Trans. Inf. Syst . 20 ( 2002 ) 422 - 446 . URL: https://doi.org/10.1145/582415.582418. doi: 10 .1145/582415.582418.