We investigate the quantification of word importance by introducing a novel self-supervised task that modifies masked language modeling. Instead of predicting masked words, our approach involves learning to identify which words were inserted. We hypothesize that resulting models will predict a higher likelihood of insertion for less important words. We experiment with two diferent insertion strategies: the List Inserting Method (LIM) and the BERT Inserting Method (BIM). We outline the process for gathering manually estimated word importance data and describe the construction of a dataset for evaluating our methods. Our results indicate that our modified language modeling surpasses baselines and is competitive with existing research in assessing word importance.
Workshop on Automata, Formal and Natural Languages 2024 (WAFNL 2024) $ adam.osusky746@student.cuni.cz (A. Osuský); javorsky@ufal.mf.cuni.cz (D. Javorský); bojar@ufal.mf.cuni.cz (O. Bojar) https://ufal.mf.cuni.cz/david-javorsky (D. Javorský); 2. Word Importance https://ufal.mf.cuni.cz/ondrej-bojar (O. Bojar)
0000-0003-2516-2535 (D. Javorský); 0000-0002-0606-0050 Word importance (WI) depends on its intended usage. (O. Bojar)© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Depending on objectives, such as text summarization or CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) 1https://github.com/adam-osusky/predicting-word-importance grammar correction, the same words may hold diferent degrees of importance. In this work, we focus on semantic importance and we define it by drawing inspiration from prior works: Kafle and Huenerfauth [17] emphasize the loss incurred by removing a word and Javorský et al. [12] focus on the meaning contribution added by a word.
We combine these perspectives into a unified definition of WI:
Importance is the measure of a word’s contribution to the overall meaning of the context, indicating the extent to which the removal of a word would diminish the information conveyed by the context.
2. Create an order for the most important ones; any unranked words will receive the last rank, and they should be considered to have similar importance. At least one word must be ranked. 3. Click on words to select them. The selection order determines their ranking. Clicking on a selected word will deselect it. The first selected word is the most important. 4. Importance is the measure of a word’s contribution to the overall meaning of the context. Indicating the extent to which the removal of a word would diminish the information conveyed by the context. 5. Contexts span five diverse domains: news, beletry, poetry, jokes, and transcribed spoken language. 6. In the transcribed spoken language domain, words may take the form of "(PERSON#NUMBER)" at the beginning of a person’s reply, indicating the speaker’s identity. These tags are non-clickable and non-rankable. Additionally, words in the form "PERSON#NUMBER" serve as references to other persons’ names within the utterance.
We aim to collect human-annotated data for word importance, and therefore we need to clearly formulate instructions for annotators. Even though the most intuitive approach would be to let annotators score each word by a real number within the range [0, 1], as it is done in other studies [17, 12], we find such task very dificult for annotators. Therefore, we represent WI as an importance ranking.
By ranking, we mean the ordering of word positions within a context, where the word position ranked as 1 is considered the most important. We rank word positions A simple annotation tool was used for data collection. because the same word can appear multiple times in a This tool allows annotators to rank words by clicking on context with varying levels of importance. them in sequence. If an annotator wants to insert a word
In our initial experiments, we observed that when an- into the middle of an already selected ranking, they must notators were unrestricted in the number of words they unselect the subsequent words and then reselect them in could rank, they tended to sequentially select key nouns the desired order. While it might seem more convenient from the subjects and objects, as well as the verbs that to allow direct insertion of a word into the middle of connect these elements in the sentences. However, this the ranking, the current approach has its benefits. By behavior does not align with our objectives. requiring annotators to reassess the subsequent rank
Our primary goal is to identify the most important ings when making changes, the process encourages a words, so the ranking can include only a subset of the more thoughtful and deliberate evaluation of the overall word positions in a given context. For a context with ranking. the length of , we aim to rank positions where 1 ≤ ≤ ⌈ 0.1 · ⌉. Word positions that are not ranked are 3. Data Collection assigned the rank of + 1, referred as the “last rank”.
We term this process as rank limit being equal to 10%.
We argue that it might facilitate the annotators’ attention on identifying only the most essential words in a given context. By restricting the number of words that can be ranked to 10% of the total word positions, we force a more selective process, making the annotators focus on the most salient words and not to be overwhelmed by numerous options of possible rankings.
To ensure diversity, we target various domains and their corresponding English datasets: News, the News Commentary dataset [21]; literature, data from [22]; poetry, data from [23]; jokes, data from [24]; and meeting transcripts, the ELITR Minuting Corpus [25]. From each domain, we manually select 10 contexts (each context possibly containing more sentences), ensuring that the contexts are around 60 words long. To achieve better granularity for certain words like “don’t” and “I’m”, the Instructions for Annotators The full set of instruc- contexts are tokenized using the Moses tokenizer [26]. tions provided to our annotators is as follows: The dataset statistics are outlined in Table 1. 1. Arrange the words in descending order by their Each of the 50 contexts is annotated by three annotaimportance. You can rank at most 10 percent of tors who are non-native English speakers. These conthe words, or choose to rank fewer if desired. texts with annotations form the Word Importance Dataset (WIDS). We make the dataset available at [27].
Count mean± std 2565 256.5± 26.1 2207 220.7± 17.1 1776 177.6± 27.5 1938 193.8± 25.4 2432 243.2± 26.6 10918 218.4± 38.5
words randomly from a predefined list. This list is generated by splitting the base corpus into words by white space. Consequently, words that appear more frequently in the corpus are more likely to be inserted, mirroring the original distribution.
To assess the similarity of the annotations, we compute insert words that do not fit well in the sentence. This inter-annotator agreement using Cohen’s kappa [28]. We is achieved by leveraging the capabilities of another simplify the calculation by classifying each word position instance of the BERT model [20].2 Because BERT predicts in every context as either “selected” or “not selected” by the words without any information except the text itself, annotators. In Section 5.2, we present metrics to incorpo- we assume that they should not alter the sentence’s rate the order of selection for a more nuanced analysis. meaning significantly. In this method, mask tokens are
The computed Cohen’s kappa values are shown in Ta- placed at the selected insertion positions within the text ble 2. It is unsurprising that one out of the two domains and BERT is then used to predict the masked tokens. with the least agreement than poetry. An intriguing ob- We prohibit predictions of neighboring tokens (those servation is that literature displays slightly lower agree- immediately before and after the masked token) and ment as poetry. The domains with the highest agreement sub-word tokens, i.e. tokens that are not a beginning are jokes and meeting transcripts. We find these findings of a word. After filtering out these unwanted tokens, in line with our intuition: There is often very clear what we select the prediction with the highest logit probability. words make jokes funny and speech in meetings may contain many objectively unimportant words, e.g. filler words, hesitations, false starts etc.
Our approach involves fine-tuning a pre-trained BERT model [20] using automatically generated data. Specifically, we generate training text data by inserting words into existing text and then use the modified text as training data. The objective of fine-tuning is to predict which words were inserted. We hypothesize that this task will require the model to understand the importance of each word and its contribution to the overall meaning of the context, ultimately leading the model to assign higher likelihoods of insertion to less important words. This enables us to create a ranking of words in a test input For both methods, possible positions for insertion include places before existing words and one additional position at the end of the text. These words are obtained by splitting the text by white space to determine the potential insertion positions. We insert at most one word in each position, ensuring that words are not inserted consecutively.
The positions for insertion are selected randomly. The insertion rate is defined as the ratio of the number of words to be inserted to the total number of words in the original text sample. For instance, if a text sample contains 10 words and we use an insertion rate of 0.5, we insert 5 new words into this text.
In our experiments, our goal is to efectively compare the two insertion methods, LIM and BIM, as well as evalu2https://huggingface.co/google-bert/bert-base-uncased In the 1970s, failures. Wheeler a became increasingly the forgetful and Olympic came to rely largely dwarf on his Seattle assistant, Molly rumored Myres, Poesaka to killed organise Police his Army affairs. souls Amid increasing ill about health, the in September 1973 pairs he moved in full-time into Mills, Myres's house in Byzantines. Leatherhead, time Surrey, although said he claim continued to use (RCA his EMS central Hill London "When flat during details day-trips Essendon. to the He city. There, he Morison authored a final was book, My Archaeological Mission who to India Studios and as Pakistan, largely although polygyny much for of the text was with culled as from his previous publications; the it was veterans published none by Thames was and Hudson in 1976. After Charles suffering that a stroke, on Wheeler went died The at On Myers' Religious home on Kierszenbaum 22 July eldest 1976. Archives' In bridge memoriam, the British Academy, Royal Academy, and Royal Society Ukraine flew their flags cites at half-mast. membership Wheeler's from funeral was player held with military review trappings in at a was local demolished. crematorium, while a contacting larger memorial service was held a in south St (April James's aid Church, area, Piccadilly in November. late In the 1970s, " Wheeler became " increasingly forgetful and came personal to " rely personal largely " on his " assistant, late Molly back Myres, country to organise his " affairs. apartment Amid occasional increasing visit ill health, in later September 1973 east he moved full-time which into original Myres's two house in Leatherhead, Surrey, although he posthumously continued press to use shortly his central London major flat finally during " day-trips to " the prestigious city. There, birmingham he authored a final book, My bafta Archaeological other Mission to all India " and Pakistan, although much of " the state text was culled also from his ceremonial previous publications; " it nearby was published much by " Thames " and catholic Hudson in 1976. new After john suffering a back stroke, Wheeler ( died at solely Myers' " home " on 22 July 1976. " In memoriam, the " British Academy, john Royal Academy, also and " Royal Society " flew their flags capital at half-mast. first Wheeler's research funeral was " held with military directly trappings what at a alone local crematorium, while a john larger george memorial service also was held in St James's michael Church, " Piccadilly in November. 4.1. Example Text with Inserted Words In Figure 1, we illustrate an example of text from our preprocessed WikiText dataset (Section 5.1), where words have been inserted using both the LIM and BIM methods.
The inserted words in the BIM method often appear superfluous, adding information to the sentences. Notably, there is a frequent insertion of apostrophes, occurring more often than desired. To investigate this phenomenon further, we conducted a simple experiment on a subset tion marks, such as “(April” or “area,”, as seen in the last sentence on the left in Figure 2.
5.1. Training Details We detail the preprocessing methods applied to the WikiText dataset and outline the construction of the training regime. of 100 examples to analyze how the frequency of apos- words, we have selected the WikiText dataset [29]. This trophes changes with varying insertion rates. Refer to of apostrophes converges to approximately 0.22 when the criteria specified by Wikipedia editors at the time of dataset comprises articles from Wikipedia3 that are clasthe insertion rate is at least 0.5.
Conversely, in the LIM method, the inserted words sometimes introduce information that seems out of con- Face datasets library [2]. Each example in the dataset is text. Additionally, some inserted words include punctua- either a paragraph or a title. For our specific use case, Frequency of Apostrophe Insertions vs. Insertion Rates Paragraphs Characters Characters mean ± std Words Words mean ± std k-inter Another perspective on rankings is to consider words that do not have the last rank and disregard their specific order. By doing this, we view the rankings as indicators of which words are important, allowing us to measure the extent of the intersection between diferent rankings. We thus propose a new metric, k-inter, where we filter the ranking and keep only word positions that 5.2. Evaluation for WI ranking do not have the last rank. We then compute the fraction Our trained models predict logits for the probability of of context pairs where the intersection of their filtered word insertion. We construct the ranking by ordering rankings has at least elements. We examine values of BERT-tokens in one context in ascending order of their ∈ {1, 2, 3}. 5https://huggingface.co/google-bert/bert-base-uncased
Model BIM-0.25 LIM-0.25 BIM-0.5 LIM-0.5 BIM-0.75 LIM-0.75
Models We create six datasets using both the LIM and BIM techniques, each with insertion rates of 0.25, 0.5, and 0.75. Subsequently, we train six models, each corresponding to a distinct combination of insertion method and rate.
To ensure a fair comparison and avoid introducing bias due to diferences in hyper-parameters, we use identical settings for all models. Hyper-parameters were selected empirically based on initial experiments.
We use a learning rate of 0.0032, batch size of 256, Adam optimizer [30] with default betas (0.9,0.999), and a linear learning rate scheduler with a linear warmup of 350 steps. Starting from the BERT5 pre-trained model, we fine-tune on each of the datasets for 5 epochs.
The performance of individual models in the classification task is shown in Table 4. The LIM models consistently outperform the BIM models. Given this discrepancy and the distribution of inserted punctuation marks discussed in Section 4.1, it indicates that the BIM data present a more challenging task, as the inserted words blend more seamlessly with the context. insertion probabilities. Since Word Importance Dataset (WIDS) is pretokenized by the Moses tokenizer, we use the logits of the first BERT-token to score the original Moses-token if a Moses-token is split into multiple BERTtokens.
For the human reference, we calculate the average rank of each token based on the rankings provided by all three annotators and then order the words according to these average ranks. With only three annotators, a majority of words still fall into the lowest rank, leading to inconsistencies between model predictions and averaged annotations, as they can result in diferent lowest ranks.
To ensure consistency in evaluation, we apply the 10% rank limit to both the averaged annotations and model predictions.
Since 90% of the positions fall into the lowest rank, this creates challenges in designing efective evaluation metrics. To address these issues, we propose three metrics, each progressively refining and incorporating desired properties to better align with our evaluation goals.
Pearson correlation The simplest and well-known approach is to calculate the sample Pearson correlation coeficient on the ranks of word positions over all positions and all contexts in the dataset. However, this method is not ideal because 90% of word positions within a given context fall into the lowest rank. Our primary focus is on achieving higher agreement within the top 10%, which is not adequately emphasized by this correlation measure.
We first evaluate the pair-wise agreement between annotators using these metrics, which we present in Table 5.
This evaluation complements Cohen’s kappa from SecTable 5 tion 3. The order of annotator pairs remains consistent oMneWtriocrsdfrImompoSretactnicoenD5a.2tacsoemt.puted between our annotators for both the Pearson correlation and the overlap metric. The -inter values for values of 1 and 2 are relatively high compared to Pearson correlation or the overlap, inDomain Pair 1-2 Pair 1-3 Pair 2-3 Average dicating that the annotators agree on the selection of the LNietewrsature 00..221816 00..222407 00..534101 00..235478 most important words but not that well on their order. Poetry 0.189 0.301 0.220 0.237 This supports our decision to let annotators focus only on Jokes 0.484 0.475 0.462 0.474 the most important words and not make them mentally Transcripts 0.450 0.354 0.437 0.413 overloaded by the vast amount of options. In Table 6, we Table 6 further present the overlap between annotators within Overlap computed between our annotators on Word Impor- individual domains of WIDS. Annotators show the hightance Dataset, but on individual domains. est similarity in the jokes domain and the lowest in the poetry domain. This observation aligns with the results Model Pearson 1-inter 2-inter 3-inter Overlap in Table 2. For other metrics on individual domains, see Random 0.256 0.54 0.13 0.01 0.061 Appendix A.
PI 0.321 0.78 0.40 0.08 0.114 Finally, we evaluate the performance of all six of our TF-IDF 0.309 0.66 0.20 0.04 0.121 trained models. Additionally, we include random predicBBIIMM--00..2755 00..333451 00..8726 00..3420 00..1124 00..113215 tions as a baseline for our metrics and the average human LIM-0.5 0.328 0.72 0.40 0.12 0.137 performance from Table 5 as an upper bound. LIM-0.75 0.352 0.80 0.48 0.18 0.142 As an additional baseline, we include term freBIM-0.5 0.344 0.70 0.42 0.14 0.143 quency–inverse document frequency (TF-IDF), computed NLI 0.374 0.90 0.56 0.22 0.150 on the Word Importance Dataset without any preprocessLIM-0.25 0.376 0.82 0.52 0.14 0.178 ing. Furthermore, we include two models, PI (Paraphrase Humans 0.563 0.91 0.73 0.45 0.346 Identification) and NLI (Natural Language Inference), deTable 7 veloped by Javorský et al. [12]. We obtain rankings from Evaluation of models from Section 6 on the Word Importance all models by ordering the words according to their sigDataset. The “Random”category represents the average met- nificance scores. rics of 100 random predictions, while “Humans” denotes the The results are presented in Table 7, indicating that average of human metrics from Table 5. The metrics are de- our models are performing reasonably well. They surfined in Section 5.2. pass random predictions and TF-IDF across all metrics and are comparable to the NLI model. Notably, LIM-0.25 even exceeds the NLI model in both the overlap and Pearson correlation metrics. Metrics that consider the order of selected words show our models are approximately halfway to achieving human-level performance. They are approaching human performance in terms of 1-inter but lag significantly in higher -inter metrics.
It is quite surprising that LIM approach is superior to BIM, suggesting that simple methods are sometimes more eficient. We hypothesize that inserted words by BERT are so well suited to the surrounding context that it is very dificult to detect them, which efectively decreases the useful learning signal from them.
For readers interested in a detailed view of all metrics across individual domains, refer to Appendix A.
Overlap The limitation of -inter is that it does not consider specific rank values, only if the words are in the top 10%. We aim to assign more weight to agreements on specific rank values, prioritizing the match on higher-ranked agreements over lower-ranked ones.
We thus propose to use the average overlap metric, as described by Webber et al. [31]. First, we derive an ordered list of words from the ranking. The agreement between lists and at depth is defined as (, , ) = |: ∩ :|/, where : represents the first elements of the list. The average overlap at depth is then (, , ) = 1 ∑︀=1 (, , ). For context pairs of rankings, we compute the average overlap for each pair and then average these values, which we refer to as overlap. The depth is chosen diferently for each pair: for a context with length , the depth is set to ⌈0.1 · ⌉, to be consistent with our rank limit of 10%.
In this paper, we define word importance, collect annotations for a small multi-domain word-importance dataset in English, propose metrics for its evaluation and introduce a novel self-supervised machine learning method: The goal is to predict inserted words in the text. Our results demonstrate that our method outperforms baseline models and is comparable to prior work on word importance.
Possible future work might benefit from more experiments when using BIM or combining LIM and BIM, potentially leading to more competitive results. Experimenting with smaller insertion ratios can be another potential avenue.
Limitations One of the primary limitations of our study is the size of the Word Importance Dataset, since it includes only 50 relatively short contexts that consists of approximately 60 words. Varying lengths of context might contribute to better generalization. The study compares importance scores to only one other indicator of word significance and it also lacks the evaluation of importance scores on a downstream task.
Another limitation is the small number of annotators. With a larger pool of annotators, the data in the Word Importance Dataset would likely exhibit lower variance. This would result in higher quality averaged rankings that are more closely aligned with the true distribution.
Finally, the work does not provide the evaluation of importance scores on the word-importance dataset collected by Kafle and Huenerfauth [17].
The work has been partially supported by the grants 272323 of the Grant Agency of Charles University, 1926934X (NEUREM3) of the Czech Science Foundation and SVV project number 260 698.
Computational resources were provided by the eINFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.
Intelligence and Security Workshops (CISW 2007), Ghosal, O. Bojar, Elitr minuting corpus: A novel IEEE, 2007, pp. 152–155. dataset for automatic minuting from multi-party [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, meetings in english and czech, in: Proceedings of L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, At- the Thirteenth Language Resources and Evaluation tention is all you need, Advances in neural infor- Conference, 2022, pp. 3174–3182.
mation processing systems 30 (2017). [26] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, [16] S. Serrano, N. A. Smith, Is attention interpretable?, M. Federico, N. Bertoldi, B. Cowan, W. Shen, arXiv preprint arXiv:1906.03731 (2019). C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, [17] S. Kafle, M. Huenerfauth, A corpus for modeling E. Herbst, Moses: Open source toolkit for statistical word importance in spoken dialogue transcripts, machine translation, in: S. Ananiadou (Ed.), Proin: N. Calzolari, K. Choukri, C. Cieri, T. Declerck, ceedings of the 45th Annual Meeting of the AssoS. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mar- ciation for Computational Linguistics Companion iani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, Volume Proceedings of the Demo and Poster SesT. Tokunaga (Eds.), Proceedings of the Eleventh In- sions, Association for Computational Linguistics, ternational Conference on Language Resources and Prague, Czech Republic, 2007, pp. 177–180. URL: Evaluation (LREC 2018), European Language Re- https://aclanthology.org/P07-2045. sources Association (ELRA), Miyazaki, Japan, 2018. [27] A. Osuský, D. Javorský, Word importance dataset, URL: https://aclanthology.org/L18-1016. 2024. URL: http://hdl.handle.net/11234/1-5520, [18] S. Kafle, P. Yeung, M. Huenerfauth, Evaluating the LINDAT/CLARIAH-CZ digital library at the benefit of highlighting key words in captions for Institute of Formal and Applied Linguistics (ÚFAL), people who are deaf or hard of hearing, in: Pro- Faculty of Mathematics and Physics, Charles ceedings of the 21st International ACM SIGACCESS University.
Conference on Computers and Accessibility, 2019, [28] J. Cohen, A coeficient of agreement for nominal pp. 43–55. scales, Educational and psychological measurement [19] L. Shen, X. Zhang, S. Ji, Y. Pu, C. Ge, X. Yang, 20 (1960) 37–46.
Y. Feng, Textdefense: Adversarial text detection [29] S. Merity, C. Xiong, J. Bradbury, R. Socher, based on word importance entropy, arXiv preprint Pointer sentinel mixture models, 2016. arXiv:2302.05892 (2023). arXiv:1609.07843. [20] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, [30] D. P. Kingma, J. Ba, Adam: A method for stochasBert: Pre-training of deep bidirectional transform- tic optimization, arXiv preprint arXiv:1412.6980 ers for language understanding, arXiv preprint (2014).
arXiv:1810.04805 (2018). [31] W. Webber, A. Mofat, J. Zobel, A similarity mea[21] J. Tiedemann, Parallel data, tools and inter- sure for indefinite rankings, ACM Transactions on faces in OPUS, in: N. Calzolari, K. Choukri, Information Systems (TOIS) 28 (2010) 1–38. T. Declerck, M. U. Doğan, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), European Language Resources Association (ELRA), Istanbul, Turkey, 2012, pp. 2214– 2218. URL: http://www.lrec-conf.org/proceedings/ lrec2012/pdf/463_Paper.pdf . [22] R. Nagyfi, Dataset card for project gutenber english language ebooks, https://huggingface.co/ datasets/sedthh/gutenberg_english, 2023. Accessed: 2024-03-28. [23] A. Parrish, Github repository for gutenbergpoetry-corpus, https://github.com/aparrish/ gutenberg-poetry-corpus, 2018. Accessed: 2024-0328. [24] SocialGrep, Dataset card for one-million-redditjokes, https://huggingface.co/datasets/SocialGrep/ one-million-reddit-jokes, 2021. Accessed: 2024-0328. [25] A. Nedoluzhko, M. Singh, M. Hledíková, a. T.
In Table 9, we present our proposed metrics computed for individual domains within the Word Importance Dataset between human rankings. Notably, the poetry domain exhibits relatively high -inter values, whereas the Pearson correlation and overlap metrics are low. This indicates that humans agreed more on which words are important rather than on the order of their importance.
In Table 8, we present the overlap of the models from Section 6 across individual domains within the WIDS. Our models outperform the TF-IDF baseline in all domains except for the news domain. In a few cases and metrics, Random ranking outperforms some methods. It is worth noting that each domain includes only 10 examples, which may lead to signicfiant variability in the results. Despite this, human performance consistently exceeds that of the models across all domains.
In Table 10, we present all of our proposed metrics computed for models from Section 6 on individual domains within the Word Importance Dataset. For these evaluations, the TF-IDF was created using text solely from the respective individual domain. It is apparent that the performance ordering of the models is not consistent across the diferent domains, likely due to each domain having only 10 examples.
An interesting observation is that TF-IDF performs best on the news domain, whereas it is under performing in the other domains.
PI TF-IDF BIM-0.75 BIM-0.25 LIM-0.5 LIM-0.75 BIM-0.5 NLI LIM-0.25
NLI LIM-0.75 LIM-0.5
PI BIM-0.25 BIM-0.75 BIM-0.5 LIM-0.25 tf-idf tf-idf
LIM-0.75 LIM-0.25 LIM-0.5 BIM-0.5 PI BIM-0.75 BIM-0.25 NLI tf-idf
BIM-0.75 LIM-0.75 LIM-0.5 LIM-0.25 NLI
BIM-0.75 BIM-0.25 LIM-0.5 NLI LIM-0.25 LIM-0.75
PI NLI BIM-0.75 tf-idf BIM-0.25 LIM-0.5 LIM-0.75 BIM-0.5 LIM-0.25 0.80 0.60 0.40 0.52 0.80 0.50 0.70 0.30 0.70 0.60 0.60 0.55 0.80 0.70 0.80 0.90 0.70 1.00 0.90 0.90