This paper proposes a semantic novelty measurement model for scientific papers using a large language model to generate question and method words semi-supervisedly. LoRA and prompt words enhance keyword generation accuracy and structural measurement. The model achieves 66.0% recall, 63.6% precision, and 65.9% sum, improving with more training samples. At 3000 samples, the training set is costeffective. The proposed method, leveraging fine-tuned large language models, is effective and robust.
Instruct represents the description of the keyword generation task, Example represents the instance, Input represents the input text, and Output represents the output result requirement. In this paper, the prompt engineering design template is designed to clearly and specifically describe the task objectives and task contents in "Instruct", and the specific paper abstract and expected keyword generation results are given in "Example".
After early data acquisition, processing and fine-tuning of LLaMA3 large language model, this study summarizes the keywords of each sample paper generated, takes the publication year of the sample paper as the reference point, and uses the fine-tuning large language model LLaMA3 to calculate the semantic similarity. Specifically, We compared the occurrence frequency of keywords in other research literature in the same research field earlier than the publication time of the current sample papers. This comparison process uses the fine-tuned trained large language model LLaMA3 to ensure the accuracy and validity of the comparison. We record the frequency of these keywords and mark them as specific reference data. Let's call it. Substituting into the calculation formula, the score obtained is the semantic novelty score of the sample papers and the paper novelty measure is shown below.
Novn= |Q| 1 ∑ a=1 ln [ n ( Qk )+1]+1 |Q| ( 3 )
Where, Novn represents the novelty score of the paper to be tested, n ( Qk ) Represents the frequency of occurrence of keywords in the paper to be tested compared with that in the paper before publication.
This paper uses the Web of Science core set as a data source, selecting scientific and technological
papers from top-level Computer Science disciplines published in 2018-2019. The search criteria
retrieved 15,348 papers. To evaluate semantic novelty, it obtained a complete database of the field,
including paper titles, abstracts, citation frequencies, and JCR partitions. The study divided the
sample dataset into a training set
The current representative general large language models Gemma2, Phi3, GPT-4 and LLaMA3 are selected for comparison with the fine-tuning model in this paper, and the results are shown in Table 2.
Conclusion: LoRA fine-tuning and hint word fine-tuning can significantly improve the generation effect of the model in this paper.
Through the above empirical analysis, it is proved that the fine-tuned LLaMA3 can effectively improve the keyword generation effect of the paper, and further optimize the semantic novelty measurement of the paper.
This research is supported by grants from the National Social Science Foundation of China (22BTQ097).
During the preparation of this work, the author(s) used LLaMA3 in order to: The text data is processed and the abstract keywords are extracted. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. [5] Peter D. Turney. 2000. Learning algorithms for keyphrase extraction. Inf. Retr. 2, 4 (May, 2000), 303-336. DOI: https://doi.org/10.1023/A:1009976227802.