The cybersecurity landscape evolves rapidly and poses threats to organizations. To enhance resilience, one needs to track the latest developments and trends in the domain. For this purpose, we use large language models (LLMs) to extract relevant knowledge entities from cybersecurity-related texts. We use a subset of arXiv preprints on cybersecurity as our data and compare diferent LLMs in terms of entity recognition (ER) and relevance. The results suggest that LLMs do not produce good knowledge entities that reflect the cybersecurity context.
using a variety of common LLM-based entity extractors highly sensitive to the embedding choice. © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License (cs.CC, cs.LO, cs.DS, cs.IT, cs.CL, and cs.AI) as compariAttribution 4.0 International (CC BY 4.0). to evaluate the relevance of extracted entities to document understanding tasks, using as a proxy the relevance of arXiv to cybersecurity (https://arxiv.org)
in large part thanks to public trials of conversationally ifne-tuned LLMs [ 10, 11, 12], modern large language models pre-trained on large amounts of data trace their roots back to ELMo LLM, first released in 2018 [ 13]. The Joint Workshop of the 4th Extraction and Evaluation of Knowledge Entities from Scientific Documents and the 3rd AI + Informetrics (EEKE1.
Methods The complete code to replicate the results presented here with instructions is available at https://github.com/technometrics-lab/0_LLM-based_ entity_extraction_CySec.
late 2022, initially collected by [22]. We focused on the cs category, specifically on the cs.CR and cs.NI listings - Cryptography and Security and Network and Internet
spaCy Large* spaCy Transformer
Refs [23] [23] [24] [25] [26, 27] [26, 28] [16] [16] [29, 30] [31, 32] [16, 32] [16, 32] [15, 32] [17, 32] [33, 32] [16, 34]
99.3 ± 6.93 99.3 ± 6.97 19.9 ± 1.97 99.3 ± 7.25 96.9 ± 14.6 76.4 ± 27.7 44.7 ± 24.0 43.3 ± 23.3 36.4 ± 23.4 39.9 ± 25.0 41.7 ± 24.9 33.5 ± 23.3 37.7 ± 24.8 28.7 ± 21.1 26.0 ± 19.5 99.6 ± 6.11
son domains. The selected listings represented 5000 to fine-tuned on a language identification dataset, avail20000 preprints each. able at https://huggingface.co/papluca/xlm-roberta-base
For each of the preprints in the listings, all documents language-detection. Following that, the preamble of the with < 1000 words and not in English were removed. preprint prior to the ”Abstract” keyword and the bibliogTo achieve the latter, we used an XLM-Roberta model raphy following the ”References” keyword were removed.
Following that, we applied models described in Ta- from preprints would allow arXiv listing identification. ble 1 to the documents. Specifically, four major classes To allow the interpretation of the results, we subsampled of models were used: Noun Extractrs (NnE), Keyphrase 100 papers from each listing and, due to high processing Extractors (KPE), Named Entity Recognition (NER), and time, excluded spaCy Transformer (cf Figs. 2, 3; addi
mented: number recognition (NER + NUM) and concept recognition (NER + CON R). Exact model names and sources are available in the code repository. 2. Results and Discussion
attention window. If the number of extracted entities ex- Our first result is that in computer science bibliometrics, ceeded 100, only 100 entities with the highest activations a variety of entity extraction models perform similarly, were retained. Samples of extracted entities are available with performance being mostly defined by their base in the code repository. architecture, task, and dataset used to fine-tune them
We compare the similarity of extractors’ outputs on all (Fig. 1. Given that base architectures are predominantly documents by embedding entities extracted from each BERT and RoBERTa [16, 17] and fine-tuning datasets are document with spaCy and calculating the average cosine general texts, notably Conll03 newswire [32], we should similarity between extractors. A hierarchical clustering not expect general LLM-based entity extraction models on cosine similarity was then used to create Fig. 1. to perform well on scientific articles. LLM fine-tunes
To visualize connections between the extracted en- are sensitive to the training data, and only KBIR-inspec tities from diferent listings, we used common embed- was fine-tuned using a scientific dataset, consisting of dings (spaCy [23], GloVe [35], BERT-Large [16], GPT- annotated 1998-2002 article abstracts from Computers 2 [36], Fasttext [37], and word2vec [38]) and four low- and Control and Information Technology journal [41, 28]. dimensional projection algorithms (linear, spectral, t-SNE Given the pace of the evolution of computer science, such [39], UMAP [40]) to investigate if the entities extracted ifne-tunes are unlikely to still be relevant today, which is supported by the lack of thematic clusters in entities is not due to 2d projection algorithms by calculating clusextracted by it (Fig. 2 ; NER organized structures are non- tering coeficients (intergroup dispersion vs. intra-group informative) suggest that they are indeed not relevant dispersion) across listings in each embedding. Hence, anymore. We hence hypothesize that non-LLM-based attention is warranted when using word embeddings in
could be essential for addressing these issues, especially given that they already give radically diferent results compared to LLM-based extractors. Acknowledgments
LLM-extracted entities does not perform well for concept- AK is supported by the CYD Campus, armasuisse W+T, oriented bibliometrics in computer science. Even 2D VBS grant (ARAMIS CYD-C-2020015). projection algorithms known for their tendency to overfit local clusters, t-SNE, and UMAP [42], fail to separate References diferent listings (Fig. 2, 3), with the exception of NERs in spaCy and GPT2 embeddings due to shared theorem [1] D. Percia David, Three Articles on the Economics of names and chapter annotations. We provide interactive Information-Systems Defense Capability. Material-, 2d projection plots in the code repository for readers to Human-, and Knowledge-Resources Acquisition for validate this claim. Critical Infrastructures, Ph.D. thesis, Université de
Our third result is that cosine similarity is highly Lausanne, HEC Lausanne, 2020. dependent on the embedding used to infer entity related- [2] R. Anderson, Security engineering: a guide to buildness (Figs. 2, 3). While present here only UMAP 2d projec- ing dependable distributed systems, 3 ed., Wiley, tion with spaCy and GloVe, the embedding-dependence 2020. of similarity is present across embeddings, which we [3] T. Daim, H. Yalçin, Digital transformations: new present in the code repository. We also validate that this [20] OpenAI, Gpt-4 technical report, CoRR [28] D. Sahrawat, D. Mahata, M. Kulkarni, H. Zhang, abs/2303.08774 (2023). URL: https://arxiv.org/abs/ R. Gosangi, A. Stent, A. Sharma, Y. Kumar, 2303.08774. arXiv:2303.08774. R. R. Shah, R. Zimmermann, Keyphrase ex[21] D. Ganguli, D. Hernandez, L. Lovitt, A. Askell, traction from scholarly articles as sequence laY. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, beling using contextualized embeddings, CoRR N. Elhage, S. E. Showk, S. Fort, Z. Hatfield-Dodds, abs/1910.08840 (2019). URL: http://arxiv.org/abs/ T. Henighan, S. Johnston, A. Jones, N. Joseph, J. Ker- 1910.08840. arXiv:1910.08840. nian, S. Kravec, B. Mann, N. Nanda, K. Ndousse, [29] A. Ushio, J. Camacho-Collados, T-NER: an allC. Olsson, D. Amodei, T. Brown, J. Kaplan, S. Mc- round python library for transformer-based named Candlish, C. Olah, D. Amodei, J. Clark, Predictabil- entity recognition, in: D. Gkatzia, D. Seddah ity and surprise in large generative models, in: (Eds.), Proceedings of the 16th Conference of the FAccT ’22: 2022 ACM Conference on Fairness, European Chapter of the Association for ComAccountability, and Transparency, Seoul, Repub- putational Linguistics: System Demonstrations, lic of Korea, June 21 - 24, 2022, ACM, 2022, pp. EACL 2021, Online, April 19-23, 2021, Associa1747–1764. URL: https://doi.org/10.1145/3531146. tion for Computational Linguistics, 2021, pp. 53–62. 3533229. doi:10.1145/3531146.3533229. URL: https://doi.org/10.18653/v1/2021.eacl-demos.7. [22] D. Percia David, L. Maréchal, W. Lacube, S. Gillard, doi:10.18653/v1/2021.eacl-demos.7.
M. Tsesmelis, T. Maillart, A. Mermoud, Measuring [30] E. H. Hovy, M. P. Marcus, M. Palmer, L. A. Ramshaw, security development in information technologies: R. M. Weischedel, Ontonotes: The 90% solution, in: A scientometric framework using arxiv e-prints, R. C. Moore, J. A. Bilmes, J. Chu-Carroll, M. SanderTechnological Forecasting and Social Change 188 son (Eds.), Human Language Technology Confer(2023) 122316. ence of the North American Chapter of the Asso[23] M. Honnibal, I. Montani, S. Van Landeghem, ciation of Computational Linguistics, Proceedings, A. Boyd, spaCy: Industrial-strength Natural Lan- June 4-9, 2006, New York, New York, USA, The Asguage Processing in Python, Zenodo (2020). doi:10. sociation for Computational Linguistics, 2006. URL: 5281/zenodo.1212303. https://aclanthology.org/N06-2015/. [24] R. Campos, V. Mangaravite, A. Pasquali, A. Jorge, [31] K. Clark, M. Luong, Q. V. Le, C. D. Manning, ELECC. Nunes, A. Jatowt, Yake! keyword extraction TRA: pre-training text encoders as discriminators from single documents using multiple local features, rather than generators, in: 8th International ConInformation Sciences 509 (2020) 257–289. ference on Learning Representations, ICLR 2020, [25] M. Grootendorst, Keybert: Minimal keyword ex- Addis Ababa, Ethiopia, April 26-30, 2020, OpenRetraction with bert., 2020. URL: https://doi.org/10. view.net, 2020. URL: https://openreview.net/forum? 5281/zenodo.4461265. doi:10.5281/zenodo.4461265. id=r1xMH1BtvB. [26] M. Kulkarni, D. Mahata, R. Arora, R. Bhowmik, [32] E. F. T. K. Sang, F. D. Meulder, Introduction to Learning rich representation of keyphrases from the conll-2003 shared task: Language-independent text, in: M. Carpuat, M. de Marnefe, I. V. M. named entity recognition, in: W. Daelemans, M. OsRuíz (Eds.), Findings of the Association for borne (Eds.), Proceedings of the Seventh ConferComputational Linguistics: NAACL 2022, Seat- ence on Natural Language Learning, CoNLL 2003, tle, WA, United States, July 10-15, 2022, As- Held in cooperation with HLT-NAACL 2003, Edsociation for Computational Linguistics, 2022, monton, Canada, May 31 - June 1, 2003, ACL, 2003, pp. 891–906. URL: https://doi.org/10.18653/v1/2022. pp. 142–147. URL: https://aclanthology.org/W03findings-naacl.67. doi:10.18653/v1/2022.findings- 0419/.
naacl.67. [33] N. Goyal, J. Du, M. Ott, G. Anantharaman, A. Con[27] L. Marujo, A. Gershman, J. G. Carbonell, R. E. Fred- neau, Larger-scale transformers for multilingual erking, J. P. Neto, Supervised topical key phrase masked language modeling, in: A. Rogers, I. Calextraction of news stories using crowdsourcing, ixto, I. Vulic, N. Saphra, N. Kassner, O. Camburu, light filtering and co-reference normalization, in: T. Bansal, V. Shwartz (Eds.), Proceedings of the N. Calzolari, K. Choukri, T. Declerck, M. U. Do- 6th Workshop on Representation Learning for NLP, gan, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis RepL4NLP@ACL-IJCNLP 2021, Online, August 6, (Eds.), Proceedings of the Eighth International Con- 2021, Association for Computational Linguistics, ference on Language Resources and Evaluation, 2021, pp. 29–33. URL: https://doi.org/10.18653/v1/ LREC 2012, Istanbul, Turkey, May 23-25, 2012, Eu- 2021.repl4nlp-1.4. doi:10.18653/v1/2021.repl4nlpropean Language Resources Association (ELRA), 1.4. 2012, pp. 399–403. URL: http://www.lrec-conf.org/ [34] S. Ishizaki, D. Kaufer, Computer-aided rhetoriproceedings/lrec2012/summaries/672.html. cal analysis, in: Applied natural language processing: Identification, investigation and resolution, IGI Global, 2012, pp. 276–296. URL: https: //www.igi-global.com/chapter/content/61054. [35] J. Pennington, R. Socher, C. D. Manning, Glove:
chitti, B. Pang, W. Daelemans (Eds.), Proceedings of the 2014 Conference on Empirical Methods in
ber 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, ACL, 2014, pp. 1532–1543. URL: https://doi.org/10.3115/v1/d141162. doi:1 0 . 3 1 1 5 / v 1 / d 1 4 - 1 1 6 2 . [36] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei,
vised multitask learners, OpenAI blog 1 (2019) 9. [37] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, Trans. Assoc. Comput. Linguistics 5 (2017) 135–146. URL: https://doi.org/10.1162/tacl_a_00051. doi:1 0 . 1 1 6 2 / t a c l \ _ a \ _ 0 0 0 5 1 . [38] T. Mikolov, K. Chen, G. Corrado, J. Dean, Eficient estimation of word representations in vector space, in: Y. Bengio, Y. LeCun (Eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL: http://arxiv.org/abs/1301.3781. [39] L. Van der Maaten, G. Hinton, Visualizing data using t-sne., Journal of machine learning research 9 (2008). [40] L. McInnes, J. Healy, UMAP: uniform manifold approximation and projection for dimension reduction, CoRR abs/1802.03426 (2018). URL: http: //arxiv.org/abs/1802.03426. a r X i v : 1 8 0 2 . 0 3 4 2 6 . [41] A. Hulth, Improved automatic keyword extraction given more linguistic knowledge, in: Proceedings of the 2003 conference on Empirical methods in natural language processing, 2003, pp. 216–223. [42] T. Chari, J. Banerjee, L. Pachter, The specious art of single-cell genomics, BioRxiv (2021) 2021–08.