Web Content Filtering Through Knowledge Distillation of Large Language Models Tamás Vörös1 , Sean Bergeron1 and Konstantin Berlin1 1 Sophos AI Abstract We introduce a state-of-the-art approach for URL categorization that leverages the power of Large Language Models (LLMs) to address the primary objectives of web content filtering: safeguarding organizations from legal and ethical risks, limiting access to high-risk or suspicious websites, and fostering a secure and professional work environment. Our method utilizes LLMs to generate accurate classifications and then employs established knowledge distillation techniques to create smaller, more specialized student models tailored for web content filtering. Distillation results in a student model with a 9% accuracy rate improvement in classifying websites, sourced from customer telemetry data collected by a large security vendor, into 30 distinct content categories based on their URLs, surpassing the current state-of-the-art approach. Our student model matches the performance of the teacher LLM with 175 times less parameters, allowing the model to be used for in-line scanning of large volumes of URLs, and requires 3 orders of magnitude less manually labeled training data than the current state-of-the-art approach. Depending on the specific use case, the output generated by our approach can either be directly returned or employed as a pre-filter for more resource-intensive operations involving website images or HTML. Keywords Machine Learning, Web Content Filtering, Large Language Models 1. Introduction Web content filtering is crucial for maintaining network security and regulatory compliance in organizations[1, 2]. The aim of a web content filtering system is to prevent employees from accessing inappropriate content that violates regulatory requirements or company policies, and by filtering out high-risk content categories, such as pornography and weapons, it helps to avoid legal liability, reduces the risk of legal or ethical issues arising from exposure to unsuitable content, and promotes a professional work environment. Unlike security classification, which detects hosted malware and phishing attacks, content filtering models address a more general problem that is independent of the attack mechanism. In this work, we address the problem of web content categorization. Traditional approaches to website categorization have relied upon creating and main- taining domain-to-category mappings, which are lists of domains grouped by their manu- ally assigned categories [3]. A natural extension to list-based URL categorization is to en- CAMLIS’23: Conference on Applied Machine Learning for Information Security, October 19–20, 2023, Arlington, VA $ tamas.voros@sophos.com (T. Vörös); sean.bergeron@sophos.com (S. Bergeron); konstantin.berlin@sophos.com (K. Berlin) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings hance them with signatures created by analysts, that would generalize better than exact string matching. In the case of web content filtering, the most straightforward signature- based approach is to propagate labels based on domains and subdomains, although more complex rules may be applied [4, 5, 6]. An example of this kind of label propagation is to maintain a list of known domains with predetermined labels, such as labeling “online- shop.com” as an e-commerce site and “news-site.com” as a news site. All URLs under these domains inherit the label. For instance, any URL under “online-shop.com” such as “online-shop.com/products/clothing”, “online-shop.com/products/electronics”, and “online- shop.com/cart” can be labeled as e-commerce. Similarly, any URL under “news-site.com”, such as “news-site.com/politics”, “news-site.com/technology”, and “news-site.com/entertainment” can be labeled as news. In the manuscript, we focus on domain label propagation signatures for acquiring ground truth for the sake of simplicity, but it could be trivially extended to longest prefix matching of the URL for ambiguous websites. To provide comprehensive customer teleme- try coverage for organizations, one of the most resource-effective manual methods involves ranking domains by frequency and labeling them in descending order. This approach maximizes coverage by prioritizing the labeling of a single domain. As new websites emerge daily and with over a billion existing websites, maintaining and scaling signature approaches manually for the long tail has become increasingly challenging. This necessitates the integration of machine learning into the classification pipeline[7, 8, 9]. Figure 1 illustrates the telemetry coverage of a large security vendor, with the space above the bars representing the infrequently seen long tail distribution of domains not already covered by domain labeling and label propagation signatures. Figure 1: Labeling drop-off. The plot visualizes the proportion of analyst-labeled domains across different popularity levels of a large security vendor. The domain popularity is represented on the 𝑥-axis using a logarithmic scale, where higher values indicate more popular domains. Each bar in the plot corresponds to a specific popularity bin, with the height of the bar illustrating the proportion of labeled domains within that bin. Maintaining domain-to-category mapping lists and extending them with signatures remains critical in the early stages of security pipelines [10]. These labels serve as initial shortcuts in the filtering pipeline to prevent catastrophic false positives, and provide low latency on more commonly seen websites. Websites like ’stackoverflow.com’ are well-known and need not be evaluated by a model whereas a potential false positive would translate to a negative impact on the productivity of an organization. In this work, we focus our evaluations on the long tail of the distribution, which aligns with actual deployment scenarios and emphasizes the need for machine learning to address the challenges associated with classifying this ever-growing subset of domains. In addition to acting as a pre-filter, domain-to-category mapping lists and label propagation signatures are often used to create the training sets for machine learning models. However, machine learning algorithms tend to memorize patterns rather than understand underlying concepts [11, 12], thus learning from already labeled URLs is insufficient for accurate content classification in the long tail of the URL distribution. A model whose parameters are configured to memorize the head of the distribution is undesired as signatures already cover such domains without risking false positives. Therefore, our objective is to identify models with superior generalization capabilities for out-of-distribution samples. For unknown or new domains, the model must infer a description from the URL. It is useful to view URL classification, especially for web content filtering, as a natural language processing task, considering URLs as semi-sentences. For a fair amount of our categories, the URL will frequently have explicit words to advertise its content, specifically semantically related keywords for the given category. For example, a site selling weapons will often contain keywords such as “armaments” or “glock” or “gun”. The current state-of-the-art in URL detection and our chosen baseline, URLTran [9], frames URL detection as a natural language processing task and fine-tunes a pre-trained BERT model [13] to detect phishing URLs. The BERT model is an early example of the transformer architecture [14] which has since been refined and scaled, giving rise to large language models. Large language models (LLMs) are state-of-the-art on natural language tasks [15]. LLMs are first pre-trained on large amounts of unlabeled textual data in a task-agnostic manner, learning a general understanding of language such as syntax and semantics [15]. Once pre-trained LLMs can effectively generalize to new tasks upon fine-tuning or few-shot prompting with much smaller amounts of data [15]. The amount of data needed for LLMs to generalize to new tasks is often several orders of magnitude less than the amount of data needed to fully train a smaller model. Direct use of LLMs for URL content classification in production is prohibitive due to cost considerations at scale [16]. Fine-tuning smaller LLMs that have lower inference costs results in a loss of performance. Through knowledge distillation [17], the LLM-labeled long tail data enables a smaller student model to improve its performance while maintaining the necessary computational efficiency for production. Turc et al. [18] proposed an approach that utilizes knowledge distillation from the teacher’s predictive distribution (soft labels) followed by super- vised fine-tuning of the student model. In the domain of web content classification, we combine the steps of distillation and fine-tuning and our computationally efficient student matches the performance of the teacher model. Instead of a predictive distribution, we distill the teacher using hard labels. The student model has a low inference cost and is well-suited for the purposes of web content filtering in production. The main contributions of this paper are as follows: • We demonstrate that when fine-tuned on data labeled with domain propagation signatures, large language models outperform standard deep learning models by 9% in terms of accuracy on the long tail categorization problem. • We demonstrate that we can fine-tune a large language model using 10000 samples to achieve better performance than the current state-of-the-art approach trained on 10 million samples. • We showcase the effective application of knowledge distillation from a fine-tuned LLM to boost the performance of a smaller, more computationally efficient model, specifically for web content filtering tasks. We attain performance levels comparable to the original LLM using a model that is 175 times smaller, decreasing from 770 million parameters to just 4 million. This reduction in size makes the model more suitable for production and enables practical deployment across various contexts, such as serving as a general pre-filter for all incoming network traffic in firewalls. • We propose a novel validation approach for the community to adopt, which more ac- curately assesses model performance in realistic scenarios where it works alongside a domain-to-category mapping list of ground truth labels, extended via domain label propagation signatures. In this setting, the model analysis focuses on labeling the long tail, focusing on a more relevant metric. Our paper is structured as follows: In Section 1 we introduce the research problem and elucidate the motivation behind our proposed approach. In Section 2 we review relevant literature and prior work in the field. In Section 3, we provide a comprehensive description of our methodology, encompassing the dataset and experimental setups. In Section 4 we present our results, which include a comparison of our approach’s performance against the current state-of-the-art, an analysis of the benefits of LLMs in terms of accuracy and sample efficiency, as well as an exploration of deployment challenges and our proposed solution utilizing knowledge distillation for more compact and computationally efficient models. Lastly, In Section 5 we conclude the paper, outlining potential avenues for future research in this domain. 2. Related Work Previous work in this field has primarily focused on security classification rather than content classification and filtering. Since machine learning approaches to security classification can be readily reformulated from binary classification to multi-class classification through modification of the last layer in the neural network, approaches to security classification are relevant to the task of content classification. We will compare and build upon security publications as they are better studied. Early work on URL-only classification for phishing detection using manually derived feature sets employed both generic features and features meant to detect certain obfuscation techniques such as obfuscation of the host with another domain [19]. The features were divided into four groups: Page Based, Domain Based, Type Based, and Word Based. The authors focused on manual feature engineering and only applied logistic regression as their classifier. A range of machine learning models, including Random Forests, Logistic Regression, Support Vector Machines, Naive Bayes, and Gradient Boosting, have been applied to detect phishing URLs using manually extracted feature sets [7, 20]. Feature sets may be entirely lexically derived such as the length of the URL, the number of digits in the primary domain, and the number of special characters in the path [21]. In addition to lexical features, domain-specific features such as the number of passive DNS changes or the remaining time of the SSL certificate may be incorporated [22]. Manual features may also be extracted from the retrieved information of lookups (Whois, GSB Reporting, Google Ranking, and Selenium Rendering) [23]. The manual feature extraction approach is difficult to maintain as adversaries tend to adapt obfuscation methods to avoid detection so models have shifted to a featureless approach based on the raw string as input. Deep learning methods learn and then automatically extract the feature set from the raw URL during training. The use of automatically extracted features does not preclude the inclusion of manual features however as the optimal input combination of manual and automatic features can be optimized with genetic algorithms [24, 25]. Automatic feature extraction can be done on various levels of granularity starting at the character level. Saxe et al. [8] encode a URL by replacing each character with its corresponding ID whereby features are extracted from the encoded URL with sequential embedding and convolutional layers. This approach outperformed a baseline which uses a manual feature set. Learning meaningful context-independent representations is difficult when using character-level tokenization as a character token doesn’t carry the same meaning that a word does. More recent approaches like subword-level and word-level tokenization have been developed in natural language processing in order to make it easier for models to maintain semantic meaning in common subwords and learn more meaningful context-independent representations. The application of word-level tokenization to URL classification was first proposed by Le et al. [26] who extracted both character-level and word-level features. Each feature set is fed through its own series of sequential embedding and convolutional layers before being fused. Tajaddodianfar et al. [27] expand on this approach by first training the word embeddings in an unsupervised manner via FastText [28]. The word and character convolutional stems include several convolutional layers in parallel with dilated convolutions allowing the model to adaptively grow in depth and width, extracting N-grams of various lengths. In addition to using both character and word-level feature models, Bu et al.[29] apply a triplet network structure in order to address class imbalances and better learn the similarity between URLs. In addition, to feature set selection, the choice of model architecture plays a large role in the performance of a URL classification model. Transformers have achieved state-of-the-art results in many natural language processing tasks making them a good candidate for URL classification after fine-tuning or even custom pre-training [30, 31, 9, 32, 33]. In addition to the URL, a transformer can leverage tokenized features of the HTML [34]. A URL classification system might employ different architectures in parallel, fusing the output of models with a convolutional architecture and a transformer architecture [35]. Instead of fusing model outputs, a system may employ an ensemble of different architectures including Decision Trees, LSTMs, and transformers for URL classification [36]. Other architectures applied to URL classification include graphical networks [37, 38] and GANs [39, 40]. AutoEncoders have proven useful against zero-day attacks [41]. In addition to the URL and HTML sequences, but beyond the scope of this paper, images of the webpage may be incorporated [42, 43]. The task of classification may be reformulated by approaching detection from a reinforcement learning perspective [44] or from the perspective of thwarting an adversarial opponent [45, 46]. The current state-of-the-art for URL-only classification for phishing detection, URLTran [9], utilizes the transformer architecture underpinning LLMs. Maneriker et al. fine-tune a pre- trained BERT model on Microsoft’s Edge and Internet Explorer production browsing telemetry. Parallel to URLTRan is the Unified Text-to-Text Cybersecurity (UTS) model. Pal et al. [47] train a multi-task encoder-decoder LLM on cybersecurity data that includes URL phishing detection. Although Pal et al. introduce LLMs to URL phishing detection, they do not explore the few shot capabilities of LLMs in the URL domain nor test the capabilities of LLMs at scale. Compared to URLTran, UTS does not consider a methodology which would allow its large model to be used in production and reports a lower F1 score on a random split compared to URLTran’s evaluation on the industry standard time split. Therefore, URLTran will act as our baseline to which all of our results will be compared. 3. Methodology In this section, we describe our methodology for collecting data and constructing training, validation, and test sets. We also explain our experimental setup and provide a detailed account of how we trained our model. 3.1. Data We obtained our dataset from a large security vendor’s customer telemetry data sourced from its firewall and endpoint products over a period spanning July 1, 2022 to December 23, 2022. We track 30 categories in our dataset. These categories were defined by a team of expert analysts to be representative of the most common internet content categories as well as the most impactful, which we define as the potential to impact productivity, the degree of liability for the organization, and the degree of associated ethical concerns. The categories include: “Chat”, “Games”, “Shopping”, “Sports”, “News”, “Job Search”, “Search Engines”, “Alcohol”, “Gambling”, “Weapons”, “Porn”, “Banking”, “Business”, “Education”, “Entertainment”, “Food and Dining”, “Government”, “Health and Medicine”, “Motor Vehicles”, “Peer to Peer”, “Real Estate”, “Religion”, “Travel”, “Translators”, “Computer and Internet”, “Hunting and Fishing”, “Marijuana”, “Radio and Audio Hosting”, “Social Networking”, and “Video Hosting”. The majority of websites in our dataset belong to categories such as “Computer and Internet”, “Search Engines”, and “Business”, while niche categories such as “Hunting and Fishing” and “Marijuana” have fewer instances. Figure A5 shows the distribution of categories in our dataset. We define our categorization task as a closed-world problem, meaning every URL belongs to one of the 30 categories. It’s important to note that, due to limitations in the domain-to-category database, we only consider a single category per URL, even though some pages may realistically have multiple category labels. 3.2. Training Sets To construct our training dataset which spans the period from July 1, 2022 to August 19, 2022, we uniformly sampled 10 million distinct URLs, out of the billions of URL lookups, that have been labeled using a domain-to-category mapping database with label propagation. Additionally, we sampled 10 million URLs from this period that did not correspond to a signature (unlabeled). The unlabeled URLs were set aside for training augmentation purposes. 3.3. Validation and Test Sets We sampled an evaluation dataset spanning from August 19, 2022 to December 23, 2022 and divided it into two validation and test sets to assess our model’s performance in different scenarios. The validation sets were based on data first seen between August 19, 2022 and November 24, 2022 while the test sets included data first seen between November 24, 2022 and December 23, 2022. We created a domain and time split to simulate a long tail deployment setting. We separated the data based on the first-seen time of the URL and the first-seen time of the URL’s domain. The first-seen time of a domain refers to the earliest instance of a URL from that domain. Meaning, there is no domain overlap between the training, validation, and test sets. This approach allowed us to better approximate the unlabeled part of the telemetry. To compare our results with the industry-standard evaluation methodology we also created a time split. This split was sampled from the same time span as the domain and time split but without the constraint of dividing based on the domain’s first-seen time. For the domain and time split, the validation set was comprised of 79,313 unique URLs from 30,897 unique domains, with a maximum of 5 URLs per domain. The test set included 110,624 unique URLs from 43,996 domains. For the time split, we sampled 183,935 URLs from 62,961 domains. To compare the various splits, we display the most common domains and their frequencies for the labeled training data, both test splits, and the unlabeled training data in Table 1. The labeled training set and the time split are dominated by common domains such as “google.com”. The domain and time split is most similar to the unlabeled long tail of the data where the desired value of machine learning resides. Table 1 Domains and their frequencies in the train and test sets. Training Set Time Split Test Set Domain and Time Split Test Set Unlabeled Data Domain Frequency % Domain Frequency % Domain Frequency % Domain Frequency % google.com 20 google.com 33 tomcleaneraddon.com <1 wymondhamcollege.org 2 microsoft.com 6 microsoft.com 4 ammdx.com <1 mfa.cloud 1 googleapis.com 5 gstatic.com 2 ogp.me <1 dimmittisd.net <1 cedexis-radar.net 3 googlesyndication.com 2 officested.com <1 qq.com.cn <1 gvt1.com 3 doubleclick.net 2 dimensionu.com <1 gnsmat.co.uk <1 facebook.com 2 msn.com 2 shreemaruti.com <1 headlandentertainment.com <1 zeotap.com 2 googleusercontent.com 2 wbe-eindhoven.nl <1 murray.edu <1 youtube.com 1 youtube.com 1 vapornodes.finance <1 pcbid.top <1 amazonaws.com 1 amazonaws.com 1 trendingtrck.com <1 stoughtonwi.com <1 sharepoint.com 1 cloudfront.net 1 bluedrop360.com <1 aveha.com <1 To quantitatively assess the disparities between the domain distributions of the time split and the domain and time split, which more closely models the long tail, we employed the Kullback-Leibler (KL) divergence as a metric for measuring the dissimilarity between token distributions. The KL divergence values were calculated between each validation split and the training dataset as the reference. We tokenized all the URLs in the training dataset using BERT tokenization and then combined all of the tokens to define the distribution of the base training dataset. We tokenized all the URLs in both validation splits using BERT tokenization and each token sequence was converted into probability distributions by computing normalized histograms. The KL divergence between the token probability distribution of each URL and the reference distribution was then determined using the entropy function. Figure 2 illustrates that the token distribution of the domain and time split displays substantially higher KL divergence values from the reference compared to the time split. This observation highlights the distinct nature of the domain distributions in the two validation splits and the similarity between the unlabeled part of the customer telemetry and the domain and time split. Figure 2: KL Divergence over BERT tokens. The 𝑥-axis represents the possible range of KL divergence values over BERT tokens, while the 𝑦-axis represents the estimated probability density of these values. The plot quantifies the difference between validation split URLs as compared to the training set token distribution, with higher KL divergence values indicating greater differences between the BERT token distributions of the base training set and validation split. 3.4. Experiments The primary objective of our experiments was to identify the best-performing model in terms of accuracy on our dataset, while using as few training labels and being as small as possible. To achieve this, we varied the training set size as a hyperparameter for each LLM, compact model, and the baseline. We explored training set sizes ranging from few-shot to large-scale learning, increasing the sample size from 10 samples per category to 5 million total samples, growing by an order of magnitude at each step. For a given sample step size 𝑁 , the exact samples per category were determined by the minimum of 𝑁 and the total labeled instances in that category. Our next goal was to refine the top-performing large language model (LLM) configuration into a more compact student model. We achieved this by using labels generated by the best- performing LLM to train smaller models. We labeled 10 million unlabeled URLs from our dataset using the best-performing LLM, utilizing them as hard labels. This resulted in a total of 20 million training set with the 10 million signature-labeled base training set and an additional 10 million labels generated by the LLM. We then investigated the impact of combining these labels using various mixing ratios of labeled samples from the base training set and LLM-labeled samples. Each compact student model and baseline were trained on a variety of dataset configurations, each containing a total of 10 million samples. We began with a 10-million base training set, incorporating LLM-generated labels at 0.0, 0.25, 0.50, 0.75, and 1.0 ratios. The 0.0 ratio used only the base training set, while the 0.25 ratio included 7.5 million base URLs and 2.5 million LLM-generated. At 0.5, the sources were evenly split with 5 million each. The 0.75 ratio contained 2.5 million base and 7.5 million LLM URLs, and the 1.0 ratio relied solely on LLM-generated labels. By varying the mixing ratios, we were able to assess the effectiveness of our knowledge distillation process and compare the contributions of LLM-generated labels to simply using signature-generated labels. We trained and compared the performance of five models: BERT-based URLTran as the baseline[9], which demonstrated state-of-the-art performance for URL classification, eXpose [8] and BERTiny [48] as the student models, and T5 Large [49] and GPT-3 Babbage [15] as the teacher models. The size configurations of our teacher models were limited by budgetary constraints, precluding larger configurations such as GPT-3 Davinci and T5-11B. Our student models were chosen for the following reasons: BERTiny is the smallest pre-trained configuration of the baseline and the inclusion of eXpose allows us to demonstrate the improvements of the transformer architecture over convolutional models for natural language tasks, specifically web content categorization. Unless otherwise noted, all experiments were evaluated on the test set of both validation splits. The GPT-3 Babbage model was not fine-tuned on 5 million samples due to cost considerations. 3.5. Training For all models, we pre-processed the data by splitting at the first occurrence of the “?” character and removing the query parameters. The query is assumed to be noisy and without any meaningful information. All URLs were truncated to a fixed length of 128 characters as we have seen no improvement in further increasing the size. The base pre-trained models and tokenizers for all T5 Large, BERT, and BERTiny configurations were the HuggingFace defaults [50]. For all reported T5 configurations, we fine-tuned all weights of a pre-trained T5 Large model using the Adafactor optimizer [51]. Early stopping was applied by monitoring performance on the validation set of the domain and time split. For all reported GPT-3 configurations, we fine-tuned the Babbage model using the OpenAI API. T5 and GPT-3 are generative models that can utilize semantic relationships between class labels and keywords in a URL for making predictions. Consequently, we employed literal class labels as our prediction target. When reporting aggregate metrics, out-of-vocabulary (OOV) predictions are not considered as a separate class, and they were considered as misclassfication for every class. Additionally, any unlabeled data for which LLM generates an OOV prediction is excluded from the distillation process. For GPT3 the temperature was set to 0 to ensure deterministic results upon inference. The logit bias for tokens associated with the class labels were set to 100 to ensure exclusive selection of expected tokens. Finally, the stop token was set to the stop sequence seen during training. For the student models, we trained a 1D convolutional eXpose model and fine-tuned all weights of a pre-trained BERTiny model. We fine-tuned all weights of a pre-trained BERT model to reproduce the architecture of URLTran as our baseline. No custom vocabulary was created for the BERT-based models. Hyperparameter configurations for T5, BERTiny, BERT, and eXpose may be found in Tables A7, A4, A5, and A6 respectively. 4. Results Figure 3: The results for scaling and augmentation are dependent on the domain and time split. Left: Scaling results: illustrates the performance of various models in relation to the logarithm of training sample size. Right: Augmentation results: compares the top-performing LLM and baseline configurations with the performance of different student models as a function of the mixing ratio for LLM-generated labels. The GPT-3 Babbage model was not fine-tuned on 5 million samples due to cost considerations. In this section, we present the key findings and results of our two sets of experiments. We report the results in terms of accuracy, with additional metrics for both experiments provided in the Appendix. The performance of various models as a function of the log of the training sample counts is displayed in Figure 3. The top-scoring configuration for each model is detailed in Table 2. On the domain and time split, the best performing model, T5 Large, achieves 46.3% accuracy after being fine-tuned on 10,000 samples. GPT-3 Babbage attains 44.4% accuracy after fine-tuning on 10,000 samples. Both LLMs surpass the best baseline configuration, which achieves 38.3% accuracy. BERTiny and eXpose achieve 35.7% and 30.2% accuracy, respectively, when trained on 5 million samples. On the time split, eXpose achieves 92.8% accuracy when trained on 5 million samples. BERTiny, fine-tuned on 5 million samples, attains 97.6% accuracy. The best configurations for the baseline, GPT-3 Babbage, and T5 Large achieve 97.1%, 98.14%, and 97.5% accuracy, respectively. Additional metrics for the time split are reported in Table A10, and for the domain and time split in Table A9 for all experiments. On the domain and time split, the best performance was achieved with T5 on 10,000 training samples, so we selected it as our teacher model. For the domain and time split, we found the best ratio to be 1.0, where every training sample had 10 million previously unlabeled URLs labeled by T5. Training eXpose on all of them increases the accuracy from 31.5% to 45%. Fine-tuning BERTiny on all 10 million LLM labels, compared to the 10 million base training set, improves the accuracy from 37.5% to 46.2%. Finally, fine-tuning URLTran on all 10 million LLM labels, compared to the 10 million base training set, raises the accuracy from 41.6% to 46.8%. For the traditional time split, the augmentation at a ratio of 0.75 also increased the performance, albeit marginally. The performance of the students and the baseline trained via knowledge distillation is shown in the augmentation plot of Figure 3 as a function of the LLM label ratio in the training data. The top-scoring distillation configuration for each student model is detailed in Table 2. Table 2 The best performing model configurations are presented, including the distilled versions of the students and baseline. The accuracy for each model’s top configuration is displayed, along with the model’s parameter count relative to the best performing LLM. For the Time and Domain split, the LLM label ratios correspond to 1.0, and for the Time split, the ratio is 0.75, as these were consistently the best across the models. More detailed results can be found in Table A9 and Table A10. Accuracy Accuracy Parameter Count Parameter Count Relative Model Time and Domain Split Time Split in millions to the Teacher (%) Training Samples Count eXpose (Conv) 0.30 0.93 3.3 5 × 106 BERTiny 0.36 0.97 4.4 5 × 106 URLTran (BERT) 0.38 0.97 110 1 × 105 T5 Large 0.46 0.97 770 1 × 104 GPT3 Babbage 0.45 0.98 6700 1 × 105 eXpose + T5 Labels 0.45 0.98 3.3 0.42 1 × 107 BERTiny + T5 Labels 0.46 0.98 4.4 0.57 1 × 107 URLTran + T5 Labels 0.47 0.99 110 14.29 1 × 107 4.1. Discussion A comparison of the models’ performance on the two evaluation splits reveals that results on the time split, the traditional validation approach, are overly optimistic. Small models such as BERTiny, trained merely on signature-driven data, exhibit performance comparable to T5 and GPT-3. The disparity in model performance between the domain and time split versus the time split, particularly for small models, underscores that signature-sourced data is repetitive and can be memorized with just a few million parameters. Time split validation measures a model’s ability to match the signature distribution while in a production setting the primary concern within the context of the overall pipeline is a model’s capacity to generalize to new data from the long tail that falls outside the coverage of signatures. When considering the domain and time split—which aligns more closely with real-world performance on unlabeled data—small models no longer match the performance of LLMs, as seen in Figure 3. Beyond 10,000 samples, LLMs show minimal to no performance gains when scaling up further. Conversely, the performance of small models and the baseline has not yet converged at 5 million training samples. This demonstrates the sample-efficiency of LLMs in the domain of website content categorization. LLMs outperform student models in terms of performance, but they still fall short of perfection when applied to domain and test splits. This discrepancy can be attributed to two main factors. First, due to dataset limitations, web content classification is framed as a single-label classification problem. Table 3 displays a set of LLM misclassifications on domain and time split, highlighting Table 3 Examples of LLM (T5) misclassifications. Comparison of LLM performance on the domain and time split, highlighting the impact of the single-label strategy and keyword-absent URLs. Domain LLM Label True Label citytocoastneurosurgery.com.au HEALTH AND MEDICINE BUSINESS twittodon.com SOCIAL NETWORKING COMPUTER AND INTERNET robinsonmalls.com/mall-info SHOPPING BUSINESS online-weinshop.at SHOPPING ALCOHOL www.fourbakery.com BUSINESS FOOD sargenttoolsonline.com BUSINESS SHOPPING praeyforthegods.com RELIGION GAMES www.hygiene-3d.com HEALTH AND MEDICINE SHOPPING beta.x9zb.live COMPUTING AND INTERNET GAMBLING www.857zb6.com ENTERTAINMENT SPORTS www.lxf.cz BUSINESS SHOPPING g11.178tiyu.com ENTERTAINMENT SPORTS that a URL could potentially belong to multiple categories. In the first three samples, the analyst opted for the more generic label, while the model choose the more generic labels in the following three samples. Both predicted and true labels could be considered correct in all six cases, suggesting that the true performance is likely better than the metrics indicate because of the single-label limitation. This trade-off between equally correct specific and general labels becomes evident when examining the confusion matrix, displayed in Figure A4 in the Appendix, for a T5 Large model’s performance on the domain and time split. As we can see on the confusion matrix, the LLM tends to generate class labels that are more specific than manual labels. The second factor occurs when a URL lacks keywords or context related to its category, as demonstrated by the last six entries in Table 3. For the middle two URLs, the model was misled by a prominent keyword in the URL, which was unrelated to its content. The final four URLs contain no apparent signal. Consequently, the large-scale pre-training of LLMs struggles to effectively transfer knowledge to a URL from the long tail. This means that if a URL lacks clear or strong indicators of its category, the LLM may not accurately classify it, leading to misclassifications. Our results reveal that mixing in the LLM-generated labels significantly enhances the perfor- mance of student models BERTiny and eXpose as we can see on Figure 3. Through this simple form of augmentation, we nearly matched the 46.3% accuracy of T5 Large, the best-performing LLM, with a transformer model that has a parameter count several orders of magnitude smaller (0.57% of the teacher) and could reasonably be deployed in-line in production. 5. Conclusion In conclusion, our paper contributes to the field of web content classification with the develop- ment of lightweight models distilled from fine-tuned LLMs. We have demonstrated that LLMs, when fine-tuned on data labeled with domain propagation signatures, significantly outperform the current state-of-the-art approach on the long tail categorization problem. Our teacher- student training approach enables the distillation of LLMs into models 175 times smaller without sacrificing accuracy, thus making deployment practical in a wide variety of new contexts. The amount of manual labels required to finetune the teacher LLM is orders of magnitude smaller than what is required for convergence of the current state-of-the-art approach. Furthermore, we have proposed a new validation approach that better measures model performance in more realistic scenarios, which should be adapted by the community to improve generalization capabilities to unseen data. Expanding beyond web content classification, the cybersecurity field could greatly benefit from proven methods of distilling large language models (LLMs) into more compact versions. This approach is particularly valuable when dealing with large data volumes and expensive training samples, especially when the model is applied to out-of-distribution cases. For ad- dressing web content classification tasks specifically, we suggest future work should focus on augmenting the training data and feature space with HTML and image data, utilizing GPT-4 as a teacher, allowing URLs to have more than one label, and re-working signatures for the assignment of general categories. References [1] F. García, Web content filtering. advances in computers., 2009. https://www.academia.edu/ 11471179/Web_Content_Filtering. [2] D. S. K. Ankur Baishya, A review on web content filtering, its technique and prospects, http://www.ijcstjournal.org/volume-7/issue-3/IJCST-V7I3P5.pdf (2022). [3] S. Sheng, B. Wardman, G. Warner, L. Cranor, J. Hong, C. Zhang, An empirical analysis of phishing blacklists, roceedings of Sixth Conference on Email and Anti-Spam (CEAS) (2009). [4] Q. Chen, P. Snyder, B. Livshits, A. Kapravelos, Improving web content blocking with event-loop-turn granularity javascript signatures, arXiv preprint arXiv:2005.11910 (2020). [5] C.-Y. Huang, S.-P. Ma, W.-L. Yeh, C.-Y. Lin, C.-T. Liu, Mitigate web phishing using site signatures, in: TENCON 2010-2010 IEEE Region 10 Conference, IEEE, 2010, pp. 803–808. [6] S. Haruta, F. Yamazaki, H. Asahina, I. Sasase, A novel visual similarity-based phishing detection scheme using hue information with auto updating database, in: 2019 25th Asia-Pacific Conference on Communications (APCC), IEEE, 2019, pp. 280–285. [7] J. Ma, L. K. Saul, S. Savage, G. M. Voelker, Beyond blacklists: learning to detect malicious web sites from suspicious urls, in: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, 2009, pp. 1245–1254. [8] J. Saxe, K. Berlin, expose: A character-level convolutional neural network with embeddings for detecting malicious urls, file paths and registry keys, arXiv preprint arXiv:1702.08568 (2017). [9] P. Maneriker, J. W. Stokes, E. G. Lazo, D. Carutasu, F. Tajaddodianfar, A. Gururajan, Urltran: Improving phishing url detection using transformers, in: MILCOM 2021-2021 IEEE Military Communications Conference (MILCOM), IEEE, 2021, pp. 197–204. [10] G. Apruzzese, H. S. Anderson, S. Dambra, D. Freeman, F. Pierazzi, K. A. Roundy, "real attackers don’t compute gradients": Bridging the gap between adversarial ml research and practice, https://doi.org/10.48550/arXiv.2212.14315 (2022). [11] J. Zhang, H. Chao, A. Dhurandhar, P. Chen, A. Tajer, Y. Xu, P. Yan, When neural networks fail to generalize? a model sensitivity perspective, https://doi.org/10.48550/arXiv.2212.00850 (2022). [12] P. Bhargava, A. Drozd, A. Rogers, Generalization in nli: Ways (not) to go beyond simple heuristics, 2021. arXiv:2110.01518. [13] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polo- sukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). [15] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [16] , How many websites are there, 2023. https://siteefy.com/how-many-websites-are-there/. [17] G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531 (2015). [18] I. Turc, M. Chang, K. Lee, K. Toutanova, Well-read students learn better: The impact of student initialization on knowledge distillation, CoRR abs/1908.08962 (2019). URL: http://arxiv.org/abs/1908.08962. arXiv:1908.08962. [19] S. Garera, N. Provos, M. Chew, A. D. Rubin, A framework for detection and measurement of phishing attacks, in: Proceedings of the 2007 ACM workshop on Recurring malcode, 2007, pp. 1–8. [20] A. Oshingbesan, C. Ekoh, C. Okobi, A. Munezero, K. Richard, Detection of malicious websites using machine learning techniques, arXiv preprint arXiv:2209.09630 (2022). [21] A. Joshi, L. Lloyd, P. Westin, S. Seethapathy, Using lexical features for malicious url detection–a machine learning approach, arXiv preprint arXiv:1910.06277 (2019). [22] C. Hajaj, N. Hason, A. Dvir, Less is more: Robust and novel features for malicious domain detection, Electronics 11 (2022) 969. [23] A. Abuadbba, S. Wang, M. Almashor, M. E. Ahmed, R. Gaire, S. Camtepe, S. Nepal, Towards web phishing detection limitations and mitigation, arXiv preprint arXiv:2204.00985 (2022). [24] S.-J. Bu, H.-J. Kim, Optimized url feature selection based on genetic-algorithm-embedded deep learning for phishing website detection, Electronics 11 (2022) 1090. [25] K.-W. Park, S.-J. Bu, S.-B. Cho, Evolutionary optimization of neuro-symbolic integration for phishing url detection, in: International Conference on Hybrid Artificial Intelligence Systems, Springer, 2021, pp. 88–100. [26] H. Le, Q. Pham, D. Sahoo, S. C. Hoi, Urlnet: Learning a url representation with deep learning for malicious url detection, arXiv preprint arXiv:1802.03162 (2018). [27] F. Tajaddodianfar, J. W. Stokes, A. Gururajan, Texception: a character/word-level deep learning model for phishing url detection, in: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, pp. 2857– 2861. [28] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, T. Mikolov, Fasttext.zip: Compress- ing text classification models, arXiv preprint arXiv:1612.03651 (2016). [29] S.-J. Bu, H.-J. Kim, Learning disentangled representation of web address via convolutional- recurrent triplet network for classifying phishing urls, in: 2021 International Conference on Electronics, Information, and Communication (ICEIC), IEEE, 2021, pp. 1–4. [30] E. M. Rudd, A. Abdallah, Training transformers for information security tasks: A case study on malicious url prediction, arXiv preprint arXiv:2011.03040 (2020). [31] E. M. Rudd, M. S. Rahman, P. Tully, Transformers for end-to-end infosec tasks: A feasibility study, in: Proceedings of the 1st Workshop on Robust Malware Analysis, 2022, pp. 21–31. [32] W. Chang, F. Du, Y. Wang, Research on malicious url detection technology based on bert model, in: 2021 IEEE 9th International Conference on Information, Communication and Networks (ICICN), IEEE, 2021, pp. 340–345. [33] H. Shirazia, K. Haynesb, I. Raya, Towards performance of nlp transformers on url-based phishing detection for mobile devices, Journal of Ubiquitous Systems and Pervasive Networks, Volume 17,No. 1(2022) pp. 35-42 (2022). [34] Q. Hu, H. Zhou, Q. Liu, Phishing website detection based on multi-feature stacking, in: 2021 2nd International Conference on Artificial Intelligence and Computer Engineering (ICAICE), IEEE, 2021, pp. 716–720. [35] C. Wang, Y. Chen, Tcurl: Exploring hybrid transformer and convolutional neural network on phishing url detection, Knowledge-Based Systems (2022) 109955. [36] S. Venugopal, S. Y. Panale, M. Agarwal, R. Kashyap, U. Ananthanagu, Detection of malicious urls through an ensemble of machine learning techniques, in: 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), IEEE, 2021, pp. 1–6. [37] S. Ariyadasa, S. Fernando, S. Fernando, Combining long-term recurrent convolutional and graph convolutional networks to detect phishing sites using url and html, IEEE Access 10 (2022) 82355–82375. [38] T. Bilot, G. Geis, B. Hammi, Phishgnn: A phishing website detection framework using graph neural networks, Conference: SECRYPT 2022At: Lisbon (2022). [39] S. A. Kamran, S. Sengupta, A. Tavakkoli, Semi-supervised conditional gan for simultaneous generation and detection of phishing urls: A game theoretic perspective, arXiv preprint arXiv:2108.01852 (2021). [40] J. Geng, S. Li, Z. Liu, Z. Cheng, L. Fan, Effective malicious url detection by using generative adversarial networks, in: International Conference on Web Engineering, Springer, 2022, pp. 341–356. [41] S.-J. Bu, S.-B. Cho, Deep character-level anomaly detection based on a convolutional autoencoder for zero-day phishing url detection, Electronics 10 (2021) 1492. [42] J. Yuan, G. Chen, S. Tian, X. Pei, Malicious url detection based on a parallel neural joint model, IEEE Access 9 (2021) 9464–9472. [43] R. Liu, Y. Lin, X. Yang, S. H. Ng, D. M. Divakaran, J. S. Dong, Inferring phishing intention via webpage appearance and dynamics: A deep vision based approach, in: 30th {USENIX} Security Symposium ({USENIX} Security 21), 2022, p. 1. [44] O. Lavie, A. Shabtai, G. Katz, A transferable and automatic tuning of deep reinforcement learning for cost effective phishing detection, arXiv preprint arXiv:2209.09033 (2022). [45] Z. Peng, Y. He, Z. Sun, J. Ni, B. Niu, X. Deng, Crafting text adversarial examples to attack the deep-learning-based malicious url detection, in: ICC 2022-IEEE International Conference on Communications, IEEE, 2022, pp. 3118–3123. [46] T. Kim, N. Park, J. Hong, S.-W. Kim, Phishing url detection: A network-based approach robust to evasion, arXiv preprint arXiv:2209.01454 (2022). [47] K. K. Pal, K. Kashihara, U. Anantheswaran, K. C. Kuznia, S. Jagtap, C. Baral, Exploring the limits of transfer learning with unified model in the cybersecurity domain, arXiv preprint arXiv:2302.10346 (2023). [48] P. Bhargava, A. Drozd, A. Rogers, Generalization in nli: Ways (not) to go beyond simple heuristics, 2021. arXiv:2110.01518. [49] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Research 21 (2020) 5485–5551. [50] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45. [51] N. Shazeer, M. Stern, Adafactor: Adaptive learning rates with sublinear memory cost, in: International Conference on Machine Learning, PMLR, 2018, pp. 4596–4604. A. Supplementary Plots and Tables Figure A4: Normalized confusion matrix on the domain and time split for the best T5 Large configuration Figure A5: Class Distribution. Displaying the 30 classes and their distribution over the datasets. Table A4 BERTiny Parameters Parameter Value model_name prajjwal1/bert-tiny hidden_size 128 hidden_act gelu initializer_range 0.02 vocab_size 30522 hidden_dropout_prob 0.1 num_attention_heads 2 type_vocab_size 2 max_position_embeddings 512 num_hidden_layers 2 intermediate_size 512 attention_probs_dropout_prob 0.1 maximum sequence length 128 learning rate 1e-4 batch size 49160 optimizer Adam maximum training epochs 20 Table A5 BERT Parameters Parameter Value model_name bert-base-uncased hidden_size 768 hidden_act gelu initializer_range 0.02 vocab_size 30522 hidden_dropout_prob 0.1 num_attention_heads 12 type_vocab_size 2 max_position_embeddings 512 num_hidden_layers 12 intermediate_size 3072 attention_probs_dropout_prob 0.1 maximum sequence length 128 learning rate 1e-4 batch size 2048 optimizer Adam maximum training epochs 20 Table A6 eXpose Parameters Parameter Value vocab_size 76 filter_size 128 dropout 0.05 learning rate 1e-3 batch size 49160 optimizer Adam maximum training epochs 20 Table A7 T5 Parameters Parameter Value model_name t5-large d_ff 4096 d_kv 64 d_model 1024 dropout_rate 0.1 initializer_facto 1.0 layer_norm_epsilon 1e-06 n_positions 512 num_heads 16 num_layers 24 relative_attention_num_buckets 32 vocab_size 32128 maximum sequence length 128 learning rate 1e-3 batch size 60 optimizer Adafactor Adafactor scale_parameter False Adafactor relative_step False Adafactor warmup_init False maximum training epochs 15 Table A8 Adafactor Parameters Parameter Value scale_parameter False relative_step False warmup_init False Table A9 Model performance on Domain and Time Split Model Samples Accuracy Macro F1 Macro Recall Macro Precision BERTiny 10 0.051 0.045 0.077 0.06 BERTiny 100 0.089 0.083 0.152 0.089 BERTiny 1000 0.117 0.079 0.123 0.119 BERTiny 10000 0.251 0.098 0.109 0.137 BERTiny 100000 0.348 0.208 0.185 0.317 BERTiny 1000000 0.333 0.213 0.196 0.279 BERTiny 2500000 0.348 0.232 0.208 0.313 BERTiny 5000000 0.357 0.241 0.221 0.304 BERTiny + LLM Labels (0.0 Ratio) 10000000 0.375 0.253 0.23 0.335 BERTiny + LLM Labels (0.25 Ratio) 10000000 0.457 0.34 0.296 0.456 BERTiny + LLM Labels (0.5 Ratio) 10000000 0.456 0.336 0.293 0.452 BERTiny + LLM Labels (0.75 Ratio) 10000000 0.449 0.323 0.279 0.442 BERTiny + LLM Labels (1.0 Ratio) 10000000 0.462 0.347 0.299 0.473 GPT-3 Babbage 10 0.267 0.235 0.362 0.214 GPT-3 Babbage 100 0.316 0.265 0.384 0.232 GPT-3 Babbage 1000 0.375 0.315 0.379 0.294 GPT-3 Babbage 10000 0.444 0.364 0.354 0.402 GPT-3 Babbage 100000 0.449 0.36 0.328 0.428 GPT-3 Babbage 1000000 0.444 0.364 0.33 0.465 T5 Large 10 0.211 0.254 0.284 0.324 T5 Large 100 0.273 0.277 0.374 0.282 T5 Large 1000 0.379 0.34 0.382 0.367 T5 Large 10000 0.463 0.368 0.332 0.477 T5 Large 100000 0.427 0.372 0.353 0.426 T5 Large 1000000 0.428 0.296 0.256 0.434 T5 Large 5000000 0.438 0.329 0.286 0.454 URLTran (BERT) 10 0.135 0.055 0.092 0.096 URLTran (BERT) 100 0.186 0.176 0.285 0.168 URLTran (BERT) 1000 0.304 0.261 0.308 0.261 URLTran (BERT) 10000 0.379 0.278 0.266 0.322 URLTran (BERT) 100000 0.383 0.246 0.224 0.325 URLTran (BERT) 1000000 0.358 0.264 0.254 0.333 URLTran (BERT) 5000000 0.377 0.288 0.278 0.357 URLTran + LLM Labels (0.0 Ratio) 10000000 0.416 0.317 0.307 0.364 URLTran + LLM Labels (0.25 Ratio) 10000000 0.465 0.357 0.326 0.446 URLTran + LLM Labels (0.5 Ratio) 10000000 0.465 0.364 0.338 0.443 URLTran + LLM Labels (0.75 Ratio) 10000000 0.466 0.364 0.329 0.453 URLTran + LLM Labels (1.0 Ratio) 10000000 0.468 0.375 0.338 0.496 eXpose (Conv) 10 0.033 0.023 0.05 0.049 eXpose (Conv) 100 0.062 0.052 0.107 0.062 eXpose (Conv) 1000 0.097 0.059 0.093 0.074 eXpose (Conv) 10000 0.223 0.051 0.063 0.105 eXpose (Conv) 100000 0.256 0.042 0.049 0.129 eXpose (Conv) 1000000 0.262 0.107 0.103 0.175 eXpose (Conv) 5000000 0.302 0.153 0.143 0.229 eXpose + LLM Labels (0.0 Ratio) 10000000 0.315 0.168 0.157 0.237 eXpose + LLM Labels (0.25 Ratio) 10000000 0.444 0.316 0.266 0.45 eXpose + LLM Labels (0.5 Ratio) 10000000 0.436 0.308 0.259 0.445 eXpose + LLM Labels (0.75 Ratio) 10000000 0.422 0.28 0.231 0.426 eXpose + LLM Labels (1.0 Ratio) 10000000 0.45 0.318 0.268 0.453 Table A10 Model performance on Time Split Model Samples Accuracy Macro F1 Macro Recall Macro Precision BERTiny 10 0.206 0.104 0.27 0.128 BERTiny 100 0.307 0.155 0.401 0.15 BERTiny 1000 0.355 0.135 0.301 0.176 BERTiny 10000 0.432 0.177 0.228 0.197 BERTiny 100000 0.912 0.713 0.722 0.714 BERTiny 1000000 0.967 0.78 0.765 0.802 BERTiny 2500000 0.973 0.821 0.804 0.844 BERTiny 5000000 0.976 0.856 0.838 0.882 BERTiny + LLM Labels (0.0 Ratio) 10000000 0.987 0.867 0.858 0.878 BERTiny + LLM Labels (0.25 Ratio) 10000000 0.971 0.835 0.814 0.86 BERTiny + LLM Labels (0.5 Ratio) 10000000 0.98 0.858 0.841 0.878 BERTiny + LLM Labels (0.75 Ratio) 10000000 0.982 0.877 0.859 0.902 BERTiny + LLM Labels (1.0 Ratio) 10000000 0.608 0.546 0.546 0.605 GPT-3 Babbage 10 0.429 0.304 0.593 0.269 GPT-3 Babbage 100 0.603 0.409 0.716 0.343 GPT-3 Babbage 1000 0.771 0.62 0.793 0.534 GPT-3 Babbage 10000 0.969 0.818 0.831 0.809 GPT-3 Babbage 100000 0.973 0.837 0.829 0.849 GPT-3 Babbage 1000000 0.984 0.855 0.839 0.874 T5 Large 10 0.394 0.35 0.497 0.344 T5 Large 100 0.475 0.375 0.61 0.328 T5 Large 1000 0.688 0.542 0.715 0.489 T5 Large 10000 0.927 0.794 0.796 0.8 T5 Large 100000 0.959 0.821 0.824 0.821 T5 Large 1000000 0.966 0.772 0.73 0.838 T5 Large 5000000 0.975 0.835 0.807 0.874 URLTran (BERT) 10 0.122 0.093 0.192 0.166 URLTran (BERT) 100 0.396 0.272 0.598 0.238 URLTran (BERT) 1000 0.636 0.508 0.773 0.432 URLTran (BERT) 10000 0.944 0.757 0.812 0.72 URLTran (BERT) 100000 0.947 0.778 0.791 0.774 URLTran (BERT) 1000000 0.971 0.817 0.806 0.836 URLTran (BERT) 5000000 0.956 0.82 0.802 0.873 URLTran + LLM Labels (0.0 Ratio) 10000000 0.992 0.914 0.897 0.94 URLTran + LLM Labels (0.25 Ratio) 10000000 0.985 0.859 0.851 0.869 URLTran + LLM Labels (0.5 Ratio) 10000000 0.988 0.889 0.884 0.896 URLTran + LLM Labels (0.75 Ratio) 10000000 0.991 0.908 0.896 0.927 URLTran + LLM Labels (1.0 Ratio) 10000000 0.74 0.676 0.672 0.706 eXpose (Conv) 10 0.125 0.075 0.165 0.116 eXpose (Conv) 100 0.296 0.168 0.362 0.164 eXpose (Conv) 1000 0.306 0.193 0.327 0.215 eXpose (Conv) 10000 0.565 0.18 0.204 0.26 eXpose (Conv) 100000 0.549 0.156 0.147 0.399 eXpose (Conv) 1000000 0.904 0.539 0.478 0.744 eXpose (Conv) 5000000 0.928 0.662 0.601 0.784 eXpose + LLM Labels (0.0 Ratio) 10000000 0.978 0.838 0.809 0.878 eXpose + LLM Labels (0.25 Ratio) 10000000 0.964 0.78 0.748 0.823 eXpose + LLM Labels (0.5 Ratio) 10000000 0.971 0.823 0.796 0.858 eXpose + LLM Labels (0.75 Ratio) 10000000 0.976 0.842 0.811 0.886 eXpose + LLM Labels (1.0 Ratio) 10000000 0.49 0.413 0.397 0.477