Given the vast scale of the Web, crawling prioritisation techniques based on graph traversal, popularity, link analysis, and textual content are frequently applied to surface documents that are most likely to be valuable. While these techniques have proven efective for keyword-based search, retrieval methods and user search behaviours are shifting from keyword-based matching to natural language semantic matching. Semantic matching and quality signals have been applied during ranking with great success, and recently, researchers have proposed to exploit them also to prioritise the frontier of Web crawlers. To investigate more on this, we propose two novel neural policies with the goal of surfacing content that is semantically rich and valuable for modern search needs, ultimately aligning the crawler behaviour with the recent shift towards natural language search. Our experiments on the English subset of ClueWeb22-B and the MS MARCO Web Search and Researchy Questions query sets show that, compared to existing crawling techniques, neural crawling policies significantly improve harvest rate during the early stages of crawling.
The efectiveness of search engines heavily depends on the indexed corpus: if the corpus is incomplete
or filled with low-quality pages, search results could be irrelevant [
Although all these prioritisation policies work well for keyword-based queries, they either (i) ignore
the textual content of pages, (ii) use it for keyword matching w.r.t. topic keywords, or (iii) use it for
query-driven focused-crawling, relying on relevance signals based on term frequency and inverse
document frequency, ignoring the semantics of texts. Recent advancements in contextualised Large
Language Models (LLMs), have shifted search on the Web toward conversational and complex,
questionbased queries rather than simple keywords [
In this work, we argue that crawlers should also adapt to this shift. We hypothesise that, by using
LLMs to estimate quality during crawling, we can improve the ability of crawlers to surface documents
valuable for search tasks, particularly for natural language search. Building on the approach by Pezzuti
et al. [
Our experiments on the English subset of ClueWeb22-B [
Let denote a Web page, and denote the set of outlinks from , i.e., the pages that links to. Let ℱ denote the frontier, which stores the URLs of the pages yet to be crawled. The frontier is composed of (, ) pairs, where is a URL and is its priority. A crawling policy is composed by: (i) a priorityassignment function : ↦→ , typically based on heuristics such as PageRank, (ii) an update policy, that defines how priorities are updated when discovering a new link to a page whose URL is already in the frontier, and (iii) a selection policy, that decides which page to crawl next, according to its priority. For instance, the well-known Breadth-First Search (BFS) crawling policy uses a constant priority-assignment function, a First-In-First-Out selection policy, and an update policy that does not change priorities upon rediscovery. In contrast, the Best-First (BF) policy employs a priority assignment function based on a link-based quality estimation heuristic such as PageRank, a maximum priority selection policy, and advanced update policies. Inspired by the link-based nature of PageRank, we propose two neural BF crawling policies leveraging an LLM-based quality estimation heuristic to prioritise Web pages with high semantic quality during the crawling process, called QFirst and QMin.
At the core of our proposed neural BF crawling policies is the use of a LLM-based heuristic function
: ↦→ R, parametrised by and optimised to distinguish high semantic quality pages from
lowquality ones. In particular, we aim at exploiting this neural heuristic in the priority-estimation function,
ideally using : ↦→ () to prioritise a page whose URL is . However, in real-world crawling
settings we cannot access the textual content of a Web page before its download. Indeed, using the ideal
semantic quality as the priority when enqueueing pages into ℱ is only feasible in a theoretical scenario
where an oracle function has access to a page text prior its download. We refer to this oracle-based
crawling policy as QOracle [
In the absence of an oracle, we can reasonably assume that the quality of a page is related to the quality of the pages it is connected to. Indeed, existing literature shows that the quality of a page is positively correlated with that of its linking neighbours. If this relationship holds, we can efectively propagate quality via link structure by using as a proxy estimate of the quality of a page that of one of its ancestors, i.e., the page that linked to it.
In our first neural crawling policy, referred to as QFirst, when processing a page and encountering the outgoing URL ˜ of a new page ˜ ∈ for the first time, we insert ˜ into ℱ with priority ˜ = () and we never update it.
In our second neural crawling policy, referred to as QMin, we additionally assume that if a page is linked to a low-quality page, it is highly unlikely to be of high-quality. If this holds, we could postpone the crawling of low-quality pages by decreasing their priority whenever a link from a low-quality ancestor is discovered. In doing so, we aim to boost the prioritisation high-quality Web pages while deprioritising low-quality ones. To implement the QMin policy, when processing a page and reencountering an already enqueued URL ˜ ∈ ℱ of a page ˜ ∈ , we update its priority as the minimum between the current priority and the quality of the new ancestor , i.e., ˜ ← min {˜, ()}.
We do not to propose a QMax policy, as our preliminary experiments, consistent with prior
research [
We perform a single-threaded simulation of the crawling process on the English subset of ClueWeb22-B
(CW22B-eng), which contains 87 head Web pages from ClueWeb22 (CW22) [
Queries. We use MS MARCO Web Search (MSM-WS) [
Neural Quality Estimation. In our experiments, we use the QT5-Small2 neural quality estimator [
Efectiveness. There exist several approaches to compute the efectiveness of crawling policies [
(, ) = |ℛ|/, where ℛ is the subset of relevant pages crawled up to time . As noted before, the page crawl time is our time unit, and corresponds to the crawling of pages. This metric measures the crawl ability to maximise the number of crawled relevant pages while minimising that of irrelevant ones.
Baseline. We compare our policies against BFS, the simplest yet efective policy. We do not compare
with PageRank since prior research showed that on small graphs PageRank is not accurate and BFS is
stronger [
3.00 R H 2.00 0
To investigate if using neural policies during crawling helps a crawler find relevant pages earlier than BFS, Figure 1 shows the HR over time for our oracle neural policy QOracle, our two practical neural policies QFirst and QMin, and BFS, on both the MSM-WS and RQ datasets.
From the figure, we note that the QOracle policy initially exhibits superior performance w.r.t. all other policies in surfacing pages relevant to keyword queries in MSM-WS, and is immediately followed by the QFirst policy. On natural language queries, all our neural crawlers substantially outperform BFS, and the QMin policy attains almost the same HR as QOracle.
The fact that QMin exhibits comparable performance to QOracle on RQ, while being unexpectedly on par with BFS on MSM-WS, suggests that most pages valuable for natural language search can be easily reached by postponing the exploration of low-quality links, while some of the pages valuable for keyword-oriented search may only be reachable through low-quality links. Thus, reluctance in following these links may hamper the discovery of valuable Web pages located deeper in the Web graph.
Meanwhile, QFirst, our simplest policy, significantly outperforms BFS on both query sets, and achieves competitive performance w.r.t. the other methods without introducing excessive overhead. Unlike QOracle and QMin, which rely on a greedier prioritisation and favour exploitation, QFirst is more exploration-oriented as it relies on noisier estimates. As a result, it has higher chances of discovering valuable pages only reachable throughout local minima.
To conclude, our experiments show that, w.r.t. to BFS, our QMin policy can significantly improve crawling efectiveness for natural language queries from RQ by up to +149% in early HR, while remaining competitive for keyword queries from MSM-WS, with up to +20% in early HR.
In this paper, we proposed two neural policies for neural crawling, both leveraging neural quality estimators to prioritise the early crawl of semantically high-quality pages.
We compared our proposed policies with an oracle policy, and with the well-established BFS baseline in terms of crawling efectiveness. Our findings reveal that, especially for natural language queries, we can markedly improve the early efectiveness of the crawler by using a neural policy in place of a traditional one. While our results show the promise of our approach, we recognise several limitations of this work that open up meaningful directions for future research. First, our experiments were conducted in a controlled, simulated setting; the efectiveness of our approach has not been validated in real-world, multi-threaded environments, which are typically subject to practical constraints like politeness policies, host reachability issues, and others. Second, further investigation is needed to better understand the potential biases introduced by neural quality estimators, particularly in terms of fairness and transparency. Third, we our proposed policies may be vulnerable to adversarial manipulation, and their robustness to such attacks should has yet to be explored.
We leave for future work experiments on other Web corpora and query sets, as well as experiments with other policies and other baseline comparisons.
Jennifer Neville, and Nikhil Rao. Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents. arXiv:2402.17896, 2024. [21] Artsiom Sauchuk, James Thorne, Alon Y. Halevy, Nicola Tonellotto, and Fabrizio Silvestri. On the
Role of Relevance in Natural Language Processing Tasks. In Proc. SIGIR, pages 1785–1789, 2022. [22] Khan Tajmir, Rashid Umer, and Rehman Abdur. End-to-end pseudo relevance feedback based vertical web search queries recommendation. Multimedia Tools and Applications, 83(31):75995– 76033, 2024. [23] Johanne R. Trippas, Sara Fahad Dawood Al Lawati, Joel Mackenzie, and Luke Gallagher. What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild. In Proc. SIGIR, pages 2703–2707, 2024. [24] Lalit Kumar Tyagi, Anish Gupta, and Vibhash Singh Sisodia. A New Era of Web Mining: Innovative Approaches in Focused Web Crawling for Domain-Specific Information. In Proc. ICTACS, pages 1–6, 2023. [25] Ryen W. White. Advancing the Search Frontier with AI Agents. Commun. ACM, 67(9):54–65, 2024. [26] Yutong Xie, Zhaoying Pan, Jinge Ma, Luo Jie, and Qiaozhu Mei. A Prompt Log Analysis of Text-to-Image Generation Systems. In Proc. WWW, pages 3892–3902, 2023.