The rise of generative AI as a primary information source presents a paradigm shift from traditional web search. This paper presents a large-scale empirical study quantifying the fundamental diferences between the results returned by Google Search and leading generative AI services. We analyze multiple dimensions, demonstrating that AI-generated answers and web search results diverge significantly in their consulted source domains, the typology of these domains (e.g., earned media vs. owned, social), query intent, and the freshness of the information provided. We then investigate the role of LLM pre-training as a key factor shaping these diferences, analyzing how this intrinsic knowledge base interacts with and influences real-time web search when enabled. Our findings reveal the distinct mechanics of these two information ecosystems, leading to critical observations on the emergent ifeld of Answer Engine Optimization (AEO) and its contrast with traditional Search Engine Optimization (SEO).
The paradigm of information retrieval is undergoing its most significant transformation in decades. For years, users seeking information online have turned to search engines like Google, which act as gateways to a curated list of web documents. This process, governed by the principles of Search Engine Optimization (SEO), requires users to actively scan, evaluate, and synthesize information from multiple sources. The recent advent of generative AI-powered answer engines promises a radical alternative: delivering direct, synthesized answers to user queries.
While this shift ofers unprecedented convenience, it also raises critical questions about the underlying mechanics and biases of this new information ecosystem. How do the answers provided by generative AI services fundamentally difer from the results of a traditional web search? The answer lies not just in the presentation, but in the provenance, typology, and recency of the sources they consult, as well as the intrinsic knowledge acquired during their Large Language Model (LLM) pre-training phase. Pre-training on vast, static web corpora creates a latent knowledge base that may prioritize established information over fresh content and influence which external sources are deemed necessary to cite for a given query.
In this paper, we present a large-scale empirical study to dissect these diferences. We systematically
compare the outputs of GPT-4o, Claude 4.5 Sonnet, and Perplexity Sonar Pro against Google Search
and Gemini 2.5 Flash across two critical axes: querying for popular entities with abundant pre-training
data and niche entities with limited data. Our analysis focuses on (1) the domains consulted and their
classification; (2) the freshness of the cited URLs; and (3) the demonstrable efect of LLM pre-training on
the final answer. Through this multi-faceted comparison, we reveal the distinct logics of “web search”
and “AI search.” We conclude by contrasting the emergent strategies for Answer/Generative Engine
Optimization (AEO/GEO) [
Domain overlap. To quantify how AI-powered answer engines diverge from traditional web search,
we begin by measuring the domain-level overlap between their cited sources and those returned by
Google. This overlap provides a first-order indicator of whether the two paradigms draw from a shared
information ecosystem or surface distinct domain spaces. We evaluate 1,000 ranking-style queries (e.g.,
“Top 10 most reliable smartphones,” “Best reviewed airlines this season”), an important and growing
commercial use case for AI search [
Queries were generated using 100 fixed ranking-oriented templates (e.g., “Rank the best {topic} from 1 to 10”, “Experts’ ranking of the best {topic}”, “Best {topic} for most consumers”), each instantiated with the ten consumer topics, yielding 1,000 total queries. All systems were queried using identical prompts without personalization, and results were collected within the same time window to minimize temporal drift. For each query, we extract the top-10 URLs returned by Google Search and all cited URLs returned by each AI system, normalize them to their registrable domains, and compute the Jaccard overlap between each model’s domain set and Google’s top-10 results, then average the resulting values across all queries. Figure 1(a) summarizes the cross-system overlap.
Across systems, we observe uniformly low domain-level overlap with Google’s results, highlighting that generative engines surface markedly diferent domain ecosystems. GPT-4o shows the lowest mean overlap at 4.0% (std = 6.6%), followed by Gemini 2.5 Flash (11.1%, std = 10.2%), Claude 4.5 Sonnet (12.6%, std = 12.2%), and Perplexity Sonar Pro (15.2%, std = 11.6%). All pairwise diferences in mean overlap between systems are statistically significant under paired bootstrap resampling over the same query set (10,000 iterations; all < 0.001), confirming that these divergences are consistent across queries. GPT-4o’s substantially lower overlap indicates that its cited domains diverge the most from Google’s top-ranked domains. Notably, the median overlap for GPT-4o is 0.0%, indicating that for more than half of the queries, no domain overlap with Google’s top-10 results is observed; in contrast, median overlaps for Gemini 2.5 Flash, Claude 4.5 Sonnet, and Perplexity Sonar Pro are 8.5%, 8.7%, and 14.3%, respectively. Together, these results establish that AI and traditional search operate over distinct source landscapes, motivating a deeper examination of these diferences across query types and source characteristics. Efect of entity popularity. We compare 216 entity-comparison queries (108 popular, 108 niche) to examine domain overlap variations. Popular queries consist of comparisons between two globally recognized consumer brands (e.g., “Nike or Adidas: which is better? Answer with one brand name.”), typically asking for an overall preference between the two entities. In contrast, niche queries compare 1smartphones, athletic shoes, skin care, electric cars, streaming services, laptops, airlines, hotels, credit cards, and smartwatches. two less mainstream or more specialized brands within a specific use case (e.g., “Aeropress or Chemex: which is better for cofee? Answer with one brand name.”). All queries follow the same comparison frame (“A or B: which is better?”), with niche queries adding a task-specific qualifier when needed (e.g., “for X”). Both popularity categories span a range of consumer domains, such as consumer electronics and home appliances.
Domain overlap is computed using the same normalization and Jaccard protocol described in the Domain overlap analysis above. In addition to AI–Google overlap, we also compute pairwise domain overlap between each AI system and Gemini 2.5 Flash.
Figure 1(b) presents low overall overlap, but niche queries increase alignment by 3-4 percentage points for most models. GPT-4o shows minimal increase (1.3% to 1.9%) but maintains the lowest overlap overall, indicating distinct sourcing patterns. This increase is statistically significant for Claude 4.5 Sonnet, Gemini 2.5 Flash, and Perplexity Sonar Pro under bootstrap resampling over queries within the two popularity groups (10,000 iterations; all < 0.01), while GPT-4o exhibits a smaller and statistically non-significant increase.
The niche query overlap increase reflects narrower topical scope, concentrating sources to specific review sites and discussion threads. This is supported by declining unique-domain ratios (74.2% to 68.6%) and slight cross-model overlap increases (+1.1%). AI engines thus rely on concentrated, shared sources for niche entities rather than converging toward Google’s ranking logic. Similar observations hold when comparing to the results of Gemini 2.5 Flash.
Beyond domain overlap, we analyze source types: brand (oficial or company-owned sites), earned (independent media and review outlets), and social (user-generated or community platforms). We categorize sources from 300 consumer-electronics queries evenly distributed across three intent categories (100 informational, 100 consideration, 100 transactional). Informational queries are knowledge-seeking (e.g., “How do OLED TVs work?”), consideration queries reflect comparative evaluation (e.g., “Best budget noise-canceling headphones under $200”), and transactional queries are purchase-oriented (e.g., “Buy Apple AirPods Pro 2 near me”). Unlike the ranking-style queries in Section 2.1, these queries were designed to reflect realistic search formulations within a single topical domain rather than following a ifxed template.
For each cited source, we classify its source type using GPT-4o (temperature = 0) under a standardized labeling prompt restricted to the three categories: Brand, Earned, and Social. Brand refers to oficial company-owned domains (e.g., apple.com), Earned to independent media and review outlets (e.g., forbes.com), and Social to community or user-generated platforms (e.g., reddit.com). In addition to model-based labeling, links from predefined social media platforms are automatically assigned to the Social category to ensure consistency. A manual spot-check of a random subset confirmed high agreement with the automated labels. Figure 2 shows source type distributions by intent and system. Aggregate source composition. Google shows balanced sourcing (41% earned, 34% social, 26% brand), while AI engines favor earned over social content. Claude 4.5 Sonnet concentrates most heavily (65% earned, 1% social), followed by GPT-4o (57% earned, 8% social). Perplexity Sonar Pro (50% earned, 39% brand) and Gemini 2.5 Flash (46% each earned/brand) are more balanced. Notably, Claude 4.5 Sonnet initially returned no links for most informational and transactional queries without explicit search prompting, despite being queried in web-enabled mode.
Variation by user intent. Source composition shifts systematically by intent. For informational queries, AI models vary widely in earned vs. brand emphasis, while Google maintains balance with more social content. In consideration queries, AI engines converge toward earned dominance (59-86%), contrasting Google’s social-focused approach (41%). For transactional intent, all AI systems sharply increase brand citations (52-68%) versus Google’s mixed approach. Interpretation. Together, these results indicate that while overlap analyses demonstrate how AI and traditional search diverge, source typology helps explain why: generative engines systematically privilege earned and brand-owned content while under-representing social and community perspectives, and their source composition varies far more sharply across intents than Google’s relatively stable profile. This intent-adaptive sourcing behavior highlights a fundamental diference in relevance logic between generative and traditional search. The next section examines whether these typological diferences extend to when sources are cited, that is, their temporal freshness.
We compare the temporal freshness of sources returned by three answer engines, Claude 4.5 Sonnet, GPT-4o, and Perplexity Sonar Pro, against Google across two high-interest verticals: consumer electronics and automotive. For each vertical, we issue a fixed set of 100 curated ranking-style queries and collect up to 10 URLs per query and engine. For Google, these are the top- ranked results returned by the search API; for answer engines, these are the URLs explicitly cited in the response.
We canonicalize URLs (strip fragments and normalize redirects when available) and deduplicate within each (engine, vertical) before computing summary statistics. We then extract a page date from HTML metadata and structured fields, including <meta> tags, Schema.org JSON-LD (e.g., datePublished, dateModified), <time> tags, and date strings in the visible body text. When multiple candidates are present, we select a single best estimate by preferring explicit publication-time signals over modificationtime signals; if no usable date can be extracted, the URL is marked undated. We compute article age as the diference (in days) between the crawl timestamp and the selected date.
We report (i) extraction coverage, defined as the fraction of collected URLs for which a date could be extracted, and (ii) the distribution of ages over dated URLs (Figure 3). Because age distributions are heavy-tailed, Figure 3 clips ages at 365 days for readability; all reported summary statistics use the unclipped ages. To compare engines when coverage difers, we also report a coverage-adjusted freshness score adj = × coverage, where = 1 ∑︁
1 =1 1 + age (1) is computed over dated URLs only.
Claude 4.5 Sonnet achieves the highest date extraction coverage (0.925; 745/805 dated URLs) and returns the freshest median content at 62.3 days. GPT-4o has comparable coverage (0.930; 623/670) with a median age of 79.8 days. Perplexity Sonar Pro dates fewer sources (0.630; 383/608) and has an older median age of 90.4 days. Google’s median age is 130.4 days with coverage 0.615 (579/941). Under the coverage-adjusted freshness score, GPT-4o ranks first, narrowly ahead of Claude 4.5 Sonnet, with Perplexity Sonar Pro third.
Automotive exhibits a stronger long tail of older sources across all systems. Claude 4.5 Sonnet returns the freshest median content at 148.0 days (coverage 0.609; 515/845). GPT-4o follows with a median of 162.2 days (coverage 0.734; 477/650). Perplexity Sonar Pro both dates fewer sources (0.426; 280/657) and returns older median content at 216.6 days. Google is substantially older, with a median age of 492.9 days and coverage 0.443 (413/932). Under the same coverage-adjusted freshness score, Claude 4.5 Sonnet ranks first, followed by GPT-4o and Perplexity Sonar Pro.
Across both verticals, the answer engines return newer cited material than Google on the median, with the gap widening in automotive (148–217 days vs. 493 days). Coverage varies materially by engine and vertical, which matters because freshness estimates are computed over dated URLs only. We therefore report age distributions in Figure 3, coverage in Figure 4(a), and median-age summaries in Figure 4(b), and include both raw age summaries and a coverage-adjusted freshness score when making cross-engine comparisons.
Large language models (LLMs) often rely on a mixture of retrieved evidence and pre-existing world knowledge when generating factual responses. To disentangle these two influences, we examine how pre-training bias afects both the stability of generated rankings and the model’s use of citations under (a) Date-extraction coverage (dated URLs / collected (b) Median article age (days) with 95% bootstrap confiURLs) by engine and vertical dence intervals computed over the dated URLs for each engine and vertical diferent prompting and grounding conditions. Unlike Section 2, which compares several systems, this section is a single-model case study using GPT-4o so that the efects of prompting and evidence perturbations can be isolated cleanly.
To investigate how pre-training bias interacts with retrieved evidence, we designed a controlled experimental pipeline that exposes the model to progressively perturbed inputs. We focus on ranking-oriented queries (e.g., “best SUVs to buy in 2025”) because they provide an interpretable testbed for reasoning consistency across entities. All experiments were executed using the same large language model conifguration ( gpt-4o) under deterministic settings to minimize stochastic variance. The retrieved web snippets served as the evidence corpus for each query.
Given a query , we first call gpt-4o-search-preview with web search enabled and a JSON-only prompt that returns a ranked "list" of candidate entities and a "snippets" array of verbatim excerpts with source URLs. This yields the evidence set = {( , )}=1. We then pass and to gpt-4o with a ranking prompt that asks the model to produce the baseline ranked list of entities. In this baseline setting, the model has access to the retrieved snippets but is not explicitly forbidden from using prior knowledge. We refer to this condition as Normal Grounding.
To examine contextual robustness, we repeatedly modify the evidence or its presentation: 1. Snippet Shufle (SS) randomizes the order of snippets in and re-ranks. This tests whether the order of the search snippets, as influenced by generic web search ranking, afects the model’s ifnal decision. 2. Strict Grounding adds an instruction that restricts reasoning to the provided snippets only and prohibits the use of prior knowledge. This aims to dampen the impact of pre-training knowledge. 3. Entity-Swap Injection (ESI) randomly chooses two entities (, ) and swaps every mention of their names across all snippets before re-ranking. This tests whether the provided context directly influences the model’s decision. If the model relies on the provided context, swapping entity mentions should afect the entity ranking.
Each perturbation yields a new ranking . We compute the mean absolute rank deviation as Δ = 1 ∑︁ |rank () − rank ()| || ∈ (2) and report the average over all runs (10 runs per condition), Δavg = 1 ∑︀ Δ.
To measure the alignment between direct and decomposed reasoning, we derive an alternate ranking
′ through exhaustive pairwise judgments. For each entity pair (, ), the model is asked: “Between
and , which is better for this query given the same documents?” Each entity’s final score equals the
number of pairwise wins. We then compute Kendall’s (, ′) to quantify the correlation between the
holistic and pairwise-derived rankings, using ′ as a proxy ranking [
To measure how pre-training knowledge afects ranking behavior, we use a diverse set of ranking queries over entities. We group queries into two categories: Popular Entities and Niche Entities. Popular Entities are widely known, so we expect substantial exposure during pre-training (e.g., “top 10 SUVs for a family”). Niche Entities are less likely to appear in pre-training data (e.g., “top 10 family law firms in Toronto”).
For each query category, we run trials. In trial , we sample a query , retrieve an evidence set , and generate a baseline ranking base using model . For each perturbation type, we modify the evidence or prompting conditions to produce a perturbed ranking ̃︀. We compute the trial-level sensitivity relative to the baseline and average over trials to obtain the aggregate sensitivity Δavg.
To assess ranking consistency, we also derive a pairwise ranking ′ by summing per-entity wins from pairwise comparisons. We then compute consistency as Kendall’s (, ′).
For queries involving widely recognized entities (e.g., “best SUVs to buy in 2025”), we expect the model’s rankings to be largely governed by its pre-trained knowledge rather than by the retrieved evidence. We pose hundreds of such queries, conduct perturbations, and report our findings below.
In the snippet shufle experiment, in which we randomly shufle the order of retrieved snippets, the model’s ranking remains highly stable despite random reordering. Normal Grounding refers to the case where we ask the model to produce a ranking without imposing an evidence-only restriction. In this case, both pre-training knowledge and the retrieved snippets in the context window are available to the model.
As shown in Table 1, popular-entity rankings are only mildly afected by snippet order or entity substitutions. The overall ranking order remains consistent, indicating that the internal entity hierarchy cultivated during pre-training is relatively invariant to lexical and contextual perturbations.
In the pairwise comparison experiments, the pairwise-derived ranking ′ was nearly identical to the one-shot direct ranking for popular entities.
For example, for the query “most reliable electric cars in 2025”, the correlation is near perfect in the popular setting (Table 2). This indicates that the model’s underlying conceptual representation of entities is already stable and internally consistent, suggesting that its decision-making relies heavily on pre-trained priors. Log analysis across hundreds of related queries shows that, on average, 16% of entities appearing in the generated rankings did not occur in any retrieved snippet. As shown in Table 3, mainstream entities such as Toyota and Honda were almost always supported by evidence, whereas others (e.g., Cadillac, Infiniti) frequently appeared without being present in citations. This reinforces that for well-known entities, the model supplements retrieval with stored priors. Interpretation. For popular entities and other high-coverage domains, the model uses retrieved evidence primarily to reinforce pre-existing representations rather than to acquire new information. The retrieval context functions as confirmation, not discovery. As a result, even aggressive manipulations such as snippet reordering, entity swaps, or strict grounding constraints produce only minor deviations in the final ranking, underscoring the dominance of pre-training bias in answer generation for familiar entities.
For queries involving less-established or domain-specific entities (e.g., law: “top lawyers in Toronto that specialize in family law”), the model’s reliance on retrieval evidence becomes substantially stronger. Unlike popular categories, we expect these queries to fall outside the model’s well-formed pre-training priors, leading to observable diferences in reasoning behavior and ranking dynamics.
In the snippet shufle experiment, the model’s ranking exhibits pronounced sensitivity to the order of evidence when applied to niche or less-popular entities, as summarized in Table 1. This large disparity demonstrates that unconstrained reasoning is highly susceptible to contextual presentation, while strict evidence-only grounding markedly stabilizes outcomes. In the same setting, ESI produces even larger movement for niche entities, confirming that for low-coverage queries the induced ranking is directly driven by the literal content of the retrieved snippets rather than by pre-training priors.
In the pairwise comparison experiments, the alignment between one-shot ranking and the pairwisederived ranking ′ drops sharply for niche entities (Table 2). For example, for a family-law query law, we observe the reduced Kendall correlation reported in the table, reflecting a pronounced inconsistency between one-shot ranking and pairwise reasoning. Unlike the popular-entity case, the model lacks a stable internal hierarchy; its judgments fluctuate as it reassesses entity relationships on a per-comparison basis.
Interpretation. For niche or low-coverage subjects, the model enters a knowledge-seeking mode, relying heavily on provided snippets to compensate for missing or uncertain priors. Retrieved evidence exerts a direct influence on the final ranking, and grounding constraints materially alter outcomes. This behavior suggests that retrieval is essential not merely for citation reinforcement but for the construction of knowledge itself when pre-training coverage is sparse.
As user engagement increasingly shifts toward AI-powered search platforms, understanding the underlying ranking mechanisms of these services becomes essential. Traditional web search spawned the Search Engine Optimization (SEO) industry, dedicated to optimizing content for superior organic ranking. Our results suggest that for AI search, once a document is included within the model’s context window, a factor that upstream retrieval can influence, its absolute position within that context may be less critical for certain query types. This underscores the importance of understanding principles for Answer and Generative Engine Optimization (AEO/GEO).
Content freshness emerges as a particularly important ranking factor in AI search ecosystems. Furthermore, current AI search behavior indicates that specific source types, particularly earned and owned media, contribute more strongly to search presence than others. The efects of model pretraining also prove important for certain queries, making it critical to understand when new content can materially impact diferent query categories. Consequently, developing analytical strategies that dissect query patterns to generate actionable content creation and placement plans will become increasingly vital for optimization success.
The author(s) did not use Generative AI tools to write or edit this paper.