In the recent years, the publication of structured data inside HTML content of Web sites has become a mainstream feature of commercial Web sites. In particular, e-commerce sites have started to add RDFa or Microdata markup based on schema.org and GoodRelations vocabularies. For many potential usages of this huge body of data, we need to crawl the sites and extract the data from the markup. Unfortunately, a lot of markup can be found in very deep branches of the sites, namely in the product detail pages. Such pages are difficult to crawl because of their sheer number and because they often lack links pointing to them. In this paper, we conduct a small-sized experiment where we compare the Web pages from a popular Web crawler, Common Crawl, with the URLs in sitemap files of respective Web sites. We show that Common Crawl fails to detect most of the product detail pages that hold a majority of the data, and that an approach as simple as a sitemap crawl yields much more product pages. Based on our insights gained in this paper, we conclude that a rethinking of state-of-the-art crawling strategies is necessary in order to cater for e-commerce scenarios.
Nowadays, evermore vendors are offering and selling their products on the Web. The spectrum of online vendors ranges from a few very large retailers like Amazon and BestBuy featuring an ample amount of goods to many small Web shops with reasonable assortments. Consequently, the amount of product information available online is constantly growing.
While product descriptions were mostly unstructured in the past, the
situation has meanwhile changed. In the last few years, numerous Web shops have
increasingly started to expose product offers using structured data markup
embedded as RDFa, Microdata, or microformats in HTML pages (cf. [
Unfortunately, for regular data consumers other than search engines like Google, Bing, Yahoo!, or Yandex2, it is difficult to make use of the wealth of product information on the Web. Web crawling is an open problem for it is known to be expensive due to the sheer size of the Web, and because it requires deep technical expertise. Hence, motivated by these challenges and the idea to provide easy and open access to periodic snapshots of industrial-strength Web crawls, the Common Crawl project was launched in 2007.
The Common Crawl corpus with its hundreds of terabytes’ worth of gathered
data constitutes an incredibly valuable resource for research experiments and
application development. At almost no cost3, it can offer unprecedented insights
into several aspects of the Web. Among others, researchers found it to be useful
to analyze the graph structure of the Web [
One popular and persistent misconception about Common Crawl, however,
is to think that it is truly representative for the Web as a whole. The FAQ
section on commoncrawl.org is further contributing to this fallacy, as it literally
terms Common Crawl as “a copy of the web” [
The main motivation for this paper was to answer the question whether Common Crawl is complete enough for use cases around structured e-commerce 2 Shop owners often proactively direct popular search engines to their Web pages. 3 The only costs that incur are those for storage and processing of the data. 4 http://www:slideshare:net/davelester/introduction-to-common-crawl 5 http://lists:w3:org/Archives/Public/public-vocabs/2012Mar/0095:html and http://lists:w3:org/Archives/Public/public-vocabs/2012Apr/0016:html
1 2 3 4 5
1 2 3 4 5
data. For e-commerce applications e.g., it is crucial to be able to rely on a solid and comprehensive product information database, which we think Common Crawl cannot offer to date. Even if it might not be very surprising that Common Crawl misses out a considerable amount of product data in commercial Web sites, it remains so far unclear by how much.
In this paper, we investigate the representativeness of Common Crawl with respect to e-commerce on the Web. For this purpose, we rely on a random sample of product-related Web sites from Common Crawl, for which we measure the coverage of (1) Web pages in general, and (2) Web pages with structured product offer data. We further analyze the crawl depth of Common Crawl. As our baseline, we use sitemaps to approximate the actual URL distribution of Web sites, as depicted in Fig. 1.
The rest of this paper is structured as follows: Section 2 reviews important related work; in Sect. 3, we outline our method, the relevant data sources, and design decisions that we made; Section 4 reports the results of the analysis of Common Crawl; in Sect. 5, we discuss our findings and emergent shortcomings; and finally, Sect. 6 concludes our paper. 2
Considerable literature work deals with crawling structured data from the Web.
In contrast to conventional Web crawling of textual content, structured data
poses special challenges on the organization and performance of crawling and
indexing tasks [
Surprisingly, none of the aforementioned approaches address the question
where the structured data actually resides in a Web site, albeit this
information can be useful for tuning the crawling process. [
In this paper, we concern with the problem of crawling e-commerce data from the Web, and in particular we focus on caveats of using Common Crawl. This section describes the data sources and the methodology for the inspection of the Common Crawl relative to e-commerce. 3.1
Datasets and APIs
Each of the Common Crawl corpora represents a huge body of crawled Web
pages which to process and analyze in detail is not a straightforward task. The
December 2014 Common Crawl corpus has a compressed file size of 160 terabytes
and is reported to contain 15:7 million Web sites and a total of over two billion
Web pages [
Web Data Commons. The Web Data Commons dataset series contains all
structured data extracted from the various Common Crawl corpora [
Common Crawl Index Query API. In order to access the huge Common Crawl corpora, it is no longer necessary to get hold of the entire datasets. In 2015, the Common Crawl Index was announced, which offers an index query API to look up entries by URL (possibly with wildcards). The Common Crawl Index for the December 2014 crawl snapshot is available at http://index:com moncrawl:org/CC-MAIN-2014-52. For the Python programming language, there further exists an API client to conveniently access the Common Crawl indices7. Sitemaps. In addition to the data sources related to Common Crawl, we obtained Web page URLs and corresponding metadata from the XML sitemaps of selected Web sites. 6 http://webdatacommons:org/structureddata/2014-12/stats/schema_org_subse ts:html 7 https://github:com/ikreymer/cdx-index-client
Method In any e-commerce Web site, the most distinctive entity type is the product offer, which describes the commercial details and terms and conditions related to the transfer of property rights for a product item. Product offers are represented in schema.org by the class s:Offer. Within the scope of this research, we used the respective class-specific subset of Web Data Commons that was extracted from the December 2014 snapshot of Common Crawl.
We extracted all available domains from the Web Data Commons subset related to s:Offer. From the 109651 domains obtained this way, we drew a random sample of 100 Web sites. For each of these Web sites, we (a) captured the available URLs in the Common Crawl Index, and (b) fetched their XML sitemaps (if existing) to collect the contained URLs. In order to locate the sitemaps, we tested whether they are referenced from a robots.txt file or, alternatively, placed in the root directory of the Web server – which is regarded as the simplest and default method. Then we (recursively) iterated over these sitemaps (or sitemap indices) and extracted the Web page URLs.
After that, we matched the Web pages found in the Web Data Commons subset to the URLs found in the respective sitemaps that contain structured product offer data. For this purpose, we conducted a deep crawl based on the URL collection in the sitemaps. We limited the number of URLs to 10000 per sitemap, otherwise the crawling process would have been overly resource-intensive. 3.3
Design Decisions To find out how deeply the Common Crawl spider is reaching within Web site structures, we rely on a basic property of URLs, which is that URL paths are organized hierarchically [14, Section 1.2.3]. For all URLs in both datasets, we thus analyzed their URL structures. In particular, we classified them by matching their URL paths against the following string patterns lev0: /{0} lev1: /{0}/{1} lev2: /{0}/{1}/{2} ...
The placeholders (indicated by numeri.c values between curly braces) describe character sequences of arbitrary length. The sequence may contain any character but the slash symbol. In order to pertain to a specific category, the patterns must be matched exactly. The URL scheme together with the authority (often the host) constitute level 0 URLs (labeled as category “lev0”) because the URL path is empty, e.g. http://www:example:org/. Other URLs with a path in the root directory also belong to level 0, e.g. http://www:example:org/index:html or http://www:example:org/index, whereas http://www:example:org/index/ would be classified as a level 1 (“lev1”) URL.
Prior to the analysis, the data needed to go through some pre-processing steps. In particular, we dropped duplicate URLs and then, for the purpose of generating statistics about the URL depth, we truncated query strings from URLs. The execution order of these two steps was essential, because otherwise we would have inadvertently discarded Web pages that only differ based on query string parameters. The reason we abstract from URLs with query string parameters is because we are only interested in the hierarchical characteristic of URL identifiers. This way we prevent the false classification of edge cases like http://www:example:org/?path=/some/path/, which should be treated equivalent to http://www:example:org/.
The random sample of product domains that we obtained was initially not very useful. We noticed some unexpected difficulties with various domains, e.g. many Web sites are part of large hosting platforms that are divided into several subdomains (e.g. masstube.uptodown.com and www.uptodown.com). As we could observe, the sitemap files of these domains often point at the main Web site (i.e. the subdomain starting with “www”). We thus decided to only consider Web sites starting with “www”, by which we could reduce the problem considerably. In addition, it has the nice side-effect that possible www-subdomains are consolidated with their non-www-counterparts. 4
From the 100 domains that we have examined, 68 featured a sitemap (15 a sitemap index)8 according to the search criteria mentioned in the previous section. Out of these 68 remaining Web sites, we crawled all but seven domains that had no more than 10000 URLs. We then compared the numbers related to pages (URLs) with and without structured product offer data in the 61 domains with those from the Common Crawl corpus from December 2014. 4.1
Dataset Statistics 8 Download date as of June 23, 2015 9 We obtained the statistics using a string match “ schema.org/Offer” against the HTML content, which is a simple heuristic to match schema.org in Microdata. 01 www.abloomaboveflorist.com 02 www.acuderm.com 03 www.antik-zentrum-alling.de 04 www.askariel.com 05 www.bellyarmor.com . . . 21 www.hamiltonparkhotel.com . . . 57 www.therealthingonline.co.za 58 www.waterbedbargains.com 59 www.wedoboxes.co.uk 60 www.windberflorist.com 61 www.xpradlo.sk 9996 In accordance with the assumption from Fig. 1, we compared the URLs in the Common Crawl corpus to the collection of URLs in the 68 sitemaps at varying URL depth levels. To accomplish that, we relied on the hierarchical part of the Web page URLs, as described in Sect. 3.2. Table 2 contrasts the domainspecific median and mean values of Common Crawl and sitemaps for the first twelve URL hierarchy levels. The median in the Common Crawl decreases across different levels, whereas for sitemaps there is first an increase from level 0 to level 1. Although the median value is then suddenly decreasing, it is still higher at level 2 than at level 0. n i a om 10000 d r e p saeg l)ed 1000 froebp l-(scoag 100 m u n raeg 10 e v A
mean
In Fig. 3, we show a boxplot of the data that was reported in Table 2. The figure graphically represents the distribution of URLs across Web sites for the first five URL depth levels. 4.3
Coverage by URL Depth of Web Pages with Structured Product Offer Data In addition to measuring the coverage of URLs, we compared the Web pages that contain structured product offer data. The comparison was conducted among the 61 crawled sitemaps and the respective subset of Web Data Commons. Once again we did this comparison for the individual URL hierarchy levels. Table 3 outlines the median and mean values of URLs in sitemaps that contain structured
mean n i a m o rd 1000 e p saeg l)ed product offer data, namely entities of type s:Offer. In comparison to Table 2, we added a rightmost column that shows by how much the mean values differ across Common Crawl and the sitemaps. E.g., the mean value between Common Crawl and the sitemaps differs by a factor of close to 67 at level 0 (the root directory). For level 1 it is 27:55, for level 2 53:22, and so on. This extrapolation factor gives an estimate of the missing coverage of structured e-commerce data in Common Crawl. In other words, if we would solely rely on sitemaps, then we could collect x times as many Web pages with structured product offer data at a URL depth of y than found in the Common Crawl dataset.
In Fig. 4, we show two boxplots for the URL collections in Common Crawl and in the sitemaps. They contrast the numbers of Web pages with structured product offer data at five different URL depth levels. For the evaluation of our results, we analyzed by how much the coverage of URLs with structured product offer data deviates between the Common Crawl corpus and the URL collections from the sitemaps. More precisely, if we did encounter level 0 URLs in the Web Data Commons dataset, then we should also be able to find level 0 URLs within the sitemaps of the corresponding domains. Otherwise, the averages reported in the results would be meaningless due to originating from distinct domains. We detected a considerable overlap between Common Crawl and sitemaps across various URL depth levels. The results are outlined in Table 4. 5
In this section, we discuss the results of our analysis. Furthermore, we want to point at potential shortcomings of our approach.
Sitemaps of Web sites attain a better coverage of URLs than Common Crawl. A side-by-side comparison of the boxplots in Fig. 2 reveals differences between Common Crawl and sitemaps concerning (a) the average number of URLs in Web sites and (b) the average number of Web pages with product offers in Web sites. The two boxplots have a similar distribution, albeit the numbers differ significantly as the scale of the y-axis used for the sitemaps is 100 times larger than for Common Crawl. A comparison of the median values also indicates a larger share of Web pages with structured product offers in the sitemaps.
Common Crawl does not crawl as deep as the URL depth available in the sitemaps. Yet, Table 2 and Fig. 3 indicate that Common Crawl is well able to reach the deep pages within a Web site. However, in most cases it stops after the second URL depth level.
Against our expectations, a large amount of product offers is encountered at the upper three levels of the URL path rather than at the deeper hierarchy levels (see Fig. 4). The maximum URL depth in our datasets was twelve levels (see Table 2), although the deepest URL path with structured product offer data was eight (see Table 3). This might be due do different uses of hierarchical identifiers.
Interestingly, while the average number of structured product offers tends to decrease in the sitemaps, it stays relatively stable for the first seven levels in the Common Crawl, as shown in Table 3. This finding suggests that Common Crawl is able to enter Web sites at any URL depth levels, but instead of crawling additional URLs, it leaves Web sites prematurely – likely because of the prioritization strategy inherent to many Web crawlers.
In the course of this work, we have identified some possible limitations: – There does not only exist s:Offer for entity types, but also concepts from other vocabularies and formats for annotating product offers on the Web (gr:Offering or microformats). We decided to rely on schema.org because it is by far the most prevalent Web vocabulary. – Between the Common Crawl snapshot (December 2014) and the sitemap crawl (June 2015) was a time gap of half a year, which was unavoidable due to the lagged publication of the Web Data Commons extraction results. – Sitemaps are useful for crawling large Web sites very quickly, but they are not necessarily complete as they are often webmaster-crafted. However, we do not see a disadvantage in here, rather would it in the best case further support our arguments. – We compared the coverage of structured product offer data based on a sample of domains featuring sitemaps. Thus, our dataset could exhibit a potential bias towards higher-quality Web sites. This issue needs further investigation. – We assumed a hierarchical organization of URLs, despite some Web sites often use flat hierarchies for URL identifiers – or hierarchical identifiers in the absence of hierarchies, e.g. for tagging. Furthermore, the URL pattern used to classify URLs into various levels of URL depth is brittle because, depending on the server configuration, trailing slashes might be handled differently, e.g. http://www:example:org and http://www:example:org/. – Finally, given the population size of 109651 Web sites, our sample size of 100 domains was relatively small which could be enhanced in future work. 6
In this paper, we have sought evidence for that a popular Web crawler like Common Crawl fails to detect the product detail pages that hold a majority of the product data, which limits the usefulness of such a crawler for e-commerce.
For a random sample of 100 Web sites from Common Crawl with structured product offer data, we have detected that the majority of them (i.e., 68) offer sitemap files. We have shown that Common Crawl covers only a small fraction of the URLs available in these sitemaps. Moreover, the amount of structured product offer data was generally lower for the Web sites considered in this work. Yet, the URL depth for sitemaps was only marginally higher than the crawl depth of Common Crawl.
Through the insights gained in this paper, we conclude that we need other crawling strategies that reach beyond what Common Crawl is currently able to offer. For example, to increase the overall coverage, we could combine Common Crawl with a deep sitemap crawl. Still, there remains the problem that a crawler like Common Crawl might miss the relevant Web pages that hold the product information within Web sites. Our findings presented in this paper apply to ecommerce sites in general as well as to e-commerce sites with structured markup. Nonetheless, other application domains might draw useful conclusions from the results of our analysis.