We provide an analysis of the adoption of metadata standards on the Web based a large crawl of the Web. In particular, we look at what forms of syntax and vocabularies publishers are using to mark up data inside HTML pages. We also describe the process that we have followed and the di culties involved in web data extraction.
Embedding metadata inside HTML pages is one of the ways to publish structured data on the Web, often preferred by publishers and consumers over other methods of exposing structured data, such as publishing data feeds, SPARQL endpoints or RDF/XML documents. Publishers prefer this method due to the ease of implementation and maintenance: since most webpages are dynamically generated, adding markup simply requires extending the template that produces the pages. Consumers such as search engines are already accustomed to processing HTML and extraction ts naturally in their processing pipelines. The close coupling of the raw data and the HTML presentation of the data has other advantages, among others it makes sure that the the raw data and the end-user presentation show the same.
In this paper, we describe the method by which we extracted metadata from a large web corpus and present some statistics. Results from similar experiments have been already published, so we also discuss the di culty in comparing numbers across the various studies.
Previous studies have reported results on the usage of embedded metadata, including Bizer et al. at http://www. webdatacommons.org/. We also published an earlier analysis on a di erent corpus collected by Yahoo! Search 1. There 3.
We take as our starting point a su ciently large sample of the web crawl produced by Bing's web crawler during January, 2012. After retaining information resources with a content type that includes text=html, we get a data set of 3,230,928,609 records with only the three elds required for analysis, the URL of the page, the content type and the downloaded content. In case the crawler arrived to a page by following a (chain of) redirects, we considered the target of the redirect as the URL.
We perform our analysis in two steps. First, we use regular expression patterns to detect metadata in web pages. We use the same patterns proposed by Bizer et al., but we strengthen the pattern for detecting RDFa. In the form proposed by the authors it allows any page that contains about followed by whitespace and an equal sign;we limit this pattern to require that the equal sign be followed by whitespace and a single or double quote. We also introduce a new pattern to speci cally detect webpages using the Open Graph Protocol Second, identi ed by the word property followed by optional whitespace, single or double quote, optional whitespace and og:. For this analysis, we lter out pages larger than 3MB and where the character set can not be identied. The total number of URLs in the output is thus slightly lower than in the input.
Table 1 shows the prevalence of each format both in terms of URLs that use that format, and in terms of e ective toplevel domains (eTLD), sometimes called pay-level domains (PLD)2. For computing PLDs, we used the Guava library version 11.0.2. For a small number of URLs we failed to determine the PLD, e.g. because they contain an IP address instead of a domain name, but we believe this does not in uence the results signi cantly.
In a second step, we actually extract RDFa data from these pages using the Any23 library (version 0.7) as suggested by Bizer et al., and using the same set of extractor plugins. We use this library with the default con guration except for setting metadata nesting3 to o , because microformat extraction generates a substantial number of additional triples in the default setting. Before passing the content to Any23, we read the char set of the page from the content-type and recode the page content to UTF-8 (we exclude pages where the character set can not be identi ed). We also modify each input page that we expect to contain OGP markup to de ne the og pre x. Without this, much of OGP data would not be extracted by Any23's RDFa parser and there is also no speci c extractor for OGP data. To speed up the process of extraction, we exclude some extreme cases: webpages larger than 3 MB, pages , pages containing more than 200 VCard objects, and also pages where the result of the extraction exceeds 64 MB. We write the data in a quintet format: subject, predicate, object, context and the name of the extractor that produced that quad.
To read the data, we use the same NxParser library that we use to write the data. Unfortunately, there are invalid lines in the output that we are not able to read back (various exceptions reported by NxParser). Further, some input lines cause the parsing to enter an in nite loop. As a temporary measure until we nd the source of these bugs, we run the parser in a separate thread and terminate this thread after 500ms. We also limit the size of each input line to 5KB and do not even attempt to parse lines longer than that. Due to these problems, we loose some data: the output contains 671,454,122 URLs compared to 973,539,519 URLs that we would expect to contain some data based on regular expressions. In total, we extract 17,443,606,947 triples. Tables reftbl:topsites-rdfa and 3 and 4 show the top 10 sites as measured by the number of triples using RDFa, microdata, or hcard, respectively. The number of triples is an aggregate that re ects both the number of indexed pages in the crawl (a proxy for the importance of the domain) and the amount of data published per page. Again, we note that these lists are not exclusive. For example, youtube.com uses both microformats, microdata and RDFa within the same pages.
In terms of vocabulary usage, we show the most commonly used namespaces in RDFa data in Table 5. We also show the most frequently used classes in terms of the number of URLs and PLDs in Table 6 and Table 7, respectively. We omit the http protocol identi er, because all namespaces start with this protocol identi er, except for a facebook namespace that appears with both http and https. The rst table con rms that the vast majority of RDFa data on the Web is due to Facebook's OGP markup. Unfortunately, OGP does not always conform with the letter and intent of RDFa. For example, type information in OGP is given using the og:type predicate, and not the RDF built-in rdf:type predicate. This explains the di erence between Table 5 vs Table 6 and Ta2http://en.wikipedia.org/wiki/Public_Suffix_List 3any23.extraction.metadata.nesting Site facebook.com tabelog.com venere.com yahoo.com tripadvisor.co.uk tripadvisor.it tripadvisor.com tripadvisor.fr tripadvisor.jp tripadvisor.es tripadvisor.de answers.com myspace.com tripadvisor.in daodao.com tripadvisor.com.tw tripadvisor.ru imdb.com youtube.com bestbuy.com
Abs PLD 1,306,827 1,140,880
93,463 1,755,733 1,700,377 30,809,476 ble 7: most OGP data does not de ne instances of any RDF class. As already mentioned above, most users of OGP also ignore the declaration of the og pre x (a problem we deal with in the extraction) and we can also see a number of variations to the current standard namespace (a problem we have not dealt with). Further, OGP assigns additional meaning to the RDFa syntax that is not re ected in the RDFa standard. As an example, the order in which triples are written on the page matters in OGP, but not in RDFa. For all these reasons, we believe that Any23 should be extended with a speci c processor for OGP markup that is able to deal with these peculiarities.
Besides OGP, a smaller amount of data can be attributed to e orts by Google's Rich Snippet program and Yahoo's retired SearchMonkey program. Social markup in the form of FOAF and SIOC is also present in a large number of domains as shown in Table 7. The fact that these vocabularies do not show up as prominently in Table 6 means that they are used more in the less deeply crawled part of the web.
For microdata, we only list the top namespaces in Table 8 and Table 9, because Any23's microdata extractor incorporates the class name into the namespace. In microdata, only two vocabularies (schema.org and Google's datavocabulary.org ) have gained signi cant traction so far, and the latter is expected to be replaced by the former.
It holds for both RDFa and microdata that the types of objects that are marked up is biased by the use case of search engine optimization, i.e. site owners prefer to mark up data that is used by the search engines to enrich search result presentation (e.g reviews, business listings). Schemas for these types of objects have also existed longer. We also observe a natural preference to mark up simple types of objects (e.g. breadcrumbs), though we did not formally investigate the relationship between the complexity of markup and its adoption. 4.
We presented metadata statistics from the analysis of a large, recent sample of the Web, which has been extracted from the crawl of a search engine and therefore provides a search-engine centric view on the Web. Current web search engines are biased toward authoritative, head sites with valuable textual content, and are not speci cally looking for data on the Web. We expect that a search engine speci cally built for data would give less weight to authority and textual content and perform deeper crawling on sites that provide large and valuable data, by some measure of quantity and quality.
Nonetheless, our work shows an impressive progress in the adoption of markup on the Web with over 30% of our collection containing some microformat, RDFa or microdata markup. Microformats and RDFa are the most popular choices of syntax. The level of microformats usage seems to be at, while RDFa adoption has grown signi cantly compared to previous studies. This is due almost exclusively to OGP markup, though there is a variety of usage in the long tail, in particular social vocabularies. On the other hand, the adoption of microdata is driven so far only by the success of schema.org.
There is signi cant future work to be done in order to
evaluate the quality and practical usefulness of data
embedded in HTML, with respect to some existing or novel tasks.
In previous work, we have looked at the extent to which
embedded metadata could be used to enrich web search results
[
Namespace ogp.me/ns# www.facebook.com/2008/ www.w3.org/1999/02/22-rdf-syntax-ns# rdf.data-vocabulary.org/# purl.org/dc/terms/ https://www.facebook.com/2008/ mixi-platform.com/ns# ogp.me/ns/fb# creativecommons.org/ns# www.w3.org/2006/vcard/ns# http:// http://www.facebook.com/ http://www.w3.org/2000/01/rdf-schema# http://developers.facebook.com/schema/ http://search.yahoo.com/searchmonkey/commerce/ http://purl.org/dc/elements/1.1/ http://opengraphprotocol.org/schema/ http://search.yahoo.com/searchmonkey/media/ http://oexchange.org/spec/0.8/rel/ http://xmlns.com/foaf/0.1/
URLs 493,443,016 150,246,016 26,402,165 19,413,470 16,424,800 7,472,815 6,323,861 4,636,260 4,622,272 4,205,037 3,881,321 3,126,045 3,042,839 2,720,567 2,664,743 2,642,796 2,293,024 2,095,577 2,034,467 1,837,749