This paper presents the results of a study of the quality of archived websites using support tickets from the Internet Archive's Archive-It (AIT) service, currently the most widely used web archiving service. The study uses grounded theory to create a definition of quality for archived websites that is composed of three dimensions: correspondence, relevance, and archivability. The focus of this paper is on website archivability, which I redefine as the intrinsic properties of a website that make it easier or more dificult to archive. I argue that archivability is a latent construct of information quality expressed by the manifest variables of correspondence and relevance, a ifnding not previously seen in the literature. The definition is independent of the technology currently in use to create web archives, making it suitable to a wide variety of platforms, preservation contexts, and situations. Furthermore, the paper describes how low levels of website archivability influences the quality of web archives as historical records.
Since the 1990s, many cultural heritage institutions, such as museums, archives, and libraries, have
undertaken the task of creating a historical record of the web through the practice of web archiving.
Web archivists who preserve websites are concerned both with quantity and with quality. Their goal is
to create archived websites that are as close as possible in appearance and functionality to the original,
live website. Failing to adequately capture a website might mean a flawed or incomplete historical
record of it. The importance of creating high-quality web archives was echoed by the results of the 2016
survey on web archiving in the United States, conducted by the National Digital Stewardship Alliance
(NDSA). When asked what were their top concerns when developing a web archiving program at their
respective institutions, 52% of participants cited quality as a top issue. Quality was the third most-cited
concern for respondents, after cost and access and use (60% each) [
The most popular web archiving service is the Internet Archive’s Archive-It (AIT), which helps
organizations build and manage their own web archives. It currently has over 800 clients partners
consisting of universities, archives, museums, and libraries in over 24 countries [
Despite the popularity of the Archive-It and the Internet Archive, web archiving is a field with few
conceptual tools or theoretical definitions. In a previous work [
A web archive can be said to have high-quality if it has high correspondence, high relevance, and high archivability. Due to the heavy presence of the dimension of correspondence in the data set, I asserted that it is the most important facet of quality in web archive. This work builds on the theory of IQ for web archives advanced earlier, but turns its attention to archivability, another important dimension. The purpose of this paper is to elaborate on and deepen the original definition of archivability, place it in a human-centred context, and predict how it will afect the future of web archiving. This goal leads to the following research questions:
When deploying crawlers to capture a website, some crawl engineers pay special attention to embedded
resources. Embedded resources are files, such as images, videos, or CSS stylesheets, that are present
and referenced in a website. A user might not notice their presence, but embedded resources play a
key role in ensuring the website looks and operates in the correct way. To this end, crawl engineers
might calculate a percentage of missing embedded resources in an archived website and use it to
estimate the overall quality of the site. Brunelle, Kelly, SalahEldeen, Weigle, and Nelson [
In their iPres paper “CLEAR: A Credible Method to Evaluate Website Archivability”, [
Each of these facets has several components, or criteria, each with its own significance. Criteria with high significance are more important to the archivability of a website, and if they are not met, can cause problematic web archiving results or even prevent the website from being archived at all. Medium-significance criteria are not critical but are still important, while low-significance criteria are considered minor issues.
[
[
Other researchers have also focused on the notion of archivability and attempted to operationalize it.
In their paper “The impact of JavaScript on archivability”, [
However, obtaining a perfect copy of the original is an onerous process, made more dificult by the
widespread use of the JavaScript programming language. The use of JavaScript, in the form of small
pieces of code called scripts, has made websites more personalized and interactive. Unfortunately, it
has also made websites more dificult to archive. As the authors state, today’s archival tools, such as
the Heritrix web crawler employed by the Internet Archive, are unable to fully capture and render this
complexity [
A website that contains JavaScript, such as Google Maps, functions diferently from a traditional,
HTML-only website. Typically, a web browser requests a website from a server, then proceeds to load
the basic elements, such as HTML code and images. After the initial page is loaded, the JavaScript code
is executed, This code will then request additional components to be loaded onto the page, such as
the panning and zooming functions of an interactive map or geographic location features. [
To study the impact of JavaScript on archivability, the researchers compiled two sets of archived
URLs: some taken from the social media platform Twitter and others from the Internet Archive’s
Archive-It service [
= (| − |, | − |) = |ℎ| +
2 = ∑︁ ∈ (1) (2) (3) (4) − =
−
Unlike [
It is worthwhile to note that the research published by [
In 2016, [
[
In [
This paper focuses on the specific dimension of relevance in web archives, and was part of a larger
project to build a comprehensive theory of IQ in web archives. The methodology described in this
section is the same as the one employed in my previous work “Correspondence as the Primary Measure
of Quality for Web Archives: A Grounded Theory Study” [
Barney Glaser and Anselm Strauss created the methodology of Grounded Theory (GT), which they
defined as “the discovery of theory from data - systematically obtained and analysed in social research’
[
The Internet Archive’s Archive-It (AIT) is a subscription-based web archiving service that helps
organisations build and manage their own web archives. Archive-It is currently the most popular web
archiving service, with over 600 clients (called “partners’) consisting of universities, state libraries and
archives, museums, and national libraries in several countries [
AIT support tickets are a rich source of information regarding quality problems in web archives. They contain the opinions and views of individuals who are experienced creators of web archives, well-versed in web archiving processes, and familiar with institutional web preservation goals, whether they be clients or the partner specialists themselves. They contain rich descriptions of how quality problems are detected, analysed, and addressed, and are thus an ideal dataset for studying quality in all its dimensions.
The first step was to obtain Archive-It support tickets in order to analyse them. Since these tickets
belonged to the Internet Archive, I negotiated a researcher agreement with the organisation to obtain
support tickets from the years 2012 through 2016. The tickets received comprised a wide variety of
institutions reflecting AIT’s client base, from national libraries, to private organisations, to universities
and museums from Europe, North America, and Asia. After the tickets were cleaned, I randomly selected
the same amount of tickets for each year from 2013 through 2016. This randomisation approach was
taken to minimise the selection bias that might have occurred if I had manually chosen which tickets to
analyse. The final dataset of 645 tickets was then imported into the NVivo software package, a popular
program for performing qualitative data analysis [
Among other conditions, the research agreement stipulates that the researcher anonymise any personal or institutional information present in the tickets, as well as any other potentially identifying information. In order to comply with the terms of this agreement, all the information presented in this paper has been anonymised: identifying elements such as personal names, names of institutions, and website addresses have been removed or changed.
The tickets collected were Level 1 support tickets that had been submitted by AIT client. They included the initial question submitted by the client, the response given by the AIT partner specialist, and any subsequent communication between the two. It is important to note that not all the AIT tickets deal with issues of quality in a web archive. Quite a few deal with collection management issues, such as how to manage user accounts for a collection of web archives, storage limitations, and questions about the privacy or public access to archived content. This research focuses on tickets in which the client discusses a perceived flaw in an individual archived website or an entire web archive. From prior experiences, I had seen that these types of tickets are the most likely to deal with issues of quality.
Support tickets not pertaining to quality issues were classified as such and separated from the main data of interest. Each ticket analysed consisted of the original ticket submitted by the client, the response sent by the AIT partner specialist, and any subsequent interactions between them. Tickets could be quite brief, consisting of three interactions (the original client ticket, the partner specialist’s response, and the client’s response), or they could have many interactions over time, spanning weeks or even months. A total of 305 tickets and 2544 interactions were analyzed.
These support tickets were analysed using the GT techniques of open coding and theoretical memos to identify the main concepts and categories present in the data. According to the precepts of GT, after several rounds of coding, the researcher will reach saturation, a state when nothing new is being extracted from the data. Per the guidelines of Grounded Theory, only the core categories (that is, the ones that explain most of the variation in quality) are part of the final theory. In order to increase the quality and rigour of the study, I engaged in purposeful peer review. University professors were periodically invited to audit the entire research project, including the codebook, preliminary findings, and core categories. In addition to peers, employees of the Internet Archive were also invited to see the ifndings and comment on them.
As previously discussed, the notion of archivability has already received some attention from academic
researchers. It was defined by [
Table 1 presents examples of the first situation. The marker C: refers to the original query presented by the AIT client, while the marker AIT: refers to the answer given by the web archivist. Many websites routinely change the way the content is delivered to the user, thus a website can go from being easily archivable to practically unarchivable fairly quickly. As one AIT employee said: “The web, and specifically social networking sites can be a moving target.’ When websites change their internal functionality, it can result in the archived website looking diferent from the original (tickets 08 and 129) and missing content (tickets 08 and tickets 258).
Cases where archivability was negatively impacted by the heavy presence of dynamic content are
shown in Table 2. Generally, sites that utilize technologies such as JavaScript, Flash, and streaming
audio and video are dificult to capture and render like the original. This finding is consistent with the
work of [
C: Our Facebook page didn’t get archived. When I viewed what was crawled, all the came up was basically a blank Facebook page.
AIT: Both Facebook and Twitter have made some changes recently to the way they set up their sites, which requires a little bit of work on our end to catch up.
1. For Facebook, your site was archived, there is just an issue that is keeping the archived page from displaying normally. Our engineers are working on this and it should be fixed this week. I will let you know as soon as I have further information. 2. For Twitter, they recently removed the “more” button from twitter feeds and instead users access older tweets by scrolling down the page. The way this feature is set up makes it dificult for our crawlers to access the older content that is not displayed automatically.
C: I am getting an error on the following Facebook crawl.
AIT: Facebook made a change to the settings for their stylesheets C: in the "Township of " collection I am trying to capture this facebook site: http://www.facebook.com/pages/township AIT: We are still generally able to capture the initial content on a Facebook timeline; however the most recent change from Facebook has made it one again dificult to capture dynamically loading content as a user scrolls down through the page search-driven, such as library catalogs, web forms, or search engines. As the AIT employee explains, these are elements that depend on a myriad of complex, dynamic interactions that cannot be replicated in an archived website.
Sometimes websites will have unique or unusual ways of rendering content, which can negatively afect archivability, as seen in Table 3. For example, some content management systems can create endlessly repeating directory structures (such as http://somesite.com/news, http://somesite.com/news/ news, and http://somesite.com/news/news/news). The presence of these will cause the crawler to go into infinite loops (crawler traps) in an attempt to capture all levels of the website. This can lead to poor-quality archived websites, stalled or incomplete crawls, and large amounts of unnecessary data.
Archivability is not a dimension of quality that is directly perceived by most AIT clients. AIT employees, who have a deep knowledge of and experience with the technical process of archiving websites, were much more likely to perceive a quality problem as an archivability problem. Instead, AIT clients framed archivability issues as correspondence or relevance problems that negatively afected the quality of an archived website or of an entire web archive. For example, in Table 1, ticket 8 describes a situation where the AIT client is writing because she sees a blank Facebook page, while it is the AIT employee that is able to determine that the blank page is being caused by Facebook changing how it delivers its content. That is, an archivability problem is being framed in terms of a problem with visual correspondence, since the archived website does not look like the original. Similarly, in Table 2, in ticket 100, the AIT client frames the problem as one of interactional correspondence, that is, the archived site does not "flip through the slides" in a slideshow, and so does not behave as the original website does. Table 4 lists the support tickets analysed, as well as the IQ dimension in which they are framed.
Website archivability can thus be seen as a latent dimension, because it is hidden from most people, and framed in terms of other quality problems. The archivability of a website can only be perceived after its archived counterpart exhibits a quality problem.
In Social Sciences research, the term manifest variables is used to describe variables that we can
directly observe [
I advance that website archivability is one such latent construct, because it cannot be directly
measured until the website is actually archived. Any proposed archivability measurement that is taken
before the website is actually archived, such as those presented by [
As can be seen in the findings, a website with low archivability can negatively impact the quality of its archived version by causing correspondence or relevance problems. An archived website with low correspondence will look diferent from the original, behave in a diferent, degraded manner, and have missing elements. An archived website with low relevance will have content that is unrelated to that of the original website, or will be much larger in size than the original. Low archivability in a website leads to low-quality web archives, and low-quality web archives lead to low-quality historical records.
The degree of archivability of a website can be estimated a priori by calculating how much of it is composed of dynamic content, such as JavaScript. However, its true archivability of a website can only be determined a posteriori by detecting correspondence or quality problems. This points to the nature of web archiving as a reactive practice instead of a proactive one. Any substantial change in web technologies, standards, or platforms necessitates a change in web archiving practice in order to "catch up" and create high-quality web archives that result in a high-quality historical record.
Many parts of the web have always been out of reach: websites that are database-, form-, or
searchdriven have always been impossible to capture. The increasing reliance on dynamic, client-side
technologies such as JavaScript has also done much to decrease website archivability [
This paper makes the following contributions: 1. It presents a detailed, comprehensive definition of website archivability, one of the most important dimensions of information quality in web archives. 2. This definition of archivability is human-centred and grounded in how web archivists perceive quality in web archives. 3. The definition is independent of the technology currently in use to create web archives, making it suitable to a wide variety of platforms, preservation contexts, and situations. 4. It characterizes archivability as a latent construct of information quality, expressed by the manifest variables of correspondence and relevance, a finding not previously seen in the literature. 5. It describes how low website archivability influences the quality of web archives as historical records.
As historian Ian Milligan stated “web archives do not provide a perfect representation of the past...but neither do traditional archives, which have had to be very selective with what they select, appraise, and preserve”[18]. Though this paper paints a sobering picture of decreasing website archivability and its implications for the historical record of the web, perhaps it is not web archivists, but historians, who will be the most able to deal with the errors and omissions of the past web. //onlinelibrary.wiley.com/doi/abs/10.1111/ijpp.12600. doi:https://doi.org/10.1111/ijpp. 12600. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/ijpp.12600. [18] I. Milligan, Historiography and the Web, Sage Publications Ltd, Los Angeles, CA, USA, 2018, pp.
16–29.
C: In reviewing our crawls, I have noticed a few of our pages do not display correctly.
The problem pages either utilize flash or javascript. I know javascript can be problematic. The following pages are the not displaying correctly: 76 369 2884 30 3481 3458 • http://www. .edu/ Video does not display/does not flip through slides • http://www. edu/y79.xml Does not flip through slides • www. .edu/y55333.xml Does not display additional photographs • http://www. .edu/y213.xml Does not display photograph slides • http://www. .edu/BBQ.xml Does not display photograph slides • http://www. .edu/toc.xml Does not display photograph slides AIT: flash and Javascript can be dificult to capture or display sometimes C: From the crawl report, it looks like a reasonable number of urls were captured, but also a good number show up in the “out of scope” column. Then, when I checked the Wayback version I got a message that the archived site cannot be displayed within a frame (see screenshot). I’m not sure what this means or if it is possible to adjust the crawl to fix it. Do you have any suggestions for the best way to capture these videos? AIT: streaming video can be dificult to archive sometimes C: The athletics department has their game day programs online. I see to be able to view the sections but can’t see a way to capture printer-friendly formats from their link. Is this possible? AIT: It looks like the site uses a fair bit of javascript to generate those “printer friendly’ pages, but I’m not sure how feasible capture is C: under the About Us tab, under Press Room, the tabs other than News Releases ( in the news, Annual report, Media Kit, and Social Media) do not work: AIT: Regarding the tabs on the Press Room URL, I am not sure if we will be able to capture this content due to the dynamic way in which these links are generated
Special case: Websites that are database and form or search-driven C: Much of this content is located in databases, so, in general I’m curious about how Archive-It will handle these databases. Here are two examples: • http://www.ourhistory.org/search.asp#index The results of a search come up on
a site with a URL with the same seed, but will Archive-It crawl this database? • http://www.ourhistory.org/iq/register/welcome.asp Same as the site above, the
results remain on the same seed, will Archive-It crawl the database? AIT: if database driven parts of sites have direct links to the content, the crawler will capture those, however the crawler can’t enter search terms or interact with forms, so if that is the only way to access the database content, the crawler likely will not automatically be able to access that content C: This site has a database backend and is queried via text input fields on the phonebook homepage...Could I get some help on how to do this successfully? AIT: Because of their interactive nature, search boxes cannot operate in an archived website in the same way as they would on the live web C: I would like to know if there is any way I can capture the search feature of the website AIT: Search boxes are something that will not behave in an archived site like they do on the live web. We can archive content that would be returned by using the search function (as you noticed with the “Browse All Projects’ button) however, the crawler is not able to archive the database or search engine that the live site search runs of of 464 3423 3001 86
C: When you’re navigating through the catalogs themselves, you also come across the same issue of not being able to get from 1 page to 2 page or 3 or 4 or 5. However, all of the lots are captured, but you have no way of accessing them through the site AIT: The way that this site does it’s navigation is significantly more complicated than your average site due to the form based dropdowns that you notice to the right of the pagination at the top of the list. The “Sort” and “per page” options are actually forms, so instead of simply clicking on links to subsequent or previous pages (the way that most sites do pagination), the crawler would actually have to select an option from the dropdown and submit a form each time, in order to get content back. These are types of interactive behavior the crawler does not perform by default, so it will require additional development...Because this site is so uniquely complicated in the way it has implemented pagination, any work our engineers put into developing a new crawling feature to capture it would be very specific to this site and likely not transferrable to other examples C: I’ve done a test crawl on all ".stateu.edu" while it has captured thousands of pages it also seems to determine many "stateu.edu" pages to be "out of scope". These pages are not blocked by a robots.txt. Why would that be happening? AIT: We do see these types of repetitive URLs from time to time, and they appear to be generated by code in certain implementations of content management systems like Drupal C: We’re having some trouble limiting the URLs on one of crawls to a reasonable number. We’re actually getting the content we’d like, but we’re also getting a ton of extraneous URLs that are either bad content or aliases to content we’re already capturing with another URL.
AIT: After taking a look at the queued URLs for this host, it appears that the crawler is running into a trap that we see from time to time on some websites (including some Drupal sites) where the site generates links with repeating directories C: We ran another test crawl on this site and now we seemed to have opened a can of worms for the main site we wanted to crawl. It looks like we have the Flickr URL under control and may just put a limit on the number captured. The main URL that we want, captured 83, 600 with over 1 million in the que AIT: The issue with your http://www.pl.gov/tef/ site is one that we see from time to time, where something in the way the site is put together creates urls with repeating directories that all point back to the same page
Information Quality Dimension and Sub-Dimensions 8, 129, 100, 76 2884, 464 258, 369, 30, 3481, 3458 3423 3001, 86, 76