We have previously shown that the mobility, or ease of travel, of visually impaired Web users is reduced since most Web pages are usually designed for visual interaction[7]. Therefore, in a visually impaired person's environment objects that support travel are missing or inaccessible altogether. We aim to enhance the experience of visually impaired Web travellers by using annotation and Semantic Web technologies. In [17], we have proposed a semi-automated tool which encodes techniques for the support of travel upon the Web. The main goal of this tool is to analyse Web pages to identify objects that support mobility and travel, discover their roles, annotate them and transform pages based on these annotations to enhance the provided mobility support. This paper mainly presents the annotation part of the tool and provides some transformation examples which are based on the annotations. The main message of this paper is that visually impaired Web users could also benefit from the Semantic Web technologies and here we demonstrate a possible approach to achieve that.
Our main goal is to improve the mobility of visually impaired
Web users by providing tool support for the provision of mobility.
The travel analysis framework which is the foundation for the tool
is introduced in [
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Copyright 2003 ACM 0-89791-88-6/97/05 ...$5.00. shown in Figure 1 and particularly, the focus of this paper is on the stage 3 of the architecture; in which Web pages are annotated with concepts from our authoring ontology and translated to mobility concepts which are derived using logical rules (heuristics).
Visually impaired people usually access Web pages either by
using screen readers[
The home page of 2nd International Semantic Web Conference
(ISWC2003) can be used to illustrate the problem (see Figure 7
part labelled as C). The page is visually laid out into two columns
with the main content in the right column. Since the screen reader
renders pages based on the tag order in the HTML code, visually
impaired users have to read the entire left column in order to access
the right column. The page is quite long and therefore it takes an
unacceptable length of time to read the whole page. Accessibility,
and in particular mobility, is not only about the provision of
alternative text for images, but also about how easy it is for a traveller
to complete a successful journey. For example, if the user wants
to directly access the “register now” part of the page, the only way
is to read almost the entire page (see Figure 7 part labelled as C).
Therefore, the whole journey experience becomes frustrating and
unsatisfactory. Further problems also exist when trying to gain an
overview of the page. Some screen readers, for instance Jaws [
Web page 1 2
Auto.
In fact, the page is semantically organized into chunks, but there is no mechanism for visually impaired users to access those chunks randomly or glance through the chunks. Whereas sighted users can change their focus easily and access the chunks randomly. Some screen readers provide a function for accessing the list of headings in the page based on the heading tags in the HTML source, which means that the page needs to be properly tagged. It allows users to scan the page rather than reading the entire page. These are crucial techniques for the mobility of the user, but very much dependent on proper HTML tagging. Our aim is to extend such techniques by using annotations and semantic Web technologies to improve the mobility experience of visually impaired Web travellers.
One of the goals of the Semantic Web vision is to make knowledge accessible not only to humans but also to agents. In this framework, our goal is to make the role of the objects that support travel and mobility explicitly interpretable by agents rather than just being visually interpretable by humans. Therefore, it is necessary to associate metadata with the objects (machine-readable vs machineunderstandable). One mechanism for associating such metadata is annotation.
Our aim is to annotate objects with semantic metadata that provides some indication of their role. This could be considered as another dimension of semantic annotation since in this case the annotation is not about the meaning of the resources, but it is rather about the role of the resources. This could be still considered as a further step along the way from simple textual annotation, as the intention within the Semantic Web context is that this information will be accessible to agents. The aim is still making the implicit knowledge explicit and making it accessible not only to humans but also to software agents.
The notion of travel and mobility on the Web is introduced to
improve the accessibility of Web pages for visually impaired and
other travellers by drawing an analogy between virtual travel and
travel in the physical world [
The ontology that we are proposing to use to annotate pages consists of three parts. The first part encapsulates the knowledge about the travel objects from real world mobility studies– mobility concepts. The second part holds information about including hypermedia concepts and vocabularies used in previous work on transcoding– authoring concepts. The last part holds information about the context of a journey. The annotation process is encoded in an annotation pipeline. The first two parts of the ontology play important role in this annotation pipeline. Authoring concepts can capture the knowledge about how the objects are presented in the environment and mobility concepts can capture the knowledge about how the authoring concepts are used in a journey. Therefore, the combination of these two parts of the ontology, could provide extensive knowledge to perform the transformations of the Web pages to ease the travel.
The paper is structured as follows: Section 3 explains the model
of travel that is the backbone of the mobility support tool[
Travel in the Web world is likened to travel in the physical world
[
Real world mobility studies also suggest that visually impaired
people travel a journey in a different way, using a number of
different cues, to sighted people. For example, visually impaired people
use simple information more frequently than complex information
(for detailed information, please refer to [
The mobility ontology serves two purposes: a representation of a shared conceptualisation of knowledge about the mobility of visually impaired people, and a controlled, shared vocabulary that can be communicated across applications. In the context of the semiautomated tool that we aim to provide for the travel support, the ontology will be used as the controlled vocabulary to drive the transformations. Fundamentally, the mobility ontology encodes three groups of concepts which hold information about: ² The context of a journey; ² The role of the travel objects: objects can have a journey role which depends on the context of the undertaken journey and can also have one (or more) environmental role(s) (see Figure 3); ² The authoring concepts encapsulate the knowledge from hypermedia design and previous work on transcoding (see Figure 2).
Figure 3 and 2 show the plain hierarchical representation of the concepts according to the groups above. These concepts mainly aim to provide a framework to encapsulate the information about a journey. Since we do not have the space to explain all the concepts in the ontology, please refer to http://augmented.man.ac. uk/ontologies/TravelOntology.daml for the complete ontology and detailed definitions. However, here we will introduce some of the important concepts from each group.
A Web journey can take place in different contexts. For example, the purpose of the journey can effect the way user travels through the Web; if the user is searching or browsing effects the objects they use in a Web page. Visually impaired Web users usually use assistive technologies such as screen readers with conventional browsers (e.g., Internet Explorer) to access pages and the assistive technologies can render the Web pages differently and provide a different environment which effects the undertaken journey. Therefore, all these aspects are important for the context of a journey.
The second part of the ontology holds information about the
travel objects. Objects might have a specific role in an
environment and based on the context, they might have another journey
role. The journey role is context dependent, for example a graphic
site map could be a cue to a sighted user but it could be an obstacle
to a visually impaired user. An obstacle is an object that directly or
indirectly obstructs the progress of a traveller to a specific
destination and a cue is an object that orientates and encourages onward
navigation[
Authoring concepts hold information about the hypermedia concepts and previous work on transcoding. In this case, we do not consider the roles of the objects in the travel framework but we are more interested in how the objects are presented in the Web landscape. The Web landscape is defined as the combination of the page and the agent (e.g, browser and assistive technologies such as screen readers). For example, there are different ways of presenting boundaries between objects including distinctive colour, space and line and colour boundaries are usually invisible to visually impaired users. In the mobility framework, these objects can have different roles, for example boundaries are way edges.
Based on this ontology, there could be three different approaches for annotating Web pages, and in consequence, transforming them regarding the annotations for enhancing the provided travel support– Web pages could be annotated with a basket of ontologies. These approaches could be as follows: 1. We could use the mobility journey roles such as obstacle and cue (see Figure 3), and also the concepts that provide information about the context of the journey. In this case the contextual information is crucial because they determine the journey role of an object, such as the current travel purpose. If enough annotation could be provided to determine the journey role of an object, then we could try to turn obstacles into cues or remove them completely to provide a better journey support for visually impaired Web users. 2. We could use the mobility environmental roles (see Figure 3) to annotate the extracted travel objects such as identification point and decision point. These annotations would make the implicit environmental knowledge explicit and enable our agent to have the understanding of the objects in the environment. By the help of the transformation agent, objects will then be able to play their intended roles. Moreover, these annotations could be used to provide different views of a page and provide better support for the orientation and navigation, and thus for the mobility and travel. 3. Finally, we could use the authoring concepts (see Figure 2) and use a set of rules to map these authoring concepts to mobility environmental roles– knowing the authoring concept, the mobility environmental role could be inferred from that knowledge. Then based on the mobility environmental roles of the travel objects, we could transform the pages so that the objects can play their intended roles. Since the authoring concepts could be considered as an extension to structures supported by HTML, the rules would also address the basic HTML elements. Table 1 shows some example mappings; these mappings are turned into a set of heuristics. This approach is important for automating the annotation process. Automation could be done in two levels: first obtaining the physical role of the travel objects based on the HTML structural elements and then based on the authoring concepts, we could infer the environmental roles.
In this paper, we particularly explore the third approach – which is crucial for automating the annotation process – and provide further explanation in the next section. In the rest of the paper, we explain how we use the mobility ontology to annotate Web pages and how we transform the pages based on the provided annotations.
AuthoringConcepts Chunk Atom
Col ection
Node
Our system uses a pipeline approach to maintain flexibility in the basket of possible annotation formats it can handle. By using a pipeline we provide for the possibility that annotations can arrive from many desperate sources and in many different forms (RDF, DC, RSS, CMS, etc.). The pipeline approach (see Figure 4) allows us to: 1. Receive different types of inputs from different sources. 2. Harmonize these inputs into a canonical form based on a uniform annotation framework. 3. Recruit annotations manually, semi-automatically, and fullyautomatically. 4. Translate between annotation vocabularies associated with hypermedia concepts and ‘expansive’ annotation vocabularies associated with mobility concepts. 5. Better realise – and simplify – the complex transcoding activity associated with our final goal based on these now expansive mobility annotations.
Figure 4 shows the annotation flow and relates the flow to the architecture of our semi-automated annotation tool which is illustrated in Figure 1 (See Figure 1 for the parts labelled as 3 and 4 on Figure 4). Annotations are received in different formats and translated into a AssociativePoint
Cur entPoint canonical form. To do this we define a simple vocabulary of authoring concepts; associated with previous work on transcoding and commonly used terms found in content management systems and site description languages. We also include an ‘expansive’ annotation vocabulary associated with mobility concepts. These vocabularies are encoded into an ontology (see previous section) which semantically links the authoring and mobility concepts. We can of course bypass this translation by using COHSE and our authoring, or mobility, concepts to directly annotate the page. Next we use a set of heuristic mapping rules in combination with the ontology to create an enhanced DOM annotated with mobility concepts. This new DOM is now in a suitable format for transcoding and the usually complex process of transcoding is dramatically simplified. To demonstrate the necessity of our authoring annotation and the associated heuristic mapping rules we will focus on the more interesting part of the pipeline, the translation from authoring to mobility concepts. Table 1 shows some sample mappings from the examples in Section 6 (Figures 6 and 7). We can see that the first column shows an annotated authoring concept (in this case a Heading). The second column shows its physical characteristics taken from the DOM and we can see that in reality it is defined as bold text with a hyperlink and a different background colour to the default page colour. In effect this heading would be missed if it was not annotated as there is no way to distinguish it as a heading because the correct HTML element is not used. Figure 5 shows how this annotation along with a knowledge of the additional physical properties are used to automatically annotate the item with more Content Mgmt Translation
to a Canonical Annotation 3 Annotated
DOM COHSE descriptive mobility concepts. The annotation and physical properties are passed through a set of horn clauses which translate the simple authoring concept to an expansive set of mobility concepts2.
We have used the browser plug-in3 version of the COHSE
annotator to annotate the Web pages with the mobility ontology. There
are a number of annotation tools available including MnM and
OntoAnnotate4. We used the COHSE annotator chiefly developed by
our group which gives us an opportunity to develop the tools we
need. Additionally, because of its compatibility with Mozilla which
is our annotation delivery environment. We have implemented a
prototype transformation tool as a plug-in to Mozilla, and using
both plug-ins can create a single environment for authoring and
publishing the annotations. In addition, the browser can take care
of malformed HTML documents. By using a plug-in approach, the
transformer, as well as the annotator can access the DOM object
built by the browser and can base the transformations and
annotations on that[
Fundamentally, COHSE aims to combine an open hypermedia
architecture with ontological services in order to provide an
architecture for Semantic Web[
3Versions have been implemented based on Mozilla and Internet Explorer and in this project we have used the Mozilla version. 4See http://annotation.semanticweb.org/tools for the list of available annotation tools. stored in an annotation service or RDF repository. Ontology service gives access to the ontologies that are used to annotate Web resources and the annotation service maintains mappings between resources and concepts.
The COHSE annotator works as follows: first the ontology is selected which is already uploaded to the ontology service, then a part of the document is highlighted and by selecting the concept, the annotation can be created and stored in the annotation service. It is easy and straightforward process. The COHSE annotator uses XPointer-like expressions to identify the region of the document. Work is underway to bring this into line with the W3C XPointer working draft5. Annotation service could be used to retrieve the stored annotations. 6.
This section introduces some transformation heuristics and their applications based on the external annotations provided. We have created a prototype browser plug-in into Mozilla to perform the transformations. Since the annotations done using the plug-in version of the COHSE annotator, the transformation prototype is also implemented using the same browser6. Annotations are stored externally based on the internal DOM tree of the browser, therefore, using the same DOM tree is important for consistency.
The transformation process also raised a number of issues concerning the usage of XPointer and external annotations. Since there
(http://www.w3.org/TR/xptr/). 6Mozilla (http://www.mozilla.org). W3C
1. Heading 5. LinkMenu Heading
Link Colour boundary Heading ^ Link
IdentificationPoint ^ WayPoint NavigationPoint ^ WayPoint WayEdge ^ WayPoint IdentificationPoint ^WayEdge WayPoint ^ NavigationPoint are a number of transformation heuristics that we want to apply, applying one after the other could change the DOM tree and invalidate the existing XPointers in the annotation store. Therefore, in the current prototype, before the transformation process, we have included an intermediate stage to transform the external annotations to internal annotations by using the internal DOM tree of the browser. In this way, we are not actually modifying the original document, but we only add the external annotations to the internal DOM representation of the browser. This intermediate stage is hidden from the user. Since it is not a difficult process we have used an ontology as a controlled vocabulary. However, we still have the problem of dynamically changing pages. Some Web pages change their content and layout almost every day, therefore, even though the annotations are created and stored, they could be easily invalidated. Therefore, we envision incorporating the annotations and mobility ontology either with the content management systems or within the designing process. However, the annotations and the created prototype of the transformation tool could be considered as a proof of concept; we would like to demonstrate that the annotations and transformations could improve the mobility of visually impaired Web users.
We return to the home page of the 2nd International Semantic Web Conference (ISWC2003) to demonstrate the implementations of some transformation heuristics based on our annotations (see Figure 7 part labelled as C). This is used because the design is simple, yet good enough to demonstrate some of the issues concerning the mobility support provided by the page. Figure 6 shows the annotations. The page is originally annotated with the simple concepts from the authoring concepts part of the mobility ontology (see Figure 2). Then the mobility environmental roles are inferred automatically from these annotations and the underlying source code, through the process encoded in the annotation pipeline which is explained in Section 4. Figure 6 shows some example translations from simple authoring concepts to an extended set of mobility environmental roles.
We propose to use the annotations to provide techniques for
overview as it is explained in the introduction, mainly for
enhancing the mobility of the user. For instance, we could use the
annotated identification points to provide a kind of table of contents
(TOC) (see Figure 6). The TOC could be considered as a way of
providing the bird’s eye view of the page. The annotated
identification points can be considered to represent the chunks in the page.
We add links from TOC to identification points and also back to the
TOC. This could actually be considered as logical fragmentation of
the page. Based on the identification points and way edges in the
page, we logically fragment the page and allow user to have the
preview of these logical fragments. These logical fragments aim to
represent the implicit chunks within the page. This is a technique
to improve the intra7 mobility support (mobility support within the
page), but once we improve the intra mobility support, this could
effect the inter7 (mobility support between the pages) and
collection wide7 mobility support (mobility support within the site). We
could also physically fragment the page by creating separate pages
based on the chunks in the page and allow the user to move from
TOC to these pages and back. These two approaches have pros
and cons. For example, in the logical fragmentation, the user can
continue to read the next chunk without returning back to the TOC.
However, the number of links in the page (from/ to TOC) might be
too many and difficult for the user to manage them. The extra added
links could increase the cognitive demand and maintenance
problem. Fragmentation of the Web page is important for good mobility
for visually impaired users. It is well known that visually impaired
people orientate themselves frequently; returning back to the TOC
could be used as an orientation technique, and their route is broken
into a greater and more complex number of stages; moving from
TOC to chunks and v.v. could increase the number of stages to
access the chunks. Fragmentation divides the environment into more
manageable and easy to travel units. Moreover, it makes the
environment more regular, increases the information flow and supports
granularity[
Skip links are popular for enhancing the navigation, and thus the mobility, support provided by the page for visually impaired users. They are mainly used at the top of the page to provide a link to the main content, so that the user does not have to read the information until the main content of the page. This is mainly for avoiding repetitions, so that whenever visually impaired users access the page, they do not have to read the information at the top to reach to the main content. Therefore, we have a set of heuristics concerning the addition of skip links and particularly deciding upon their targets. For example, if there is a decision point closer to the top of the page, then we add a skip link at the top of the page pointing the first element just after the decision point (see Figure 7 part labelled as A). This heuristic is derived by analysing a number of pages and observing that usually Web pages have a decision point on the left hand side and closer to the top of the page. Adding a skip link can be considered as a simpler version of creating a TOC.
We have some heuristics particularly concerning decision points. 7Please refer to the ontology at http://augmented.man. ac.uk/ontologies/TravelOntology.daml for detailed explanation.
A
B WP- Way Point ID- Identification Point DP- Decision Point WE- Way Edge NP- Navigation Point TM- Travel Memory TA-Travel Assistant Authoring Concepts H- Heading M- LinkMenu L- Link For instance, specifying the number of items explicitly just before the object is one of them (see Figure 7 part labelled as B). This could be important for the predictability and overview of the object. Based on this information, the user could have an idea about the size of the object and how long it will take to read the entire object. Sighted users can easily glance through the object and have an idea about the size, here we try to provide implicit information explicitly somehow to simulate the glancing activity. It could also be useful for the orientation of the user; when he (she) starts to access the items by knowing the size could help user to know where he (she) is.
There are also some heuristics generally applicable to all of the travel objects, for example checking the repetitions. We try to eliminate repetitions within specified objects, in order to reduce the cognitive overload and avoid the confusion the user might have when they have to decide which one to choose. Moreover, the aim is to try to make the page concise and task focused rather than verbose. For example, the sponsors section (see Figure 7 part labelled as B) is actually divided into three sections; platinum, gold and silver. However, the way that this part of the page is designed cause problems to visually impaired users because of several reasons. They are all images and only the images in the gold section have alternative tags, so the platinum and silver sections are invisible to the user. Additionally, images are repeated and the part with gold is rendered as “gold gold gold [network inference]”, which might be confusing to the user. Since they are annotated as identification points, the repeated images are removed and only one of them is left. We also have heuristics for missing Alt tags, such as using the file name of the image. Sometimes the file name might not appropriate but at least it gives an idea about the image. Figure 7 (part B) shows how the sponsors section is transformed.
Essentially, the heuristics and transformations that we have explained here are all simple but have high impact on the provided mobility support of the page and they illustrate how the annotations could derive the transformation of the pages.
The goal of annotations for Web content transcoding is to
provide better support either for audio rendering, and thus for visually
impaired users, or for visual rendering in small screen devices. The
problem of rendering Web pages in audio has some similarities to
the problem of displaying Web pages on small-screen devices. For
example, in both cases, only the small portion of the page is
viewable at any point. However, there are major differences and
requirements. Although the amount of information that could be accessed
at once in a small-screen device is also limited, the interaction is
still visual. The provided visual rendering is still persistent[
[
For small-screen devices, [
Another approach for content adaptation is page clipping[
This paper has first presented an ontology that aims to encapsulate the knowledge from real world mobility studies, previous work on transcoding and information about hypermedia concepts. Then, it has discussed several possible annotation and transformation approaches based on this ontology. One of these strategies has been explored and presented in detail. In particular, an annotation pipeline is introduced which can be considered as the core of this approach. The annotation pipeline can be used to annotate Web pages by using different parts of the ontology. Some annotation and transformation scenarios are also explained here to illustrate the appllication and usage of this pipeline.
Our main goal is to improve the mobility support for visually impaired Web users and by using the proposed ontology and also the annotation pipeline, we expect to achieve our goal. The work presented here is still preliminary and much is to be done, in particular an evaluation of the annotation pipeline and the transformation process.