Ontology Based Semantic Annotation for Enhancing Mobility Support for Visually Impaired Web Users Yeliz Yesilada, Simon Harper, Carole Goble and Robert Stevens Information Management Group Department of Computer Science University of Manchester Oxford Road Manchester M13 9PL, UK yesilady@cs.man.ac.uk 1. ABSTRACT shown in Figure 1 and particularly, the focus of this paper is on the We have previously shown that the mobility, or ease of travel, of stage 3 of the architecture; in which Web pages are annotated with visually impaired Web users is reduced since most Web pages are concepts from our authoring ontology and translated to mobility usually designed for visual interaction[7]. Therefore, in a visually concepts which are derived using logical rules (heuristics). impaired person’s environment objects that support travel are miss- Visually impaired people usually access Web pages either by us- ing or inaccessible altogether. We aim to enhance the experience of ing screen readers[10] or specialist browsers[3]. If the Web pages visually impaired Web travellers by using annotation and Seman- are properly designed and laid out in a linear fashion, these assis- tic Web technologies. In [17], we have proposed a semi-automated tive technologies work satisfactorily. Particularly, nowadays some tool which encodes techniques for the support of travel upon the screen readers access the HTML source code rather than solely Web. The main goal of this tool is to analyse Web pages to iden- reading the screen, which enables them to provide better support. tify objects that support mobility and travel, discover their roles, However, not many pages are properly designed; the focus is usu- annotate them and transform pages based on these annotations to ally on the visual presentation which makes audio interaction al- enhance the provided mobility support. This paper mainly presents most impossible. Furthermore, chunking the page into several parts the annotation part of the tool and provides some transformation and presenting it in a nonlinear fashion is becoming popular which examples which are based on the annotations. The main message makes the provided functionalities of these assistive technologies of this paper is that visually impaired Web users could also bene- insufficient. There are also guidelines to aid the designers in creat- fit from the Semantic Web technologies and here we demonstrate a ing accessible pages[2], unfortunately, few designers follow these possible approach to achieve that. guidelines and therefore Web accessibility is still a problem. The home page of 2nd International Semantic Web Conference (ISWC2003) can be used to illustrate the problem (see Figure 7 2. INTRODUCTION part labelled as C). The page is visually laid out into two columns Our main goal is to improve the mobility of visually impaired with the main content in the right column. Since the screen reader Web users by providing tool support for the provision of mobility. renders pages based on the tag order in the HTML code, visually The travel analysis framework which is the foundation for the tool impaired users have to read the entire left column in order to access is introduced in [17]. The aim of this tool is to analyse the travel the right column. The page is quite long and therefore it takes an support offered within a Web page and semi-automate the process unacceptable length of time to read the whole page. Accessibility, of: and in particular mobility, is not only about the provision of alter- 1. Extracting travel objects; native text for images, but also about how easy it is for a traveller to complete a successful journey. For example, if the user wants 2. Discovering their roles; to directly access the “register now” part of the page, the only way is to read almost the entire page (see Figure 7 part labelled as C). 3. Annotating the extracted objects; Therefore, the whole journey experience becomes frustrating and 4. Transforming the page with respect to annotations. unsatisfactory. Further problems also exist when trying to gain an overview of the page. Some screen readers, for instance Jaws [10], This paper presents the annotation part (3 above) of the tool[17] and provide information for overview when the user first accesses a discusses the associated challenges. The architecture of this tool is page. This overview information includes, for example, the num- ber of headings in the page based on the heading tags in the source code. However, if the page is not appropriately designed, such in- formation could be misleading. For example, when ISWC2003 is accessed by Jaws, then it says “one heading”. This information Permission to make digital or hard copies of all or part of this work for is misleading; although the page is quite long and heavy in the personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies content, the user might expect to access a small page. This is a bear this notice and the full citation on the first page. To copy otherwise, to typical problem because the designers use visual effects to specify republish, to post on servers or to redistribute to lists, requires prior specific the headings in the page rather than the proper tags to specify the permission and/or a fee. semantics of the structure. Copyright 2003 ACM 0-89791-88-6/97/05 ...$5.00. Analysts Web page Heuristics edge of how visually impaired people actually travel gives a context for their travel on the Web[7]. The ontology that we are proposing 1 2 User Agent to use to annotate pages, aims to encapsulate that knowledge. This 4 ontology could be considered as an assistive mechanism for ap- Enhanced DOM Auto. Manual DOM Transcoding plying physical travelling metaphors to movement around the Web. Web page Our domain of interest is the mobility of visually impaired users Travel Analysis and this ontology will be used as a controlled vocabulary for the Annotation COHSE transformation part of the tool. The COHSE1 annotator[5] will be Heuristics 3 used to annotate pages with this ontology. The annotations will Annotation Ontology be stored externally and accessed by the transformation part of the Service Service tool. We also present some example transformation heuristics to illustrate how we will use the annotations. Annotation DB The ontology that we are proposing to use to annotate pages consists of three parts. The first part encapsulates the knowledge Mobility Ontology about the travel objects from real world mobility studies– mobil- ity concepts. The second part holds information about including hypermedia concepts and vocabularies used in previous work on transcoding– authoring concepts. The last part holds information Figure 1: The basic architecture of the mobility support tool. about the context of a journey. The annotation process is encoded in an annotation pipeline. The first two parts of the ontology play important role in this annotation pipeline. Authoring concepts can capture the knowledge about how the objects are presented in the In fact, the page is semantically organized into chunks, but there environment and mobility concepts can capture the knowledge about is no mechanism for visually impaired users to access those chunks how the authoring concepts are used in a journey. Therefore, the randomly or glance through the chunks. Whereas sighted users can combination of these two parts of the ontology, could provide ex- change their focus easily and access the chunks randomly. Some tensive knowledge to perform the transformations of the Web pages screen readers provide a function for accessing the list of headings to ease the travel. in the page based on the heading tags in the HTML source, which The paper is structured as follows: Section 3 explains the model means that the page needs to be properly tagged. It allows users to of travel that is the backbone of the mobility support tool[17] and scan the page rather than reading the entire page. These are crucial then introduces the ontology that encapsulates the knowledge pro- techniques for the mobility of the user, but very much dependent vided by this model and by the mobility of visually impaired peo- on proper HTML tagging. Our aim is to extend such techniques by ple. Section 4 introduces our annotation pipeline. Section 5 de- using annotations and semantic Web technologies to improve the scribes COHSE annotator and how the mobility ontology is used to mobility experience of visually impaired Web travellers. annotate pages. Some example scenarios are explained in Section 6 One of the goals of the Semantic Web vision is to make knowl- that demonstrate how the annotated pages will be used in the trans- edge accessible not only to humans but also to agents. In this frame- formation stage of the mobility support tool. Section 7 presents work, our goal is to make the role of the objects that support travel and discusses some related works. Finally, Section 8 offers some and mobility explicitly interpretable by agents rather than just be- conclusions. ing visually interpretable by humans. Therefore, it is necessary to associate metadata with the objects (machine-readable vs machine- understandable). One mechanism for associating such metadata is 3. MOBILITY ONTOLOGY annotation. Travel in the Web world is likened to travel in the physical world Our aim is to annotate objects with semantic metadata that pro- [7]. There has been extensive work undertaken in the mobility of vides some indication of their role. This could be considered as visually impaired people in the physical world, which can be trans- another dimension of semantic annotation since in this case the an- ferred to the Web world. In order to transfer and adapt real world notation is not about the meaning of the resources, but it is rather metaphors to Web world, a model of travel is introduced[7] and ex- about the role of the resources. This could be still considered as a tended in[17]. In order to complete a successful journey, travellers further step along the way from simple textual annotation, as the use or may need to use travel objects. These objects are mainly intention within the Semantic Web context is that this information grouped into three broad categories: way points, orientation points will be accessible to agents. The aim is still making the implicit and travel assistants. Way points are points within a journey at knowledge explicit and making it accessible not only to humans which a decision may be made that directly facilitates onward jour- but also to software agents. ney. Knowledge about orientation suggests that a person needs in- The notion of travel and mobility on the Web is introduced to formation about location, distance and direction in order to be ori- improve the accessibility of Web pages for visually impaired and ented in a journey and the objects that provide such an information other travellers by drawing an analogy between virtual travel and are categorised as orientation points. Sighted or visually impaired travel in the physical world [8]. Travel is defined as the confident travellers experience problems in orienting themselves from time navigation and orientation with purpose, ease and accuracy within to time in an unfamiliar or familiar environments, they use differ- an environment. Therefore, travel is not only about Web navigation ent strategies to re-orientate themselves. The objects that they use but it is about the whole journey experience. Mobility is the ease in these strategies are grouped as travel assistants[17]. Fundamen- and confidence at which travel can be accomplished. Environment tally, a traveller navigates and orientates by consulting, detecting is the context in which the traveller journeys through and includes the way the landscape is rendered and perceived [7]. Travel objects 1 The Conceptual Open Hypermedia Project (COHSE) (http: are environmental elements that are used during a journey; in the //cohse.semanticweb.org), particularly, Mozilla plug-in Web, they are supplied by the page design and the browser. Knowl- version of the COHSE annotator is used. and identifying all these travel objects. Consultation, detection and identification points identify an object, a place or a person in the identification are accomplished through the mobility instruments of environment (for detailed explanations, please refer to [17]). in-journey guidance, previews, probes and feedbacks. These com- Authoring concepts hold information about the hypermedia con- ponents form the model of travel[7]. cepts and previous work on transcoding. In this case, we do not Real world mobility studies also suggest that visually impaired consider the roles of the objects in the travel framework but we are people travel a journey in a different way, using a number of differ- more interested in how the objects are presented in the Web land- ent cues, to sighted people. For example, visually impaired people scape. The Web landscape is defined as the combination of the use simple information more frequently than complex information page and the agent (e.g, browser and assistive technologies such as (for detailed information, please refer to [7]). Knowledge of these screen readers). For example, there are different ways of presenting differences and how visually impaired people travel provide a con- boundaries between objects including distinctive colour, space and text for their travel on the Web. Here, we introduce an ontology line and colour boundaries are usually invisible to visually impaired which aims to capture the knowledge about the mobility of visu- users. In the mobility framework, these objects can have different ally impaired people. The encoded information in the ontology roles, for example boundaries are way edges. then could be used to provide better support for the provision of Based on this ontology, there could be three different approaches mobility. for annotating Web pages, and in consequence, transforming them The mobility ontology serves two purposes: a representation of a regarding the annotations for enhancing the provided travel support– shared conceptualisation of knowledge about the mobility of visu- Web pages could be annotated with a basket of ontologies. These ally impaired people, and a controlled, shared vocabulary that can approaches could be as follows: be communicated across applications. In the context of the semi- automated tool that we aim to provide for the travel support, the on- 1. We could use the mobility journey roles such as obstacle tology will be used as the controlled vocabulary to drive the trans- and cue (see Figure 3), and also the concepts that provide formations. Fundamentally, the mobility ontology encodes three information about the context of the journey. In this case groups of concepts which hold information about: the contextual information is crucial because they determine the journey role of an object, such as the current travel pur- • The context of a journey; pose. If enough annotation could be provided to determine the journey role of an object, then we could try to turn obsta- • The role of the travel objects: objects can have a journey cles into cues or remove them completely to provide a better role which depends on the context of the undertaken journey journey support for visually impaired Web users. and can also have one (or more) environmental role(s) (see Figure 3); 2. We could use the mobility environmental roles (see Figure 3) • The authoring concepts encapsulate the knowledge from hy- to annotate the extracted travel objects such as identifica- permedia design and previous work on transcoding (see Fig- tion point and decision point. These annotations would make ure 2). the implicit environmental knowledge explicit and enable our agent to have the understanding of the objects in the environ- Figure 3 and 2 show the plain hierarchical representation of the ment. By the help of the transformation agent, objects will concepts according to the groups above. These concepts mainly then be able to play their intended roles. Moreover, these an- aim to provide a framework to encapsulate the information about a notations could be used to provide different views of a page journey. Since we do not have the space to explain all the concepts and provide better support for the orientation and navigation, in the ontology, please refer to http://augmented.man.ac. and thus for the mobility and travel. uk/ontologies/TravelOntology.daml for the complete ontology and detailed definitions. However, here we will introduce 3. Finally, we could use the authoring concepts (see Figure 2) some of the important concepts from each group. and use a set of rules to map these authoring concepts to mo- A Web journey can take place in different contexts. For example, bility environmental roles– knowing the authoring concept, the purpose of the journey can effect the way user travels through the mobility environmental role could be inferred from that the Web; if the user is searching or browsing effects the objects they knowledge. Then based on the mobility environmental roles use in a Web page. Visually impaired Web users usually use assis- of the travel objects, we could transform the pages so that tive technologies such as screen readers with conventional browsers the objects can play their intended roles. Since the authoring (e.g., Internet Explorer) to access pages and the assistive technolo- concepts could be considered as an extension to structures gies can render the Web pages differently and provide a different supported by HTML, the rules would also address the basic environment which effects the undertaken journey. Therefore, all HTML elements. Table 1 shows some example mappings; these aspects are important for the context of a journey. these mappings are turned into a set of heuristics. This ap- The second part of the ontology holds information about the proach is important for automating the annotation process. travel objects. Objects might have a specific role in an environ- Automation could be done in two levels: first obtaining the ment and based on the context, they might have another journey physical role of the travel objects based on the HTML struc- role. The journey role is context dependent, for example a graphic tural elements and then based on the authoring concepts, we site map could be a cue to a sighted user but it could be an obstacle could infer the environmental roles. to a visually impaired user. An obstacle is an object that directly or indirectly obstructs the progress of a traveller to a specific destina- In this paper, we particularly explore the third approach – which is tion and a cue is an object that orientates and encourages onward crucial for automating the annotation process – and provide further navigation[9]. The concepts under environmental role aim more to explanation in the next section. In the rest of the paper, we explain capture the knowledge of what kind of objects traveller use or may how we use the mobility ontology to annotate Web pages and how need to use to complete a successful journey. For example, deci- we transform the pages based on the provided annotations. sion and identification points are way points. Decision points are the choice points where alternative paths of travel are possible and AuthoringConcepts Chunk Atom Collection Node Advertisement Section Summary Paragraph Footer Header DataTable List Heading Link Boundaries SpecialGraphic Footnote NavigationalList Caption Title SectionHeading SkipLink Banner Icon Logo SiteMap Toolbar TableOfContents Index LinkMenu TableCaption FigureCaption PageTitle FrameTitle TitleBanner Figure 2: Hierarchical overview of the concepts encoding information about authoring concepts. Mobility JourneyRole EnvironmentalRole OutOfView Obstacle Cue TravelAssistant WayPoint Signage OutOfRange Invisible ExistanceObstruction NonexistenceObstruction TravelSupport InformationPoint TravelMemory TravelAid Alert NavigationPoint Attention IdentificationPoint WayEdge DecisionPoint ReferencePoint Location&Position Distance Direction AssociativePoint CurrentPoint Figure 3: Hierarchical overview of the concepts encoding information about objects that support travel and mobility. 4. THE ANNOTATION PIPELINE canonical form. To do this we define a simple vocabulary of author- Our system uses a pipeline approach to maintain flexibility in ing concepts; associated with previous work on transcoding and the basket of possible annotation formats it can handle. By using commonly used terms found in content management systems and a pipeline we provide for the possibility that annotations can arrive site description languages. We also include an ‘expansive’ anno- from many desperate sources and in many different forms (RDF, tation vocabulary associated with mobility concepts. These vocab- DC, RSS, CMS, etc.). The pipeline approach (see Figure 4) allows ularies are encoded into an ontology (see previous section) which us to: semantically links the authoring and mobility concepts. We can of course bypass this translation by using COHSE and our authoring, 1. Receive different types of inputs from different sources. or mobility, concepts to directly annotate the page. Next we use a set of heuristic mapping rules in combination with the ontology to 2. Harmonize these inputs into a canonical form based on a uni- create an enhanced DOM annotated with mobility concepts. This form annotation framework. new DOM is now in a suitable format for transcoding and the usu- ally complex process of transcoding is dramatically simplified. 3. Recruit annotations manually, semi-automatically, and fully- To demonstrate the necessity of our authoring annotation and the automatically. associated heuristic mapping rules we will focus on the more in- teresting part of the pipeline, the translation from authoring to mo- 4. Translate between annotation vocabularies associated with bility concepts. Table 1 shows some sample mappings from the hypermedia concepts and ‘expansive’ annotation vocabular- examples in Section 6 (Figures 6 and 7). We can see that the first ies associated with mobility concepts. column shows an annotated authoring concept (in this case a Head- ing). The second column shows its physical characteristics taken 5. Better realise – and simplify – the complex transcoding ac- from the DOM and we can see that in reality it is defined as bold tivity associated with our final goal based on these now ex- text with a hyperlink and a different background colour to the de- pansive mobility annotations. fault page colour. In effect this heading would be missed if it was Figure 4 shows the annotation flow and relates the flow to the archi- not annotated as there is no way to distinguish it as a heading be- tecture of our semi-automated annotation tool which is illustrated in cause the correct HTML element is not used. Figure 5 shows how Figure 1 (See Figure 1 for the parts labelled as 3 and 4 on Figure 4). this annotation along with a knowledge of the additional physical Annotations are received in different formats and translated into a properties are used to automatically annotate the item with more Mobility Authoring Concepts Concepts & RSS Mapping Authoring Rules Concepts Translation Content to a Mgmt Canonical Enhanced Annotation Enhanced DOM Mobility DOM 4 Document RDF Annotation Document annotated with Transcoding User Processor annotated with Annotated Authoring Agent DOM Mobility Concepts 3 concepts Mobility Annotation Concepts DB COHSE & Authoring Concepts Figure 4: The annotation pipeline (see Figure 1 for the parts labelled as 3 and 4). Authoring Concepts Physical characteristics Mobility Concepts 1. Heading Bold text, colour boundary and link Identification point, way edge, way point and navigation point 2. Heading Bold text, colour boundary Identification point, way point 3. Link A text link Travel memory , travel assistant , navigation and way point 4. Link An image link Navigation and way point 5. LinkMenu A list of text links Decision, navigation and way point, travel memory and travel assistant 6. LinkMenu A list of image links Decision, navigation and way point Table 1: Some examples of mapping authoring concepts to mobility concepts. The table should be read in conjunction with the explanation in the text (see Section 4) and Figure 5 which describes the heuristic rules. descriptive mobility concepts. The annotation and physical prop- stored in an annotation service or RDF repository. Ontology ser- erties are passed through a set of horn clauses which translate the vice gives access to the ontologies that are used to annotate Web simple authoring concept to an expansive set of mobility concepts2 . resources and the annotation service maintains mappings between resources and concepts. 5. THE COHSE ANNOTATOR The COHSE annotator works as follows: first the ontology is selected which is already uploaded to the ontology service, then a We have used the browser plug-in3 version of the COHSE anno- part of the document is highlighted and by selecting the concept, tator to annotate the Web pages with the mobility ontology. There the annotation can be created and stored in the annotation service. are a number of annotation tools available including MnM and On- It is easy and straightforward process. The COHSE annotator uses toAnnotate4 . We used the COHSE annotator chiefly developed by XPointer-like expressions to identify the region of the document. our group which gives us an opportunity to develop the tools we Work is underway to bring this into line with the W3C XPointer need. Additionally, because of its compatibility with Mozilla which working draft5 . Annotation service could be used to retrieve the is our annotation delivery environment. We have implemented a stored annotations. prototype transformation tool as a plug-in to Mozilla, and using both plug-ins can create a single environment for authoring and publishing the annotations. In addition, the browser can take care 6. SOME ANNOTATION AND TRANSFOR- of malformed HTML documents. By using a plug-in approach, the MATION SCENARIOS transformer, as well as the annotator can access the DOM object built by the browser and can base the transformations and annota- This section introduces some transformation heuristics and their tions on that[5]. applications based on the external annotations provided. We have Fundamentally, COHSE aims to combine an open hypermedia created a prototype browser plug-in into Mozilla to perform the architecture with ontological services in order to provide an archi- transformations. Since the annotations done using the plug-in ver- tecture for Semantic Web[6]. The COHSE annotator facilitates the sion of the COHSE annotator, the transformation prototype is also two supporting services of the COHSE agent: ontology service and implemented using the same browser6 . Annotations are stored ex- annotation service. The annotator interacts with a Web browser ternally based on the internal DOM tree of the browser, therefore, and a collection ontologies, providing annotations which are then using the same DOM tree is important for consistency. The transformation process also raised a number of issues con- 2 An example of the menu translation is also provided. cerning the usage of XPointer and external annotations. Since there 3 Versions have been implemented based on Mozilla and Internet 5 Explorer and in this project we have used the Mozilla version. XML Pointer Language (XPointer) W3C Working Draft 4 (http://www.w3.org/TR/xptr/). See http://annotation.semanticweb.org/tools 6 for the list of available annotation tools. Mozilla (http://www.mozilla.org). 1. Heading 5. LinkMenu LinkMenu DecisionPoint ^ WayPoint ^ Heading IdentificationPoint ^ WayPoint NavigationPoint Link NavigationPoint ^ WayPoint TextLink NavigationPoint ^ WayPoint ^ Colour boundary WayEdge ^ WayPoint TravelMemory ^ TravelAssistant LinkMenu ^ TextLink DecisionPoint ^ WayPoint ^ Heading ^ Link IdentificationPoint ^WayEdge NavigationPoint ^ TravelMemory WayPoint ^ NavigationPoint ^ TravelAssistant IdentificationPoint DecisionPoint , NavigationPoint, NavigationPoint WayPoint , TravelMemory, WayPoint TravelAssistant Figure 5: Some example rules for mapping Authoring concepts to Mobility concepts (also see Table 1). are a number of transformation heuristics that we want to apply, page, we logically fragment the page and allow user to have the applying one after the other could change the DOM tree and inval- preview of these logical fragments. These logical fragments aim to idate the existing XPointers in the annotation store. Therefore, in represent the implicit chunks within the page. This is a technique the current prototype, before the transformation process, we have to improve the intra7 mobility support (mobility support within the included an intermediate stage to transform the external annota- page), but once we improve the intra mobility support, this could tions to internal annotations by using the internal DOM tree of the effect the inter7 (mobility support between the pages) and collec- browser. In this way, we are not actually modifying the original tion wide7 mobility support (mobility support within the site). We document, but we only add the external annotations to the internal could also physically fragment the page by creating separate pages DOM representation of the browser. This intermediate stage is hid- based on the chunks in the page and allow the user to move from den from the user. Since it is not a difficult process we have used TOC to these pages and back. These two approaches have pros an ontology as a controlled vocabulary. However, we still have the and cons. For example, in the logical fragmentation, the user can problem of dynamically changing pages. Some Web pages change continue to read the next chunk without returning back to the TOC. their content and layout almost every day, therefore, even though However, the number of links in the page (from/ to TOC) might be the annotations are created and stored, they could be easily inval- too many and difficult for the user to manage them. The extra added idated. Therefore, we envision incorporating the annotations and links could increase the cognitive demand and maintenance prob- mobility ontology either with the content management systems or lem. Fragmentation of the Web page is important for good mobility within the designing process. However, the annotations and the for visually impaired users. It is well known that visually impaired created prototype of the transformation tool could be considered as people orientate themselves frequently; returning back to the TOC a proof of concept; we would like to demonstrate that the annota- could be used as an orientation technique, and their route is broken tions and transformations could improve the mobility of visually into a greater and more complex number of stages; moving from impaired Web users. TOC to chunks and v.v. could increase the number of stages to ac- We return to the home page of the 2nd International Semantic cess the chunks. Fragmentation divides the environment into more Web Conference (ISWC2003) to demonstrate the implementations manageable and easy to travel units. Moreover, it makes the envi- of some transformation heuristics based on our annotations (see ronment more regular, increases the information flow and supports Figure 7 part labelled as C). This is used because the design is sim- granularity[7]. ple, yet good enough to demonstrate some of the issues concerning Skip links are popular for enhancing the navigation, and thus the the mobility support provided by the page. Figure 6 shows the mobility, support provided by the page for visually impaired users. annotations. The page is originally annotated with the simple con- They are mainly used at the top of the page to provide a link to the cepts from the authoring concepts part of the mobility ontology (see main content, so that the user does not have to read the information Figure 2). Then the mobility environmental roles are inferred au- until the main content of the page. This is mainly for avoiding rep- tomatically from these annotations and the underlying source code, etitions, so that whenever visually impaired users access the page, through the process encoded in the annotation pipeline which is they do not have to read the information at the top to reach to the explained in Section 4. Figure 6 shows some example translations main content. Therefore, we have a set of heuristics concerning the from simple authoring concepts to an extended set of mobility en- addition of skip links and particularly deciding upon their targets. vironmental roles. For example, if there is a decision point closer to the top of the We propose to use the annotations to provide techniques for page, then we add a skip link at the top of the page pointing the overview as it is explained in the introduction, mainly for enhanc- first element just after the decision point (see Figure 7 part labelled ing the mobility of the user. For instance, we could use the an- as A). This heuristic is derived by analysing a number of pages and notated identification points to provide a kind of table of contents observing that usually Web pages have a decision point on the left (TOC) (see Figure 6). The TOC could be considered as a way of hand side and closer to the top of the page. Adding a skip link can providing the bird’s eye view of the page. The annotated identifica- be considered as a simpler version of creating a TOC. tion points can be considered to represent the chunks in the page. We have some heuristics particularly concerning decision points. We add links from TOC to identification points and also back to the 7 TOC. This could actually be considered as logical fragmentation of Please refer to the ontology at http://augmented.man. the page. Based on the identification points and way edges in the ac.uk/ontologies/TravelOntology.daml for detailed explanation. Mobility Concepts WP- Way Point ID- Identification Point DP- Decision Point WE- Way Edge NP- Navigation Point TM- Travel Memory TA-Travel Assistant Authoring Concepts H- Heading M- LinkMenu L- Link Figure 6: The home page of the 2nd International Semantic Web Conference (12-July-2003) showing the annotations embedded into the page. A C B Figure 7: The home page of the 2nd International Semantic Web Conference (12-July-2003) when transformed. For instance, specifying the number of items explicitly just before travel objects, for example checking the repetitions. We try to elim- the object is one of them (see Figure 7 part labelled as B). This inate repetitions within specified objects, in order to reduce the cog- could be important for the predictability and overview of the object. nitive overload and avoid the confusion the user might have when Based on this information, the user could have an idea about the they have to decide which one to choose. Moreover, the aim is to size of the object and how long it will take to read the entire object. try to make the page concise and task focused rather than verbose. Sighted users can easily glance through the object and have an idea For example, the sponsors section (see Figure 7 part labelled as B) about the size, here we try to provide implicit information explicitly is actually divided into three sections; platinum, gold and silver. somehow to simulate the glancing activity. It could also be useful However, the way that this part of the page is designed cause prob- for the orientation of the user; when he (she) starts to access the lems to visually impaired users because of several reasons. They items by knowing the size could help user to know where he (she) are all images and only the images in the gold section have alter- is. native tags, so the platinum and silver sections are invisible to the There are also some heuristics generally applicable to all of the user. Additionally, images are repeated and the part with gold is rendered as “gold gold gold [network inference]”, which might be approach. The annotation pipeline can be used to annotate Web confusing to the user. Since they are annotated as identification pages by using different parts of the ontology. Some annotation points, the repeated images are removed and only one of them is and transformation scenarios are also explained here to illustrate left. We also have heuristics for missing Alt tags, such as using the the appllication and usage of this pipeline. file name of the image. Sometimes the file name might not appro- Our main goal is to improve the mobility support for visually priate but at least it gives an idea about the image. Figure 7 (part impaired Web users and by using the proposed ontology and also B) shows how the sponsors section is transformed. the annotation pipeline, we expect to achieve our goal. The work Essentially, the heuristics and transformations that we have ex- presented here is still preliminary and much is to be done, in partic- plained here are all simple but have high impact on the provided ular an evaluation of the annotation pipeline and the transformation mobility support of the page and they illustrate how the annotations process. could derive the transformation of the pages. 9. REFERENCES [1] Annotation of web content for transcoding, 1999. W3C Note, 7. RELATED WORK http://www.w3.org/TR/annot/. The goal of annotations for Web content transcoding is to pro- [2] Web content accessibility guidelines 1.0, 1999. http: vide better support either for audio rendering, and thus for visually //www.w3.org/TR/1999/WAI-WEBCONTENT/. impaired users, or for visual rendering in small screen devices. The [3] Chieko Asakawa and Takashi Itoh. User interface of a home problem of rendering Web pages in audio has some similarities to page reader. In Proceedings of the Third International ACM the problem of displaying Web pages on small-screen devices. For Conference on Assistive Technologies, pages 149–156, 1998. example, in both cases, only the small portion of the page is view- [4] Chieko Asakawa and Hironobu Takagi. Annotation-based able at any point. However, there are major differences and require- transcoding for nonvisual web access. In Proceedings of the ments. Although the amount of information that could be accessed Fourth International ACM Conference on Assistive at once in a small-screen device is also limited, the interaction is Technologies, pages 172–179. ACM Press, 2000. still visual. The provided visual rendering is still persistent[14], screen acts as an external memory, as opposed to audio rendering [5] Sean Bechhofer and Carole Goble. Towards annotation using which is transient. Additionally, compared to visual rendering, au- daml+oil. In K-CAP 2001 workshop on Knowledge Markup dio is less focused and more serial in nature[15], the user cannot and Semantic Annotation, Victoria B.C, October 2001. easily and quickly shift the focus. It is then the aim of this section [6] Carole Goble, Sean Bechhofer, Leslie Carr, David De Roure, to discuss related work based on these two themes. and Wendy Hall. Conceptual open hypermedia = the [4, 16] propose a proxy-based system to transcode Web pages semantic web? In SemWeb2001, The Second International based on the external annotations for visually impaired users. The Workshop on the Semantic Web, Hong Kong, May 2001. main focus is on extracting visually fragmented groupings, their [7] Carole A. Goble, Simon Harper, and Robert Stevens. The roles and importance. Eight different roles such as proper con- travails of visually impaired web travellers. In Proceedings tent, header and footer are proposed for annotation. These roles are of the Eleventh International ACM Conference on Hypertext mainly at abstract level and are not rich enough to fully annotate and Hypermedia, pages 1–10, 2000. the page to enhance the mobility support. They do not support deep [8] Simon Harper. Web Mobility for Visually Impaired Surfers. understanding and analysis of pages, in consequence the supported PhD thesis, The University of Manchester, 2001. transcoding is constrained by these proposed roles. [9] Simon Harper, Robert Stevens, and Carole Goble. Web For small-screen devices, [12] proposes a system to transcode an mobility guidelines for visually impaired surfers. Journal of HTML document by fragmenting it into several documents. The Research and Practice in Information Technology Special transcoding is based on an external annotation framework[1]. Since Issue on HCI, 33(2), July 2001. the focus is the small-screen devices, physical and performance [10] Henter-Joyce, Inc. Jaws. http://www.hj.com. constraints of the devices need to be considered, such as screen [11] Nichelle Hopson. Websphere transcoding publisher: size, memory size, and connection bandwidth. However, these are Html-to-voicexml transcoder, 2002. IBM developerWorks. not the main requirements of the users accessing Web pages in au- [12] Masahiro Hori, Goh Kondoh, and Kouichi Ono. dio and there are differences as explained above. Annotation-based web content transcoding. In Proceedings Another approach for content adaptation is page clipping[13]. of the Nineth International World Wide Web Conference, The approach is annotating pages with elements such as keep (con- pages 197–211, 2000. tent should be preserved) and remove, and then at content delivery, [13] Masahiro Hori, Kouichi Ono, Teruo Koyanagi, and Mari filter the page based on these annotations. This approach is also Abe1. Annotation by transformation for the automatic used for converting HTML to VoiceXML[11]. This is simple and generation. In Pervasive 2002, pages 267–281, 2002. could be an efficient approach, however, our main goal is to iden- [14] Jakob Nielsen. Voice interfaces: Assessing the potential. tify the roles of the objects in a page and transform accordingly, Alertbox, January 2003. rather than doing some kind of filtering. [15] Ian Pitt and Alistair Edwards. Design of Speech-Based Devices - a Practical Guide. Springer, London, UK, 2003. 8. SUMMARY [16] Hironobu Takagi and Chieko Asakawa. Transcoding proxy This paper has first presented an ontology that aims to encap- for nonvisual web access. In Proceedings of the Fourth sulate the knowledge from real world mobility studies, previous International ACM Conference on Assistive Technologies, work on transcoding and information about hypermedia concepts. pages 164–171. ACM Press, 2000. Then, it has discussed several possible annotation and transforma- [17] Yeliz Yesilada, Robert Stevens, and Carole Goble. A tion approaches based on this ontology. One of these strategies has foundation for tool based mobility support for visually been explored and presented in detail. In particular, an annotation impaired web users. In Proceedings of the Twelfth pipeline is introduced which can be considered as the core of this International World Wide Web Conference, 2003.