Smart Style on the Semantic Web ∗ Jacco van Ossenbruggen Lynda Hardman CWI CWI P.O. Box 94079 P.O. Box 94079 1090 GB AMSTERDAM 1090 GB AMSTERDAM The Netherlands The Netherlands Jacco.van.Ossenbruggen@cwi.nl Lynda.Hardman@cwi.nl ABSTRACT constraints imposed by the access device. In addition, the Web publishing systems have to take into account a plethora generated presentation should conform as much as possible of Web-enabled devices, user preferences and abilities. Tech- to the preferences of the user and the author [6]. These two nologies generating these presentations will need to be ex- types of adaptation may lead to an explosion of potential plicitly aware of the context in which the information is delivery contexts with which current stylesheet technology being presented. Semantic Web technology can be a fun- is unable to deal. damental part of the solution to this problem by explicitly In previous work, we describe our prototype multimedia modeling the knowledge needed to adapt presentations to presentation generation system Cuypers [21]. Cuypers gen- a specific delivery context. We propose the development of erates multimedia presentations adapted to the constraints a Smart Style layer which is able to process metadata that of a specific delivery context. We claim that the particular describes content and use this metadata to improve the pre- solutions deployed within Cuypers realize a level of adap- sentation of the content to human users. In the paper, we tivity that should become generally available on the Web. derive the requirements of such a Smart Style layer by con- This introduces new challenges since the solutions need to sidering Web design from both the document engineering be embedded within the current Web infrastructure. In this and graphic design perspectives. In addition, design trade- paper, we introduce the concept of Smart Style: an intelli- offs made by human designers have to be taken into account gent presentation adaptation layer for the Web that builds for the automated process. After stating the requirements upon two fundamental technologies: for a Smart Style layer, we discuss to what extent the cur- 1. Web document engineering technology, including de- rently available Web technology can be used and what its livery formats such as HTML [30], SMIL [29], SVG [10] limitations are. The limitations are illustrated with exam- and XSL [28], and style and transformation languages ples of potential future extensions. such as CSS [4] and XSLT [7]. Keywords: Semantic Web, Device Independent Author- ing, Document Engineering, Graphic Design. 2. Semantic Web knowledge representation and metadata Word count: 7150 technology, including RDF [24], RDF Schema [25], DAML+OIL [19] and CC/PP [26]. 1. INTRODUCTION Currently, Semantic Web technology is primarily deployed to improve Web-based information gathering and brokerage, As the Web continues to grow not only in size but also in with little attention to improving information presentation. complexity, the increasingly varying needs of the intended Our vision is, however, that the Semantic Web infrastruc- audience marks the end of the “one size fits all” era. Delivery ture should not only play a key role in finding information contexts [27] can be characterized in terms of specific user on the Web, but also in presenting this information in the preferences and abilities, capabilities of the access device most appropriate way to each individual reader. Our pro- and available network resources. Given this heterogeneity, posed Smart Style layer will deploy Semantic Web technol- any single message needs to be adapted to a particular set ogy to improve the presentation’s adaptation, aiming for an of circumstances. As a minimum requirement, the author’s optimized design of the presentation that suits the specific intended message needs to be conveyed to the user given the requirements of the user’s delivery context. ∗ This is a revised version of CWI Technical Report INS- In this paper, we derive the requirements for realizing the R0201 Smart Style layer. In section 2, we specify the key design ingredients of a Web-based presentation from two perspec- tives: a document engineering and a graphic design perspec- tive. These allow internal trade-offs to be made in the design Permission to make digital or hard copies of all or part of this work for of a presentation. In addition, external forces that influence personal or classroom use is granted without fee provided that copies are the decisions made during the design process are discussed not made or distributed for profit or commercial advantage and that copies in section 3. Both sections contribute to a set of require- bear this notice and the full citation on the first page. To copy otherwise, to ments for Smart Style. Section 4 states to what extent the republish, to post on servers or to redistribute to lists, requires prior specific requirements are met in terms of the current Web infras- permission by the authors. Semantic Web Workshop 2002 Hawaii, USA tructure, identifies gaps and gives suggestions for extending Copyright by the authors. the current Web infrastructure. 2. DESIGN PERSPECTIVES to knowledge-driven or model-driven sites (See for example, ON WEB PRESENTATIONS [12]). Furthermore, the current Web infrastructure, with its large number of XML-related tools, is well equipped to In this section we compare two different perspectives of support this process. creating a presentation: document engineering and graphic design. The former assumes that the authoring process can be broken down into a sequence of sub-processes which are able to operate independently to generate the end result. 2.2 Graphic Design Perspective The latter assumes a content-provider with a message to be Despite the advantages of the document engineering ap- communicated to a target audience, both of which the de- proach, it also has significant limitations. Specifically, in our signer has to understand exactly before creating the appro- own work on automatically adapting multimedia presenta- priate mix of graphics and text to effectively communicate tions to a variety of delivery contexts, generic XML tools the client’s message. Both perspectives are valid and need to proved to be inadequate (see [21] for details). Current tools be understood before distilling the requirements for a Smart are unable to deal with multimedia content for which it is Style layer. not known a priori which transformation and stylesheet are suitable for displaying the content in a particular context. In 2.1 Document Engineering Perspective online multimedia databases, for example, multimedia pre- From a document engineering perspective, it is important sentations can be generated from the media items returned to separate content from style information. The underly- by a database query. Since information about the media ing principle is that the essence of the message is contained items such as quantity, type, size and size is not known in in the (XML-structured) text and remains unchanged when advance, template-based solutions cannot be used for de- style parameters, such as screen width or font size, are var- termining a suitable presentation structure. Several solu- ied. This principle allows the creation of an infrastructure tions for this problem have been proposed and include: the where the file containing the content, the XML file, can be use of large numbers of templates, where selection of the created and maintained separately from a style file, such as correct template becomes a problem [9]; constraint-based a CSS stylesheet. The advantages of this approach are well- approaches, using grammars [31], planners [1] or logic pro- known within the Web community. These include the reuse gramming [21] to generate the constraints; and other model- of the same content in different contexts and the enforce- driven approaches to automatic presentation generation [3, ment of consistent styles across different sets of content [20]. 13]. A presentation, however, involves more than applying an Presentation structure plays a much more important role appropriate style to the selected content. A third, and es- in multimedia than in text-based applications. Multime- sential, ingredient is the structure of the presentation. The dia users experience presentation structure primarily as the simple separation of content and style as described above sequential arrangement of the constituent media items in suffices only when the presentation structure is similar to time, as the spatial arrangement on the screen and as paths the content structure in the underlying XML. If this is not of navigational hyperlinks. The presentation structure of the case, then a transformation step, such as enabled by multimedia is more difficult to determine automatically by XSLT, is needed to convert the content structure to the de- stylesheets. For text, stylesheets may change the layout sired presentation structure. For example, the lexical order (e.g. switch from single column to two column, or change in a source HTML document might need to be transformed the margins) while preserving the semantic integrity of the to the order that is most appropriate in the text-flow of the underlying message. For multimedia, changes in the spatio- target HTML presentation. Alternatively, a more structural temporal arrangement will often have a large impact on the process may be needed, such as a transformation of an XML perceived semantics of the presentation [22]. Multimedia document into an XSL formatting object tree. formats such as SMIL [29] address this problem by allowing The document engineering process of creating Web pre- the author to specify the presentation structure explicitly in sentations can be summarized in three steps: the document. This is required because in multimedia the message is conveyed not only by the individual media items 1. select or create the content (typically structured using but also by the spatio-temporal and navigational arrange- XML); ments of the media items in the presentation. In multimedia the presentation structure and content are in general not in- 2. define a mapping from content to the presentation dependent. structure that defines, among other things, the most The document engineering approach thus needs to be re- appropriate order (e.g. by using XSLT); fined for media-centric applications, in which the assump- tion that content, presentation structure and style are inde- 3. (optionally) refine this presentation structure by ap- pendent is false. In contrast to the content-centric approach plying preferred style parameters (e.g. by using CSS). in most of the document engineering literature, most of the Essential in this approach is the assumption that the three graphic design literature features a more balanced perspec- steps can be carried out independently. Content can be en- tive on the relation between content, presentation structure tered into a database by a content-provider. This content and style and the roles these three ingredients all play in can then be extracted from the database in the desired order conveying the overall message of the presentation. Under- by a server-side script written by a Web-site programmer. standing these roles and their dependencies is crucial for Finally, the preferred style parameters can be determined determining the requirements of a Smart Style layer. (server-side) by a graphic designer’s and/or (client-side) by Figure 1 illustrates how decisions made in any of the three the end-user’s stylesheet. For many (database) content- sections can influence the other two. We give examples of driven Web sites, this assumption holds. The same applies how each part of the figure influences the other two parts. e n ur io ct tat Grouping determines layout style ru n St st ese yl Layout aesthetics e Pr determine grouping Top−level order/grouping Overall style determines affects local arrangements local style of media items ite by ms Gr se ong Gr nes ia dia ed ou m m mi med ou se item of me in am pin an ed ed erm pin le dia tio nes ite n of g d tic r ia i ms g d ctio s lct et me lec mi ep ela tem se yle d ete n ter en tio s de r− e St ds ns s le on Sty Content Figure 1: Dependencies between content, presentation structure, and style. Presentation structure depends on content. The partic- ture among sections, is often determined first and the con- ular selection of content items can be used to determine the tent created to fit into it. It is difficult, however, to make presentation structure of the items. For example, suppose these dependencies explicit. that a number of media items have been selected for presen- tation. The items fall into three categories, and the presen- Style depends on presentation structure. The style can tation structure reflects these categories by first displaying also depend on the presentation structure. For example, if all items in a single category before displaying the items the presentation structure uses spatial alignment for convey- from another category. For printed graphics, Williams [32] ing grouping relationships in the content, then the designer advocates the use of spatial layout to express grouping re- needs to choose a particular alignment style (e.g. left-, cen- lationships in the underlying content. In multimedia time, tered or right-aligned). In figure 2, for example, the title is space and links can all be used to communicate the under- centered above the text to convey the grouping relation, in lying grouping relationships [18]. this case that the title applies both to the explanatory text and to the example image. Note that the same presenta- Style depends on content. The content can also influence tion structure could have been conveyed using a longer title the overall style of the presentation. For example, suppose a aligned left with the chiaroscuro explanation. number of images are selected to convey the message. They happen to share a number of color characteristics, which Content depends on style. In the document engineering lead to the choice of particular colors for the background perspective, style is often perceived as the “add-on” after and main text colors of the presentation. Other aspects, the “important” decisions have been made. The “more fun- such as image texture, could, for example, also influence the damental” choice of content may, however, also depend on selection of appropriate font type faces. Availability of ap- the style. For example, in order to preserve the visual unity propriate content may influence the choice of style through, in a presentation, relevant images may be selected for pre- e.g., using an image as a background for the rest of the pre- sentation only if their color histograms or clarity of images sentation. fit in with the style of the presentation as a whole [17]. For Web-sites aiming for a strong visual effect (e.g. for branding Content depends on presentation structure. In the doc- purposes), the look-and-feel of the site is typically designed ument engineering perspective, presentation structure is de- first. The content is selected, manipulated or created to fit rived from the original content structure. In practice, how- in with the chosen style. ever, when a Web site is created, the presentation struc- ture, in particular the spatial layout and navigation struc- Presentation structure depends on style. Similarly, an the user’s personal preferences (e.g. user could prefer visual established style may prescribe certain limits on the spatial to textual information, dislike fast cuts in video material, grid and pacing of the presentation. Ideal groupings and prefer soft colors to primary colors). orderings of selected content may have to be put aside for Given a good understanding of the type of forces that play reasonable alternatives which fit in with consistent use of a role, it is the task of the designer to come up with a design margin widths and item alignments throughout the presen- that best matches the needs of the content provider and the tation. Similar effects and tensions are possible for temporal user. In addition to the forces originating from the content- layout. A rhythmic presentation gives a certain desired ef- provider and the user, there are additional forces originat- fect but may clash with specific durations needed to express ing from the designer, whose resources are also limited and the message at different points in the presentation. might also have personal preferences. Many of these forces could give rise to conflicts and will require the designer to . In addition to the mutual dependencies between each of make balanced trade-offs. For example, the designer might the three aspects, local presentation structure or style can decide not to use the soft colors of the organization’s com- depend on more global presentation structure or style. This pany logo for users that need to fill in Web-forms while work- can be used to provide continuity and consistency through- ing in bad lighting conditions. out the presentation. The role of an intelligent automatic adaptation mecha- In summary, in the graphic design perspective of creat- nism is very similar to that of the human designer. Au- ing a presentation, aspects of content, presentation struc- tomatic adaptation also has to deal with forces originat- ture and style depend on each other in ways that are gener- ing from content-provider and user, as well as with forces ally ignored in the document engineering perspective. This originating from the adaptation process itself (e.g. limited is not to say that document engineering tools are not use- computing resources, or personal preferences of the devel- ful, but rather that the extra dependencies which make the oper of the adaptation system). For example, in figure 2 task of good design so complex require more complex so- the preferred design centers the title across the width of the lutions. Since automated adaptation requires finding solu- screen. In figure 3, however, the client’s platform display is tions within this design space, the three aspects of content, shorter and the images are not to be scaled — a condition presentation structure and style need to be expressed and imposed by the copyright holder to preserve image quality. manipulated explicitly. This forced our Cuypers system to search for an accept- Being aware of the internal mutual dependencies is neces- able layout design alternative within the given device and sary but insufficient for finding a satisfactory solution in the content-provider constraints. We do not claim that we can design space. The adaptation also needs to fulfill external build an automated system that could make such decisions requirements of, e.g., the user and content provider. In the as well as a professional designer. Intelligent adaptation sys- following section we discuss potential external influences on tems such as Cuypers can, however, make acceptable design the design process. decisions when dealing with these types of trade-offs. Their intelligence is based on explicit knowledge about the design space dependencies and external constraints, combined with 3. EXTERNAL FORCES an adequate search strategy. These characteristics require ON THE DESIGN PROCESS that adaptation be more than the application of a simple mapping from source to destination format. Rather, it re- The previous section explains the mutual dependencies quires heuristic reasoning to find an optimal solution to bal- that play a role when making decisions about content, pre- ance the forces involved. sentation structure and style. So far, we have limited the The requirements for automated adaptation following from discussion to the dependencies that are internal to the pro- the discussions here and in section 2 can be stated as follows. cess of designing a Web presentation. In this section we dis- cuss the dependency of design decisions on external forces. • Explicit knowledge reflecting the internal design de- The external forces that influence the design originate di- pendencies among content, presentation structure and rectly from the different interests of the parties involved. style discussed in section 2, and external influences To determine the requirements of an automated system, we of the delivery context specifying the user’s resources, use the following motivating example, based on a typical preferences and needs, and the content provider’s server scenario with three main parties: a content-provider who context. wishes to effectively communicate a message to a user, aided by a skilled designer. • A transformation method which can take the above Examples of forces that originate from the content provider knowledge into account, to make an informed choice in include the mission of the content provider’s organization the internal design space while balancing the external (e.g. making profit by selling books online), the limited avail- trade-offs. Note that this transformation method will ability of resources (e.g. the amount of time and money the not be based on simple mappings. organization is willing to spend on the design, the amount of disk space or bandwidth that is available at the server), and In the next section we discuss the implications of these re- the content provider’s preferences (e.g. the use of company quirements for extending the current Web infrastructure. colors in the Web forms). Examples of forces that originate from the user include the user’s needs (e.g. the desire to buy a book), the limitations 4. TOWARDS INTELLIGENT STYLESHEETS imposed by the user’s delivery context (e.g. the user could The type of adaptation that can be found on the Web be driving a car, have a low bandwidth connection, have today may seem to be a far cry from the type of intelligent physical disabilities, or have strict time constraints), and adaptation discussed in the previous sections. To a certain Figure 2: Example SMIL presentation (generated by Cuypers for display on large screen). extent, however, the current Web infrastructure already pro- automatic adaptation. A good example is the work on auto- vides a good basis upon which a smarter adaptation layer matic linearization of SVG documents, to allow synthesized could be built. Making this layer work in practice, however, speech browsing of SVG [16, 11]. requires the specification of new standards, and — arguably The appropriate use of the techniques sketched above al- more difficult — a sophisticated and seamless embedding of ready requires human designers to deal with the design de- these new techniques in the current Web framework. pendencies discussed in section 2 and trade-offs discussed in In this section, we give a short overview of how current section 3. For example, multimedia authors have to make Web standards relate to automatic adaptation. We then trade-offs between function and resources for each switch discuss the requirements of a Smart Style layer illustrated by element in their SMIL presentations, because they need to examples of potential extensions of current Web technology. match the functionality of their media items against the We use the current Web Recommendations as a basis, and available bandwidth and others resources that are required incorporate, as much as possible, other W3C activity that by these media. The trade-offs between function and pref- is still work in progress. erences in the company color scheme example are similar to the decisions made when using CSS !important rules, where designers need to think to what extent the prefer- 4.1 Current Web adaptation techniques ences of the end-user’s stylesheet are allowed to override the To a certain extent, a Web site can already build its defaults determined in the document’s stylesheet. own server-side adaptation techniques by deploying generic These current techniques suffice for applications in which Web technology, such as CSS and XSLT stylesheets, which a human designer has, during authoring time, sufficient in- can be used to adapt and style XML and HTML content. formation for making the required design decisions and trade- These techniques can be combined with commonly used so- offs. For applications for which this information is only avail- lutions such as filling Web templates with material stored able at the time of the user request, the decisions need to be in databases. made by the adaptation system. The current Web infras- Today’s Web formats also allow client-side adaptation. tructure is, however, insufficient for making run-time design Many delivery formats, including HTML, feature basic func- decisions. In order to move towards the intelligent adapta- tionality to improve accessibility. A well-known example is tion and styling advocated in this paper, we need to extend the alt tag that can be used to provide an alternative, tex- the current Web framework. tual description of the role of an image within an HTML page. SMIL features a more sophisticated example in the switch element, which can be used by multimedia authors to 4.2 Communicating delivery contexts provide alternatives for parts of the presentation depending The first ingredient we need is a commonly agreed upon on the delivery context. The XSL vocabulary [28] includes way to communicate the information upon which we will features that allow similar client-side adaptation, including base our design decisions. A key requirement is the ability the role property and the multi-switch formatting object. to communicate delivery contexts. Delivery contexts are re- The use of metadata also has a huge potential for improving quired in order to provide information about the client-side Figure 3: Same SMIL presentation, but adapted to the smaller height of the user’s display. resources that are available and about the personal pref- ... erences of the user. Assuming that at least a part of the adaptation will need to take place on the server, it is essen- tial to standardize the communication of delivery contexts: clients need to be able to send the information in a way that 640 the server understands. A machine-readable description of 480 a delivery context that can be sent to the server is often ... called a profile. Within W3C, work on a common ground for delivery contexts is currently in progress. CC/PP [26] ... provides an RDF-based framework for defining the vocab- ularies that are needed to define profiles. In addition, it also provides a small vocabulary that can be reused across different profiles. A typical example of a CC/PP profile is Figure 4: Example fragment of a delivery context the User Agent Profile developed by the WAP Forum [33]. specified using CC/PP. This profile provides a commonly agreed upon mechanism to communicate the (technical) capabilities of mobile phones to servers and proxies. The CC/PP framework, however, is 4.3 Supporting metadata sufficiently flexible to allow the definition of profiles that for content description focus on more user-centered aspects of a delivery context. Clients need to be able to communicate delivery contexts, From a technical point of view, CC/PP is built on top but in itself this is insufficient. Many design decisions will of RDF. CC/PP profiles use RDF statements to describe also depend on information that is available at the server- the relevant client-side capabilities and preferences. For ex- side. Even when this information is not intended to be pub- ample, figure 4 shows a fragment of a delivery context that lished on the Web, having commonly used and standardized uses CC/PP to inform the server that the client platform solutions for describing and processing it will greatly reduce features a 640x480 display. the development effort needed to implement a smart, adap- CC/PP profiles are, at the time of writing, hardly used tive Web site. on the Web (the WAP industry forms a notable exception). Intelligent adaptation systems will need some knowledge Communicating delivery contexts between client and server of the function of the content they are adapting. To make needs to become standard practice, which is more than an this type of knowledge explicit, appropriate use of meta- implementation issue. Additional CC/PP vocabularies need data will be of key importance. Within and outside W3C, to be provided, not only to describe the capabilities of the a large amount of work on metadata standardization is cur- hardware and software of the user’s device, but also to de- rently in progress, and in most of this work RDF plays a scribe the needs, environment and personal preferences of central role. For example, work on RDF Schema aims at the user. adding functionality that allows RDF vocabularies to be de- fined in a standardized way. Ontology languages, such as DAML+OIL, built on top of RDFS, add features while still allowing efficient implementations that are able to reason about metadata information. is using the technique “Chiaroscuro”, an adaptation engine While the current focus of this type of Semantic Web tech- might decide to add, for non-expert users, a link to the page nology is on the use of metadata to achieve a more intelligent describing this technique. This requires an adaptation pro- model for Web-based information retrieval (e.g. improving cess that takes into account both the delivery context (be- search engines), the use of metadata in our Cuypers system cause it needs to know that the user is a non-expert) and shows that there is also a huge potential in applying this type metadata (because it needs to know in which conditions it of technology for improving the adaptation and presentation should add a link). Based on our experience with Cuypers, process. Through the use of metadata to make the intended we found that most metadata is used for content descriptions semantics and function of the content explicit, adaptation that are defined in terms of the application domain. This systems are able to make informed decisions during the de- may be sufficient for most information retrieval purposes, sign process. For example, suppose an online museum site but not for information presentation. Metadata that, for has developed an RDF Schema1 for the metadata2 used to example, identifies the potential role the content could play annotate their Web site. Also suppose the site features an in the presentation is hard to find. In the example above it HTML page describing a work by the painter Rembrandt was hard to predict on the basis of the metadata whether van Rijn, focusing on the use of chiaroscuro (the painting the textual description of the “Chiaroscuro” technique is technique that uses strong contrasts of light and dark paint- suitable for non-expert users or not. For the images, it was ings). Figure 5 shows a fragment of the HTML version of hard to determine to what extent images could be resized to the earlier SMIL presentation. fit the presentation without compromising the information that was intended to be conveyed. In general, to improve intelligent adaptation and presen-

tation, metadata annotations of Web content is required.

Musical Allegory

Annotation should, however, not be confined to information

retrieval, but also facilitate information presentation.

This is hardly just an ordinary group of musicians. The figures are too exotically dressed in oriental 4.4 Processing delivery contexts ... and content descriptions

Assuming that the information upon which we base our design decisions will be available from the Web through the Figure 5: Example XHTML 1.0 fragment from a use of standard Semantic Web technologies such as CC/PP page about a Rembrandt painting. and RDF, the next ingredient needed for building a Smart Style layer is an efficient set of tools that allows this in- From an XML markup perspective, all we know is that we formation to be taken into account during the adaptation have a fragment with a first level heading, an image and a process. As described above, many of the current gener- text paragraph. The underlying semantics, however, could ation W3C Recommendations already have some features be explicitly added by the use of RDF metadata, as shown that address adaptation issues. A first step is to make the in figure 6. current generation presentation-oriented Web technology in- teroperable with the next-generation Semantic Web technol- ogy. For example, CSS stylesheets are currently not able to take CC/PP profiles into account. CSS has, however, a fea- Rembrandt ture that is closely related to CC/PP, and allows the spec- Harmenszoon van Rijn ification of device dependent style rules: the @media rule. Figure 7 shows an example3 of a stylesheet that uses bigger fonts on computer screens than on paper printouts of the same document. Musical Allegory Chiaroscuro @media print { body { font-size: 10pt } } @media screen { Figure 6: RDF metadata of XHTML 1.0 fragment. body { font-size: 12pt } } This explicitly states that our HTML fragment is an in- stance of a class Painting, with a title property “Musical Allegory”, and that there is a Painter instance that has a Figure 7: Device dependent style rules as already painted relation with the painting at hand. supported in CSS2. Given such semantic information about the content, and the explicit descriptions of the delivery context, adaptation A first step towards a CSS syntax that allows more de- engines should be able to make better decisions about how to tailed queries is suggested in [15]. In this syntax, queries to adapt the presentation to a particular situation. For exam- specific device features are allowed. For example, the CSS ple, because the metadata explicitly states that the painting media rule for screen display above could be further refined 1 by adding constraints on the minimum width of the screen, Museum schema example adapted from [14]. 2 3 Metadata example adapted from [23]). Example taken from the CSS2 Specification [4]. as shown in figure 8. Using the constraints, stylesheets could Using the current generation CSS and XSLT engines to pro- take into account the information provided by profiles such cess general metadata it is, however, not practical to match as the example in figure 4. on the semantic properties of metadata: for CSS and XSLT processors, RDF is just XML. As a result, it is very hard to write, for example, a rule that matches on all alterna- @media screen and (min-width: 640px) { tive XML serializations that are allowed for RDF. A more body { font-size: 14pt } serious problem, however, is that it is impossible to write } CSS or XSLT rules that make use of the structural rela- @media screen and (min-width: 800px) { tions of RDF and RDF Schema, for instance a style rule body { font-size: 16pt } that applies to all objects that are instances of a specific } RDFS (sub)class. Neither is it possible to write rules for all objects that have a certain DAML+OIL-defined ontological relation, etc. Model-driven Web site management systems Figure 8: Detailed media queries using a CSS3 ex- such as OntoWebber [12] are thus forced to develop their tension (work in progress). own solutions to associate presentation design elements to their RDF (and DAML+OIL) data, because CSS and XSLT Even from this extended CSS syntax, however, it is still a are currently not applicable to RDF. long way to fully CC/PP aware style engines. CC/PP fea- Future, Semantic Web-aware, selector mechanisms could tures that will affect style application include the ability to allow specification of style rules in terms of the RDF seman- define new profile vocabularies, inheritance mechanisms for tics expressed in the metadata. This would extend the cur- specifying default values and the description of the capabil- rently used CSS and XPath selectors, that are based on the ities of transcoding proxies. Style engines need to be able XML syntax encoding the semantics. Consider the extended to deal with these features in order to take full advantage of XSLT example rule in figure 10, which uses the RDF-aware the information specified in CC/PP delivery contexts. query language RQL [14] for its selector, instead of XPath. Note that the need to take CC/PP information into ac- count also applies to XSLT transformation engines. While the full details of how this could affect future versions of ample, imagine an extension4 of XSLT’s mode concept. For ... example, transformation rules could be selected in a way similar to that of the media rules in CSS. In such a hy- pothetical extension (see figure 9) one could, for instance, define a rule for creating a two column layout only if the Figure 10: Semantic matching of XSLT rules using output medium is print and the paper is wider than 17cm. RQL selectors (tentative syntax). Artifact and its subclasses. Since our Museum RDF Schema ... defines Painting as a subclass of Artifact, the rule above would match on the semantics and structure of the RDF ... metadata describing the painting shown in figure 6, irrespec- tive of the XML serialization syntax used to encode these semantics [5]. Figure 9: Device dependent rules by extending 4.5 Beyond CSS and XSLT style XSLT modes (tentative syntax). and transformation rules Above, we suggested extensions to CSS and XSLT that In addition to taking information about delivery contexts would allow stylesheets to take into account delivery con- into account, stylesheets also need to take into account the texts as specified by CC/PP and content semantics as ex- semantic information that is contained in the metadata as- pressed by RDF metadata. While taking this type informa- sociated with the content. Currently, style selector mech- tion into account is a prerequisite for a Smart Style layer, anisms only match on the syntactic properties of the un- this is in itself not sufficient. derlying (XML) document hierarchy. This applies both to Adaptation engines need to be able to search in the de- the selector mechanism used by CSS and to the XPath [8] sign space sketched in section 2, and make the trade-offs selectors used by XSLT. discussed in section 3. This type of decision process is hard In all examples above, the rules were intended to match on to define using the simple “if selector matches then apply the element of an HTML document. Similar rules rule body” type of current style and transformation rules. could be written to match on the syntactic properties of In addition, our experiments with the Cuypers system [21] metadata, e.g. on the XML element and attribute names allowed us to analyze the adaptation process of multimedia that are used to encode the RDF statements in figure 6. presentations for which the quantity, type and size of the me- dia items were not known until run-time. We found that for 4 We are not advocating a specific syntax, but are only claim- these applications, automatic adaptation also requires the ing that future XSLT transformations need to be able to take ability to verify the presentations that result from applying CC/PP-like information into account a set of transformation rules. When designing transforma- tion rules for dynamic multimedia, one cannot, at author- cient for dealing with a limited number of delivery contexts ing time, guarantee that the resulting presentation indeed per stylesheet but is inadequate for adapting content to the meets the “hard” constraints imposed by the available re- plethora of delivery contexts for different devices, network sources. We have used the Cuypers system to experiment resources and user groups. To solve this problem the Web with a transformation engine that can evaluate the multi- is currently missing three key ingredients. media presentations it generates. The system employs back- tracking to search for alternative rules when the end result 1. Common vocabularies for describing delivery does not meet the constraints imposed by the available re- contexts Web applications need to be able to com- sources. For example, even when a specific rule is applied municate their capabilities and the preferences of their only for target screens with a certain width, that condition users so that transformation engines are able to make in itself will not guarantee that the presentation resulting informed choices during the presentation generation from applying the rule to media content of unknown size or transcoding process. CC/PP already provides a will indeed meet the maximum width constraints. What is framework for defining such vocabularies. Commonly needed is a means of evaluating the actual width of the final agreed upon vocabularies will be needed for defining presentation, and a means of trying alternative rules in case user preferences, device capabilities, network charac- the presentation did not meet the constraints. teristics etc. While CSS and XSLT rules cannot be used to specify the 2. Intelligent transformation methods Transforma- required search strategies, this type of processing is vital for tions need to be able to take into account a wide variety intelligent adaptive behavior on the Web. The Web thus of delivery contexts to generate a presentation corre- requires more sophisticated ways of transforming the com- sponding to a particular delivery context. While it is bined information provided by delivery contexts, metadata unrealistic to expect that even an intelligent stylesheet and the content into meaningful presentations. In future would be sufficiently powerful to cater for any given research, we want to explore how, and to what extent the delivery context, our claim is that the current trans- combined search, transformation and evaluation techniques formation technologies can be significantly improved used within our Cuypers system could be made generally in order to allow a substantial increase in flexibility. available on the Web. 3. Explicit metadata and design knowledge Given 4.6 Beyond atomic style properties the vocabularies for describing delivery contexts, and In addition to improved transformation processes, we also given an appropriate transformation method, in the- need to develop better abstractions to reason about the ory we would be able to develop adequate intelligent “soft” constraints imposed by the preferences of the parties stylesheets. In practice, however, these stylesheets involved. This type of reasoning requires explicit knowledge would implicitly contain a large amount of design knowl- of the dependencies discussed in section 2. Taking these edge and domain knowledge. This type of knowledge preferences and the associated dependencies into account should preferably be made explicit and specified declar- will have a large impact on the perceived overall quality atively, in a similar manner to the explicit and declar- and design of automatic Web presentations. Currently, style ative delivery contexts. RDF Schema [25] and rules work only on the basis of individual style properties. DAML+OIL [19] already provide a framework for en- For example, one can specify the font type or color of a spe- coding this type of knowledge. To what extent vocab- cific XML element. To what extent the application of these ularies for this type of knowledge can be standardized individual rules yield the desired overall result is hard to remains to be seen, since they may be highly domain predict in advance, especially when dealing with more com- and application specific. plex publishing systems that feature dynamic content, XSLT As these three ingredients build directly upon Semantic Web transformations, transcoding proxies and CSS stylesheets. technology, we believe that only by a synthesis of (future) After this process, the font and color of two XML elements Semantic Web tools with the presentation-oriented tools of positioned together in the final presentation might not go the (current) Web, can we hope to address the adaptation well together. Within the graphic design profession, style problems discussed. guidelines and checklists have been developed that can be This brings us to the first Achilles’ heel of our Smart Style used to avoid such design mistakes (see, for example, [32, layer: the large amount of current and future W3C Recom- 17]). It should be possible to build on this body of knowl- mendations that currently exist. Many of the Recommen- edge, and at least check the overall presentation against the dations can be used to address part of the problem, but it most common design flaws. In addition to graphic design, is not clear how they can be used in concert to solve the similar checks could be developed for checking the design of overall problem. This paper derives the requirements for an the overall temporal flow of, and synchronization within, the ambitious goal: automatic adaptation of dynamic text and presentation [2], and for checking the design of the naviga- multimedia content to the requirements of the individual tion and interaction schemes that the presentation exposes user’s delivery context, while respecting the integrity of the to the end user. semantics of the content. If we reduce our ambition levels, however, and “only” aim for taking into account process- 5. CONCLUSIONS ing context information, this alone would still have major Current Web technology addresses the problem of mul- consequences. To prevent CC/PP from becoming a stand- tiple delivery contexts through the use of CSS and XSLT alone W3C recommendation that can only be processed with stylesheets. These can be used for transforming presenta- proprietary tools, we need to clearly define how other rec- tion-independent XML documents to specific presentation ommendations, including CSS, XSLT, XHTML, SMIL and formats, such as XHTML, SVG or SMIL. This is suffi- SVG operate in the context of CC/PP. From CC/PP-aware Web transformations, another step is required towards Se- [4] Bert Bos, Håkon Wium Lie, Chris Lilley, and Ian mantic Web-aware transformations that also take metadata Jacobs. Cascading Style Sheets, level 2 CSS2 semantics into account. This will require tools that can ab- Specification. W3C Recommendations are available at stract from the underlying XML syntax and operate directly http://www.w3.org/TR, May 12, 1998. on the semantics of languages such as RDF, RDFS and [5] J. Broekstra, A. Kampman, and F. van Harmelen. DAML+OIL. Realizing this level of interoperability among Sesame: An Architecture for Storing and Querying W3C Recommendations will be a huge effort. It should be RDF Data and Schema Information. In D. Fensel, clear that the examples given in this paper serve only to il- J. Hendler, H. Lieberman, and W. Wahlster, editors, lustrate the derived requirements, and should by no means Semantics for the WWW. MIT Press, 2001. be regarded as readily applicable solutions to achieve the re- [6] D.C.A. Bulterman, L. Rutledge, L. Hardman, and quired interoperability. Making the current Web infrastruc- J. van Ossenbruggen. Supporting Adaptive and ture interoperate seamlessly with the upcoming Semantic Adaptable Hypermedia Presentation Semantics. In Web will be a huge challenge and a long term effort. The 8th IFIP 2.6 Working Conference on Database Finally, the other Achilles’ heel of our Smart Style layer is Semantics (DS-8): Semantic Issues in Multimedia the large amount of high quality design and domain knowl- Systems, Rotorua, New Zealand, 5-8 January 1999, edge that it requires. Smart Style does not aim at replacing 1999. human designers, but strives for providing applications with [7] James Clark. XSL Transformations (XSLT) Version sufficient design knowledge when design decisions cannot be 1.0. W3C Recommendations are available at make by humans. It will require a large amount of human http://www.w3.org/TR/, 16 November 1999. effort to make this knowledge explicit and it will require even [8] James Clark and Steve DeRose. XML Path Language more work to maintain it and keep it up to date. Given the (XPath) Version 1.0. W3C Recommendations are problems most authors already have when they are forced to available at http://www.w3.org/TR/, 16 November move from the “what you see is what you get” paradigm of 1999. desktop publishing to the “structured document” paradigm [9] Isabel F. Cruz, Kimberly M. James, and David C. of XML-based Web publishing, this will not be an easy job. Brown. Integrating Layout into Multimedia Data Having said this, we should also realize that we do not have Retrieval. In AAAI Fall Symposium on Using Layout to build it overnight: just as the current Web, we can build for the Generation, Understanding, or Retrieval of the Semantic Web with its Smart Style layer incrementally, Documents, Cape Cod, Massachusetts, 1999. by building new layers on the XML and RDF-based frame- work that is ready to be used now. Content-providers will [10] Jon Ferraiolo. Scalable Vector Graphics (SVG) 1.0 start to use these new layers as soon as there are sufficiently Specification. W3C Recommendations are available at large economic (e.g. attracting more customers by making http://www.w3.org/TR/, 4 September 2001. their site accessible from new mobile devices) or legal (e.g. [11] Ivan Herman. Follow up on SVG Linearization and laws that require sites — including multimedia content — Accessibility. Work in progress. Document available at to be accessible for users with disabilities) incentives. http://www.w3.org/2001/svgRdf/, 19 July 2001. [12] Yuhui Jin, Stefan Decker, and Gio Wiederhold. Acknowledgments OntoWebber: Model-Driven Ontology-Based Web Many of the issues discussed were originally inspired by the Site Management. In Semantic Web Working W3C workshop on Device Independent Authoring in Bris- Symposium (SWWS), Stanford University, California, tol, October 2000. The research has been funded by the USA, July 30 – August 1, 2001. European ITEA/RTIPA and the Dutch Dynamo and To- [13] Th. Kamps. Diagram Design : A Constructive Theory. KeN2000 projects. Examples are taken from a ToKeN2000 Springer Verlag, 1999. demonstrator, and all media content has been kindly pro- [14] Gregory Karvounarakis, Vassilis Christophides, vided by the Rijksmuseum in Amsterdam. Frank Cornelis- Dimitris Plexousakis, and Sofia Alexaki. Querying sen, Patrick Schmitz, Jeen Broekstra and Frank van Harme- Community Web Portals. len, and our CWI colleagues Joost Geurts, Oscar Rosell and http://www.ics.forth.gr/proj/isst/RDF/RQL/rql.html. Lloyd Rutledge provided many valuable insights. [15] Håkon Wium Lie and Tantek Ĉelik. Media queries. Work in progress. W3C Working Drafts are available 6. REFERENCES at http://www.w3.org/TR, 17 March 2001. [1] Elisabeth André, Jochen Müller, and Thomas Rist. [16] Guillaume Lovet. SVG Linearizer tools. Internship WIP/PPP: Knowledge-Based Methods for Fully report and software download. Work in progress. Automated Multimedia Authoring. London, UK, 1996. Document available at [2] Frederic Bes, Muriel Jourdan, and Farid Khantache. A http://www.w3.org/WAI/ER/ASVG/, 25 August Generic Architecture for Automated Construction of 2000. Multimedia Presentations. In Proceedings of the [17] Mark Oldach. Creativity for Graphic Designers. Narth International Conference on Multimedia Modeling Light Books, Cincinnati, Ohio, 1995. 2001 (MMM01), pages 229–246, CWI, Amsterdam, [18] Lloyd Rutledge, Jacco van Ossenbruggen, Lynda The Netherlands, November 5-7, 2001. Hardman, and Dick C.A. Bulterman. Structural [3] M. Bordegoni, G. Faconti, M.T. Maybury, T. Rist, Distinctions Between Hypermedia Storage and S. Ruggieri, P. Trahanias, and M. Wilson. A Standard Presentation. In Proceedings of ACM Multimedia, Reference Model for Intelligent Multimedia pages 145–150. ACM Press, November 1998. Presentation Systems. Computer Standards & [19] Frank van Harmelen, Peter F. Patel-Schneider, and Interfaces, 18(6-7):477–496, December 1997. Ian Horrocks. Reference description of the 443–451, San Francisco, October 15 - 20, 1994. DAML+OIL (March 2001) ontology markup language. [32] Robin Williams. The Non-Designer’s Design Book. http://www.daml.org/2001/03/reference.html. Peachpit Press, 1994. Contributors: Tim Berners-Lee, Dan Brickley, Dan [33] Wireless Application Group. WAP-174: WAG Connolly, Mike Dean, Stefan Decker, Pat Hayes, Jeff UAPROF User Agent Profile Specification, 1999. Heflin, Jim Hendler, Ora Lassila, Deb McGuinness, Lynn Andrea Stein, ... [20] Jacco van Ossenbruggen. Processing Structured Hypermedia — A Matter of Style. PhD thesis, Vrije Universiteit, Amsterdam, The Netherlands, April 10, 2001. Also available on http://www.cwi.nl/∼jrvosse/thesis/. [21] Jacco van Ossenbruggen, Joost Geurts, Frank Cornelissen, Lloyd Rutledge, and Lynda Hardman. Towards Second and Third Generation Web-Based Multimedia. In The Tenth International World Wide Web Conference, pages 479–488, Hong Kong, May 1-5, 2001. IW3C2. [22] Jacco van Ossenbruggen, Lynda Hardman, and Lloyd Rutledge. Integrating Multimedia Characteristics in Web-based Document Languages. Technical Report INS-R0024, CWI, December 2000. [23] Jacco van Ossenbruggen, Lynda Hardman, and Lloyd Rutledge. Hypermedia and the Semantic web: A research agenda. Technical Report INS-R0105, CWI, 2001. [24] W3C. Resource Description Framework (RDF) Model and Syntax Specification. W3C Recommendations are available at http://www.w3.org/TR, February, 22, 1999. Editied by Ora Lassila and Ralph R. Swick. [25] W3C. Resource Description Framework (RDF) Schema Specification 1.0. W3C Candidate Recommendations are available at http://www.w3.org/TR, 27 March 2000. Edited by Dan Brickley and R.V. Guha. [26] W3C. Composite Capability/Preference Profiles (CC/PP): Structure and Vocabularies. Work in progress. W3C Working Drafts are available at http://www.w3.org/TR, 15 March 2001. Edited by Graham Klyne, Franklin Reynolds, Chris Woodrow and Hidetaka Ohto. [27] W3C. Device Independence Principles. Work in progress. W3C Working Drafts are available at http://www.w3.org/TR, 18 September 2001. Edited by Roger Gimson, co-edited by Shlomit Ritz Finkelstein, Stéphane Maes and Lalitha Suryanarayana. [28] W3C. Extensible Stylesheet Language (XSL) Version 1.0. W3C Recommendations are available at http://www.w3.org/TR/, 15 October 2001, 2001. [29] W3C. Synchronized Multimedia Integration Language (SMIL 2.0) Specification. W3C Recommendations are available at http://www.w3.org/TR/, August 7, 2001. Edited by Aaron Cohen. [30] W3C. XHTML 1.1 - Module-based XHTML. W3C Recommendations are available at http://www.w3.org/TR/, May 31, 2001. Edited by Murray Altheim and Shane McCarron. [31] L. Weitzman and Kent Wittenburg. Automatic presentation of multimedia documents using relational grammars. In Proceedings of the second ACM international conference on Multimedia ’94, pages