1We describe an approach and an environment for setting up Wizard of Oz experiments to test multimodal interaction (through mobile devices) for multimedia content delivery. This is based on a metamodel and a reference client-server architecture, whose implementation is discussed. The approach allows progressive refinement of the Wizard of Oz environment into a deployed application, through the use of a single metamodel, to be progressively specialized, and a service-oriented approach which allows the substitution of implemented services to simulated ones. A visual interface is provided for the definition of the wizard interface, of the metadata relating content delivery and possible interactions on it, and of some aspects of the client interaction.
The increasing capabilities of processing power and
memory and the high resolution displays of current mobile
devices, together with the expansion of available bandwidth
and powerful compression techniques, enable their users to
enjoy advanced multimedia experiences. On the other hand,
PDAs, Tablet PCs, and even smart-phones, are embodying
different types of sensors and interaction strategies,
including pen-based interaction, VoIP support, GPS, etc. These
also allow new methods of interaction, combined in a
multimodal way, to steer the usage of multimedia material.
Interaction with these devices differs from the traditional
desktop experience with multimedia content, as it occurs
in less protected environments affected by noise, unusual
postures, disruptions in availability of certain channels, e.g.
WiFi or GPS coverage [
Wizard of Oz (WOz) techniques [
While many WOz experiments are performed using ad hoc
machinery, we propose WOEB (Wizard of Oz Experiment
Builder) as a general framework for the construction of the
WOz logic and interface in terms of services. The
deployment of the final application can proceed through the
progressive refinement of the available services, and the
substitution of the WOz services with those running the actual
application. WOEB is based on a fragment of a metamodel for
multimodal adaptive multimedia information systems and
relies on services for the identification of the relevant
content on the server-side, to be transmitted to the client-side
on the mobile device, in the format which is most suitable
to the user preferences and the device capabilities. At this
stage of design, our approach focuses only on a subset of the
possible interaction modalities typically related to
interaction with hand-held devices; for instance we do not take into
account bimanual interaction. We have tested WOEB in a
scenario where a technician must be remotely instructed on
troubleshooting hardware. We have run two experiments:
in the first one, the system provides the technician with
some predefined graphic or textual content about the
hardware, and interaction is performed by selecting special areas
(hotspots or hyperlinks) in the content, along with oral
commands. In the second, in conjunction with speech inputs, 2D
physical labels are placed directly on hardware components
[
WOz experiments can be run on both high- and
lowfidelity prototypes. In [
The development of a complete multimodal system starting
from WOz experiments is described in [
Momento [
The Ozlab [
Bunt et al. [
The applications of the suggested environment are that
of adaptive hypermedia, for which several general models
are being proposed. In [
In WOEB, the construction of WOz modules logic and interface relies on a metamodel and a reference architecture. A typical WOz environment provides a set of tools organized in a Client-Server architecture. The Server side is the Wizard module, hosting a human operator who simulates the not yet implemented functionalities of the system. The Client side is the multimodal human-computer interface deployed on the mobile device.
All packages of the WOEB metamodel import a basic Model package, a fragment of the UML 2.1.1 metamodel presented in Figure 1, which allows the definition of (simple or derived) typed properties, paired with expressions for their evaluation. A model can contain different relationships between properties, and the presence of some properties can be constrained by others. Finally, the model must provide ways to update its composition, upon receiving some triggering event.
The WOEB environment is a set of tools for the rapid setup of WOz experiments. We identified here three main actors: the WOz experiment designer, the wizard and the subject of the experiments.
The WOz Builder Module offers a GUI for specification of XML-based descriptions of the WOz Server and Client components. An XML Interpreter processes the XML description to produce the Server and Client widgets and functionalities, according to the metamodel. The WOz Server Module represents the system used by the human wizard to lead the experiments. It allows the wizard to serve users requests with a simple and intuitive interface and integrates the data collection facilities. The server module also implements the server-side communication mechanisms for sending and receiving data from the WOz Client Module, which is the multimodal application running on a mobile device used by the subject of the experiment.
In the current implementation, the WOz Builder enables the definition of the information sources, which represent the way the Client displays content. As the interaction model accommodates the combination of speech input with triggering events, the WOz Builder has to present an interface for the definition of such events.
In the first experiment, WOEB is used for developing a multimodal mobile application where interaction is based on the selection (through a pointing device) of special areas in the information source, an image in our case. We do not describe the second experiment, due to lack of space. The designer loads the source (the image itself) in a dedicated panel of the WOz Builder and defines the TriggerBase as sensitive areas. In the current implementation, an area can be a hyperlink, if the information source is a hypertext, or a polygon for a graphical information source.
By selecting an interactive area, the designer associates a description of the relative content, which will be sent to the client following a user request. This involves the specification of attribute-value pairs describing the content’s behavior relative to the user’s delivery context. Such attributes can be used to guide the adaptation of content to be delivered to the mobile device (via a DISelect engine2 which represents the realization of the ComplexRequestManager in our implementation of the metamodel). For example content manifestation admits as values audio, text or graphics, while request context defines the type of user request a particular content is designed to serve, e.g. how or what. Different request context types can be defined at the time of WOz modules definition. The builder can automatically organize the layout of the WOz Server GUI, so that the buttons are distributed over columns labeled by the request context.
In Figure 3, the information source is an image of a computer mother-board on which a technician has to intervene supported by the mobile system. The CPU socket area has been defined as a hot spot. On the client-side, a user can interact with the system selecting it and asking ”what is this?” or ”how can I use this one?”. The designer has thus defined two request context types, called ”what” and ”how”(Figure 2http://www.w3.org/TR/cselection/. 3), and assigned two ”what” or ”how” label to relative content. This information can then be used by the DISelect engine to filter a subset of content, matching this attribute. The designer can also define general purpose content, not associated with any interactive area, in order to manage oral user interactions or unpredicted events. Once all the elements are specified within the WOz Builder, definitions (in terms of interface artifacts and services) are stored in an XML archive. This platform-independent description of the logic and interface of the WOz modules can be processed by customized engines to generate the effective WOz Server and Client components. In the current implementation the XMLInterpreter co-generates both the WOz Server and WOz Client interfaces and services. 5.1
This module was automatically generated by the XMLInterpreter, starting from the definition contained in the XML archive previously mentioned. Information sources are placed in a tabbed pane including an appropriate content viewer. For each information source a dedicated panel is created for content selection. Each panel contains buttons for each trigger base element in the information source (the motherboard image in Figure 4) that can be used by the human wizard to call the associated content handler (e.g. the DISelect Engine).
In order to simplify content selection by the human wizard, buttons are organized in columns labeled with the different request contexts. Contexts are used by the DISelect engine to preliminarily filter the content to be retrieved, depending on the mobile device. This feature can be disabled during WOz module definition: for example if some components are fully implemented. The WOz Server is designed to be a starting point to build a real adaptive multimodal system. A data communication interface, handling data exchange between client and server, is implemented as well as a Timestamp Agent tracing the temporal order of the events. To deploy the final system, only services that are currently embodied by the human wizard have to be implemented: for example an Adaptive Dialog Manager, or a module to manage the multimodal fusion of pointing and speech events through the Timestamp Agent. One can also replace existing modules with customized ones: as an example the DISelect Engine can be substituted by another implementation of the Content Handler interface, simply by redefining the process of attribute definition and the content description within the WOz Builder. 5.2
In a way similar to the Server Module, the WOz Client Module is automatically generated by the XMLInterpreter. This module is essentially a data viewer, which can display content sent by the server and manage user pointing and oral interaction. Pointing interaction is captured through the trigger base, while audio inputs are processed by a Speech Capture engine. A description of a triggered event is sent, while the captured speech is streamed directly to the Dialog Manager on the server (e.g. the human wizard).
We presented WOEB (Wizard of Oz Experiments Builder), a framework and environment for setting up of Wizard of Oz experiments by defining modules interface and services. WOEB encourages the development of multimodal systems for managing multimedia information through the progressive refinement and implementation of components and services. The adoption of a common metamodel for the definition of the Wizard of Oz experiments and the deployment of the final application favors the rapid identification of problem sources and the reuse of components and services for different application scenarios. The first tests with WOEB show its efficacy in rapid setting of experiments for exploring different interaction modes. Further initial experiments have been performed, only partially reported here due to lack of space.