Web mashups are becoming increasingly popular. A mashup is a Web application that combines data from more than one source into a single integrated form. Several mashup creation tools, so-called mashup makers, exist and many of them are advertised as easy to use tools for casual users. These tools target Web users without programming background and promise that creating a Web mashup is just a matter of a few mouse clicks. However, no profound usability evaluations have been performed to justify those claims. The purpose of this PhD research work is twofold. Firstly, we want to investigate the usability of these mashup makers and verify if they indeed satisfy their promise of being easy to use. Secondly, and also for the purpose of accomplishing the first goal, we will develop a usability framework that can be used to evaluate the usability of existing and future Web mashup makers. In this paper, we discus the research objectives of the PhD work, the methodology used, related work, as well as the results achieved so far.
Mashup originally referred to the practice in pop music (notably hip-hop) of producing a new song by mixing two or more existing pieces [19]. In computer technology, a mashup is a Web application that integrates data from more than one source. A well-known example is the use of cartographic data from Google Maps to add location information to some costumer's data, thereby creating a new service that was not originally provided by either source. The most common way to develop a mashup is by accessing content via a public interface or API. That data is made available by relevant Web protocols such as REST, RSS and Web services [20]. The data is extracted from the output of these APIs, and is then passed to the mashup site where the logic resides, it could be server-side (dynamic content aggregation) and/or client-side scripting or both of them. The application then is rendered graphically and transferred to the client's Web browser.
Several mashup creation tools, so-called mashup makers, exist and many of them are advertised as easy to use tools for casual users. Our research objective is to investigate the usability of mashup makers for casual users, i.e. users without programming background. We believe that this research is very important from two perspectives. The first perspective is to check if those mashup makers indeed fulfill their promises and meet the needs of casual users (first major objective of the PhD). The second perspective concerns usability necessities for mashup makers for casual users. Our investigations and studies may reveal a considerable amount of hindrances and difficulties that casual users are facing when using mashup makers. One of the results of our usability study could be a set of minimal usability necessities for those kinds of tools. Therefore, it is also our aim to develop a set of usability criteria that can be used to measure usability of existing and future mashup makers. Together with a number of benchmarks that can be used to evaluate the usability of mashup makers and usability evaluation procedures, this will form the Usability Framework that we aim for as second major objective of the PhD.
As we want to investigate the usability of mashup makers, we first should define what we mean by mashup usability. Usability, as defined by ISO9241 part11 [15], is "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use". Applying this definition of usability to mashup makers, we define mashup usability as the extent to which a mashup maker can be used by specified users (in our case casual users with no programming background) to achieve specific goals (in our case a web mashup maker) with effectiveness, efficiency and satisfaction in a specified context of use. In other word, mashup maker usability is about user aspects and measurements when learning, creating, developing, building, using and handling mashup creation tools. This should not be confused with web usability that usually refers to how well users can learn and use a Web site to achieve their goals.
The main contribution of our research work is the development of a Usability Framework for Web mashup makers for casual users. Challenges in this research can be summarized as follows. No usability evaluation framework for web mashup makers for casual users exists. Therefore, it is necessary to investigate which criteria should be considered for evaluating the usability of these tools. Next, the existing tools are very different in the type of mashups they allow to create. Therefore, it will not be obvious to compare them; a framework for this is also needed.
The work for this PhD is divided into 7 main steps. For each step, we will give the main objectives and the actions that we will take to reach the objectives.
Step 1. Objective: To obtain an overview of existing Web mashup makers, in order to discover the main issues related to Web mashup technology and to have a concrete understanding of the possibilities and limitations of Web mashup technology. Method: A literature study on Web mashup technologies, and reading related tutorials.
Step 2. Objective: To obtain an overview of mashup usability in order (1) to discover the main issues related to mashup usability and to obtain a good understanding of Web mashup usability; (2) to check related work in the context of measuring the usability of Web mashup technology. Method: Literature study on usability, Web usability and Web mashups usability.
Step 3. Objective: To obtain a deeper understanding of the usability issues related to mashup makers for casual users Method: Selection of some Web mashup technologies for casual users and performing experiments with them in order to get practical knowledge and experiences on how these tools should be used. Here we intend to perform the experiments ourselves.
Step 4. Objective: To define a set of mashup usability criteria, i.e. usability measurement factors that can be used to evaluate the usability of Web mashup makers for the target audience (casual users). Method: Critical analysis of the results of Step 2 and Step 3, and the identification of missing and/or irrelevant usability issues. Further investigation of the relationship between the usability criteria identified and the target audience.
Step 5. Objective: Development of an experimentation environment Method: Selection of a set of representative Web mashup makers to be used in the experiments; selection of a representative set of target users; preparation of the experiments that will be performed.
Step 6. Objective: To reach the first objective of the research: verifying the usability of existing mashup makers Method: Performing the usability experiments prepared in Step 5, analyzing the results by means of statistical methods, and summarizing the results.
Step 7. Objective: To reach the second objective of the research: development of a usability framework for mashup makers for casual users. Method: Reinvestigation of recent research to keep track of new developments; evaluation of the approach used in Step 6; collecting and resuming guidelines, criteria's and benchmarks for mashup usability into a coherent usability framework.
In this section we review some works related to usability of Web mashup makers. Our literature research and first investigations have found that there is no complete and comprehensive work about Web mashup usability at this moment.
Exploring Usability Guidelines for RIA [8]. In this master thesis, desktop usability guidelines and web usability guidelines have been used as a basis to create an outline of Rich Internet Application (RIA) usability guidelines. Most of their work is focused on a comparative study of general usability guidelines. In conclusion they only formulated some so-called start guidelines for developer in the field of RIA.
MIT Potluck Usability evaluation. Potluck [9] is a project that aims at the development of an easy to use tool to mashup data for casual users. They performed a usability evaluation study to ascertain whether people could learn how to use Potluck as well as to discover usability problems. We have learned a lot from their work and we will follow some of their notes related to the usability evaluation of mashups tools.
Intel MashMaker Usability evaluation. Intel MashMaker [5] is a web-based tool to create web mashups by browsing around, without needing to type, or plan in advance what you want to do. The research team of MashMaker has performed a usability evaluation of the tool following [7] and using the Cognitive Dimension of notations (CDs) framework [3]. This evaluation has helped us directing our intended study and experiments of usability of Web mashups tools.
Environments. An internal report by Oleg Beletski [2] contains a study of some Web mashup programming environments and compares usability basic aspects of those environments. The report summarizes the usability aspects of the compared Web mashup programming environments (tools) by simply mentioning whether they are easy to use or not. The author has not mentioned how he obtained these results. [21] is an end-user programming tool for mashups that repurposes and combines existing Web content and services. Marmite is targeting users with programming backgrounds and spreadsheet skills. They have also performed a usability evaluation study. This usability evaluation has helped us understanding a new way of evaluating mashup development tools.
An end-user perspective of Mashup makers. In this internal report [18], the authors reviewed six mashup makers from the so-called End User Development perspective. There reviewing methodology is a mix between some selected dimensions of the CD´s framework [3], software engineering techniques and some concepts related to e-learning. We have tried to fairly examine their report but we found it undetermined and in our opinion, it lacks specific usability review points.
Compared to this work, our research focus on a complete and robust methodology for evaluating the usability of web mashup makers.
At the moment of writing, Step 1, 2, 3, and a part of Step 4 have been performed.
Among different usability evaluation methods that we have reviewed, we currently have selected the cognitive dimensions (CDs) of notations framework [3] [7] because of its nature as task-specific, and concentrating on the process and activities. This framework also targets visual programming tasks, which makes it very suitable for mashup makers as these tools usually use a visual language. CDs framework provides a vocabulary that enumerates concepts important to variant users. These concepts have been shown over time to be important to human problem solving [3] [7]. Some examples of cognitive dimensions are: Consistency, Hidden dependencies, Visibility. We currently are evaluating this framework for its use in the context of our research. Furthermore, we have conducted a study on 8 general purpose mashup tools: Yahoo Pipes YP [23], Microsoft Popfly MP [12], Intel Mashmaker IM [5], Openkapow Robomaker OK [13], Jackbe JB [11], IBM mashup Center IC [10], Apatar AP [1] and Dapper DA [4]. We have conducted five activities for each mashup maker (Step 3). The first activity was exploring the mashup maker from an end-user perspective taking in consideration our target user (casual users). The second, third and fourth activities are the three main steps of any mashup creation process: aggregating data, manipulate data and visualize data. The fifth activity was the creation of a mashup example. For each mashup tool and for each activity, we have given a qualitative evaluation for the different cognitive dimensions. As we did the evaluation ourselves, it was not always possible to correctly give a mark to the different dimensions, as we are not casual users. Therefore, the ranking provided in this way should not be considered as definitive and it is important to (re)do the evaluation with members from the target users (Step 6).
Despite this limitation, the study itself was very interesting as we were able to detect which of the dimensions are useful to consider in further experiments and which should be omitted. This gives us useful information for the definition of the usability criteria. Furthermore, we found that it may also be necessary to divide the target users further into groups based on their computer skills, and background in English, because all considered mashup makers provide their interfaces in English. To allow for a better comparison between the different tools in further experiments, the experimentation environment should include a common example, and should also provide learning materials and some know-how tutorials.