End-user development (EUD), the practice of users creating, modifying, or extending programs for personal use, is a valuable but often challenging task for nonprogrammers. From the beginning, EUD systems have shown that recommendations can improve the user experience. However, these usability improvements are limited by a reliance on handcrafted rules and heuristics to generate reasonable and useful suggestions. When the number of possible recommendations is large or the available context is too limited for traditional reasoning techniques, recommender technologies present a promising solution. In this paper, we provide an overview of the state of the art in end-user development, focusing on the different kinds of recommendations made to users. We identify four classes of suggestion that could most directly benefit from existing recommendation techniques. Along the way we explore straightforward applications of recommender algorithms as well as a few difficult but high-value recommendation problems in EUD. We discuss the ways that EUD systems have been evaluated in the past and suggest the modifications necessary to evaluate recommenders within the EUD context. We highlight EUD research as one area that can facilitate the transition of recommender system evaluation from algorithmic performance evaluation to a more user-centered approach. We conclude by restating our findings as a new set of research challenges for the recommender systems community.
H.1.2 [Models and Principles]: User/Machine Systems – enduser development, recommender systems.
Computing devices are ubiquitous in today’s professional environments and are increasingly invading our homes and mobile lives. Unfortunately, a deep understanding of these systems, and the ability to modify them, remains confined to the realm of the specialist. While the human-computer interaction community has made great strides in improving software usability, it has devoted far less attention to making systems customizable by end-users [22]. We argue here that existing recommender techniques can make a meaningful contribution toward increasing the usability, and therefore the acceptance and proliferation, of customizable software.
The current state of the art in user-centered software design is to
engage users in an iterative design process and to test systems
with users to refine the interaction design. Customization, if
available, is built into the system at design time as a bounded
number of user-selectable options. However, as workflows and
processes change over time, even customized software
applications need updates. Currently, the burden of these updates
falls on the shoulders of professional developers, but the pool of
users needing customizations is expected to grow much faster
than the supply of professional software engineers [
One promising approach is end-user development (EUD), the
practice of users creating, modifying, or extending programs for
personal use [
As such, EUD research mainly focuses on approaches for lowering the barrier of entry to software development. Such approaches cover a wide spectrum, from enhancing the macros and spreadsheets that millions use every day to sophisticated algorithms that create programs by example without ever exposing the user to textual code [22]. While the technology behind these approaches may vary a great deal, there are some crosscutting techniques that seem to improve usability across the spectrum of EUD systems. In this paper, we will discuss one particular mechanism for improving user performance—systemgenerated recommendations.
As early as 1991, EUD systems like EAGER were using simple
proactive suggestions as a component of their user interaction [
When the system detects that the user is performing an iterative task, it suggests a sequence of actions for completing the iteration automatically. In this case, the recommendation algorithm is straightforward, owing to the fact that there is only one recommendation type. If the system can complete the iteration it makes the recommendation, otherwise it does not.
Nearly twenty years later, advances in EUD systems have greatly expanded the opportunities for offering recommendations; however, most approaches continue to rely on constrained forms of recommendation that use handcrafted rules to make limited types of suggestions. In this paper, we explore the different classes of recommendations made in EUD systems, highlighting the areas where current approaches fall short and how recommender technologies can help fill the gap. Concurrently, we emphasize the ways in which these systems have been evaluated to date and discuss the ways in which such approaches will need Copyright © 2010 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes.
This volume is published and copyrighted by its editors: Knijnenburg, B.P., Schmidt-Thieme, L., Bollen, D. modification to successfully evaluate integrated recommenders.
We believe that by evaluating recommender systems in a userfocused context like EUD, researchers can facilitate the transition of recommender system evaluation from a focus on algorithm performance evaluation to a more user-centered approach.
A core tenet of user-centered design is to create systems that
mesh with users’ existing workflows and environments [
One of the earliest approaches to EUD was programming by
demonstration (PBD), also known as programming by example
[
A more general approach to automation within the user’s
workflow relies on activity recognition to observe what the user
is doing and infer what that user is trying to accomplish. In
combination with some mechanism for determining appropriate
assistance, the system can use activity recognition to assist with
the completion of a task. For example, Lumière uses Bayesian
user models to offer context-dependent assistance [
EUD systems that make automation recommendations within a user’s workflow provide a gentle transition from the user’s core workflow into the world of programming, as users are generally not even aware that programs are being created or selected for execution. While current systems already provide useful EUD assistance, the use of recommender technology raises the possibility of further improvement.
EUD systems such as WARP and Task Assistant, which utilize a potentially unbounded set of automated procedures, offer the greatest opportunity for the application of recommender technology. Consider a shared procedure repository for the members of an organization. As the library of assistive procedures grows, handcrafting the rules or patterns for recommendation becomes more difficult. The problem is exacerbated when there are multiple criteria for evaluating procedure quality or applicability within a given context. As new procedure sources (e.g., web services, new EUD systems) continue to proliferate, the problem of finding and automating procedures and associating them to user tasks and workflows will only increase in frequency and importance. For example, suppose the system determines that a user wants to schedule a meeting with some coworkers. The procedure repository may contain dozens of meeting scheduling procedures—some idiosyncratic to a given organization, some specific to certain types of meetings, and some buggy or outdated. Even if the procedure repository was constrained only to meeting scheduling, it would be difficult to craft a set of rules that covered all situations. Using recommender technology, one could imagine leveraging information about what other users have found useful (or not) to make better decisions about what procedures to recommend. The challenge is in incorporating sufficient context from the user’s current activity into the recommendation algorithms.
Instead of applying recommender technology only after determining what the user is doing, one could also imagine a system like WARP applying a recommender within the activity recognition algorithm itself. In this case, the algorithm might be able to narrow its search space to the activities that similar people have automated or been assisted with in the past. Such an enhancement would be particularly valuable in the case where the number of identifiable activities is very high.
To evaluate a recommender’s ability to improve an EUD
system’s level of integration into a user’s workflow, system
designers must take a more user-centered approach than is
traditional in recommender system evaluation. Instead of
focusing on algorithmic performance of independent predictions
involving a single, primitive task, EUD evaluation must situate
itself in the context of a user performing a task, often comprising
multiple subtasks. For Dynamic Macro and APE, this meant
considering task performance time and user acceptance as part
of the success criteria for the application [24,28]. Other systems
may be able perform post-hoc analysis on user logs and avoid
addressing the user’s workflow directly, but regardless of
methodology, the type and complexity of the user’s task to
automate will have some effect on the evaluation results [
A final consideration for evaluating increased automation and
activity recognition is that recommendations may occur
infrequently in comparison to the entire duration of the user’s
workflow. In this situation, longitudinal study protocols are
appropriate. In the PBD space, Dynamic Macros and LAPDOG
both worked with logs collected over an extended period of time
[
The cognitive burden on end-user developers can be further reduced, and their task performance improved, by providing recommendations that help them make better programming decisions. This approach can be of particularly high impact in EUD systems that require a mixed-initiative interaction with the user—with appropriate recommendations, the user and system can move through their dialog more quickly and with fewer errors. By using the recommendations to sort lists of programming decisions by likelihood of correctness, EUD systems can help users make the right decisions in an unobtrusive, easy-to-override fashion.
The Integrated Task Learning (ITL) system provides a
procedure editor that takes advantage of recommendations to
produce sensible defaults [
Another task in EUD where sensible default behavior is
important is the generalization performed by PBD systems
[
For this class of problem, recommender systems would likely sit behind the scenes, transparently organizing the plethora of options that systems currently present to the user. Potentially, this input could even cause systems to suppress options that are highly unlikely to be to be useful.
For a system like ITL, the primary limitation for reasoning is
that for large procedures, the number of valid suggestions can be
large, and if the user’s desired edit does not appear high in the
list of alternatives, that user’s task performance can suffer. As
with procedure recommendation, in large repositories,
collaborative or social recommender techniques could also be
used to improve recommended edits. However, a key difference
is that the possible edits are information sparse compared to
procedures. Algorithms would need to utilize the
recommendation context even more to make appropriate
suggestions, i.e. “other users in this situation tended to make
change A” rather than “other users generally like you tended to
make change A.” Leveraging this context presents a challenge
that recommender systems are only beginning to explore [
The approach of presenting users with candidate generalizations and letting them choose the correct generalization is appealing because it transforms the difficult problem (for the system) of divining user intent into the relatively easier problem (for the user) of recognizing the correct generalization. However, with very few demonstrations, the number of alternative generalizations can become prohibitively large and determining the best ones may be difficult. PBD systems can thus benefit from mechanisms to organize the space of options and provide guidance toward reasonable generalizations.
Collaborative or social recommendations may help determine what and how to generalize. For example, in SMARTedit, a recommender system could speed up learning by suggesting a higher level of generalization based on how other similar actions were generalized in past procedures. And in LAPDOG, the preferences over different explanations could be derived from the explanations selected to explain similar actions in procedures developed by colleagues within an organization.
When determining the extent to which recommendations help
users to make appropriate decisions, there are two phases in
which a rigorous evaluation can be helpful: design time and
implementation time. Post-implementation evaluations are
common in the EUD literature but often fail to provide insight
into the utility of the system in the field. Design evaluations are
less common, although they are discussed for some systems,
including ITL and CoScripter [
Much as is the case with professional programmers, end-user
programmers spend a large proportion of their time debugging
[
Ko’s Whyline provides a novel approach to supporting
hypothesis-driven debugging by suggesting “Why?” questions
for the user to explore when debugging (Figure 3) [
Whyline generates its questions and answers automatically using
heuristics and program slicing [
Another approach to error handling is to take the act of
debugging out of the user’s hands entirely by automatically
verifying the program for correctness before something goes
wrong. For some classes of programming problems, EUD
systems can detect errors and provide suggested fixes without
forcing the user to search for problems. The ITL procedure
editor takes this approach when possible (Figure 4) [
Whyline uses highly dynamic techniques to generate its hypothesis recommendations; however, it is still limited by its output and familiarity heuristics. Both exist to limit the number of questions and reduce time necessary to perform program slicing; however, as with all heuristics, they may not be correct all the time. By reducing the question space using recommendations rather than static heuristics, Whyline could find more bugs without sacrificing the processing time required to slice the program for every possible output primitive.
Users do not always test their programs on every possible execution path, so the observed output heuristic captures only a subset of the bugs that actually exist. A recommender that works over a community pool of debugged programs could suggest potential problem areas outside of the observed output based on problem areas in similar programs. This approach requires algorithms to be able to determine what features indicate programmatic similarity, a more difficult task than many traditional recommender applications. Nevertheless, programs are information-rich—the core problem is identifying what information is most relevant to determining similarity.
It is likely that users will be less successful debugging code with
which they are unfamiliar. The familiarity heuristic weeds out
libraries that the user cannot modify without source code access,
but it does not actually address the root cause of the
unfamiliarity that leads to coding problems. A recommender of
relevant code examples for problem application programming
interfaces (APIs) could go a long way toward helping users
debug calls into unfamiliar source. In fact, Brandt suggests that
“opportunistic programming,” based on web searches of code
examples and tutorials, constitutes a primary programming
approach for novice programmers [
Once errors are identified, recommending a subsequent fix is likely to be a highly relevant challenge for the recommender community. Here, current systems tend to pre-engineer relatively simple solutions to straightforward errors. If one could instead capture the ingenuity of an entire EUD system user community, it may be possible to propose solutions for more complex classes of problems. By creating a data set out of the problems and solutions faced by previous end-user developers, recommenders could suggest, and with some extension possibly apply, novel solutions to programming problems initially unanticipated by the creators of EUD systems. Such an advance would make EUD systems much more robust to unexpected programming challenges.
Whyline’s evaluation consisted of a task-focused controlled experiment of debugging approaches [17]. Such a protocol also lends itself to the evaluation of error-handling recommenders because it provides a solid task context while still allowing the experimenter to isolate algorithmic performance from userinterface design choices. For Whyline, a traditional breakpoint debugging system served as the control condition, and user task completion defined the success criteria. Thus, the experiment tested hypotheses about how the interaction design paradigm would affect user task performance To test the recommender opportunities above, one would instead control for user interface deviation and vary the recommendations while still measuring task completion to evaluate the success of each condition. To manage user interface variation, we again suggest following the two-step approach advocated in section 2.3, first isolating and solving confounding user interface problems qualitatively, and then comparing task performance on interface without recommendations against an interface with recommendations.
Other possible conditions to test include recommendation domains of varying size or composition. Likewise, testing tunable parameters like recommendation confidence or recommendation diversity could yield insights into exactly what sorts of error-handling recommendations provide the most value to end user developers.
End-user programmers, by definition, do not set out to create programs for others; however, unplanned sharing is a frequent side effect of EUD [23]. Even if end users do not plan on sharing their programs, they can often benefit from using someone else’s code. While many early EUD systems lacked a community and, by extension, lacked sharing, several systems now leverage this powerful capability.
Two systems that benefit from sharing are ITL, which provides
a demonstration-based EUD environment for collaborative
military command and control, and Task Assistant, which allows
users to explicitly encode best-practice workflows in a sharable
way [
This sort of aid does not go far enough because users generally do not have sufficient familiarity with large codebases to know when code that solves their problems already exists [32]. Ye et al. suggest that “information push” (proactive recommendation) is one mechanism to assist users in such situations.
The CodeBroker system is one example of proactive development support. It recommends API calls to the user based on programming context (Figure 5) [32]. While it is not focused on “end-user” developers, it is straightforward to imagine using similar mechanisms for a code repository like the one used by CoScripter. CodeBroker generates its recommendations by gathering context from code comments. Then it applies Latent Semantic Analysis (LSA) to determine what components in the code base are conceptually similar to the concepts described by comments [19]. It further reduces the suggestion space by applying signature matching to only suggest API calls that are close type matches to the current calling context.
In the case that the above technique does not produce satisfactory results, the programmer can create a “discourse model” to manually identify suggestions that were not relevant [32]. Such irrelevant suggestions are not displayed in future queries. The system expands this concept to also include “user models” that manually identify code with which the user is already very familiar. Following the intuition that users do not want to get suggestions for familiar code, user models also serve to reduce the suggestion space for a given user. It is important to note, however, that in CodeBroker users manually create both the discourse and user models and must know a specific syntax for doing so.
Community has not always been a central focus in EUD, but as recent systems push the boundaries toward more and more collaboration, community-based recommender systems seem to be a good fit. More users typically means more possible recommendations, and recommender algorithms can help EUD systems to cope with this increased scale.
LSA has been integrated previously into a collaborative filtering
algorithm, so one could imagine implementing similar
recommendation techniques within CodeBroker [
Higher hurdles remain for EUD systems that are actually focused on novice programmers. Not only do these programmers not know what API calls exist, but they are also likely to misuse such calls. As discussed in Section 4.2.2, recommendations that also include code examples can be useful to teach novice opportunistic programmers how to reuse code.
Suggesting code reuse is only half of the battle. End-user
developers are unlikely to aim to create reusable code when their
core goal is to program for personal use [
Sharing is a particularly difficult user need to evaluate because a realistic evaluation must be situated within a community of users. CoScripter was able to overcome this difficultly by selecting users to interview based on indicators gathered from broad-based logging of user activities [20]. Unfortunately, this approach is difficult to replicate without a critical mass of users such as the one CoScripter enjoys. Without such a user pool, the CodeBroker evaluation protocol falls back on extrapolating code sharing efficacy from precision and recall of the method calls suggested [32]. While these measures are valuable, they cannot tell the whole story of how a community-based sharing system will operate in practice.
In a situation where a large user pool is not available, we suggest an approach that combines a longitudinal protocol with user satisfaction surveys at regular intervals. In this case, users have a longer time in which to benefit from sharing, and the evaluation design explicitly addresses the user’s perceived benefit from this sharing.
As a mechanism to improve usability, system-created recommendations have been a part of end-user development systems for a very long time. Unfortunately, few systems today take advantage of recommender technologies to cope with the increased scope of recommendation within EUD. Instead, handcrafted rules limit the types of suggestions that today’s systems can make. As such, usability and adoption suffer. We identify four main classes of recommendation that could be improved by using recommender technology: • •
• •
Each of these classes of recommendation could make appearances in a variety of EUD systems, and as a whole they address user needs across the entirety of an end-user development workflow, from conceiving of a needed customization, to programming it, to sharing it with others. In the space of inserting automation, recommenders could improve suggestions over shared repositories and perhaps directly improve activity recognition algorithms. Here, the major research challenges involve incorporating the user’s operating environment into the recommendation algorithm to make the recommendations appropriately context-aware.
For improving user decision making, recommenders can help to organize the decision space, providing sensible defaults for a number of programming decisions, such as performing dataflow changes in an editor or choosing a best generalization in a PBD system. Challenges here include making recommendations about information-sparse decisions like simple edits. Again, context awareness could help to solve this problem.
When handling errors, opportunities for recommender systems are numerous, including suggesting potential problems, code examples, and possible fixes. While suggesting code examples is a fairly straightforward recommendation application, identifying potential problems and fixes exposes recommender algorithms to a nontraditional set of features to reason over. Learning exactly what features allow recommenders to identify similar programs is an interesting challenge we pose to the community. Finally, supporting unplanned sharing allows recommender systems to apply their ability to leverage communities to the world of end-user development. Systems could help users to identify reusable components, especially aiding novices. Further, automatic identification of best practices could improve performance even for advanced end-user developers. In all of these cases, end-user development provides a natural environment in which to evaluate recommendations in the context of real user workflows. Many evaluations in the enduser development space already combine user-centered evaluation methods with algorithm performance evaluation; user-centric evaluation of recommenders in this space simply requires evaluators to extend existing protocol designs to isolate the features unique to recommenders.
In conclusion, EUD systems have just begun to scratch the surface of how recommendations can improve user experience. By increasing communication between the EUD and recommender communities and creating recommendations that meet real user needs, we can move development out of the realm of the specialist and into the real world.
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8750-07-D-0185/0004. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA) or the Air Force Research Laboratory (AFRL).