Adaptive voice applications supported by conversational agents (CAs) are increasingly popular (i.e., Alexa Skills and Google Home Actions). However, much work still remains in the area of voice interaction design and evaluation. In our study, we deployed a voice crawler to collect responses from the 100 most popular Alexa skills within 10 diferent categories. We then evaluated these responses to assess their compliance to 8 selected design guidelines published by Amazon. Our findings show that design guidelines requiring basic commands support are the most followed ones while those related to personalized interaction are relatively less. There also exists variation in design guidelines compliance across diferent skill categories. Based on our findings and real skill examples, we ofer suggestions for new guidelines to complement the existing ones and propose agendas for future HCI research to improve voice applications' user experiences.
• Human-centered computing → HCI design and evaluation methods; Interactive systems and tools; Systems and tools for interaction design.
conversational agents; voice user interface design; user experience evaluation;
Voice-powered conversational agent (CA) devices have recently
achieved significant commercial success. In the U.S.A., 47.3 million
(19.7% of) households now own CA devices (March 2018), an
increase from less than 1% two years ago [
One key characteristic that makes this new generation of CA
devices adaptive is their API platform for third-party developers.
Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
Here, developers design and build voice applications and publish
them on a marketplace with the potential to reach millions of users.
Amazon’s Alexa skills [
There is a huge body of literature in HCI that propose design
guidelines to educate practitioners in the field who want to design
and develop an application for a wide range of interactive
technologies (i.e. web readability design [
An example of a well-designed Alexa skill is Would You Rather for Family. This skill is an interactive Q&A game that exhibits several design features following Amazon’s guidelines, including remembering where the last interaction ends, giving a personalized opening prompt to users, and speaking naturally. Deservedly, this skill has a high average rating – 4.9 out of 5 stars based on 3209 user reviews. In contrast, an example of a poorly-designed Alexa skill is AccuWeather. This skill’s average rating is low – 2.2 out of 5 stars based on 182 user reviews. By interacting with this skill, we can tell that the skill’s design violates several design guidelines, such as handling errors properly. These violations are also complained by some users in their reviews. By analyzing a large number of skills like these, we can gain insights into design guidelines’ adoption pattern. We want to ask: Among the design guidelines for voice applications, which are followed or violated more often by developers in the wild? (RQ1). Another phenomenon we observed is the high variance in user ratings across app categories. For instance, we found the average user rating of top 10 popular skills in the Games category is 4.5, comparing to 2.6 for those in the Food & Drink category. Motivated by this phenomenon, a second research question can be opened: Could this high degree of variability among categories be related to whether certain guidelines are followed or not followed? (RQ2).
To study these questions, we decided to limit the scope to Alexa skills in this paper. We selected a sample of 100 most popular Alexa skills from ten diferent categories and evaluated whether their designs follow the a selected subset of Amazon’s oficial design guidelines. Note that our scoping decision does not imply an acknowledgement of Amazon’s design guidelines as the gold standard nor an endorsement of Amazon’s products. Rather, the decision is based on where we might be able to gather the most data, which platform has the largest number of developers in the wild, and which set of design guidelines are most likely read by these developers (which is unlikely an academic paper). To automate our data collection process, we deployed a voice skill crawler to collect responses from these skills under diferent commands input. We then analyzed the collected responses to determine whether or not certain guidelines are followed. Regarding the first research question, an example of key findings is that basic commands support are the most obeyed guidelines while personalized service-related guidelines are relatively less obeyed. Regarding the second research question, an example of key findings is that skills in the Games category on average obey the most guidelines while skills in the Entertainment category the least.
Furthermore, previous research (e.g. [
Recent years’ advances in speech technology have led to voice user
interfaces’ (VUIs) improved accessibility and they have been studied
in the HCI literature in a wide variety of application contexts (e.g.
assistive services [
Based on the results of our literature survey, we noted several
existing user experience evaluation methodologies that could be
applied to VUIs or VAs. Traditional usability studies are very useful
in gathering feedback and conducting evaluation analysis. In [
Specifically for Alexa skills’ user experience evaluation, although
Alexa provides an overall platform for developers to check their
skills before submitting to the review process, there is still no
guarantee these skills follow the published voice design guide [
In order to investigate the adoption and compliance pattern of current Alexa design guidelines, we first deployed a crawler system to collect responses from a sample of 100 Alexa skills and then manually labeled those collected responses to study whether or not they comply with the selected design guidelines. Here we elaborate our method in details. 3.1
More than 30,000 Alexa voice skills [
The most common interaction flow of an Alexa skill is the "opencommand-stop" flow. To begin interacting with a skill, A user first says "Alexa, open X", where X is a skill’s invocation name. Then, the skill typically responds with an introduction or greeting message. After that, the user starts uttering specific commands to make use of the skill’s functionality. The skill responds with its answers or follow-up questions. The conversation continues until the user says "Alexa, stop" to indicate their desire to quit. Sometimes, the skill responds with a goodbye message, but not always. In order to study our research questions, we needed to have conversations like this with each of the 100 skills in our sample, recorded how each skill responded, and analyzed whether its responses followed or violated certain design guidelines. Our initial attempt was fully manual. Given a skill, we spoke to it, listened to and wrote down its responses in an excel spreadsheet, and coded the responses with respect to their compliance with design guidelines. However, after about 20 skills, we found manual data collection time-consuming, dificult to scale to a large sample, and hard to replicate for other researchers. Thus, we were motivated to develop a method to automate certain parts of this process.
We present a crawler tool we developed to automatically
converse with a given skill and record the skill’s responses (the progress
paper of this tool was presented in [
Figure 1 provides a conceptual example of how our crawler simulates a voice conversation between users and Alexa devices. the skill’s responses. We used the Speech Recognition package for Python 2 to implement the listening ability. Given our sample of 100 skills, our crawler iterated through them, carried out a range of conversations with each skill, and listened to the skill’s responses. This automatic data collection process is described pragmatically as follows: Algorithm 1 Collect Responses to Commands by Skills 1: for skill in [1,2,..., ] do 2: speech ← TextToSpeech("Alexa, open {{skill’s name}}"); 3: play speech 4: for command in [1,2,..., ] do 5: speech ← TextToSpeech(command); 6: play speech; 7: audio ← listen; 8: text ← SpeechToText(audio); 9: save text; 10: end for 11: end for 3.3
Amazon’s voice design guide [
Basic commands support (G1, G2, G3): Three design guidelines recommend voice skills should support users to start, get help on, and end an interaction. For Alexa skills, these translate into the ability to understand the basic commands of "open", "help", and "stop" respectively. Described in more details, when a user says "Alexa, open [skill’s name]," the skill needs to remain open and wait for the user’s responses (G1); when a user asks "Alexa, help," the skill is expected to provide informative instructions such as introducing its core functionality (G2); and when a users says "stop," the skill should end the conversation naturally and gracefully with few or no words (G3). In order to evaluate the compliance situation of these 100 skills with respect to G1, G2, G3, we designed crawler loops by setting the basic commands as elicitation commands. Within one round of the crawler loop, the crawler will say "open", "help", "stop" commands and listen to the responses in turn (this crawler loop is denoted as "open-help-stop" loop in the rest of the paper). Based on the responses collected, we would get to know how many skills support basic commands and conduct our analysis.
Variety support (G4, G5): Two design guidelines recommend voice skills should provide varying responses to the "open" (G4) and "stop" (G5) commands so that the interaction can feel more natural and less robotic. To test whether a given skill complies with these guidelines, our crawler tool carried out an "open-help[commands]-stop" dialogue where commands are what the specific 2The project website is: https://pypi.org/project/SpeechRecognition/ skill can support. This dialogue was repeated times where is default to three so that we can detect variations in the skill’s responses to "open" and "stop" command, if any. Particularly, in the ifrst run of this dialogue, the tested skills was first-time enabled. The second round was run after solving all the account linking, age verifying steps. The third round was run after the skill has been fully explored with a list of commands. The list of commands were automatically extracted from the skill’s response to the "help" command in the second round. For example, the skill "Examining the Scriptures Daily" responded to "help" with the message "You can say tell me my daily text for today or read me my daily text for last Monday. You can also say read me tomorrow’s daily text." We wrote a parser to extract three commands from this message: "tell me my daily text for today", "read me my daily text for last Monday" and "read me tomorrow’s daily text." Then each of the extracted commands was applied in the crawler and given to the skill.
Error handling support (G6, G7): Problem handling is one of the most important aspects of user-aware design. We chose to evaluate two guidelines regarding error handling. The first guideline is when a skill receives no answer from a user regarding a question, it should deliver a re-prompt (G6). The second guideline is the re-prompt should be reworded or with more detailed instructional information(G7). To collect responses for a given skill regarding its error handling ability, our crawler first carried out a "open-helpstop" loop and repeated the "open-command-stop" loop three times to make sure the skill was fully explored (the commands were automatically extracted similar to as how we handled G4, G5). After that , we enabled this skill again and stopped giving further command to wait for how it would respond. Our crawler then repeated this process for all skills in our sample.
Memorizing support (G8): According to the design guide, users would appreciate it if a skill can remember their past interactions and provide more personalized services (G8). In order to test this, our crawler first fully explored a skill’s capabilities (like G4 and G5), and then carried out an "open-help-stop" loop one more time to see whether the skill remembered its last interaction and personalized its responses accordingly. 3.4
Given our sample of 100 skills and 8 design guidelines to test for, our crawler automatically collected more than 1000 responses (the dataset is included in the supplementary material). We manually analyzed the data as follows. 3.4.1 Data Correction. First, we compared this dataset to a small pilot dataset of 20 skills we previously collected by hand in order to identify any discrepancy between machine and human transcribed responses. In doing so we were able to detect and correct problems brought by limitations of speech-to-text technology, such as typo and missing punctuation. 3.4.2 Data Coding. After data correction, two researchers independently coded each response’s compliance with respect to design guidelines. Afterwards, two researchers compared their coding results and resolved their discrepancies.
For basic commands support, we examined collected responses to see whether the skill successfully executed the commands. For variety support, we compared the responses across repeated dialogues. If there are variations, we would code as following G4 or G5. For error handling support, we first determined if the skill supports G6 and then compared with previous messages to determine whether the re-prompt messages were reworded or not. Finally, for memorizing support, by comparing the last and the very first "open-help-stop" loop’s responses, we judged whether the skill memorized previous interaction. The contents of the two responses were compared. If the second time’s contents include any personalized information or previous interaction information while the first time doesn’t, we would code as following G8. 3.4.3 Comparative Analysis. After we coded all the responses, we were finally able to address our research questions by comparing the results across guidelines (RQ1) and across categories (RQ2). For each guideline, we counted the number of skills that follow it and picked out both positive and negative examples for further investigation. By comparing across diferent guidelines, we were able to understand the guideline adoption pattern in the wild, that is, which guidelines are obeyed by more or fewer skills. By comparing across categories, we were able to examine whether category can be a factor associated with whether certain guidelines are followed (or violated). 4
In this section, we present our findings regarding the current compliance situation of a sample of 100 skills with respect to eight voice design guidelines, in order to address the two research questions proposed earlier in the introduction. In the following Improving Design Guidelines section, we will discuss several illuminating realworld examples, both positive and negative, we discovered during the data collection process, which serve to motivate further design recommendations for voice skills.
As described before, we initially selected 100 most popular skills from 10 diferent categories as our sample. All the responses were collected in late 2018. However, after all the data was collected and cleaned, we needed to exclude six skills for which we failed to obtain meaningful results because of issues related to account linking or access permission. Hence, our findings presented below are based on 94 skills. 3 4.1
4.1.1 Open Command (G1). According to the design guide and
Amazon’s Alexa building requirements [
What could be the reasons these skills do not support G2? One
reason is that audio/music skills are meant for passive listening, as
in the case of NPR One and Thunderstorm Sounds. Another reason is
that some skills only involve one-shot interactions where a user asks
a question or gives a command, the skill responds with an answer
or confirmation, and the interaction is complete [
Compared to basic commands support, we found variety support is provided by much fewer number of voice skills in our sample. Below we present our findings for the two relevant design guidelines we studied. 4.2.1 Variety in open responses (G4). When a customer invokes a skill without specific intents ("Alexa, open [skill name]"), the skill should deliver an opening prompt. Skills are expected to provide several variations of opening prompts including one for first-time use, one for return and personalized prompts (G4). We found 34 out of 94 skills (36%) supported opening prompt variations. Furthermore, we observed they often served three use scenarios (with overlaps). 1. Some (n=8) were daily used skills or skills with regular updates; variety in opening prompts help keep users feel fresh and updated. 2. Some (n=16) were kills that remembered previous interactions; whenever users open the skill, its opening prompts will tell users where they left of last time. 3. Some (n=13) were skills with multiple states; the opening message will always inform users the current state.
For the first scenario (daily use), one good example is the "Zyrtec" skill which can report weather, pollen count and predominant allergens in a user-defined location. When this skill was opened the ifrst time, its opening prompt was "Hello! Let’s get ahead of your allergies with today’s Allergycast based on your location. Just follow these steps. One, Open your Alexa app on your phone ... [19 more words]" For the second time, the skill said "let’s start with your city and state, then we can get ahead of those allergies by setting up your allergy test report. What’s your city and state?". For the third time, after the location was set, the skill’s opening prompt turned into "Welcome to Zyrtec. Today in xxx, the pollen count is High, at 9.2 out of 12... [34 more words]" Comparing these three opening prompts, we found that when the skill was first enabled or used, it provided instructions about setting up step by step and elaborated clearly about the location requirement. After the skill got the location permission, the opening prompt changed into daily report of pollen. The whole interaction was natural and personal for users. In contrast, a poor example is Examining the Scriptures Daily. We found the skill always responded with the same sentence: “Which day do you like to hear”. Although this opening prompt provided users with a cue to begin speaking and coached users on what to say next, the interaction could feel monotonous and less natural.
For the second scenario (remembering previous interactions), we found 26 of 94 skills (28%) could remember previous interactions but only 16 supported variations (17%). A good example is 7-min Workout in the Health & Fitness category. This skill is used to play instructions and background music for people who work out. When the skill was firstly used, its opening message was “Welcome to Seven Minute Workout. When you are ready, just say start workout.”. After a previous workout was interrupted, the skill’s opening message changed into “Welcome to Seven Minute Workout. To continue where you last left of, say ready. Otherwise, just say start workout.” In this situation, variety in opening prompts provides a personalized experience for users by allowing them to pick up where they left of.
For the third scenarios (multiple states), a good example was the
popular Magic Door skill in the Games category. This skill always
informs users the current game state in the beginning so that users
can choose to resume or restart the game. A negative example is
Categories Game skill, also in the Games category. The skill always
says “howdy. You’re playing Categories Game! For instructions, say
help me or, say start playing!” in its welcome message and have to
start the game all over again no matter how many times it has been
played.
4.2.2 Variety in stop responses (G5). According to [
Error handling is an important part in any user interface design, voice skill design is no exception. In our study, we focused on a typical error handling scenario: when a customer responds to a skill prompt with silence. Under this situation, the skill is expected to deliver a re-prompt (G6) with rewording (G7) to disambiguate or elaborate on the kind of responses supported. Our findings are as follows. 4.3.1 Re-prompting (G6) with Rewording (G7). When doing the response coding, we manually determined that 82 skills (of 94 total) should have support for G6 and G7 (some skills are not expected to support G6 and G7, like "one-shot" skills and those which are meant for passive listening). For G6, We found a high percentage of skills supporting re-prompting—74 of 82 (90%).
For G7, however, we found a low percentage of skills—23 of 82 (28%)—that reword in their re-prompts. A good example is the Amazon Story Time skill. First, the skill greets users by saying "Welcome back to Amazon Story time! Would you like to resume The Mouse and the Unicorn?". After the question, if it receives no responses from the user, it would re-prompt with "you can say yes to resume or no to play the next story". In this example, the re-prompt ofered more specific instruction for users to say yes or no. In contrast, a negative example is the Bring skill in the Shopping & Finance category, which simply stayed silent, without any instruction or hint to help users handle a potential error. 4.4
Just like conversing with a friend, users appreciate when Alexa remembers what happened previously and what was said, especially for frequent actions and static information. We found 27 of 94 (34%) skills that provided memorizing support.
One positive example is Lemonade Stand in the Kids category. It is a game where users can sell products and manage their income. The skill always remembers how the game ended last time. Each time a conversation began, this skill would say "Today is your twelfth day selling lemonade. Currently, it’s windy and cool with some clouds. The forecast is a very low chance it will be warm and partly cloudy. Your cost for lemonade is fifteen cents a cup. You have three dollars and fity cents . How many cups do you want to sell? " This message conveyed the key statistics to help users remember their progress. In contrast, the 5-min Plank Workout skill in the Health & Fitness category did not say anything explicitly to users that it remembered what exercises users might have done in the previous session. It always asked users to start over again, which could be frustrating. As we examined further, we identified certain legitimate exceptions past interactions were not remembered. For example, a skill like This Day in History is designed to be relevant for that day where past interactions do not matter. 5
In previous sections, we presented our findings with respect to each of the eight guidelines (i.e., G1 to G8). In this section, we will compare our findings across both guidelines and skill categories. These comparisons address the two research questions in the introduction (i.e., RQ1 and RQ2). 5.1
As shown in Table 1, we calculated the compliance rate for all 8 design guidelines and ranked them based on their rates. These results show that among the design guidelines we evaluated, some were more frequently violated than others.
Based on the ranking, open command support (G1) and stop
command support (G3) were among those followed by the most
number of skills. In contrast, Memorizing support (G8),
rewording support (G7), and stop variation support(G5), were followed
by the fewest skills. As we can see, both G8 and G5 are related to
Alexa skills’ personalized services. What could explain such
diferences in compliance rate across design guidelines? For guidelines
related to personalized services, one possible explanation of their
low adherence rate may be the dificulty in implementation, which
involves user behavioral modeling, user data analysis and other
techniques. Also, high-quality personalized services require users
to provide more personal information. It is hard to strike a good
balance between the quality of personalized services and users’
concern about their privacy [
In this part, we will make comparisons across diferent skill categories. As indicated in Table 2, we counted the number of design guidelines that a certain skill complied with and calculated the average number within one category. Based on the calculated results, we obtained a ranking where the Games category was the top-ranked one while the Health & Fitness and Entertainment categories had relatively low rankings. Moreover, we broke down the comparison into four types of design guidelines: basic commands support (G1, G2, G3), variety support (G4, G5), error handling support (G6, G7) and memorizing support (G8). We first calculated the percentage of skills that supported each design guidelines within each category and then computed the average compliance rate within each of the four types. The results are also shown in Table 2.
From the ranking shown in Table 2, we can see that the Games category had the highest compliance rate for variety support guidelines (G4, G5), memorizing support guidelines (G8) and close to the highest compliance rate for error handling support (G6, G7). Also can be seen is that the skills in the Game category were more likely to follow the rest of the guidelines. Game Skills are expected to involve more interactions with users and require more complicated user interface design, like remembering users’ previous score and provide personalized game processes. When we looked at these 10 selected skills’ user ratings on Amazon’s website, they also achieved relatively high average user ratings (4.5/5), which matched with the comparative analysis results. The skills in the Kids category also held high ranking positions in our table. One explanation is that children are considered a sensitive population that tends to have a higher requirement for design quality.
Let us now turn attention to categories with relatively low compliance rates. Several interesting patterns emerged. For example, the Entertainment category had close to the lowest compliance rate across all four guideline types. At a quick glance, this finding was Category surprising because Entertainment and Games seemed similar yet occupied the two opposite ends in the ranking. Upon closer examinations, we realized skills in the Entertainment category tend to ofer quick and instant "fun" such as telling a joke or a compliment, which do not need many user inputs and require less interaction design. Another example is the Communication and Social category that had a relatively low compliance rate with respect to variety support guidelines but a high compliance rate for error handling guidelines. One explanation could be that communication and social skills may involve users speaking longer and more intentional utterances and may be more prone to errors, which necessitates additional efort to handle errors. In conclusion, with respect to RQ2, we found evidence that design guideline compliance patterns do difer greatly across categories, which suggests associations between design guideline compliance and categories. However, we were unable to determine whether these associations are causal or correlational, which will require further studies. 6
Based on our findings and real skill examples we encountered during the evaluation process, we derived a set of new design guidelines for voice skills to complement the existing ones. In order to make our ideas clearer, simulated user-Alexa dialogues are presented for some of the points. 6.1
We found that a significant number of instances of guideline violations are associated with one-shot skills such as Cat Facts and Damn Girl. In our sample, fewer than 10% were one-shot skills but they accounted for a large number of guideline violations. Upon closer examinations, some of the violations could be excused (users do not have chances to go deeper). This observation may suggest that the oficial design guidelines need to be revised to consider the special needs of one-shot skills. Here we present two ideas for the revision informed by our findings. 6.1.1 Use Informative Invocation Name to Replace Help Command. Since a one-shot skill often exits automatically after responding to users’ commands, users may not have a chance to interact with the skill deeper. Thus, it is advisable to carefully choose an invocation name that is informative to remind users of its core functionality. A good example is the Rain Sounds skill whose name clearly indicates that this skill intends to play rain sounds for users. In contrast, a negative example is the Damn Girl skill, which carries a unusual name but gives users little information about what it does (in fact, it says a diferent compliment each time). 6.1.2 Personalize the Contents Based on User’s Interactions with Other Skills. One-shot skills’ interaction mode limits the collection of user inputs, which makes the process of personalizing their contents very dificult. In this case, a good way to solve the problem is to connect with other skills for more user inputs. For example, a one-shot skill aimed at providing basic facts about cats (such as Cat Facts) could make use of a user’s previous inquiries about a cat’s health, collected from other skills. With this personalized information, this one-shot skill could provide more relevant health facts about cats the next time the user opens it. But this approach must be implemented carefully to respect users’ privacy preferences regarding sharing data across skills. 6.2
Our findings show that there is still a room for improvement in terms of providing personalized experiences for voice skill users. During the process of analyzing our data, we noted several realworld design examples that could inform new design guidelines for voice skill developers. 6.2.1 Change Interaction Mode for Repeat Users. Personalized service should not be limited to variety in responses, it should also be reflected through variations of the whole interaction mode. For example, through analyzing a user’s interaction history, a skill could tell whether the user is a frequent user. If not, the skill could guide the user to explore its features in details. If yes, the skill could simplify or streamline the whole interaction flow to provide more personalized service. For example, repeat users could get what they want immediately or receive a list of recommended services based on interaction history. Here we present an ideal interaction mode variation example. First is the interaction mode for non-frequent users.
User: Alexa, open Dishes Delivery.
Alexa: OK, what kind of dishes do you want? ...(the skill acquire necessary information like dishes kind, price, personalized taste like dishes cooked with no peppers) Alexa: OK, got it. Your order is ready.
Next are the good and bad examples of the interaction mode for frequent users.
User: Alexa, open Dishes Delivery.
Alexa: (Bad) OK, what kind of dishes do you want? Alexa: (Good) OK, welcome back. Do you still want "A" cooked with no peppers? 6.2.2 Providing detailed information via other platforms. One limitation of a voice skill is the amount of information it can provide in a single utterance. Meanwhile, an overly long utterance in response to a user’s question is highly discouraged. In this case, we found some voice skills take advantage of other platforms such as mobile, emails, and SMS to deliver extra information. A good example is the Store Card skill. When this skill needs to tell users information that is not suitable through voice interaction, such as an URL, instead of saying it aloud, it sends the information to a user’s mobile app and explains to the user that “we just made some improvements that you need to disable the skill and then enable it again. Please use the link we just sent to your app”. This practice eliminates the need for users to listen and remember long text. Hence, we suggest that detailed information can be optionally provided via another platform. 6.3
Here we present several more design guidelines (not already covered by the oficial ones) informed by real world examples we observed, which reflected both good and bad design practices. 6.3.1 Give feedback to help users locate problems in their commands. Users might feel frustrated when their commands cannot be correctly processed by a voice skill several times in a row. Under this situation, if the skill could specifically tell users where the problems are in their input and give more specific instructions, it would more efectively help users adjust their input and receive the desired services from the skill. During the manual collection process, we found many skills just repeated the same generic sentence like "Sorry, I didn’t understand that. What would you like?" when researchers gave commands that could not be understood. Those kind of responses do not provide any information about why Alexa cannot understand the user’s command. We suggest an additional guideline that a skill should provide informative feedback such as telling users what it originally expected and why users’ voice input did not match the expectation. Here is an example dialogue contrasting a good response with a bad response with respect to this guideline. User: Alexa, open Pizza Delivery.
Alexa: OK, what city do you live in? User: My city is horse.
Alexa: (Bad) I didn’t understand that. What city do you live in? Alexa: (Good) (The skill’s logic does not think ’horse’ is a city name.) Sorry, "horse" is not a city name, can you say your city’s name again? 6.3.2 Let users know which skill they are currently interacting with. Sometimes users may mistakenly think they are interacting with a skill but in fact with another skill. They may say commands which are only meaningful for other skills but cannot be understood by this skill. In this case, a useful design guideline would be to remind users which skill they are interacting with when the skill fails to understand users. The WebMD skill is a good example following this guideline. Below is a sample dialogue that demonstartes WebMD’s informative response. (Suppose a user forgot to exit WebMD but thought he is interacting with a pizza delivery skill.) User: Alexa, order pizzas.
Alexa: (Bad) Sorry, I didn’t understand. What would you like to know? Alexa: (Good) Sorry, you are already speaking with the WebMD skill. You can ask things like "What is diabetes?" or "What are the side efects of Nexium?" What would you like to know? 6.3.3 Recognize and acknowledge problems in users’ input. During our evaluation and the process of reviewing users’ reviews, we noticed that for some skills, even if users give incorrect input, those skills still continue with the wrong information and respond to users with irrelevant answers. For example, Categories Game is a skill that presents diferent categories and asks users to come up with a word that begins with a certain letter in each category. One of the reviews said that the skill sometimes does not seem to understand the words users actually spoke and continues the game regardless. Hence, we suggest a skill should improve the ability to recognize diferent types of errors and acknowledge those errors, rather than acting if there is no error. 6.3.4 Don’t give advertisements or encouragements too often after "stop". We observed that some skills include advertisements or encourage users to give ratings in the goodbye message too often. For example, the skill Big Sky always asks users to write a review at termination. One reviewer complained “A nice, helpful app, except that it frequently ends answers to queries with ‘please consider writing a review for this skill...’, etc. I end up spending more time stuck listening to it beg for a review than, say, finding out what the temperature is.” Whenever users give the stop command, they hope to stop the skill successfully rather than listen to other bunch of sentences. In this case, if the skill could reduce the frequency of or stop giving advertisements after user says “stop”, it would provide better user experiences. Instead, skill developers should use other channels to do their advertising. 6.3.5 Don’t include questions in goodbye messages. We observed that some voice skills include a question in its goodbye message in response to the “stop” command. For example, the Alexa Prize Socialbots responds to a stop command by uttering “Thanks for chatting! Quick question. On a scale from 1 to 5 stars... how do you feel about speaking with this socialbot again?” We found this practice problematic. As mentioned before, whenever users say “stop” to a skill, they indicate strongly that they do not wish to interact with the skill anymore. But asking a question in the goodbye message would require users to continue the interaction, which may negatively afect the user experiences. Hence, we suggest that voice skills hould not include questions in goodbye messages. 7 7.1
Although current design guidelines and suggestions we ofered before have already covered many design problems developers might meet, future research is still necessary to revise the guidelines to meet new needs.
First of all, research on understanding the design space of voice
skills is critical. The design space of traditional voice user interfaces
has been proposed [
Next, we found that connecting with other platforms is important for a voice skill, especially when the skill has versions on other platforms, such as Uber, that have corresponding mobile versions. In order to improve user experiences, sharing information across diferent platforms is critical. For example, an Alexa skill could give users a concise message while detailed information could be sent to users’ mobile application. Another benefit of connecting with other platforms is to allow users to receive consistent services. Hence, future studies are needed to understand how to best support a seamless cross-platform experience.
Finally, the design of personalized voice skills is also worth
studying in the future. From our findings, we found personalized services
do not have very high support rate among the current popular skills.
This finding implies personalized design requires more attention
and further revision. Studies show that in order to achieve a high
degree of personalization, more personal or private information is
often required from users. However, due to privacy concerns, users
may want to disclose less personal information [
One contribution of this paper is the finding that there exists a high degree of variation in design guideline compliance across diferent skill categories. Thus, the variation we found suggests each skill category has its own specific requirements and design challenges. This opens up several research questions for future, such as how should design guidelines and design space be adapted for diferent categories and even further, application scenarios? We can start from understanding which design guidelines are more important and needing more attention for each category. Also, we can study the variations in interaction flows across categories and understand the diferent challenges one may face in evaluating the skills in each category. 7.3
The evaluation methodology presented in this paper has several limitations. On the technical side, our crawler is a research prototype that covers only selected design guidelines. More research is still needed to support others. Also, the speed of our tool is limited by the natural speed of a human’s voice (since our tool simulates a human’s interaction with a voice skill). In terms of data collection, we only focused on popular skills and categories. We do not yet know whether our findings can be generalized to less popular skills. At the same time, only 8 design recommendations were evaluated and we did not explore all the possible commands. Furthermore, the responses we collected were only one snapshot in time; we don’t know whether tested skills have since updated their interaction models. In terms of the crawling algorithm, we cannot cover all the possible situations, like when we tested variety (G4,G5), the crawler only repeated the same commands for three times. The possibility of variety appeared in the fourth time or later was not eliminated. In terms of responses analysis, all the response labeling was conducted manually, which leaves room for improvement.
Correspondingly, there exist several possibilities for improving
the evaluation methodologies in the future in order to better
triangulate usability issues and design guideline violations. They include
increasing the size and variety of the response data collection,
integrating log data analysis (user’s interaction history) [
With the popularity of customized voice services, evaluation on them is of more importance than ever before. In our paper, we conducted design evaluation of a sample of 100 most popular Alexa skills from ten diferent categories using a voice skill crawler. The entire evaluation was performed with respect to eight design guidelines. Our findings revealed how these selected skills followed the guidelines. Based on our findings and the real sample responses we encountered during the evaluation process, we made several suggestions for improving the design of voice skills and identified challenges as well as opportunities for future research.