A Cross-Cultural Analysis of Explanations for Product Reviews John O’Donovan Shinsuke Nakajima Tobias Höllerer Dept. of Computer Science Faculty of Computer Science Dept. of Computer Science University of California, Santa and Engineering University of California, Santa Barbara, CA, USA Kyoto Sangyo University, Barbara, CA, USA jod@cs.ucsb.edu Kyoto, Japan holl@cs.ucsb.edu nakajima@cse.kyoto-su.ac.jp Mayumi Ueda Yuuki Matsunami Byungkyu Kang Faculty of Economics 3 Faculty of Computer Science Dept. of Computer Science University of Marketing and and Engineering, University of California, Santa Distribution Sciences, Kobe, Kyoto Sangyo University, Barbara, CA, USA Japan g1245108@cc.kyoto-su.ac.jp bkang@cs.ucsb.edu Mayumi Ueda@red.umds.ac.jp ABSTRACT Keywords Cosmetic products are inherently personal. Many people User Experience, Explanation, Decision Making, User-Centric rely on product reviews when choosing to purchase cosmet- Evaluation ics. However, reviewers can have tastes that vary based on personal, demographic or cultural background. Prior work 1 Introduction has discussed methods for generating attribute-based expla- Over the last 25 years, recommender systems have attempted nations for item ratings on cosmetic products, based on as- to help users find the right information at the right time [15]. sociated text-based reviews. This paper focuses on evalu- More recently, the proliferation of e-commerce applications ating explanation interfaces for product reviews and related supports buying and selling products in the global market attributes. We present the results of a cross-cultural user with relatively little e↵ort. Increasingly, consumers are rely- study that evaluates five associated explanation interfaces ing on customer reviews to inform purchasing decisions [8]. for cosmetic product reviews across groups of participants In many cases, product reviews are presented in summary from three di↵erent cultural backgrounds. We applied a 3 form via mechanisms such as star ratings. Such represen- by 2 within subjects experimental design in a user study tations, however, typically fail to capture the subtle opin- (N=150) to evaluate e↵ects of UI design and personaliza- ions that exist in the accompanying text-based reviews. In tion on a range of user experience metrics in a cosmetics this paper, we build on recent work that automatically ex- shopping scenario. Results of the study show that 1) Ko- tracts attributes and associated ratings from online product rean and Japanese speakers chose the most complex UI more reviews [10]. In particular, we focus on understanding how often than English speakers. 2) older participants also pre- visual representations of various types of extracted item rat- ferred more options in cosmetic product selection, regardless ings impact user experience and conversion likelihoods in an of cultural background. 3) personalization of product rat- e-commerce setting, as exemplified in Figure 1. Motivated ings did not show an e↵ect on user experience. 4) Attribute- by recent research that shows the importance of user ex- based explanations were preferred over star-ratings for all perience over traditional accuracy metrics in recommender three cultures. 5) Rating propensity evaluation showed that systems [7], we conduct a user experiment to understand Japanese provided significantly higher ratings than Korean how rating display a↵ects user experience. Specifically, we or English participants, and that Females provided higher applied a 3 by 2 within subjects design (Table 1) in an online ratings than Males, regardless of background. study (N=150) to evaluate e↵ects of UI design and personal- ization on a user experience metrics in a cosmetics shopping CCS Concepts scenario, considering the following research questions: •Human-centered computing ! HCI design and evalu- R1: Do cross-cultural preference di↵erences exist for recom- ation methods; User models; User studies; mendation interfaces? If so, what are the key predictors of these di↵erences? Permission to make digital or hard copies of all or part of this work for personal or R2: Are there cross-cultural preference di↵erences for per- classroom use is granted without fee provided that copies are not made or distributed sonalized v/s non-personalized recommender system inter- for profit or commercial advantage and that copies bear this notice and the full citation faces? on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or R3: Are there cross-cultural preference di↵erences between republish, to post on servers or to redistribute to lists, requires prior specific permission traditional (star-rating) and more granular attribute-based and/or a fee. Request permissions from permissions@acm.org. recommender system interfaces? IntRS 2016, September 16, 2016, Boston, MA, USA. Copyright remains with the authors and/or original copyright holders, 2016. R4: Are there di↵erences in rating propensities across the three cultures? If so, what are the strongest predictors of backgrounds. In contrast to their study, which compared observed rating shifts? a novel UI against a list view and assessed user experience The cosmetics domain was used for this study, since they metrics, we focus on the perception of attribute ratings ver- are sold globally and are inherently personal in nature. To sus traditional less-fine grained ratings, and on the impact explore variances in opinions on the explanation interfaces of personalization on these perceptions. A second contrast across di↵erent cultural backgrounds, participant groups were to [1] is that our work explores rating propensity across the sourced from American, Japanese and Korean cultural back- di↵erent groups. grounds. These particular groups were selected as a repre- sentative sample with diverse cultures, and because they are 3 Mining Attribute Ratings among the fastest growing markets for cosmetics.1 This study builds on a recent work [10] on attribute ex- traction from online product reviews. Specifically, we posit 2 Related Work that more explanations of a given product in the form of In this study, we focus on explanations and transparency of multiple attributes with corresponding scores (on five star recommender systems and on the (associated) role of prod- rating scales), see Figure 1, can provide benefits to poten- uct attributes mined from product reviews. Here, we discuss tial customers. In the prototype of the proposed recom- several related work in these areas. mender system, both personalized information (Simgroup ratings: “Users similar to you rate this item as”) and mul- Product Attributes To understand consumer behavior in tiple product attributes extracted from a review text are economics, research has focused on the di↵erent attributes added as features. Through an online user study, we apply and uncertainties that consumers consider when purchasing both novel approaches as controlled variables to the proto- a product [8, 13]. For buyers, these attributes play impor- type design and investigate the preference of the users to tant roles when deciding to purchase a product. More im- such features across demographic backgrounds, particularly, portantly, attributes vary widely across product types and cultural backgrounds (English, Korean and Japanese). users’ personal tastes. For example, [3] study the e↵ects of search attributes and provide a comparison between tradi- 4 Interface Design tional and online supermarkets. A recent study on descrip- tion and performance uncertainty [4] focused on the diffi- culty in assessing the product’s characteristics. Building on B works such as [13] that show advantages of using fine-grained KO Good Moisturizing Lotion Good Moisturizing Lotion 당신과 비슷한 사용자의 평점 전체평점 C mean rating (B~E) product attributes in the recommendation process, we aim JP 상품 리뷰 好みが似ているユーザによる 合評 Good Moisturizing 선블록 냄새가 나지 않는 적당히 가벼운 로션, 괜찮아요! Lotion Customer Reviews Users similar to you rate this item as この商品に するレビュー to further our understanding of the role of fine-grained prod- 약간 느낌이 두텁기는 하지만 미끈거리지는 않네요. 바르고 나면 피부가 엄청 뽀송뽀송 해 ENサンブロック臭のしない，程よいトロトロ感の良い化粧水です！ 지진 않지만 2개월 사용Reviews 후에 얼룩덜룩on했던 this item 고르게 바뀌고 더 밝아졌어요. 확실 피부톤이 simgroup rating (C, E) 이 리뷰를 쓰신 분의 세부 평점 E 히 겨울철에濃厚でクリーミーなのに，アブラっぽくない．お肌がメチャクチャソフトになる感じでは 흔하게 생기는 각질이나 자외선으로 인한 손상은 늦춰주는듯 합니다. Nice, mid-weight lotion with no sunscreen smell 보습 このレビュー投稿者による項目別評 ないけれど，普通に２ヵ月間使用したら，明らかにお肌のキメが整ってきましたし，明る uct attribute ratings in consumer decisions. Thick and creamy but not greasy. 더くなってきました．間違いなく冬のお肌の Doesn't make my skin ultra-soft, but after い味方です． 피부탄력 읽기 A two months of regular use, my skin tone has very obviously evened out and glows to boot. Definitely staves off scaly winter skin. 안티에이징 This 保 reviewer rate this item as ハリ· 力 Explanation and Transparency in Recommendation Within もっと む 가격 Moisturizing アンチエイジング 천연재료사용 Tightening skin the recommender systems research community, there is an Read 강추입니다. 역시 이 회사 제품은 제가More 예상한대로네요. 구입 후 리뷰 남깁니다. 상품 패키징 퀄리티가 상당히 좋구요, 최상급 상태로 매우 빠른 배 これ良いです！ さすがはこのブランド，期待通り！ 송도 마음에 듭니다. 와이프가 며칠 사용해 봤는데 확실히 얼굴에 광택이 달라졌네요. 한 D 브랜드 コスト オーガニック 향 Anti-aging Cost ブランド 번 써볼 만 한 듯 합니다. に素敵なパッケージで送られてきました．本 に到着も早く， 態も良 この商品は本 Organic increasing understanding of the need for user-centered eval- It's wonderful. かったです．私の妻は 明るくなるのです． What I expected from this brand. 日間この商品を試したのですが，使用すると明らかに妻の顔が I was sent this to review for Alina. The product ships in a really nice package, 더 읽기 arrived extremely fast and was in excellent condition. 香り Brand Scent uations [12]. Recent keynote talks [2] and workshops [14] もっと My wife gave it a try for a couple days. It puts a noticeable shine on the face. む attribute weights (D, E) have helped to highlight the importance of this topic. In Read More this paper, we follow Knijnenburg et al.’s [9] argument for a framework that takes a user-centric approach to recom- mender system evaluation, beyond the scope of recommen- dation accuracy. In contrast to that work however, we argue Figure 1: Screenshots of the interface used in the that decision quality is an important evaluation metric that online user study. The annotations A-E show the goes beyond the user experience metrics described in [9], and items that varied in each condition, as shown in Ta- further, that it can be used to explain observed usage pat- ble 1. We designed a novel user interface for product review terns for search and recommendation tools. Garcia-Molena pages based on the feedback we received from a preliminary [6] described di↵erences and similarities between search and user study (N=100). We performed the study with a simple recommendation, and argued that interactive interfaces can design layout to test the di↵erent visual conditions outlined help users understand and use these tools in more efficient in Table 1. Participants gave feedback on their preference ways. Along the same vein, it has also been recognized for each UI in a virtual shopping scenario. They were also that many recommender systems function as black boxes, required to leave a comment on the interface design. For providing no transparency into the working of the recom- example, they reported the benefit of the new features, such mendation process, nor o↵ering any additional information as “I like the level of detail it has related to the product ”, to accompany the recommendations beyond the recommen- and suggested preferred features, such as “More alive col- dations themselves [7]. To address this issue, static or in- ors” / “More explanations and ratings”. The collection of teractive/conversational explanations can be given to im- 100 comments were manually assessed, and improvements prove the transparency and control of recommender sys- were made to the UI, including shortened review text with tems. Research on textual explanations in recommender “read more” button and breakdown of multiple attributes systems to date has been evaluated in wide range of do- extracted from the review text on stars. The revised design mains (varying from movies to financial advice [5]). From a is shown in Figure 1. cross-cultural perspective, Pu and Chen performed a related study that evaluated perceptions of di↵erent recommenda- 5 Experimental Setup tion interfaces in [1], using subjects from Chinese and Swiss Figure 1 shows an example of the refactored interface for 1 http://polishcosmetics.pl/Korean-Market-Analysis.pdf a sample product review. To test our hypotheses above, a Table 1: Overview of the controlled variables for the online user study. UI Config non-personalized personalized (no information from similar users) (with social data from similar users) review text only A: product review text review text with star B: A + mean rating on stars C: A + mean rating and the rating from rating active user’s simgroup on stars review text, star rating D: B + attribute weights computed from E: C + attribute weights computed from and attributes current review text (on stars) current review text (on stars) 6 country English value 4 Japanese Korean 2 Text Only(A) NP−Star(B) P.Star(C) NP−Attr(D) P−Attr(E) variable Figure 2: Preferred User Interface by Culture. Figure 4: Cross-cultural perspective of helpfulness of the five evaluated interfaces. Mean Rating of Frown (5 point Likert Scale) Mean Rating of Smile (5 point Likert) Neutral Expression (5 point Likert) 4.4 4.8 1.9 4.0 4.4 1.7 3.6 4.0 Figure 3: Preferred User Interface by Age. 1.5 3.2 3.6 n=53 n=23 n=53 n=23 n=53 n=23 3x2 within subjects experiment was conducted, controlling Female Male Female Male Female Male Gender Gender Gender for personalization, and rating type, as shown in Table 1. The study (N=150) was performed on the crowdsourcing Figure 5: Di↵erence in rating propensity by gender. platform, Amazon Mechanical Turk. Each participant was shown a randomly ordered set of 5 di↵erent design layouts accordingly, an increased need to explore user ratings on fine corresponding to the treatments in Table 1, and were asked grained product attributes (see Figure 3). to rank them in order of preference. They were also asked to rate the helpfulness of each. Participants were evenly Personalization and Rating Type Figure 4 shows the re- balanced across cultural backgrounds. All participants were sults of perceived usefulness of the interfaces, broken down shown with the five interfaces in random order. The content by cultural groupings. Each UI condition is shown as a group was shown in their primary language based on their cultural on the x-axis, and each group contains the mean utility score background. Overall, participants took between 5-10 min- for the three cultural groups. The x-axis groups (UI treat- utes doing the study, and were paid $1.50 for their time. ments) are also ranked from left to right based on number of Questions were added to test for user attention level and for visible features (UI complexity). This graph shows several language proficiency, including identification of di↵erences interesting e↵ects: first, there is a general preference across between UIs and simple math questions written in the ap- all groups for the attribute-based representations (groups D propriate language. After filtering our data based on these and E, on the right side), over less granular, star-ratings metrics, group sizes were 39, 25 and 12 for English, Japanese or text-based UIs. This is a promising result that indicates and Korean, respectively. Participant age ranged between that attribute extraction and visualization has a positive ef- 18-64 with an average of 26. Gender groups were not evenly fect on Ux. The second interesting result is that within the distributed, as expected for the cosmetics domain, with 70% star-rating group (2nd and 3rd group) and the attribute- female and 30% male. rating (4th and 5th) groups there is no notable di↵erence 6 Results between the personalized and non-personalized treatments. This result tells us that the granularity of presented ratings Perception and Rating Differences Figure 2 shows the has more positive impact on user experience than the percep- results for the UI ranking task, broken down by age. The tion that the ratings come from similar users. To investigate result shows a clear preference for design E in all groups, but this result in more depth, a followup experiment is planned there is a significant increase in that preference for partici- with a large corpus of product reviews collected from Ama- pants over 40 (shown on the right side). This e↵ect was also zon.com [11] 2 to compute actual similarity scores based on seen from 100 participants in the preliminary study. Inter- user profiles. This would clearly give better insight into the face E, shown in Figure 1, shows the most information, and observed e↵ect. Figure 4 also answers R2, in that there are allows users to understand how users similar to them rate no significant di↵erences between the cultural groups within individual product attributes. This e↵ect might be a result 2 of specific preferences for cosmetics developing with age, and http://jmcauley.ucsd.edu/data/amazon/ Mean of Neutral Expression (5 point Likert) step is to evaluate on real product data. The authors plan a Mean Rating of Frown (5 point Likert) Mean Rating of Smile (5 point Likert) follow-up study to compare LDA and dictionary-based ap- 4.5 proaches to product attribute extraction, and to explore how 2.5 5.0 the resulting attributes can improve explanations, and user 4.0 2.0 profiles for collaborative filtering. Additionally, a more de- 4.6 tailed evaluation of the di↵erent rating propensities across 1.5 3.5 cultures is underway using a larger number of participants 4.2 and multiple product domains. 1.0 3.0 n=39 n=25 n=12 n=39 n=25 n=12 n=39 n=25 n=12 En Jp Ko En Jp Ko En Jp Ko Culture Culture Culture 8 References [1] L. Chen and P. Pu. A cross-cultural user evaluation of product recommender interfaces. In Proceedings of the 2008 ACM Figure 6: Di↵erence in rating propensity by culture. Conference on Recommender Systems, RecSys ’08, pages 75–82, New York, NY, USA, 2008. ACM. [2] E. H. Chi. Blurring of the boundary between interactive search each UI treatment, although the Japanese showed a trend and recommendation. In Proceedings of the 20th International towards favoring the more complex UI treatments. Conference on Intelligent User Interfaces, pages 2–2. ACM, 2015. Rating Propensity For some users, rating an item with a [3] A. M. Degeratu, A. Rangaswamy, and J. Wu. Consumer choice specific number of stars can have very di↵erent meanings. behavior in online and traditional supermarkets: The e↵ects of User ratings on items serve as the basis for most collabo- brand name, price, and other search attributes. International Journal of research in Marketing, 17(1):55–78, 2000. rative recommendation techniques, but they tend to ignore [4] A. Dimoka, Y. Hong, and P. A. Pavlou. On product uncertainty such di↵erences when computing neighborhoods for recom- in online markets: Theory and evidence. Mis Quarterly, 36, mendation. Further, little work has been done to under- 2012. stand cross-cultural di↵erences in rating propensity. Since [5] A. Felfernig, E. Teppan, and B. Gula. Knowledge-based recommender technologies for marketing and sales. Int. J. these participant groupings were available our experimen- Patt. Recogn. Artif. Intell., 21:333–355, 2007. tal setup, a logical step was to evaluate rating propensities [6] H. Garcia-Molina. Thoughts on the future of recommender within each of the cultural groups, to serve as both an in- systems. In Proceedings of the 8th ACM Conference on Recommender Systems, pages 1–2. ACM, 2014. dependent result, and as a weighting factor for the analysis [7] J. L. Herlocker, J. A. Konstan, and J. Riedl. Explaining in Figure 4. Each participant was shown three randomly collaborative filtering recommendations. In ACM conference on ordered faces, showing expressions with happy, neutral and Computer supported cooperative work, pages 241–250, 2000. sad expressions. They were asked to rate the ‘happiness’ per- [8] Y. Kim and R. Krishnan. On product-level uncertainty and online purchase behavior: An empirical analysis. Management ceived in each on a five point Likert scale. Figure 5 shows Science, 61(10):2449–2467, 2015. the results by gender (for all groups). Interestingly, there [9] B. P. Knijnenburg, M. C. Willemsen, Z. Gantner, H. Soncu, is a trend for Females to rate higher than males, and the and C. Newell. Explaining the user experience of recommender di↵erence becomes more pronounced for the ‘happy’ expres- systems. User Modeling and User-Adapted Interaction, 22(4-5):441–504, 2012. sion, shown on the rightmost plot of Figure 5 with a mean [10] Y. Matsunami, M. Ueda, S. Nakajima, T. Hashikami, di↵erence of 0.7 (relative increase of 16%, p<0.005). Figure S. Iwasaki, J. O’Donovan, and B. Kang. A method for 6 shows the results of the rating propensity analysis broken automatic scoring of various aspects of cosmetic item review texts based on evaluation expression dictionary. In Proceedings down by cultural group. Again, the graphs represent mean of the 24th International MultiConference of Engineers and rating for sad, neutral and happy expression ratings from left Computer Scientists, IMECS ’16, pages 392–397. IAENG, to right, respectively. Here, we see a clear trend for higher 2016. ratings in the Japanese group across all three expressions. [11] J. McAuley and A. Yang. Addressing Complex and Subjective Product-Related Queries with Customer Reviews. ArXiv While this is only a small-scale initial study, we believe that e-prints, Dec. 2015. this is an important result for the study of recommender [12] S. M. McNee, J. Riedl, and J. A. Konstan. Being accurate is system performance across di↵erent cultures in general, and not enough: How accuracy metrics have hurt recommender a follow-up study on propensity of ratings for recommender systems. In Extended Abstracts of the 2006 ACM Conference on Human Factors in Computing Systems (CHI 2006), 2006. systems is planned to investigate this further. [13] J. O’Donovan, B. Smyth, V. Evrim, and D. McLeod. Extracting and visualizing trust relationships from online 7 Discussion and Future Work auction feedback comments. In IJCAI, pages 2826–2831, 2007. [14] J. O’Donovan, N. Tintarev, A. Felfernig, P. Brusilovsky, This study applied a 3 by 2 within subjects experimental de- G. Semeraro, and P. Lops. Joint workshop on interfaces and sign in a user study (N=150) to evaluate e↵ects of UI design human decision making for recommender systems (intrs). In and personalization on a range of user experience metrics H. Werthner, M. Zanker, J. Golbeck, and G. Semeraro, editors, in a cosmetics shopping scenario using participant groups RecSys, pages 347–348. ACM, 2015. [15] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl. from three di↵erent cultural backgrounds. Results of the Grouplens: An open architecture for collaborative filtering of study show that 1) Korean and Japanese speakers chose the netnews. In Proceedings of ACM CSCW’94 Conference on most complex UI more often than English speakers. 2) older Computer-Supported Cooperative Work, pages 175–186, 1994. participants also preferred more options in cosmetic product selection, regardless of cultural background. 3) personaliza- tion of product ratings did not show an e↵ect on user expe- rience. 4) attribute-based explanations were preferred over star-ratings for all three cultures. 5) Rating propensity eval- uation showed that Japanese had significantly higher ratings than Korean or English, and that Females provided higher ratings than Males, regardless of background. A clear next-