Business partnerships can help businesses deliver on opportunities they might otherwise be unable to facilitate. Finding the right business partner (BP) involves understanding the needs of the businesses along with what they can deliver in a collaboration. BP recommendation meets this need by facilitating the process of finding the right collaborators to initiate a partnership. In this paper, we present a real world BP recommender application which uses a similarity based technique to generate and explain BP suggestions, and we discuss how this application is enhanced by integrating a solution that 1. dynamically combines diferent recommender algorithms, and 2. enhances the explanations to the recommendations, in order to improve the user's experience with the tool. We conducted a preliminary focus group study with domain experts which supports the validity of the enhancements achieved by integrating our solution and motivates further research directions.
Strategic partnerships are important for
businesses to grow and explore more complex
opportunities [
In this paper, our focus is on BP Connector,
a real-world application that provides
company to company recommendations, where
the companies themselves become the subject
items to recommend to each other, and the
recommendations must suit the preferences
of both parties involved. This setting is
studied under the reciprocal recommender
systems research [
BP Connector has already been deployed not reflecting the current interests and acby an organization with a large ecosystem of tual expertise of the BPs, which may degrade BPs to foster collaborations among them in not only the quality of the recommendations order to create a virtuous cycle, where a suc- but also the explanations. However, the orcessful engagement between BPs promotes ganization deploying BP Connector has acthe business interest of the instigating orga- cess to data around BPs such as the histornization itself. The system defines two roles ical sales records and product certifications, for the partnership: the beneficiary and the which, if integrated into the recommender helper. Beneficiary refers to the company logic, would improve the quality of the recwho is seeking assistance in a specific ter- ommendations and the explanations, and this ritory, technology, etc., whereas the helper can help users to make better decisions [7]. refers to the company who states that it can Although using more data has benefits, provide assistance. The system allows com- one important challenge is that data around panies to first specify whether they are seek- BPs exists in diferent heterogeneous sources ing help or asking for help, and then asks and these data sources have diferent covthem to fill in a form to specify the details erage. Moreover, there is a possibility that around their interests and expertise. Both additional data sources may become availthe beneficiary and the helper complete the able over time. To handle this, hybrid recsame forms, therefore providing information ommendation approaches can be used, which around the same features. These features can essentially fuse the benefits of multiconstitute the BP profiles and are used as ple data sources and leverage the compleboth the user and the item profiles by the mentary knowledge in order to provide betunderlying recommender to generate BP rec- ter recommendations [8, 9]. Hybrid recomommendations [5]. More specifically, a ben- menders support combining diferent recomeficiary requesting a BP connection is the menders built on diferent data sources. For user who is seeking for recommendations example, one model might be a collaboraof helper BPs, where the helper BPs consti- tive filtering recommender that uses a rattute the items of this recommendation set- ings matrix including the feedback provided ting. The initial solution used a content-based by the companies regarding their previous recommender [6] which is based on the sim- partnerships, whereas another model could ilarity between the profiles of the beneficia- be a content-based recommender. In such ries and the helpers to generate both the rec- cases, it would be important to combine exommendations and the explanations, where planations generated from diferent recomthe explanations reveal the degree of the sim- menders as well, which will assist users’ in ilarity between the two profiles. Therefore, the decision making process. the quality of the recommendations depends Motivated by these discussions, in this paper, we present our solution called Multi Source Evidence Recommender, henceforth referred to as MSER, which is built to enhance the recommendation and the explanation facilities of BP Connector. MSER can ensemble diferent recommendation algorithms that are built on top of diferent data sources. Moreover, it can receive explanations from these diferent recommenders, which are presented to the user to support their decision making process. MSER can re-rank and post-process the recommendations based on pre-configured business rules.
When we developed MSER, we were aware that diferent companies may have diferent goals when seeking a partnership, where these goals strongly influence which features and which data sources may be the most relevant to support the recommendation process. For example, company A may need a local presence for a sales opportunity, therefore the location information may be the most important factor, whereas company B may be looking for an expert in a specific technology, therefore, accurate information on product certifications and sales performance could be the most important factor. To support this, we designed MSER to enable users to provide feedback around the data sources they are interested in, in order to better align the recommendations with the users’ dynamic interests.
Integrating MSER to the BP Connector application leads to substantial changes over the initial version. These changes lead us to initially formulate two research questions: 1.
What is the diference in subjective recommendation quality between the recommendations generated by a single recommender and recommendations generated by MSER? 2. How do the users perceive the explanations generated by MSER? In order to investigate these research questions, a preliminary focus group study with domain experts is conducted which motivates us for further research.
a brief review of the related art, and then describe our solution we designed for BP Connector application in order to enhance its recommendation and explanation capabilities. Then, we present the initial focus group study and discuss our findings. We conclude with proposals for future research.
BP recommendations have been studied
considering diferent sources of data and
diferent types of methods [10]. [
Our solution, MSER, orchestrates diferent recommender algorithms that run on disparate data sources, which relates our work to hybrid recommenders [18, 9]. MSER can be considered as a recommender ensemble [19], which is a particular type of hybrid recommenders in which the recommender algorithms to combine are treated as black boxes. In [20], authors present several ap- thors argue that future research should creproaches for generating an ensemble of col- ate new kinds of information, interaction, laborative models based on a single collab- and presentation styles. To this end, MSER orative filtering algorithm. In [21], authors is designed to support combining explanapresented a hybrid recommender with an tions generated by diferent recommenders interactive interface which allows users to through dynamic user feedback, and it can adjust the weights assigned to each recom- support diferent explanation styles. mender through sliders. This proposed sys- The primary contribution of this paper is tem is designed to provide recommendations to describe how the recommendation and exon media content leveraging multiple social planation generation facilities of an existsources. With the enhancements designed ing recommender application, BP Connector, for BP Connector, we aim to enable users are enhanced through designing a solution to interact with the recommenders. In this called MSER, which combines recommendaregard, our initial choice for a new interac- tions and explanations through user feedtive UI fell on a chatbot system. Among the back. possible interaction models, chatbot systems have seen a steep increase in popularity in the recent years driven by the wide adop- 3. Proposed Solution: tion of mobile messaging applications [22]. Multi-Source Evidence They also represent a natural interface for conversational recommenders which provide Recommender (MSER) recommendations to the users through dialogue [23]. The enhancements designed for BP Connec
Considering the explanations, in [24], au- tor are encapsulated within MSER, which thors reviewed the literature on explanations is designed around four main components, in decision-support systems, where they dis- Controller, Connector Layer, Rank-Combiner tinguished between variables such as the and Post-Processor, as depicted in Figure 1. length of the explanations, their vocabulary The figure shows the high-level view of the and their presentations, and they concluded components in which the components’ interthat additional studies are necessary to as- actions are labelled in sequence to show the sess the impact of these variables. In [25], execution flow. Below, we summarize the deauthors introduced the concept of recipro- tails of these components. cal explanation where the user who is look- Controller connects the client application ing for a connection is also presented with with the underlying recommender logic, an explanation on what would be the in- thus making it responsible for orchestratterest of the other party in establishing a ing the execution flow of MSER. It exposes mutual connection. Kouki et al. [26] stud- a get_recommendations method, which takes ied how to provide useful hybrid explana- two parameters: 1. query parameters, which tions that capture informative signals from specifies the properties of the recommendaa multitude of data sources, and conducted tion request, 2. recommender weights, which a crowd sourced user study to evaluate sev- determines the weights that should be aseral diferent design approaches for hybrid signed to diferent recommender algorithms, explanations. In another work [27], authors where a weight of 0 indicates that the correproposed a taxonomy that categorizes difer- sponding recommender should be excluded ent explainable recommenders and the au- from the recommendation process.
Once a recommendation request is re- mender weights it receives from the Controller ceived from the client application through (4). The ranked list is then processed by calling the get_recommendations method of the Post-Processor which applies the business the Controller (1), Controller first forwards rules (5). Avoid recommending a firm to anthis request to the Connector Layer (2) which other firm if their business needs do not coinin turn calls the configured recommender cide or if they operate in diferent geographies systems to receive the recommendations and is an example of a business rule that BP Conthe explanations (3). The responses received nector enforces. Each recommender can send from the recommenders are then handed an explanation associated with the recomover to the Rank-Combiner together with mended BP, which is also combined by Postthe recommender weights. Rank-Combiner Processor to present the final explanation in computes the ranking of the final recom- a way that is pre-configured within the solumendation list using a linear combination of tion. Lastly, the final list is returned to the the recommendation scores [9, 28], where it client application (6). adjusts the weighting based on the recom- Integrating MSER to BP Connector. The initial version of BP Connector used a sin- as a vector of weights, and then it computes gle Similarity-Based Recommender (SBR), and the similarity between these vectors using through the adoption of MSER, the solu- the Cosine Similarity metric. The web form tion has been enhanced with two additional data represents a kind of explicit user prorecommenders: Expert Recommender (ER) file [13], and SBR tries to connect a proacand Performance Recommender (PR). ER has tive user (beneficiary) with a reactive one been serving a production application in (helper), so that the reciprocal recommendathe sales domain for more than two years, tion satisfies the preferences of both sides. therefore, the existing recommender service ER formulates the recommendation probwas plugged into the BP Connector solution, lem as an Information Retrieval process [29], whereas PR is specifically designed for BP where the sales history of a BP corresponds Connector. to a document, an attribute of a sales oppor
SBR computes the similarity between the tunity is a field of the document (e.g. counfeatures that the beneficiary and the helper try, sector, product), and an attribute value specified in the initial web forms. To achieve corresponds to a term (e.g. United States for this, SBR first represents the form parameters country; banking for sector). The beneficiary request form plays the role of the query, and a user interface of BP Connector limited the TF-IDF Similarity score1 is computed for each users to following a predefined set of steps. document, which represents the proficiency We aimed to increase the interactivity bescore of the helper BP corresponding to the tween the user and the application by designdocument. ing a dialogue-based interface that sits next
PR uses a machine learning model to pre- to original interface. From this dialogue, bendict the probability of an opportunity being eficiaries can perform the following interacwon or lost by considering the expertise of tions: 1. fill in request details, 2. receive reca BP. It computes a probability score for a ommendations, 3. guide MSER to use the rehelper to win an opportunity whose char- quired recommenders, and 4. receive explaacteristics match the requirements defined nations. A sample screenshot for the third in the beneficiary request form. This rec- interaction listed is given in Figure 2. The ommender builds a Gradient Boosting Clas- dialogue is designed to be able to elicit user sifier [30] for each helper BP in the dataset preferences towards the recommendation alusing historical sales data. gorithms. It assigns a weight of 1 to a rec
Explanations. In addition to the recom- ommender if the user expresses interest in it, mendations, each of the three recommenders or a weight of 0 if the user shows no interest provide its own set of explanations which towards it. At the beginning of the conversais combined by MSER. As for the explana- tion a weight of 1 is assigned to each recomtions, SBR provides the similarity score be- mender. The dialogue is built using Watson tween the helper request and the beneficiary Assistant2, an existing service which is one of request as an explanation. Moreover, it pro- the natural language understanding services vides four other scores, which represent the for conversational question answering [31]. overlap between the beneficiary request and the helper request in terms of technology (e.g. Analytics, Cloud, Security, etc.), business 4. Evaluation need (e.g. Consulting, Marketing, Sales, etc.), industry (e.g. Banking, Education, Health- Setup and Participants. We evaluated care, etc.) and assistance type (e.g. developing MSER as the new recommender behind BP new sales relationship, creating new services, Connector with two diferent groups. The supporting new solutions, etc.). ER, on the ifrst group involved 7 domain experts, and the other hand, provides the number of deals that second group included 5 active users of the a helper had in the past in the sector, indus- application. Domain experts were employed try, country, etc. listed in the beneficiary re- by the organization deploying BP Connector quest form. PR establishes a baseline win rate and they worked directly with BPs. They opgiven the parameters specified in the benefi- erated at a global scale (2 in North America, ciary request form. As explanation, the per- 1 in Europe, 1 in Middle East and 3 in Asia). formance of a helper is provided as a relative Active users included the users of the initial increment of the win rate over the baseline’s. BP Connector before MSER deployment. DoAs it is relative to a baseline value, perfor- main experts participated in a remote briefmance can assume negative values as well. ing meeting to get information about the user User Interaction. The original form-based study. Afterwards, they filled in a survey, which was the same for all of them, and then
apache/lucene/search/similarities/TFIDFSimilarity.html
(a) Match score explanation (b) Short explanation (c) Detailed explanation participated in a remote focus group to dis- mendations and explanations using MSER. cuss the results and provide further feedback. The surveys were similar for both groups. Active users, on the other hand, answered a During the surveys, a partnership request survey personalized to their company. This was explained, and three companies were was performed through selecting one of their recommended as potential partners, where former requests made to the initial BP Con- each recommended company had one explanector and generating a new set of recom- nation accompanying it. We experimented on three types of explanations with diferent Table 1 levels of details: 1. match score, 2. short ex- Recommendation quality perceived by the explanation, and 3. detailed explanation. Match perts for each type of explanation score explanation includes only the percentage value representing how much the of- Exp. Type Very Good Neutral Bad good fer of help from a company fits the help request, which is generated by SBR, whereas Match score 0 5 1 1 short explanation and detailed explanation are Short 2 5 0 0 formed using the explanations from all three Detailed 1 4 0 1 recommenders, SBR, ER and PR. For the explanations generated by SBR, short explanation includes only the percentage of match, allows us to explore the completeness princi(same with the the match score explanation), ple as defined in [32], where each explanawhereas the detailed explanation presents the tion includes more information than the predetails of the overlap between the ofer and vious one in order to detect where informathe request of help considering the four di- tion overload starts generating a problem. mensions; technology, business need, industry Results and Discussion. To evaluate and assistance type, as discussed in Section 3. how participants perceive the recommendaFor the explanations generated by ER, short tions from MSER, we examined their evaluexplanation includes the total number of op- ation of the recommendations with each of portunities the helper BP had in the past with the explanations provided with them. Table 1 the products listed in the beneficiary request summarizes the results for the group of exform, together with the product family that perts. As can be seen from the table, the marepresents the main area of expertise of the jority of the experts ranked the recommenhelper BP. The detailed explanation, on the dations as Good independent of which explaother hand, includes the details of this exper- nation type was provided. However, when tise, specifically, the number of opportunities they were presented with more than just a for the diferent products, countries, sectors match score, their ratings improved. One and the deal sizes requested by the benefi- of the experts said "I like that I can underciary. Finally, the explanation generated by stand the size of their experience.". Users, on PR is the same for both types. Examples of the other hand, responded as Neutral when a the three types of explanations for the same match score was provided to them; however, request are given in Figure 3. If a recom- receiving either a short or a detailed explamender did not recommend a specific BP that nation helped them to build more confidence appeared in the final recommendation list, its in the recommendations. We observed that explanation was omitted from both the short evaluating recommendations without explaand the detailed explanations. nations is dificult in this context, as one
A page of the survey showed all three cannot quantify if a partnership worked or companies with the same type of explana- not after it really happens. In our evaluation. Subsequent survey pages showed dif- tion, however, we could only evaluate the ferent types. However, the order was always judgment that the users made of a potenkept the same as follows: 1. match score, 2. tial partnership; therefore, providing users short explanation, and 3. detailed explana- with valuable explanations was key to suption, since each of the next explanations adds port their decisions. more information to the previous one. This Regarding the amount of information provided (explanation completeness), the prefer- research. As a future work, we aim to evalence of short versus detailed explanations uate the scalability of the solution by enlargwas not homogeneous among participants. ing the recommender engine behind BP ConOne participant mentioned: "Of little value nector with additional recommender systems just showing a name and a percentage match" based on additional data sources such as data for the match score type, and another one said around product certifications, ratings given "I can get an idea of the experience and type of by the beneficiaries to the helpers they conwork of each partner." for the detailed type. nected with, and implicit preferences based Some declared that the detailed explanation on users’ behaviour [33] such as requests of shows too much information and is dificult connections and responses to matches. to process, whereas others mentioned that they would like to have as much information as possible to decide on future partnerships. 6. Acknowledgements This aligns with findings in [25] about how the cost of the decision influences the expla- We would like to acknowledge the support nation efectiveness. Apart from the personal and collaboration provided by IBM CAO preferences, the presentation mode was also team: Sanjmeet Abrol, Cindy Wu and Alice important for our participants. When they Chang. were asked about interaction and visualization, personal preferences played an impor- References tant role. Participants mentioned that interactivity with the system and graphical representations of the data presented for each company are desirable. The design could therefore include an interactive interface in which users initially receive a match score, ask for a short explanation, and are able to explore the detailed explanation of each dimension individually. This would allow users to find their own balance in the explanation completeness and information overload scale.
hance the recommendation and the explanation facilities of a real-world application, BP Connector that provides company to company recommendations. An initial user study revealed that the extensions enabled by MSER can improve both the recommendation and the explanation capabilities of BP Connector, and the results motivates further //doi.org/10.1145/3366424.3383295. [10] J. Bivainis, Development of business doi:10.1145/3366424.3383295. partner selection, Ekonomika 73 (2006) [4] I. Palomares, C. Porcel, L. Pizzato, 7–18.
I. Guy, E. Herrera-Viedma, Recip- [11] J. Mori, Y. Kajikawa, H. Kashima, rocal recommender systems: Analy- I. Sakata, Machine learning approach sis of state-of-art literature, challenges for finding business partners and buildand opportunities towards social rec- ing reciprocal relationships, Expert ommendation, Information Fusion 69 Systems with Applications 39 (2012) (2021) 103–127. 10402–10407. [5] J. Leskovec, A. Rajaraman, J. D. Ull- [12] P. Xia, B. Liu, Y. Sun, C. Chen, Recipman, Recommendation Systems, 2 rocal recommendation system for oned., Cambridge University Press, line dating, in: 2015 IEEE/ACM In2014, p. 292–324. doi:10.1017/ ternational Conference on Advances in CBO9781139924801.010. Social Networks Analysis and Mining [6] M. J. Pazzani, D. Billsus, Content- (ASONAM), 2015, pp. 234–241. based recommendation systems, in: [13] L. Pizzato, T. Rej, T. Chung, I. KoP. Brusilovsky, A. Kobsa, W. Nejdl prinska, J. Kay, Recon: A reciprocal (Eds.), The Adaptive Web, volume recommender for online dating, in: 4321 of Lecture Notes in Computer Sci- Proceedings of the Fourth ACM Conence, Springer, Berlin/Heidelberg, 2007, ference on Recommender Systems, pp. 325–341. URL: http://dx.doi.org/ RecSys ’10, Association for Com10.1007/978-3-540-72079-9_10. doi:10. puting Machinery, New York, NY, 1007/978-3-540-72079-9_10. USA, 2010, p. 207–214. URL: https: [7] D. Jannach, M. Jugovac, I. Nunes, //doi.org/10.1145/1864708.1864747.
Explanations and user control in rec- doi:10.1145/1864708.1864747. ommender systems, in: Proceedings [14] X. Cai, M. Bain, A. Krzywicki, of the 23rd International Workshop on W. Wobcke, Y. S. Kim, P. CompPersonalization and Recommendation ton, A. Mahidadia, Learning to make on the Web and Beyond, ABIS ’19, Asso- social recommendations: a modelciation for Computing Machinery, New based approach, in: International York, NY, USA, 2019, p. 31. URL: https: Conference on Advanced Data Mining //doi.org/10.1145/3345002.3349293. and Applications, Springer, 2011, pp. doi:10.1145/3345002.3349293. 124–137. [8] C. C. Aggarwal, Ensemble-Based [15] R. Liu, W. Rong, Y. Ouyang, Z. Xiong, A and Hybrid Recommender Systems, hierarchical similarity based job recomSpringer International Publishing, mendation service framework for uniCham, 2016, pp. 199–224. URL: https: versity students, Frontiers of Computer //doi.org/10.1007/978-3-319-29659-3_6. Science 11 (2016) 912–922. doi:10.1007/978-3-319-29659-3_ [16] C.-T. Li, Mentor-spotting: recom6. mending expert mentors to mentees [9] R. Burke, Hybrid recommender for live trouble-shooting in codementor, systems: Survey and experiments, Knowledge and Information Systems 61 User Modeling and User-Adapted In- (2019) 799–820. teraction 12 (2002). doi:10.1023/A: [17] F. Vitale, N. Parotsidis, C. Gentile, On1021240730564. line reciprocal recommendation with theoretical performance guarantees, in: doi:10.1007/s11257-017-9195-0. Advances in Neural Information Pro- [25] A. Kleinerman, A. Rosenfeld, S. Kraus, cessing Systems, 2018, pp. 8257–8267. Providing explanations for recommen[18] C. Aggarwal, Recommender dations in reciprocal environments, in: Systems, 2016. doi:10.1007/ Proceedings of the 12th ACM confer978-3-319-29659-3. ence on recommender systems, 2018, [19] R. Cañamares, M. Redondo, P. Castells, pp. 22–30.
Multi-armed recommender system [26] P. Kouki, J. Schafer, J. Pujara, bandit ensembles, in: Proceed- J. O’Donovan, L. Getoor, User ings of the 13th ACM Conference preferences for hybrid explanations, on Recommender Systems, Rec- in: Proceedings of the Eleventh ACM Sys ’19, Association for Comput- Conference on Recommender Systems, ing Machinery, New York, NY, 2017, pp. 84–88.
USA, 2019, p. 432–436. URL: https: [27] G. Friedrich, M. Zanker, A taxonomy //doi.org/10.1145/3298689.3346984. for generating explanations in recomdoi:10.1145/3298689.3346984. mender systems, AI Magazine 32 (2011) [20] A. Bar, L. Rokach, G. Shani, B. Shapira, 90–98.
A. Schclar, Improving simple collab- [28] M. Claypool, A. Gokhale, T. Miranda, orative filtering models using ensem- P. Murnikov, D. Netes, M. Sartin, Comble methods, in: International Work- bining content-based and collaborative shop on Multiple Classifier Systems, iflters in an online newspaper, 1999.
Springer, 2013, pp. 1–12. [29] A. Costa, F. Roda, Recommender sys[21] S. Bostandjiev, J. O’Donovan, tems by means of information retrieval, T. Höllerer, Tasteweights: a visual in: Proceedings of the International interactive hybrid recommender sys- Conference on Web Intelligence, Mintem, in: Proceedings of the sixth ACM ing and Semantics, 2011, pp. 1–5. conference on Recommender systems, [30] A. Natekin, A. Knoll, Gradient boosting 2012, pp. 35–42. machines, a tutorial, Frontiers in neu[22] P. B. Brandtzaeg, A. Følstad, Why peo- rorobotics 7 (2013) 21. doi:10.3389/ ple use chatbots, in: International Con- fnbot.2013.00021. ference on Internet Science, Springer, [31] D. Braun, A. Hernandez-Mendez, 2017, pp. 377–392. F. Matthes, M. Langen, Evaluat[23] D. Jannach, A. Manzoor, W. Cai, ing natural language understanding L. Chen, A survey on conver- services for conversational question sational recommender systems., answering systems, in: Proceedings of CoRR abs/2004.00646 (2020). URL: the 18th Annual SIGdial Meeting on http://dblp.uni-trier.de/db/journals/ Discourse and Dialogue, Association corr/corr2004.html#abs-2004-00646. for Computational Linguistics, 2017, [24] I. Nunes, D. Jannach, A systematic pp. 174–185.
review and taxonomy of expla- [32] T. Kulesza, S. Stumpf, M. Burnett, nations in decision support and S. Yang, I. Kwan, W.-K. Wong, Too recommender systems, User Mod- much, too little, or just right? ways eling and User-Adapted Interac- explanations impact end users’ mental tion 27 (2017) 393–444. URL: https: models, in: 2013 IEEE Symposium on //doi.org/10.1007/s11257-017-9195-0. Visual Languages and Human Centric Computing, IEEE, 2013, pp. 3–10. [33] L. Pizzato, T. Chung, T. Rej, I. Koprinska, K. Yacef, J. Kay, Learning user preferences in online dating, in: Proceedings of the Preference Learning (PL10) Tutorial and Workshop, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), Citeseer, 2010.