Call centers data represent a valuable asset for companies, but it is often underexploited for business purposes. By call center data we mean all information that can be gathered from recording calls between representatives (or agents) and customers during their interactions in call centers. These interactions can happen over multiple different channels including telephone, instant messaging, email, web forms, etc. Some information can be collected without looking at the content of the interaction, by simply logging the system used for carrying the conversation. For example, in call centers, calls duration or number of handled calls can be measured by software for telephony communication. We call these measures standard call center Key Performance Indicators (KPIs). With standard KPIs, only limited analytics can be done providing a partial understanding of the call center performance and no information whatsoever is collected about what is going on within the interaction. Call Center Analytics is aimed at solving the above issue by enabling tapping into the content of conversations. The technology for Call Center Analytics is still in its infancy and related commercial products have not yet achieved maturity. This is due to two main factors: i) it is highly dependent on quality of speech recognition technology and ii) it is mostly based on text-based content analysis.
Our approach to Call Center Analytics is based on Interaction Mining, a new research field aimed at extracting useful information from conversations. In contrast to Text Mining (Feldman and Sanger 2006), Interaction Mining is more robust, tailored for the conversational domain, and slanted towards pragmatic and discourse analysis. In particular, with our approach we were able to achieve the following four objectives:
1. Identify Customer Oriented Behaviors, which are highly correlated to positive customer ratings (Rafaeli et al. 2007); 2. Identify Root Cause of Problems by looking at controversial topics and how agents are able to deal with them; 3. Identify customers who need particular attention based on history of problematic interactions; 4. Learn best practices in dealing with customers by identifying agents able to carry cooperative conversations. This knowledge coupled with customer profiles can be used effectively in intelligent skill-based routing1
The article is organized as follows: in section 2 we review current Speech Analytics technology and make the case for Interaction Mining approach in order to address the current business challenges in call centers quality monitoring and assessment. In section 3 we present our Interaction Mining solution based on a specific kind of pragmatic analysis: the Argumentative Analysis and its implementation with the A3 algorithm. In section 4 we showcase our solution for call center analytics and the implementation of new relevant metrics and KPIs for call center quality monitoring. We conclude the article with a discussion on the achieved results and a roadmap for future work.
Call center data contain a wealth of information that usually remains hidden. Key Performance Indicators (KPIs) for call centers performance can be classified into three broad categories (Baird 2004):
1.
Interaction Mining is an emerging field in Business Analytics that contrasts the standard approach based on Text Mining (Feldman and Sanger 2006). In Text Mining the assumption made is that input is textual and can be treated as sets of content-bearing terms. This assumption is no longer valid in conversational input. Non-content words such as conjunctions, prepositions, personal pronouns and interjections are extremely important in conversations cannot be filtered out as they bear most of their pragmatic meaning. As pointed out in Pallotta et al. (2011) there are several advantages of moving to Interaction Mining for generating intelligence from conversational content. It is important to note that while the purpose is similar, namely turning unstructured data into structured data for performing quantitative analysis, Text Mining focuses on pattern extraction from documents. This is no longer the case with conversational content as the units of information in conversational content are dialogue turns and typically they are significantly shorter than documents. This means that the input has to be fully linguistically processed in order to understand its pragmatic function in the conversation. For instance, a An example of call center analytics dashboard is available at: http://demos7.dundas.com/HVR.aspx simple turn containing just one single word like "Yes" or "No" can make a substantial difference in the interpretation of a whole conversation.
Interaction Mining tools are substantially different than those employed in Text Mining. Machine learning approaches are no longer a viable option since data are very sparse and attempts have failed in providing satisfactory results so far (Rienks and Verbree 2006;Hakkani-Tür 2009). In Pallotta et al (2011) we have provided evidences that bag-of-words approach simply is not suitable for pragmatic indexing of conversations, and therefore useless for tasks as Question Answering or Summarization. Another limitation of Text Mining is in Sentiment Analysis. As we have previously shown in Delmonte and Pallotta (2011), shallow linguistic processing and machine learning often provide misleading results. Therefore, we advocated for a deep linguistic understanding of input data even for standalone contributions such as product reviews. In Interaction Mining, Sentiment Analysis issues become even more compelling because sentiment about a topic is not fully condensed in a single turn but it develops along the whole conversation. For example, it is very common that dissent is expressed toward other the opinion of other participants in the conversation rather than to the topic under discussion. Sentences like "why do you think product X is bad?" would be simply mistakenly considered as a negative attitude to product X in a bag-of-word approach.
Current approaches to Call Center Analytics are mostly based on Speech Analytics, which is fundamentally based on Search and Text Mining technology. Recorded speech is phonetically indexed and searched: phonetic transcription is more reliable and accurate than Large-Vocabulary Continuous Speech Recognition (LVCSR) and keyword queries can easily be turned into its phonetic counterpart for search. With this approach one can search for occurrence of specific words in calls. Its simplicity is at the same time its strength and weakness. On the one hand the method is fast and accurate but, on the other hand, it is limited to its applicability for generating adequate insights on calls because the context of word's occurrence is lost and it can only recovered by physically listening to the audio excerpt where the searched word occurs.
While still very high compared to human performance, the Word Error Rate (WER)5 of LVCSR systems shows a promising trend as reported by the NIST Speech-To-Text Benchmark Test History 1988-2007 (Fiscus et al. 2008). Instead of downgrading the analysis capabilities we believe it is more appropriate to make the analysis less sensitive to WER. In other words, we want a robust solution capable of delivering approximate but still sound measurements for content-based metrics. We will show in the next sections that our approach to Interaction Mining is robust and it can properly deal with output from LVCSR systems.
Another common approach to the analysis of call center data is that of automatic call categorization through supervised machine learning (Gilman et al. 2004;Zweig et al. 2006;Takeuchi et al. 2009). These methods failed in providing satisfactory results even in very broad categories. The problem lies on the fact that data is very sparse and that huge amount of training data is necessary to achieve reasonable discriminatory power.
Unsupervised learning provides better results for domain-specific classes as shown in Tang et al. (2003). However, the sensibility to domain represents a big issue. Moreover, this type of categorization -i.e. topics of calls -helps little to understand if a call is satisfactory or not. It might be better suited for retrieval and aggregation of other quality-oriented information.
Our approach to pragmatic analysis for Interaction Mining is rooted on argumentative analysis (Pallotta 2006). Argumentation is a pervasive pragmatic phenomenon in conversations. Purposeful conversations are very often aimed at reaching a consensus for a decision or to negotiate opinions about relevant topics.
The argumentative structure defines the different patterns of argumentation used by participants in the dialog, as well as their organization and synchronization in the discussion. From this perspective, we adopted in (Pallotta 2006;Pallotta et al. 2007) an argumentative coding scheme, the Meeting Description Schema (MDS). In MDS, the argumentative structure of a meeting is composed of a set of topic discussion episodes, where several issues are discussed through the proposal of alternatives, solutions, opinions, ideas, etc. in order to achieve a satisfactory decision. Proposals can be accepted or challenged through acts of rejecting or asking questions.
The core of our solution is a system that extracts the argumentative structure of conversations. This system is based on adapting and extending the GETARUNS text understanding system (Delmonte 2007;2009). Details of the Automatic Argumentative Annotation (A3) algorithm are available in Delmonte et al. (2010).
The system has been evaluated on manually transcribed conversations from the ICSI meeting corpus (Janin et al. 2001) and annotated by Pallotta et al. (2007). With a Recall of 97.53%, we computed the Precision as the ratio between the number of Correct Argumentative Labels and the number of Argumentative Labels Found, which corresponds to 81.26%. The F-score is 88.65%.
In order to check the robustness of the A3 algorithm when applied to automatically transcribed conversations, we evaluated the A3 algorithm on similar data that were transcribed using a state-of-the-art LVCSR system (Fiscus et al. 2008). We have measured the performance of our system and observed a degradation of only 11.7% of the overall performance with a LVCSR system showing an average WER of 30% (Hain et al. 2009). These results are quite promising and, coupled with expected improvements in LVCSR technology and further tuning of the system, they provide us with a solid basis for development.
One key issue with conversation is that topics are not expressed by single words but very often by compounds. Hence, quality of topic detection can be improved if the lexicon contains domain-specific multi-word expressions. We thus run a multiword expressions extraction tool (Seretan and Wehrli. 2009) to identify the most frequent compounds in the corpus and compare them with the topics detected by the GETARUNS system. The top 10 extracted multi-word expressions6 and topics are shown in Table 1. a b While there is a predictable overlapping it is interesting to see that some domaindependent terms were detected by the multi-word extraction system but they were not included in the lexicon of our system such as "gross balance" and "and available", and "direct deposit"7 . Enriching the lexicon with these terms would greatly improve the pragmatic analysis of conversations.
In this section we present the results of an experiment where we applied our Interaction Mining technology to actual call center data. The goal was to find out if the argumentative analysis, coupled with sentiment analysis, could indeed provide useful insights for Call Center Analytics. The results show, in particular, that we were able to achieve the objectives we introduced in Section 1 and therefore implement the relevant and most requested KPIs in call center quality management.
In our experiment we used a corpus of 213 manually transcribed conversations of a help desk call center in the banking domain. Each conversation has an average of 66 turns and an average of 1.6 calls per agent. This corpus was collected for a study aimed at identifying conversational behaviors that could favor satisfactory interaction with customers (Rafaeli et al. 2007) Table 2 shows the identified COBs and their distribution in the data. Unfortunately, only a very small portion of the data (2.5%) was manually annotated with COBs, which prevented us from either performing a statistically sound correlation study or train a model.
We run our A3 algorithm to the call center data and we visualized the results with off-the-shelf business intelligence tools. We used Tableau 6.08 , which revealed to be a suitable tool for getting insightful multi-dimensional aggregations and charts into dashboards for addressing the four quality monitoring objectives mentioned in the beginning of this section.
We noticed that COBs showed a high resemblance to our argumentative categories and that they might correlate as well.
As shown in Fig. 1 the "Provide Explanation/Justification" and "Suggest" categories highly correlate with COBs. Combined with additional extracted information such as Sentiment and Subjectivity (see Pallotta and Delmonte (2011) for more details), we can safely conclude that COBs can be easily predicted by our system.
By looking at controversial topics we can identify root cause of problems in call centers. We selected the worst 20 topics ranked according to frequency of negative attitudes obtained by the Sentiment Analysis module. Fig. 2. shows a dashboard that can be used to detect controversial topics and thus help in spotting unsolved issues. The cooperativeness score is a measure obtained by averaging the score obtained by mapping argumentative labels of each turn in the conversation into a [-5 +5] scale. The mapping is shown in Table 3.
Argumentative Categories Level of Cooperativeness
Suggest 4
Propose 3
Provide opinion 2
Provide explanation or justification 1
Request explanation or justification 0 Question -1
Raise issue -2
Provide negative opinion -3 Disagree -4
Reject explanation or justification -5
The mapping is hand crafted and rooted on Bales's Interaction Process Analysis framework (Bales, 1950), where uncooperativeness (i.e. negative scores) is linked to high level of criticism, which is not balanced by constructive contributions (e.g. suggestions and explanations). This mapping provides a reasonable indicator of controversial (i.e. uncooperative) conversations.
The dashboard in Fig. 3 highlights the top 10 most discussed topics and the level of cooperation of the discussions. In the main pane, rows correspond to speakers and for each topic a square is displayed whose dimension represents the number of turns and the color its cooperativeness score. The histograms show the overall cooperativeness scores.
A critical issue in this domain is that customers are not all the same and need to be treated differently according to their style of interaction. There are agents with interpersonal skills who are able to comfortably deal with demanding customers. Agents who show consistently positive cooperativeness can be assumed to be more suitable to deal with extreme cases. Customers who have already shown negative or uncooperative attitudes could be routed to more skilled agents in order to maximize the overall call center performance (i.e. customer satisfaction). We present a dashboard in Fig. 4. where problematic customers can be identified and given a particular care. With this dashboard speakers (agents or customers) are ranked according to their cooperativeness score. In the right-hand pane, also the sentiment analysis results are displayed and compared to the overall sentiment score. The analyst can then drill through a specific customer and visualize a specific customer and the calls he/she made.
This objective results from all the insights gained through the presented visualizations. In particular, Fig. 4. with Agent filtering activated allows one to visualize overall and specific agent's behavior. Best scoring agents can be taken as models and their interaction used as models. While most of available solutions for skillbased inbound call routing are based on ACD information such as area codes for agent's language selection or based on IVR 9 for option selection. Additionally, the agent selection is often based on efficiency measures in order to optimize the costs and workload (e.g. by assigning the fastest agent to the longest queue). If this strategy might maximize efficiency, they are insufficient to maximize customer satisfaction. We advocate for skill-based call routing based on interpersonal qualities and by influencing the agent selection by cooperativeness requirements.
In this article we have presented a new approach to Call Center Analytics based on Interaction Mining, contrasting Text Mining, which is currently used in Speech Analytics. We presented an Interaction Mining tool for pragmatic analysis of conversations based on argumentation theory. We showed that our system is robust enough to deal with automatically transcribed speech, as it would be the case in Call Center Analytics. We conducted an evaluation the impact of this technology to a real case by applying our tools to a dataset of call center conversations in the banking domains. We presented the extracted information in several dashboards with the goal of implementing relevant KPIs for Call Center Analytics. As for future work we would like to explore other pragmatic dimensions beyond argumentation. This might be relevant in the Call Center domain to look at COBs that are more related to emotional support or providing personalized information, which do not relate directly with argumentation. We need to consider finer granularity in argumentative analysis, for instance at clause level. This might be helpful when a single turn carries several argumentative functions. This would definitively improve the quality of the analysis. Our goal is to implement other KPIs for the Call Center domain such as adherence to scripts and corporate image exemplification. In order to achieve these challenging objectives, new types of pragmatic analysis will be required. Finally, we would like to explore the possibility of automatically learning agent and customer profiles from our analysis in order to implement more effective skill-based call routing.