Now a days every industry no matter what domain has customer service associates. Organizations with large number of customers have lot of data and events with respect to customer services. There are some unwanted events which causes customer dissatisfaction or customer attrition. These may be the rare events which may be unfavorable to the business and remain unidentified in huge data. For example, customer calls to the customer service associate and waiting in a long waiting queue, or transferred to multiple departments etc. This manuscript provides the real time algorithm to prioritize the inbound calls (inbound calls are from customer-to-customer service associate) in order to improve customer satisfaction and engagement experience by learning their behavior and detecting anomalous inbound calls. While working on fast increasing volumetric real-life data, it is impossible to tag each observation for modelling. This manuscript proposes an unsupervised machine learning algorithm for real time behavior learning and anomaly detection in inbound call data records. This is an unsupervised hybrid fuzzy algorithm which supports the real time untagged data.
In most of the business organization the biggest stakeholders are customers and it is very important to maintain the stakeholder’s interest and engagement in business. One of the biggest customer communication department is customer service department. Customer Service representatives/associates are responsible to grow, retain and improve customer experience and engagements. Call data records is one of the channels where customer directly communicate to business representatives and if something goes wrong here will result in customer dissatisfaction customer attrition. It is important to understand the call records behavior in order to ensure customer satisfaction either by prioritizing or by attend customer without delay in real time. We assume that there are rare call records which are lets say are in waiting in extremely long queue, or kept on hold for long time, call drops immediately after connection etc. The aim to detect these rare records.
With the fast growing data in every field, the possibilities of unusual activities within the data increases. These activities remain unidentified in huge data and may affect the system in long term. If the system is business, then these unusual events could cause business a huge loss. Hence, it is important to find out these rare activities.
The rare or unusual activity in the dataset is called as an anomaly. The phenomena of detecting the rare event or observation from a dataset is called as anomaly detection. To perform anomaly detection, it is required to learn the patterns of the activities.
This paper proposes a machine learning hybrid unsupervised Fuzzy Clustering algorithm for anomaly detection, which detects the unusual behavior of the inbound calls based on the various factors occur in call like hold time, queue time, department to which call was made, number of transfers within a call, whether it is a weekday/weekend, whether the time of call is day/night etc.
The anomaly detection algorithms provide the rare events which occurred in groups, as
point/singular event or with context
The input to the algorithm includes all the features in call data records. A synthetic dataset with 100k records was used to implement the algorithm. The advantage of using this methodology is that the freedom to go to granular level. For example: the hierarchy like, day of week, time of the day, department where call has been made and so on. The hierarchy is maintained for each call record and the output to the hybrid algorithm are a tagged record, where each record is tagged as anomalous or normal call. The methodology is the hybrid approach where the first step is to compute the continuous features approximate threshold values among the hierarchies and the second step is to tag the record as anomalous or normal.
A fuzzy membership function characterizes fuzziness in a fuzzy set, (whether the elements in the set are discrete or continuous) in a graphical form for eventual use in the mathematical formalisms of fuzzy set theory.
The Fuzzy Inference System (FIS) is composition of fuzzification, defuzzification member functions. Figure 1 is the high-level block diagram of Fuzzy Inference System. Fuzzy Inference System is the process of formulating input/output mappings using fuzzy logic.
Fuzzy Membership Function (FMF)
: Є 0,1 where 0,1 means real numbers between 0 and 1 (including 0,1).
There are multiple variations of fuzzy membership function. Few of them are: ……Equation (1) …..Equation (2) a. Triangular b. Trapezoidal c. S-function fuzzy membership d. Z-function fuzzy membership e. Gaussian fuzzy membership This is the first step of the hybrid algorithm for anomaly detection. This module consumes and produce the real-world continuous values also called as crisp values.
•
Fuzzification: This step takes crisp-values or the real value as the input, and returns the real
numbers between 0 and 1, see equation 1 and 2. This algorithm is implemented using the
equation 3, which denotes the S-function fuzzy membership in Figure 2.
centroid based defuzzification
= − − 0, < , ≤
≤ 1, > ……Equation (3) ……Equation (4) >16 /: ) **+ 2 & '( 7 11+ = 0.15 ℎ5)
To understand the Fuzzy Membership function, let’s have a look to an example.
Figure 4 depicts a fuzzy rule which defines vehicle speed based on the temperature and climate. Fuzzy membership function of type z, type s and triangular are used to define different temperature rules in Figure 4 (a) and fuzzy membership function of type z and triangular are used to define different weather rules in Figure 4 (b) for fuzzification and fuzzy membership function of type z and type s are used to define different speed rules in Figure 4 (c) for defuzzification. Given a temperature as 16℃ and weather as sunny 25% these rules define the speed of the vehicle is 38 M/hr. An example to understand fuzzy membership function is given in Figure 4: Fuzzy membership function: % &'( ) **+,-. /-01 ) **+ 2 & '( % &': 7 11+ 408'5 93 :';6 3',, *-,-. /-01 45-(6 2 & '( <0/ ) (/ 76''= >16 /: 45-(6 2 & ':';6'5 / 5' = 16℃ , A /6 /: ) **+ 2 & ' 400& = 0.3<0/ = 0.6 >16 /: 45-(6 2 & '7 11+ = 25% , A /6 /: ) **+ 2 & ' 7 11+ = 0.154&0 =+ = 0.35 93 <0/ = 0.6 , A /6 /: 45-(6 2 & '( 76''= = 38 G/
Cluster formation is the second step of the hybrid machine learning algorithm. This module consumes the output from the fuzzy membership function and creates a hyper spherical cluster of radius ‘r’. This parameter is a hyperparameter in the algorithm, which is assigned as any real value. The evaluation of the instances is done based on the location of the instance in multi-dimensional space. In other words, the tagging is performed by analysing the location of the instances in multi-dimensional space. There are following steps to identify an anomalous instance which is also shown in Figure 5:
Any new instance which lies in hyper spherical cluster or on the cluster’s circumference is tagged as anomalous instance.
Any instance which lies outside the hyper spherical cluster and in quadrant Q1 are anomalous instances, see figure 5.
Any instance which lies outside the hyper spherical cluster and in quadrant Q2 is anomalous with respect to feature vector on Y-axis.
Any instance which lies outside the hyper spherical cluster and in quadrant Q4 is anomalous with respect to feature vector on X-axis.
Any instance which lies outside the hyper spherical cluster and in Quadrant Q3 are tagged as normal instances.
Due to the graphical limitations 3-dimenstional space and tagging in 2-dimensional plane is shown instead of multidimensional space in Figure 5. Figure 5 shows the cluster with centroid (x, y, z) with some radius ‘r’, 4-quadrants. The centroid of the hyper-spherical cluster is computed by Fuzzy Membership Function.
This hybrid model was implemented on synthetic Call Data Records for Customer service dataset. The inference of the results was done manually because it is an unsupervised machine learning algorithm and tagged dataset is unavailable. This technique is efficient for large data while working with granular hierarchies. Granular level represents day, time and department when the call was made, call transfer, queue time, hold time in the hierarchy order. Hence suitable for industrial solutions. Since the real tags are unavailable in the dataset, the results generated by proposed hybrid fuzzy-clustering algorithm is compared with another hybrid algorithm (clustering and xg-boost). Where it has been observed that tags associated with 99,928 records out of 100k records matched and tags associated with only 72 records out of 100k records mismatched. Which means that they both generated tagged results with 0.07% of the difference in tagging.
A synthetic dataset of size 3 million rows was created. A random sample of 100k records was used for this experiment. There are mixed featuresin dataset. The sample dataset is provided in table 1.
This technique was implemented on the synthetic dataset for seven days of the week of size 100k size and findings are as follows: 1. On manual inference it has been observed that there are no false alarms for anomalous class. 2. The ratio of anomaly vs nominal is exceedingly small 0.001% that shows class imbalance which is quite challenging with supervised learning. 3. After comparing the proposed fuzzy-cluster based algorithm and another novel clustering-xg Boost algorithm it was observed that the rate of differences gradually decreases with increase in size of the dataset. 4. However, while implemented on granularity level, it was observed that this approach performs better with huge data. 5. Since the dataset is unlabelled, the inference is possible by comparing or by manual interpretations.
6. The real time solution is to train supervised probabilistic model with the resulting tagged data.
Novel proposed hybrid fuzzy-clustering machine learning algorithm has potential in multiple domains like anomalies in telecom, astronomy, automotive vehicles, performance measure of an application on a platform, call service centres etc. and industries like medical, business, airlines, logistics etc. This manuscript explains what anomalies are and anomalies can occur in everyday life like covid19 pandemic, attacks, wars, information security, servers, IT, vehicles, nature, weather etc.
This experiment has shown one real life application for behavior learning and anomaly detection. The proposed approach shows a potential machine learning solution to business gaps in industries. There is no existing robust solution for this gap. The fuzzy-clustering algorithm is completely unsupervised and hence is suitable for unlabelled real-life data. The advantages to this approach are it maintains the hierarchy and tags the instances as normal or anomalous among the hierarchy, proposed approach performs best on big data. There are two steps to this approach the first module uses fuzzy membership function, and the second module uses clustering.
The limitation to this approach is it does not perform well on small dataset and one should be careful while assigning the hyperparameters.