Natural Language Processing, briefly NLP, will lead in the next years the revolution of the Artificial Intelligence for the Insurance industry. There are several opportunities to employ NLP in the insurance activities, from claims processing to fraud detection and chatbots. In the marketing field, NLP can be used to monitor the sentiment analysis of feedback that people publish on diferent social networks to better consider insured needs or extract insight risks. Textual analysis of claims and classification can simplify the claims processing to reduce time treatment, operational errors or provide help in fraud detection. Underwriting process can be improved by a better textual assessment. The workshop had the goal to show NLP techniques on fraud detection by the disaster Tweets data set from Kaggle classification competition.
ance industry. [
-Marketing: NLP can be used to monitor the sentiment analysis of feedback to better consider insured needs, to monitor risks insight on what people are thinking about a particular product, to extract information about expected trends to improve marketing strategy.
-Underwriting: using Optical Character Recognition (OCR) and NLP is possible to extract information from medical reports and help underwriters in a better quote of the insurance coverage. NLP can categorize patients’ diseases and retrieves correlation between some symptoms and the likely cost of treatment for the Insurance Company.
-Reserving: the analysis of claim reports during the ifrst notification of loss can improve the reserving process for severe claims.
-Claims processing: textual analysis can simplify the trial reducing the time treatment of the process and reducing operational mistakes.
-Risk management: text classification is useful in the risk assessment giving help in fraud detection.
ing model, is better to understand data involved in the project, and Exploratory Data Analysis is a block between data cleaning and data modeling with the goal to undercheck relationships between variables of a data set with
Kaggle classification competition. 1
an application of NLP techniques in the Insurance Fraud
the Insurance world
CEUR htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org) https://www.kaggle.com/competitions/nlp-getting-started
”text”, ”target” and other 2 columns: ”keyword” and ”location”.
Text of the tweets are written in English for the almost whole data frame.
The data set has a length of 7.613 tweets with an average of 15 words per tweet, an average of 6 characters word length, and roughly 5 stop words per tweet.
Tweets are largely coming from the USA followed by UK and Canada. From the word cloud tool appears words “earthquake”, “deeds”, “reason”, “forest” and “fire” as the most common terms.
4.1.1. Named Entity Recognition and Part of Speech Tagging generate tree representation of the sentence. POS tagging works as a prerequisite for further NLP analysis such as chunking, syntax parsing, information extraction, machine translation, sentiment analysis, grammar analysis and word-sense disambiguation. 4.1.2. Text cleaning and N-grams
as text cleaning, to generate features, extracts patterns in text and further analysis. The aim of this process is preparing raw text for Natural Language Processing.
Text cleaning is based on several steps starting with the text normalization, proceeding with removing Unicode characters and stop words, and ending with performing stemming/lemmatization.
The analysis went ahead with N-grams, a contiguous sentence of n items generated from a given sample of text where the items can be words. N-grams can be used as features from text corpus for Machine Learning models, moreover they are useful in autocompletion of sentences, or speech recognition helping to predict the next word that should be occurring in a sequence.
In the workshop N-grams were performed by Bagof-Words (BoW) approach retrieving top occurrence of unigram, bigrams and trigrams in the data set.
Bag-of-Words [
Each word is represented by a one-hot vector, a sparse vector in the size of the vocabulary. The Bag-of-Words feature vector is the sum of all one-hot vectors of the words. 4.1.3. Word Embedding Text Analysis tools are used by Companies to extract The second level to transform raw text into numerical valuable insights from text data. features is given by Word Embedding with the goal to
The first approach followed was to explore Named En- generate vectors encoding semantic meanings:
individtity Recognition (NER) and Part of Speech (POS) Tagging ual words are represented by vectors in a predefined
by the use of “Spacy” library. vector space [
A named entity is a proper noun that refer to a specific to have a similar representation. In the workshop Word entity like location, person, date, time, organization, etc. Embedding was performed by Word2vec using “gensim” Named Entity Recognition systems are used to identify library. Word2vec is a neural network that try to maxiand segmentize named entities in text, and you can find mize the probability to see a word in a context window, them in application such as question answering, infor- the cosine similarity between two vectors. This task was mation retrieval, and machine translation. In Insurance, explored by two architectures: continuous bag-of-words NER applications can be found in customer care to clas- (CBOW) model that try to predict a word from its context sify customer complaints. and by continuous skip-gram model that try to predict
A POS Tagging is the process on marking tokens in a the context from a word. part of speech like nouns, adverbs, verbs, punctuation, etc. The purpose of a POS tagger is to assign grammatical information to each token and in this way is possible to 4.1.4. Topic Modelling
a form of unsupervised learning that works discovering hidden relationships in the text, more precisely the purpose is to identify topics in a document.
The purpose of the workshop was to discover new available and best performing tools for NLP, and for this activity was explored the use of BERTopic developed by Maarten Grootendorst. 3
There are four key components in BERTopic [
It starts generating several document embeddings to represent the meaning of the sentences using a pretrained language model: BERT (Bidirectional Encoder Representations from Transformers).
Given the huge dimension of vectors generated, is applied a general non-linear dimensionality reduction technique: UMAP (Uniform Manifold Approximation and Projection).
At this point the reduced embeddings are clustered with HDBSCAN (Hierarchical Density-Based Spatial Clustering). It finds clusters of variable densities converting DBSCAN into hierarchical clustering.
To retrieve topics from clustered document is applied a modified version of TF-IDF (Term Frequency-Inverse Document Frequency): the class-based TF-IDF procedure, where the class represents the collection of documents merged into a single document per each cluster.
Input features for BERTopic has been generated by
TF-IDF that is an extension of Bag of Words, where terms
are weighted and in this way are highlighted words with
useful information [
Interesting results are coming from Word Embedding, where the vocabulary is similar between the two architectures models with these relevant words: “earthquake”, “forest”, “evacuation”, “people”, ’wildfire’, “california”, “flood”, “disaster”, “emergency”, “damage”.
What is changing is the similarity between words, the CBOW model usually provides a similarity between words lower than the one provided by the Skip-Gram model.
The Word Embedding exploration ended reducing the vectors dimension with the application of the Principal Component Analysis giving the opportunity for words visualization into two dimensions.
After that, was the turn of the Topic Modelling with
BERTopic. The tool was used in easy way, without fine
tuning parameters, using TF-IDF features as input and
4.2. Results with the arbitrary choice of ten topics.
Participants asked how to trust in results, so the job
From the N-grams analysis the top occurrence words has been completed with the coherence score
evaluaare: “people”, ”video”, ”crash”, ”emergency”, and ”disas- tion of BERTopic and the comparison with the Latent
Dirichlet Allocation (LDA) model [
The same relevant words from BERTopic appears also The attention mechanism was introduced to improve in LDA, but with diferent probabilities. The evaluation the performance of the encoder-decoder model for mahas been done with the UMass coherence score that calcu- chine translation. The idea behind the attention mechalates how often two words appear together in the corpus. nism was to permit the decoder to use the most relevant The perfect coherence is in 0, and it usually decrease parts of the input sequence in a flexible manner, by a with the rising number of topics. The issue with the LDA weighted combination of all of the encoded input vecmodel was that the number of trained documents in each tors. chunk has been reduced to make it converge. From re- With Transformers we had reached the third level of sults BERTopic shows a number closer to 0 (roughly -14) vectorization technique in NLP, the Contextual Embedthan LDA model (roughly -18), though the result can be ding [9]. Both traditional Word Embedding (word2vec, improved tuning the model. Glove) and Contextual Embedding (ELMo, BERT), aim to learn a continuous (vector) representation for each word in the documents. 5. Text Classification The Word Embedding method builds a global vocabulary using unique words in the documents by ignoring 5.1. Activity the meaning of words in diferent context. Hence, given The approach followed in this chapter is the classifica- a word, its embedding is always the same in whichever tion task: given a target variable the aim is to predict if sentence it occurs, and for this reason, the pre-trained a tweet can be considered a “disaster” or otherwise “not Word Embeddings are static. disaster”. Twitter has become an important emergency Contextual Embedding methods are used to learn communication channel, because people by smartphones sequence-level semantics by considering the sequence are able to announce an emergency they’re observing in of all words in the documents. The embeddings are obreal-time. For this reason, more agencies and Insurance tained from a model by passing the entire sentence to Companies are interested in monitoring Twitter. More- the pre-trained model. The embeddings generated for over, it’s not always clear whether a person’s words are each word depends on the other words in a given senactually announcing a disaster, so this task can be linked tence. The Transformer based models work on attention to a Fraud Detection task. mechanism, and attention is a way to look at the relation
The approach followed was to use Transfer Learning between a word with its neighbours, and for this reason,
[
The idea behind Transfer Learning is to try to store The usual workflow of BERT consists of two stages: the knowledge gained in solving the source task in the pre-training and fine-tuning. source domain and apply it to another similar problem The attention mechanism in the Transformer allows of interest, it is the same concept of learning process by BERT to model many downstream tasks, such as sentiexperience, so the aim is to exploit pre-trained models ment analysis, question answering, paraphrase detection that can be fine-tuned on smaller task or specific data and more. In this workshop has been used the “ktrain” sets. [10] library, a low-code library developed by Arun S.
Bidirectional Encoder Representation from Transform- Maiya 4 that provides a lightweight wrapper for “Keras”, ers (BERT) is one of the most popular state-of-art NLP making it easier to build, train, and deploy Deep Learning approaches for Transfer Learning, published by Google models. in 2018 [7].
BERT is a bidirectional multi-layer Transformer model 5.2. Results that exploits the attention mechanism [8]. A basic Transformer uses an encoder-decoder architecture. The en- Thanks to the “ktrain” low coding library was easy the coder learns the representation from the input sentence implementation of BERT, just to split the data set into a and the decoder receives the representation and produces train and test set, then sent the train set as input into the “ktrain” pre-processing and Deep Learning model.
The Transformer based model shows, as expected, a good performance both on the train set and test set with an accuracy greater than 82%. Given the imbalanced data set the performance was evaluated also by F1 score with quite the same results.
Participants asked an evaluation with other models, and the job has been completed with the common classification Machine Learning model: the Logistic Regression, always with “ktrain” library.
Implementation of this model is easy, but there is a poorly performance: roughly 63% on train set and roughly 58% on validation set.
Looking at the confusion matrix, BERT model shows a large number of elements across the diagonal and small number of elements of the diagonal, so a better matrix than the Logistic Regression.
Last step was the inference of the model, testing the right prediction of test tweets, and also in this situation BERT outperformed Logistic Regression model with all right predictions.
have an overview of Natural Language Processing applications in the Insurance world, and with the disaster Tweets data set there has been the opportunity to discover NLP applications in Fraud Detection.
In this work the development of Natural Language Processing has been retracted, from Tokenization to Bag of Words, from Word Embedding to Contextual Embedding.
Transfer Learning has been explored, looking on its potential that outperforms benchmark models both on Topic Modelling and Classification prediction. [7] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). [9] Q. Liu, M. J. Kusner, P. Blunsom, A survey on contextual embeddings, arXiv preprint arXiv:2003.07278 (2020). [10] A. S. Maiya, ktrain: A low-code library for augmented machine learning, J. Mach. Learn. Res 23 (2020) 1–6.