In this article, we present the program we have developed for lung cancer detection. For making predictions, it uses comprehensive patient information, including gender, age, smoking habits, yellow fingers, anxiety, peer pressure, chronic disease, fatigue, presence of allergies, wheezing, alcohol consumption, coughing, shortness of breath, difficulty swallowing, and chest pain. We start by providing a thorough analysis of the database to identify which features have the most significant impact on the likelihood of developing lung cancer. This includes statistical evaluations and visualizations to better understand the data distribution and correlations between various attributes and lung cancer incidence. Next, we present the results of implementing several different classifiers on the dataset. Through this comparative analysis, we demonstrate that, after preliminary tests, the naive Bayes algorithm emerges as the most effective classifier. We provide the pseudocode for the naive Bayes algorithm, offering a clear and accessible explanation of its implementation. Additionally, we conduct a detailed analysis of its effectiveness, supported by charts and graphs that illustrate the algorithm's accuracy and other relevant performance metrics. Furthermore, we highlight the process of feature selection. By removing irrelevant from the database, we are able to enhance the program's speed and accuracy.
Lung cancer remains one of the most lethal forms of cancer worldwide[
This study provides an in-depth analysis of the data features that significantly influence lung
cancer risk, offering insights into their relevance and impact. We compare the performance of
various classifiers and demonstrate why the naive Bayes algorithm stands out as the most
effective after initial testing.[
Furthermore, we explore the process of refining the dataset by eliminating unnecessary information, which enhances both the speed and accuracy of the predictions. This article not only showcases the technical aspects of our program but also emphasizes its potential to revolutionize lung cancer diagnosis, offering a valuable tool for healthcare professionals in the fightagainst this devastating disease.
In order to choose the classifierthat best suits our task, we have conducted a test of three popular algorithms: k nearest neighbours classifier, naive Bayes classifier and decision tree classifier.Each of the algorithms has been run 500 times, each time with random training and test dataset. Then, we have calculated mean accuracy for all of classifiersand compared their results.
KNN (k-nearest neighbors) is one of the most basic and popular classificationalgorithm. It
measures the distance between the new sample and all points in the training set, identifies the K
nearest neighbors, and assigns the most common class label among these neighbors to the new
sample.[
• : The -th coordinate of the point y.
The Gaussian Naive Bayes classifier works by classifying a sample based on the probabilities of each class given the feature values, assuming that features follow a Gaussian (normal) distribution. It calculates the likelihood of the sample’s features for each class, combines these with the prior probabilities of the classes, and assigns the class with the highest resulting probability to the sample.
We decided on the Gaussian Naive Bayes because it had the highest efficiency 2 The formula for the conditional probability of a feature given class is: • (| ): The conditional probability of feature given class . • ,: The standard deviation of feature in class . It measures the spread of the feature values around the mean. • ,: The mean (average) of feature in class . It represents the central value of the feature for the given class. • : The value of the i-th feature.
• : The class label.
Data: training data, object to classify
Result: class to which the object belongs 1 groups = split training data into groups according to their class; 2 best_class = ""; 3 best_score = 0; 4 for group in groups do 5 score = log(number of rows in group/number of rows in all training data); 6 for column in group do 7 std = standard deviation for column; 8 mean = mean for column; 9 x = value of column from object to classify; 10 11
The Decision Tree classifier works by recursively splitting the dataset into subsets based on
feature values, creating a tree structure where each node represents a feature and each branch
represents a decision rule. It continues splitting until the subsets are as pure as possible, meaning they
contain samples predominantly from one class. The class label assigned to a new sample is
determined by traversing the tree according to the sample’s feature values until reaching a leaf
node, which represents the predicted class. [
Our database consists of 16 columns and 309 rows. Individual information includes information about the patient such as gender, age, smoking, yellow fingers, anxiety, peer pressure, chronic disease, fatigue, allergy, wheezing, alcohol consuming, coughing, shortness of breath, swallow- ing difficulty, chest pain and lung cancer, which tells us whether the person has cancer. A value of 1 means that the patient does not have a given symptom and 2 means that he does. We made a correlation matrix. We were most interested in the last row to findout which symptoms have a positive correlation with lung cancer. From it we can conclude that smoking and shortness of breath have the lowest correlation (but still positive) while allergy and alcohol consuming have the highest correlation.
We compared 4 classifiers to check which one would work best for our data. We used K Nearest Neighbours, Decision Tree, Gaussian Naive Bayes and Multinomial Naive Bayes. As you can see in the figure2, the Gaussian Naive Bayes has the highest accuracy. We then removed one column and checked how its removal would affect the accuracy of the classifier. The differences were negligible, so we decided to remove several columns at once. The best results were obtained after removing columns such as: ’WHEEZING’, ’SWALLOWING DIFFICULTY’, ’AGE’, ’COUGHING’, ’SMOKING’ where the accuracy of the model averaged 91.28%, and for the best sample of 500 was 100%
We also created an error matrix for each classifier and calculated: Accuracy, Recall, Precision, F1
and Specificity[
The following values were calculated from the formulas: • Accuracy - determines what part of all classified texts was classified correctly • Recall - determines the share of correctly predicted positive cases (TP) among all positive cases • Precision - determines how many of the examples predicted positively are actually positive • F1 - is the harmonic mean between precision and recall. The closer it is to one, the better it proves about the classification algorithm. • Specificity - determines how often the model accurately predicted falsehood when something was actually false
• TP - the sick person was correctly classified • TN - a healthy person has been correctly classified • FP - the sick person was classified as healthy • FN - a healthy person has been classified as sick
In conclusion, our study presents a novel approach to lung cancer detection through the integration of machine learning algorithms and comprehensive patient data analysis. Our research highlights the importance of feature selection in optimizing algorithm performance, leading to improved prediction accuracy and efficiency. Through comparative analysis and detailed evaluation, we have demonstrated the superiority of the naive Bayes algorithm in this context.
By facilitating early detection and intervention, our approach has the potential to significantly improve patient outcomes and contribute to the ongoing efforts to combat this deadly disease.