Technology nowadays affects every aspect of our life and makes it easier, more convenient. This research aims to integrate Artificial intelligence (AI) in the development of a smart mobile application that works as a smart tour guide. An Android application was developed to do image recognition in almost real-time by using Convolutional Neural Network (CNN). After the image is recognized, reviews for the restaurant or the hotel are retrieved. The reviews were collected from various resources on the internet and saved in a database to be retrieved after the image is recognized. This research required collecting images to retrain a CNN model called Inception v3 from Google. A technique called transfer of learning was used, which is building CNN model that is trained on a small dataset. The data size is about 100 images for each of the 9 categories of the hotels and restaurants. The scope for this application is limited to a few numbers of hotels and restaurants in Makkah city to deliver to the visitors of this Holy city a pleasant experience while they are roaming around. They could take a photo or upload a photo of the hotel or restaurant and a list of reviews are shown to help them decide to go to the best places and save their time and effort rather than searching for this online. The application is available without an internet connection to make it convenient to use by users. The image recognition and the retrieval of the reviews from the database are successfully accomplished. The accuracy various between 0.84 to 1 on different testing images.
Deep learning is a subfield of machine learning, it tries to mimic the human brain when it receives a
piece of information, and how this information is transferred from one neuron to the other one till they
form a network. Deep learning depends on having what is known as a neural network and can be
implemented in many areas. For instance, to do image recognition and sound recognition [
However, both ways are time-consuming. Technology nowadays affects every aspect of our life and makes it easier, more convenient, and efficient to do daily life’s tasks. Images of data was collected along with the reviews of some places. Moreover, this task will be done in real-time and without an internet connection by applying image recognition using CNN and the technique of transfer of learning. After the training is over, this model can be deployed on an Android application. The user can simply take a picture or upload a photo to this application, and it will be able to provide the reviews. This research is highly beneficial to the visitors of Makkah city. It is difficult to find hotels and restaurants for visitors to Makkah, where they do not know the city and suffer from searching for a suitable place to stay in or to eat. As the visitors explore the city, they can take photos of the restaurant and decide to eat there or not based on the reviews that this application will provide, therefore, comfortable and enjoyable experience in Makkah without time and effort consuming was provided to them
The objective of this research is to create an Android application that can do image recognition for the visitors to Makkah city, which makes it easier for users to find good places where the user uploads or takes a photo. For the authentication of the recognized image, it will provide the name it recognized just to show how accurate the application was to recognize that specific image and display the reviews review that people wrote about it. This research will contribute to the AI community by showing it is possible to build a smart system with limited dataset and computing resources.
The application that was developed in [
The application in [
The application mentioned in [
The application in [
The application in [
The application in [
The application in [
The application in [
The site [
The applications provided depend on searching images online using for example Google image search. All the represented applications require to be connected to the internet otherwise the application will be useless. Also, these applications’ main feature is to search for similar images, no application according to our knowledge, was able to retrieve reviews about the image that the user search for by using image recognition technique.
Image recognition is easy for humans and animals, but it is a very difficult task for computers. The task of image recognition can be done by using deep learning. CNN has a great ability to recognize the input images in this application we used already trained CNN model called Inception v3, then it is retrained to do the recognition on custom images. Python programming language was used to retrain the CNN model. The dataset was manually collected. It was divided into training dataset and testing dataset. Once the training is over, the model is smart enough to do image recognition. After that, the model was integrated into an Android application, in which users can take or upload photos. The application is using this retrained CNN model to recognize the input images from the users and retrieved the reviews from the database, the system overview is illustrated in Figure 1.
A dataset of four hotels and five restaurants was manually collected. In each category, there are about 100 images as illustrated in Table 1 and Table 2. To increase the data size and improve the performance of the model data augmentation was used, by feeding the model with images from different angles and at day or night as shown in Figure 2 a sample of the datasets. The images have been resized to 244 X 244. There are 100 images for training and 20 unique images for testing.
Building and training the CNN model from Scratch requires a huge amount of dataset to learn the important features. However, because of the small data size we have collected and limited computing resources, we used the transfer learning technique to do the image recognition task by using Inception v3[20], a model that was trained on ImageNet dataset with 1.2 million data size with more than 1000 labels. The knowledge is transferred from this model into the retrained model to recognize images for the new task which is in our case recognizing hotels or restaurants images. Recently, many CNN models have been developed. In this research, we used Inception v3 provided by Google as a pretrained model. This model has different sizes of filters in the same layer that significantly able to reduce computation complexity. The sizes of filters are 1 × 1, 3 × 3, and 5 × 5 one of these filters had to be chosen first then sent to the SoftMax layer to detect new features. This process will repeat until all features are extracted. The feature was extracted using both CNN fully connected layer and SoftMax layer to classify images. After that, new fully connected layers are added to extract the input features of the hotels and restaurants images. The top layer SoftMax that classified the original dataset was removed to train the next layer with our task. To enhance the performance of the model, 3 Dense layers were added. Two layers with Rectified Linear Unit (RELU) as an activation function and the output layer with Softmax, and categorical cross entropy loss function was used. This process lasted for almost two hours on a device with Windows 10 64-bit and Intel (R) Core (TM) i5 CPU. 3.3.
The database was implemented by using SQLite in Android Studio. This database is used to save the reviews. It has only one table in the database, the review table. The ID was set for each restaurant and hotel along with its name. After the recognition of images, the label was sent to the database to retrieve the corresponding reviews in the application.
When opening the application, it shows us an interface as shown in Figure 3. (a) through which the user can take or upload an image and when clicking on taking pictures, the image will appear in the main interface as shown in Figure 3. (b) after successfully taking the image and then clicking on reviews to do the image recognition and then show the reviews. When clicking on Upload Image, the gallery will open as shown in Figure 3. (c) after the image has been successfully uploaded, if the user clicks on Reviews, it will display the name of the recognized image and reviews as shown in Figure 3. (d). The results show that the application can recognize the image and retrieve the comments successfully with any angle the photos was taken.
Many decisions regarding the number of layers were made to improve the performance. Moreover, a key factor to build an accurate model is the dataset. The decision of choosing the appropriate dataset was based on trial-and-error approach. Firstly, the model was trained on 50 images of each category, but overfitting problem was noticed as a result of the limited data. The problem is mainly happened because the model has learned the same patterns during the training, but it is not able to generalize the features already learned during the training phase on testing data. Therefore, the number of images were increased up to 100 for each one. As a result, the model recognized the data for testing successfully to our satisfaction.
Figure 4 demonstrate the integrating of the model with the android application. After the training is completed, two files are generated, one is the retrained model and the other one is the 9 classes or categories we have. The application used these files to do the image recognition in almost real-time.
As it is illustrated in Figure 5 the loss or the error was about 24 at the beginning of the training and the error is reduced as the training continued until it reached to 0.03 by the end of epoch. Although the epoch was set to only 10, the accuracy got increased from 0.6 at the beginning of the training session to 0.99, the results are a proof of the effectiveness of using transfer of learning technique.
The figure above represented the result we got after passing to the retrained model a new test image it did not train on for a hotel called Ellaf Al Mashaer Hotel, the model perfectly identified it with 1 accuracy. In general, the accuracy various between 0.84 to 1 on different testing images. While these results are satisfactory, by increasing the dataset size and adding more layers or different activation function the results could be enhanced. Moreover, the inception v3 model is an old model a better approach to increase the accuracy of our model is by using the-state-of-art CNN models.
An Android application that works as a smart tour guide in Makkah city was developed by using a technique known as transfer of learning. This technique works well if we with limited datasets and computing resources. Inception v3 convolutional neural network model was retrained on our custom dataset. The trained model was converted from TensorFlow to TensorFlow Lite to be integrated within the Android application. SQLite database that contains all the reviews of some of the restaurants and hotels is used as well. All the functionalities was successfully implemented, and the image recognition was done in almost real-time.
As future work, an enhancement of the user interface of the application could be done, such as the way the reviews are displayed to the users. Another enhancement will be to add more hotels and restaurants that cover the important areas. Furthermore, sentiment analysis could be used in the reviews to show short comments if the hotels or the restaurants are good or bad. Finally, to improve the results a state-of-art CNN models are highly recommended to be used.