Food recognition research is growing rapidly since it is able to be applied in some applications, including leftover estimation. In previous research we built prototype, named Smart Nutrition Box for measuring the leftover automatically. However, before calculating the remaining weight of food, we need to recognize the type of food itself, since blank compartment (non-food detection) negatively affects estimation. In this paper, we propose a feature of multi-food recognition in single tray box images using Convolutional Neural Network (CNN). Automatic cropping by using rectangular contour is applied. However, for further recognition, we only have small amount of data from our own dataset as training and validation images, so a technique to deal with scarcity data using image augmentation is used. Different type of activation function in the dense layers are tested: Rectified Linear Units (ReLu), Sigmoid, and Hyperbolic Tangent (Tanh). The result shows that ReLu activation function is the best with no dropout and reaches 90% of accuracy. If dropout is applied in dense layer, the Sigmoid activation function is better with slightly different of accuracy 89% in recognition. It concludes that CNN can be applied to recognize multiple food in single image tray box.
Leftover food estimation and food recognition is becoming popular research nowadays, since every person needs
to be aware with everything they consumed. This application also useful for people who manage their diet and
control their appetite
In the previous research, we also created a prototype called as Smart Nutrition Box to estimate leftover food
Extracting features by using conventional methods, such as color and texture has complex problem when
determining the best feature to classify food
CNN represents the feature for large scale of dataset adaptively
The number of data in this paper is very limited to be used in the model of deep neural network, CNN. So that, we
implement data augmentation to boost the size of training dataset by creating more variations of images in the
dataset
A set of train and validation images are used to build an ideal model to recognize an item of food in each compartment of tray box. In order to create our own dataset, we have 6 folders related to annotate the data. It consists of 5 food items and 1 label of non-food. The number of data in each label are imbalanced as referred in Table 1. There are 212 images belonging to 6 classes as train set and 53 images belonging to 6 classes in validation set.
Generally, the entire of methodology is presented in the Figure 2. The first phase is capturing tray box image using our previous prototype, Smart Nutrition Box. A tray box containing a menu which is consisting of several foods are placed inside prototype, exactly perpendicular to the bottom of web camera. All images are saved and cleaned manually in order to take only main part of tray box, so that we remove some unused edge surrounding a tray box. We create our own benchmark related to the pattern of our tray box, then apply automatic cropping algorithm to break down a single tray box image into compartments. The proposed automatic algorithm is utilized based on identifying rectangle contour in each part to obtain some information in terms of x and y coordinate, height and width. This information is used to acknowledge the size that should be cropped in other tray box images. To classify food items, we apply CNN to create model in both train and validation set images. Good model helps good prediction as well, so that choosing parameters among CNN algorithm is also important to be considered. The result is single tray box image with detail information of what item of food in its compartments.
In order to have equal dimensions among tray box images, thus we also propose the size calibration to apply
cropping algorithm automatically
Algorithm 1. Automatic cropping algorithm from tray box image into compartments Input: Tray box image and benchmark Output: Four cropped image called as compartments Steps: 1) Coordinate and size identification from benchmark a. Read benchmark image (black and white line) b. Size identification from benchmark 2) Finding contour information in order to get the detail size of each compartment a. Color transformation from RGB image to grayscale image b. Noise removal using Gaussian filtering with 15x15 kernel size c. Get binary image using Otsu thresholding d. Finding rectangular contour e. Return list of x and y coordinates, height and width in each rectangular object 3) Cropping tray box image into compartment a. Read original image containing of food items in tray box b. Size calibration of original images into benchmark’s size c. Get list of x and y coordinates from benchmark to be applied in original input of tray box image 4) Cropped image as compartments from tray box image. Loop whole tray box images in a directory
CNN is part of Neural Networks algorithm that is widely used as it is an effective method to recognize food in large scale of image dataset. In general, thousands of images are involved to build a pre-trained model. However, the problem is when capturing image from our own dataset, we have limited number of data, so that there is an alternative to put image augmentation in order to be suitable for building a model since it can increase the number of datasets. Figure 4 shows the detail process of using CNN model with image augmentation. In this case, we use four types of image augmentation: shear range, zoom range, horizontal flip and fill mode. After that, constructing the model of CNN by using two phases of 2D convolution and max pooling is computed. In this paper, we only try to use 2 layers to extract the features. In the first convolution, we apply 16 kernels with 3x3 of its size and activation function ReLu. In the second convolution, we utilize two times of the kernels in the first convolution with the same kernel size. Both of max pooling have 2x2 kernel size in order to reducing image size by 50%. Based on the Figure 4, after 2 layers of extracting feature, then covert it into 1D of flattened matrix to be fed to the fully connected layers. Two dense layers is used, and we observe three types of dense activation functions to build different model of training. In the experiment of dense activation function, we apply three activation functions that are: ReLu, Sigmoid, and Tanh.
a) b) c) ( ) =
1 + exp(− )
ReLu is the most used activation function in deep learning since it is computationally efficient to converge the network. The formula is given in the Equation 1.
( ) = max(0, ) (1) If z is less than 0 (means that is negative), so f(z) returns 0 as well as if z is equal to zero. However, if z is greater than zero, then f(z) is equal to z. ReLu is suitable for backpropagation when z is above zero, but when if the value of z is 0 or less than 0, then the backpropagation cannot well performed. Sigmoid activation function Sigmoid activation function is one of derivative functions which is necessary for training in neural network. The activation has S curve which has range between 0 and 1. It can reduce the problem in the gradient and normalizing output in each neuron. However, sigmoid achieve convergence slowly. Equation 2 describes the Sigmoid activation function.
1 (2)
It is similar with Sigmoid activation function, but Tanh is zero centered, so it is possible to have strong value of positive and negative as model input which has range between -1 and 1. Tanh activation function formula is illustrated in Equation 3.
1 − exp(−2 ) (3) ( ) =
1 + exp(−2 )
The last part is tuning parameter in training model using Adam optimizer and it is proved that suitable for
structured dataset
There are 36 testing data from tray box images with several conditions in different color of tray box. Train and validation set are used to get model from CNN architecture as depicted in the Figure 5. From testing set image, it is acknowledged per compartment and in the last step is concatenate whole compartment in one single tray box. Based on these techniques, it detects multiple food or non-food and recognize them as one menu in tray box.
Three type of activation functions at dense layer are tested: ReLu, Sigmoid, and Tanh. In addition, to avoid some overfitting value, we also tested dropout and no dropout in the training process since we have imbalance number of data in each labels of training and validation. Dropout is also a technique of choosing randomly when selecting neurons during training process. The value of dropout is represented by using probability. We use small dropout probability: 0.5. The use of dropout in the experiments is depending on using dense activation function.
The accuracy of CNN model between training and validation with no dropout reach the best training using ReLu dense activation function with 98% accuracy of training and 96% of validation. The accuracy metrics between training and validation has slightly different, thus actually with no dropout the model is still good for recognition as stated in the Table 2. It is proven by accuracy of data testing achieves 90% of compartment images and 75% of tray box images (means that all compartment in a single tray box are correct to classify). Sigmoid is good as well but cannot overcome the performance of ReLu in this experiment which yields 85% of accuracy recognizing compartment images, although the accuracy of training and validation model are same in 98%. Sigmoid is stable dense activation function rather that ReLu and Tanh. If dropout do not include in dense layer, Tanh performance is sufficient good, but not the best comparing both of ReLu and Sigmoid. Figure 6 shows the example result of multiple food recognition in single tray box image using different dense of activation functions.
This condition is contrary when dropout is involved in the dense layer. In this case, using Tanh dense activation functions much better with raising accuracy of model in training and validation is 99% and 97%, respectively. The result of compartment image classification is the best among other dense activation functions which reaches 89% of accuracy. It is contrary when using ReLu dense activation function, the accuracy drops until 42% of tray box images consisting multiple food inside and 75% of compartment images. It means that using dropout does not have any impact to the recognition performance, except Sigmoid activation function which is converge in both situation: with dropout or without dropout. Figure 7 shows that using Tanh produces the best result to recognize multiple food in single tray box images.
Multi-food recognition in single tray box image with scarcity data using improved preprocessing and CNN is proposed. It has remarkable result with different experiment of dense activation function either using dropout and without dropout. From the entire experiment, without dropout is sufficient to classify each compartment image. The best result when using ReLu dense activation function which reaches 90% of accuracy in recognizing compartment images without dropout. When dropout is involved, then Tanh dense activation function is more suitable to use and, in this case, it reaches 89% of classifying compartment images in tray box.
The future works aims to improve the performance of preprocessing and classification since it needs to estimate the leftover in the tray box images after being consumed. Adding some training data or making it balance class is also treated, so that it expects to increase the accuracy of testing data from CNN model.
The authors give thanks to Faculty of Computer Science (FILKOM), Brawijaya University, to support this research and also Ritsumeikan University which collaborates with FILKOM to compose this conference. This research is one of the consecutive results of research in development of prototype Smart Nutrition Box which is created between Computer Vision Research Group, Faculty of Computer Science and Agricultural Department of Technology, Brawijaya University.