Currently, technological advances and developments are occurring rapidly, making work easier and providing many benefits in various fields. Technological progress is also inseparable from the agricultural sector, and is having a positive impact on it. A technological development that can be applied in agriculture is in the detection of tomato maturity. This will help and make it easier to detect and select tomatoes to be harvested subsequently. In a greenhouse Citayam, tomato ripeness is still measured manually. The weakness of this method is the varying level of accuracy. Digital image utilization is crucial to determining tomato maturity. Using digital images, tomato ripeness can be obtained based on color. Without such a system, checking is still conducted manually and time-consuming, and something may be missed and the effectiveness is low, causing tomatoes to subsequently rot.

To obtain best results, a detection process to determine tomato maturity is to classify ripe and unripe tomatoes and subsequently conduct training, yielding an overview of features that can be used as markers or indicators of tomato ripeness. Such a detection process can be developed by comparing tomatoes with images that have been classified.

Image classification is closely related to object detection. The former is the categorization of an image into certain categories. Object detection is a technology to detect an object in an image or a video. It can also be used to identify two or more objects that appear similar to each other. The accuracy of detecting similar objects can be very high if the detection process is accompanied with adequate training data. There are several methods to detect objects in an image, one of which is faster region-based convolutional neural network (Faster R-CNN) [1]. Faster R-CNN is used to detect and recognize objects in an image.

Considering the technological developments in agriculture, not many studies have used a camera to detect tomato maturity, particularly in a greenhouse Citayam, where tomato maturity is currently determined manually. To detect tomato ripeness effectively and efficiently for a greenhouse Citayam, based on the above background, this study intended to design a program and analyze the results of the comparison or the levels of accuracy obtained using Faster R-CNN, SSD, and YOLO model.

2.1 Faster R-CNN

A Faster R-CNN is used to detect objects in an image. It uses the fast R-CNN and region proposal network (RPN) methods as its main architecture. A Faster R-CNN is the same as Faster R-CNN except that an RPN remodel places the selective search section of Faster R-CNN. The RPN is placed after the CNN layer [2]. The RPN is fed to the region of interest (ROI) pooling layer followed by a classifier and a bounding box regressor. The Faster R-CNN architecture is shown in Figure 1.

2.2 SSD

An SSD recognizes or detects an object in an image using a single deep neural network, and it is faster and significantly more accurate than the previous state-of-the-art for single-shot detectors (YOLO). An SSD is as accurate as slower techniques that perform detailed region proposals and pooling (including Faster R-CNN). It only needs an input image and a ground truth box for each object during training.

Figure 2 shows that an SSD has two network architectures: the VGG-16 network and a network in the extra feature layers. The VGG-16 network is used as the base network because VGG-16 performs strongly for high-quality images. SSDs add layers of convolutional features to the end of the base network that predicts different aspect ratios [3].

2.3 YOLO

YOLO uses a new approach for object detection. Object detection is performed by framing detected objects as regression problems into spatially separated bounding boxes and associated class probabilities. It uses a single neural network to predict bounding boxes and class probabilities directly from an entire image in one evaluation. Because the entire detection pipeline is a single network, it can be optimized end-to-end directly based on the detection performance. The YOLO architecture consists of 24 convolutional layers with 2 fully connected layers (Figure 3). An alternating 1 × 1 convolutional layer reduces the feature space from the preceding layers [4].

2.4 Object Detection

Object detection is used to determine the existence of objects in an image or a video (object localization) and categorize each object (object classification). Object detection can be further classified into soft and hard detection. Soft detection only detects an object, whereas hard detection detects an object and its location [5]. It identifies the location of an object in an image and draws a bounding box around it. It typically involves two processes: object classification and drawing boxes around objects or bounding boxes.

Image classification is the classification of an image into a specific category. Image localization is the development of a task, with the resulting output being the location of the class object in the image, generally shown as a bounding box. Object detection comprises classification and localization to detect several objects in an image (Figure 4).

2.5 Previous Research

Several previous studies have used similar methods in agriculture to detect fruits. One of the five earlier studies was on the real-time detection of fruit ripeness using the YOLOv4 algorithm [6]. It succeeded in detecting banana ripeness with an accuracy of 87.6% during testing.

The second study used the deep learning architecture of Faster R-CNN for quality classification of nutmeg (Myristica fragrans Houtt) [7]. The Faster R-CNN method offers a solution to solve the problem of real-time fruit quality detection, because it can manage many classes. Using this method, higher and faster accuracy is obtained than by a CNN.

The third study used an SSD for robust cherry tomato detection in a greenhouse. The SSD was selected because of its excellent anti-interference ability and self-learning from datasets. By analyzing the experimental results, it was found that the Inception-v2 network was a better feature extractor for the SSD network for cherry tomato detection than the VGG16 and MobileNet networks [5].

In a further study [8], the Faster R-CNN method and an intuitionistic fuzzy set were used in the automatic detection of single ripe tomatoes on a plant. With accurate classification, the trained Faster R-CNN rapidly localized candidate ripe tomato regions. In a recent study, detecting tomatoes in greenhouses using SSD and YOLO models showed that the MobileNet-v2 SSD model outperforms YOLOv4. In addition, it has the lowest false positive ratio and performs computations rapidly. Although the inferring time is not sufficiently short, the YOLOv4 Tiny model also yielded excellent results and processed images in approximately 5 ms [9].

In general, the research method used in this study was divided into four stages (Figure 5). The first stage is pre-processing, which involves data collection followed by data annotation. The second stage is training, in which a model is constructed, the config file is configured, and the class of tomato ripeness is defined. The training process uses the TensorFlow object detection API framework and the Faster R-CNN, SSD, and YOLO methods. Subsequently, the introduction of tomato ripeness using training data trained on image training data itself is tested. In the last stage, the testing results are analyzed.

3.1 Data Collection

Tomato ripeness detection requires a tomato dataset. In this study, the tomato fruit dataset was collected from a smartphone camera and Kaggle. The collected tomato fruit ripeness image data comprised 400 images (Table 1). The data were divided into two data groups: training and test data.

An example of the image dataset is shown in Figure 6.

3.2 Pre-processing

The next stage was pre-processing and selection of images of tomatoes. The collected pictures of tomatoes were placed into separate folders: ripe, half-ripe, half-unripe, and unripe tomatoes. The percentages of the number of images used as training and test data were determined. The data groups were divided in a 70:30 proportion into training and test data, respectively. The training and test data used were 280 and 120 images of tomatoes, respectively. The image data used as the training and test data were annotated using the LabelImg application. Image annotation is labelling of images of a dataset to train a model to learn and provide results based on the quality of the given data. Here, labels were annotated with ripe, half-ripe, half-unripe, and unripe tomatoes. The annotation process is presented in Figure 7.

Image annotation was conducted using the LabelImg application by creating a box around an object contained in an image, which was labelled subsequently (Figure 8). Following this, the object that was boxed was defined and labelled according to its classification class (ripe, half-ripe, half-unripe, and unripe tomatoes).

The saved annotation file had an .xml extension in the well-known VOC Pascal format and contained the class name and the x and y coordinates of the bounding box position (Figure 9). Subsequently, the .xml file was converted to a .csv format, where the csv file contained class, width, high, and filename column information. The file converted into .csv was reconverted to produce tfrecord, which generated test.record and train.records files; the latter was used in the training processes of the Faster R-CNN and SSD models. In the YOLO method, the saved annotation file had a .txt extension in the well-known YOLO format and contained the index class name and x and y coordinates of the bounding box position.

3.3 Training

3.4 Flow Process of Image Prediction

Figure 15 shows the flow process of image prediction. More details can be observed in Figure 16.

Based on Figure 15, the image and video prediction process begins with inputting the images used to recognize or detect tomato ripeness. The system reads the input images for further processing to identify the objects in the images. The Faster R-CNN and SSD models use the results of the training process that are converted into a ProtoBuf file and use labelmap to obtain the label definitions for object prediction. The YOLO requires the results of the training process in the form of trained weights, YOLO config file, and obj data. Furthermore, the system reads the frame on the input images and displays the prediction results of the detected object.

4.1 Model Configuration Faster R-CNN

This section discusses the configuration conducted before the training process. The model resizes a training image to be small with a maximum size of 1024 pixel, and many objects are classified into as many as four classes. Training config is useful for setting and training model parameters. Path fine tuning is adjusted to the faster rcnn inception v2 coco directory. The number of training processes is denoted as num step and set as 200, 000.

4.2 SSD Model Configuration

The model resizes a training image to be small, with a maximum size of 300 pixels, and many objects are classified into as many as four classes. Training config is useful for setting and training the parameters. Path fine tuning is adjusted to the ssd mobilenet v2 coco coco directory. The number of training processes is marked with a num step, which is set as 200, 000.

4.3 Model Configuration using YOLO

Several lines were adjusted according to the conditions of the data to be trained. The batch was changed to 64, subdivisions to 16, width and height changed to 416, and max batch value was 8, 000. This number was chosen because the number of classes is four; therefore, the maximum batch calculation was class ×2, 000, which is < the number of training images or < 6, 000. This suggests that the literacy trains the data as much as 8, 000 times. The steps included 80% and 90% max batches; therefore, the values were 6, 400 and 7, 200. Subsequently, the convolution layer was set, and the classes were used. Four classes were used to mean the dataset to be trained on four classifications, which also affects the value of the filter. The filter value was taken from the filters= (classes +5) × 3; therefore, the filter value was 27.

4.4 Training

Data training was conducted at Google Colaboratory, which is the cloud service of Google. Google Colaboratory is accessible with a 12-hour time limit. In conducting Faster R-CNN and SSD training, checkpoint files were saved automatically every 10 minutes, whereas with YOLO, they were saved automatically after 100 literations. Using a GPU can increase the speed of training compared to that using the CPU of a computer. The training process is shown in Figure 17.

The training data produced loss values relative to the training results of each step, and yielded the average precision (AP) and mean AP (mAP) values. A loss shows inaccuracy of the model predictions. The optimal value of the loss must be at least below 0.5. A graph of loss was obtained by accessing the tensorboard data via the command prompt, as shown in Figure 17, and the results of the mAP are summarized in Table 2.

The training process required 5 hours. On the Faster R-CNN, SSD, and YOLO methods, it involved 51, 036, 14, 624, and 1, 500 iterations, respectively. The model results trained on the Faster R-CNN and SSD models were subsequently converted into graphics files and stored in the inference graph directory. The conversion process used the last checkpoint file in the training process.

4.5 Testing

At this stage, the success or accuracy of the classification results obtained from the program were tested. Accuracy testing was conducted using images from the training data taken from a smartphone camera. Objects detected by the program were marked with bounding boxes including visible objects. An example detection result is shown in Figure 19.

The accuracy of using five images each for the four annotations in the training data obtained from the dataset for each object was tested using the Faster R-CNN, SSD, and YOLO methods. The results of the training are summarized in Table 3.

It can be seen that the identification accuracies of half ripe, half-unripe, ripe, and unripe tomatoes are high. The average accuracy values of the Faster R-CNN, SSD, and YOLO models are 99.55%, 89.3%, and 94.6%, respectively.

In addition to testing using images, testing was conducted real-time using a webcam, to determine how its speed and application in daily life. The tests were conducted using a webcam with two different resolutions, and they refer to the difference in frames per second (fps) caused by each resolution, which can affect each detection process. An example test result obtained using a camera is shown in Figure 20.

The results of this test are fps values of 1280 × 720 and 720 × 480 resolutions for each model. The overall test results are shown in Table 4.

It can be concluded that the Faster R-CNN method requires a device with a higher level than the SSD and YOLO methods. Moreover, the use of a GPU is recommended to obtain improved results, and for real-time use, using YOLO and SSD methods are recommended; however, the results are not as accurate as those using the Faster R-CNN method.

This research identified objects in mages using deep learning methods. Specifically, the Faster R-CNN, SSD, and YOLO models were implemented into Jupyter Notebook. The program was written using Python and TensorFlow libraries.

The results obtained from training using a dataset with 400 images and a training model required 5 hours to produce a graph with a total loss < 2. Moreover, it was shown that the total loss value reduced, improving the prediction results. The results of the training process on several images from the training data yielded relatively good average accuracies of 99.55%, 89.3%, and 94.6% on the Faster R-CNN, SSD, and YOLO models, respectively.

For further research, using a high GPU is recommended to improve the process, effectiveness, and efficiency of the training and identification stages. The models were implemented on real-time cases (for example, in a greenhouse), requiring embedding the deep learning algorithms in the IoT.

Table 1. Image data collection.

Table 2. mAP results.

Table 3. Results of training.

Table 4. Detection speed.