A New Perspective on Detection of Green Citrus Fruit in the Grove Using Deep Learning Neural Networks

Abstract

The entire citrus industry is under attack by terminal citrus diseases such as Huanglongbing (HLB) or citrus greening (Candidatus Liberibacter asiaticus), and citrus canker (Xanthomonas axonopodis pv. citri). There is a need to develop fruit quality assessment techniques that can assess the quality of green citrus fruits during the growing season when green fruits are still on the trees. Learning technique is one of the fundamental techniques for fruit quality evaluation. Efforts to develop an efficient automated fruit classification system continued as an industry priority since fruit evaluation through human visual inspection has drawbacks such as subjectivity, high labour costs, inefficiency, and most importantly, the inconsistency caused by tediousness. Estimating citrus fruit yield at an earlier stage of fruit development can benefit growers to adjust site-specific management practices while it is possible, to increase fruit yield, and plan harvest operations on time to minimize harvesting costs. Also, to, (i) Estimate the infections and fruit defects, (ii) Estimate the number of fruits in the citrus grove and potential fruit size before harvesting, (iii) Yield prediction, and (iv) Make economic estimates such as planning of incomes, and calculation of net profit. An excellent recognition method would be the one that can separate green citrus fruits from background green leaves of citrus trees in the grove. The previous methods had difficulty in detecting young fruits and creating maps or make economic estimates because young or immature citrus fruits are green and resemble tree leaves. Some recent advances in machine learning yielded new techniques to train deep neural networks, which has the potential to successfully recognize patterns of objects such as citrus fruit's morphological structure and differentiate fruits from the main crop. This system uses forward propagation, which is a neural network way of classifying a set of images and to improve their performance.

Keywords

Artificial intelligence, Automatic fruit detection, Deep learning, Precision agriculture

The Citrus Industry

Maintaining citrus fruits quality is key to success in the fresh fruit market all over the world. The food industry is expecting high-quality products that meet the set food safety standards. As a result, there is a need for accurate, fast, and objective quality determination of citrus fruits [1]. Fruit quality assessment techniques that can assess the quality of green citrus fruits during the growing season when green fruits are still on the trees are vital [2].

Citrus quality, yield and disease management

Estimating green citrus fruit yield at an earlier stage of fruit development can benefit growers to adjust site-specific management practices; and while it is possible, to increase fruit yield, and plan harvest operations on time to minimize harvesting costs [3,4]. In addition, estimating the quantity of fruits in the citrus grove and potential fruit size before harvesting represents the basis for prediction of future fruit yield, planning of incomes, and calculation of net profit [5]. This is relevant in Florida and possibly in some regions of the world where there is a declining availability of seasonal agricultural labour, and growing pressure from worldwide competition in the citrus industry [6]. In any agrarian system, more especially the citrus industry, there is a developing need to produce and harvest more products of higher quality at a lower input cost. More especially now than any other time before, since the entire citrus industry is under attack by terminal citrus diseases such as Huanglongbing (HLB) or citrus greening (Candidatus Liberibacter asiaticus), and citrus canker (Xanthomonas axonopodis pv. citri). The citrus greening disease is a major cause of crop and tree loss in many parts of Asia and Africa [7]. This disease is a severe, highly destructive, and widely distributed disease of citrus, especially on sweet orange and mandarin varieties [8]. A phloem-restricted, the uncultured bacterium causes greening disease of citrus, which is characterized and named Liberobacter [9]. A range of primary and secondary leaf symptoms are associated with HLB, making field diagnosis difficult, unless the typical lop-sided greened fruit are present [8].

Machine learning and citrus production

Learning technique is one of the fundamental techniques for fruit quality evaluation. Du and Sun [10] indicated that this technique uses computer vision technology, which seeks to eventually substitute the human visual decision-making process with an automated procedure. Critical areas for computer vision technology application in agricultural and biological industries include fruit quality evaluation, infestations, and blemishes on fruits [11]. The automated inspection of product quality in all agrarian systems is becoming of high importance to reduce production input costs and maximize product quality [12].

Significant to this challenge is the development of an automated procedure that can mainly substitute human labour on tasks where a person is not consistent with the performance of a task in each process, or performs worse than an automated machine in terms of accuracy, and repetitiveness in a working cycle [13]. In the past, defect detection on fruits was performed mainly by humans through visual inspection and in some current sorting applications by people trained in the fruit quality standards for market purposes [14]. Together with variability associated with a human visual inspection of fruits that are easily influenced by physiological factors, this accentuates a need for a consistent measurement system [10]. Efforts to develop an efficient automated fruit classification system continued as an industry priority since fruit evaluation through human visual inspection has drawbacks such as subjectivity, high labour costs, inefficiency, and most importantly, the inconsistency caused by tediousness [14].

Machine Vision and Prospects of Innovative Techniques in Green Fruit Detection

Previous studies detected matured citrus fruit from trees in the grove using a machine vision system [13]. However, detecting matured citrus fruits from a citrus tree in the grove appears quite easy because a matured citrus fruit has a different color from green citrus leaves. Other studies used color and texture features for fruit recognition to achieve near-human levels of recognition. However, only 69.2% recognition rate for oranges was achieved on 51 trained images [15]. In agricultural and biological sciences, images are a key source of information, and for computer vision technology to achieve a near-human level of image recognition is a challenge yet to be resolved [16]. Developing an outdoor machine vision system that can exceptionally recognize green fruits on green citrus trees in the grove can be a challenge, primarily because of varying illumination conditions among images [17]. Tian and Slaughter [18] reported that lightning condition is one important factor to consider in object recognition with machine vision system because changes in lighting conditions may vary because of low-resolution images resulting in improper object recognition. In a study conducted by Stajnko, et al. [19], the method developed by the authors was not useful in distinguishing green leaves and green fruits growing deep in the tree crown. The authors recommended that future studies investigate methods of fruit recognition by shape so that it is possible to detect a hidden spherical fruit by obtaining a set of pixels that correspond to fruit boundaries. In this situation, the use of automated robotics system, and computer and machine vision is inevitable in the whole citrus industry and fruit detection [20]. That is because computers in agriculture have many potential applications, such as analyzing fruit images for automated agricultural and biological tasks [21].

Computer and machine vision attempts to replicate human behavior and after that substitute human visual sense, perform an inspection task, and make judgment capabilities by electronically perceiving and understanding an image with a video camera [22,23]. And for these reasons, machine vision system fits perfectly well in the problem-solving strategy of the citrus industry because this technology has a potential to detect defects on green citrus fruits in the grove and perhaps increase the commercial value of the fruits [24]. Other advantages of this technology are that the technology is an instantaneous non-destructive GPS operated that is comparatively low-cost, objective, and can detect fruit multi-index simultaneously, such as size, skin defects, color, shape, and to a larger extend even fruit maturity [25]. Hence automated inspection is experiencing considerable growth in the agrarian system because of its competency, reliability, superior speed, and precision [23].

To this date, there is a lot of research conducted in the field of machine vision system. However, there is a limited success about practical applications in agricultural and biological sciences [26]. In a study of machine vision methods for locating a fruit on a tree, Jiménez, et al. [13] concluded that there is a need for a new technique to reduce total occlusion to make the process feasible, more especially when more than 40% of the fruit is occluded. Kane and Lee [27], reported that a major problem with these camera systems is occlusion and grouped fruit segmentation on a tree. Scientists at Citrus Research and Education Center of the University of Florida in Lake Alfred have developed an assignment of solving the occlusion and group fruit segmentation problem, and we believe that when this problem is solved, the newly developed technique can be used earlier in the growing season. This can potentially help in acquiring information on expected citrus yield, and health earlier during the growing season, and in-field variability [27]. It is a challenge to develop machine vision system that matches the human vision and can recognize green citrus fruits in the grove; an excellent recognition method would be the one that can separate green citrus fruits from background green leaves of citrus trees in the grove [28].

Previous methods had difficulty detecting young fruits to create maps or make economic estimates because young or immature citrus fruits are green and resemble tree leaves [29]. Recognizing green citrus fruits on a citrus tree in the grove is an important procedure in estimating the infections and fruit defects, a number of fruits for yield prediction, and make economic estimates. There is a need for an innovative and sustainable method that can accurately, effectively and efficientlyable to recognize spherical fruits in natural conditions facing difficult situations: shadows, bright areas, occlusions and overlapping fruits. recognize spherical green citrus fruits in the grove with shadows, bright areas, occlusions and overlapping fruits, and green color that matches a color of the tree leaves. Most importantly, a system that does not detect a single fruit multiple times since this will overestimate fruits and negatively influences environmental, agricultural, and economic decision-making processes.

Some recent advances in machine learning yielded new techniques to train deep neural networks [30], which have a potential to successfully recognize patterns of objects such as citrus fruit's morphological structure and differentiate fruits from the main crop [31]. This system uses forward propagation, which is a neural network way of classifying a set of images and to improve their performance; we can collect larger datasets of green citrus fruits, and leaves, and learn more powerful models, and use better techniques for preventing overfitting [32]. Deep neural networks have proved to be the best in image pattern recognition and machine learning [33]. Most importantly, this method of deep learning neural network is less affected by natural variations such as changes in illumination, shadows, skewed leaves, and occluded plants as compared to other previous image classification methods [34]. For this reason, we are optimistic that identification of citrus greening on citrus leaves and fruit surfaces in the grove could be one of those agricultural problems that can potentially benefit from deep learning approach, and consequently, help to eradicate the predicaments facing the citrus industry.

References

1. Patel KK, Kar A, Jha SN, et al. (2012) Machine vision system: A tool for quality inspection of food and agricultural products. J Food Sci Technol 49: 123-141.
2. Shahin MA, Tollner EW, Gitaitis RD, et al. (2002) Classification of sweet onions based on internal defects using image processing and neural network techniques. Trans ASAE 45: 1613-1618.
3. Okamoto H, Lee WS (2009) Green citrus detection using hyperspectral imaging. Comput Electron Agric 66: 201-208.
4. Zhao C, Lee WS, He D (2016) Immature green citrus detection based on colour feature and sum of absolute transformed difference (SATD) using colour images in the citrus grove. Comput Electron Agric 124: 243-253.
5. Welte HF (1990) Forecasting harvest fruit size during the growing season. In II International Symposium on Computer Modelling in Fruit Research and Orchard Management 276: 275-282.
6. Roka FM, Burns JK, Syvertsen J, et al. (2009) Improving the economic viability of Florida citrus by enhancing mechanical harvesting with the abscission agent CMNP. The University of Florida, Institute of Food and Agricultural Sciences.
7. Da Graça JV (1991) Citrus greening disease. Annu Rev Phytopathol 29: 109-136.
8. Da Graça JV, Korsten L (2004) Citrus huanglongbing: Review, present status and future strategies. In: Diseases of fruits and vegetables I: 229-245.
9. Planet P, Jagoueix S, Bové JM, et al. (1995) Detection and characterization of the African citrus greening liberobacter by amplification, cloning, and sequencing of the rplKAJL-rpoBC operon. Curr Microbiol 30: 137-141.
10. Du CJ, Sun DW (2006) Learning techniques used in computer vision for food quality evaluation: A review. J Food Eng 72: 39-55.
11. Mahendran R, Jayashree GC, Alagusundaram K (2012) Application of computer vision technique on sorting and grading of fruits and vegetables. J Food Processing Techno S1-001.
12. López-García F, Andreu-García G, Blasco J, et al. (2010) Automatic detection of skin defects in citrus fruits using a multivariate image analysis approach. Comput Electron Agric 71: 189-197.
13. Jiménez AR, Ceres R, Pons JL (2000) A survey of computer vision methods for locating fruit on trees. Trans ASAE 43: 1911-1920.
14. ElMasry G, Wang N, Vigneault C, et al. (2008) Early detection of apple bruises on different background colors using hyperspectral imaging. LWT-Food Sci Technol 41: 337-345.
15. Arivazhagan S, Shebiah RN, Nidhyanandhan SS, et al. (2010) Fruit recognition using color and texture features. J Emerg Trends Compt Inform Sci 1: 90-94.
16. Dubey SR, Jalal AS (2012) Adapted approach for fruit disease identification using images. Int J Comput Vis Image Process 2: 44-58.
17. Choi D, Lee WS, Ehsani R, et al. (2016) Detection of dropped citrus fruit on the ground and evaluation of decay stages in varying illumination conditions. Comput Electron Agric 127: 109-119.
18. Tian L, Slaughter D (1998) Environmentally adaptive segmentation algorithm for outdoor image segmentation. Comput Electron Agric 21: 153-168.
19. Stajnko D, Lakota M, Hocevar M (2004) Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Comput Electron Agric 42: 31-42.
20. Bulanon DM, Burks TF, Alchanatis V (2009) Image fusion of visible and thermal images for fruit detection. Biosyst Eng 103: 12-22.
21. Jain AK, Dorai C (1997) Practicing vision: Integration, evaluation and applications. Pattern Recognit 30: 183-196.
22. Domenico S, Gary W (1994) Machine vision and neural nets in food processing and packaging: Natural combinations. In Proceedings of the Third Conference of Food Processing Automation III, Orlando, FL, USA 9-12.
23. Brosnan T, Sun DW (2004) Improving quality inspection of food products by computer vision-a review. J Food Eng 61: 3-16.
24. Li JB, Huang WQ, Zhao CJ (2015) Machine vision technology for detecting the external defects of fruits-a review. J Imaging Sci 63: 241-251.
25. Liming X, Yanchao Z (2010) Automated strawberry grading system based on image processing. Comput Electron Agric 71: S32-S39.
26. Kapach K, Barnea E, Mairon R, et al. (2012) Computer vision for fruit harvesting robots-state of the art and challenges ahead. Int J Comput Vis Robot 3: 4-34.
27. Kane K, Lee WS (2007) Multispectral imaging for in-field green citrus identification. ASABE Paper No. 073025. ASABE, St. Joseph, MI, USA.
28. Bulanon DM, Kataoka T, Ota Y, et al. (2002) AE-automation and emerging technologies: A segmentation algorithm for the automatic recognition of Fuji apples at harvest. Biosyst Eng 83: 405-412.
29. Kurtulmus F, Lee WS, Vardar A (2001) An advanced green citrus detection algorithm using color images and neural networks. J Agric Machine Sci 7: 145-151.
30. Kooi T, Litjens G, van Ginneken B, et al. (2017) Large scale deep learning for computer aided detection of mammographic lesions. Med Image Anal 35: 303-312.
31. Moshia ME, Newete SW (2019) Mexican poppy (Argemone mexicana) control in cornfield using deep learning neural networks: A perspective. Acta Agric Scand B-S P Sci 69: 228-234.
32. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25: 1097-1105.
33. Schmidhuber J (2015) Deep learning in neural networks: An overview. Neural Netw 61: 85-117.
34. Dyrmann M, Karstoft H, Midtiby HS (2016) Plant species classification using deep convolutional neural network. Biosyst Eng 151: 72-80