Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities
| Autor: | Eduard Gregorio, Josep Ramon Morros, Jordi Gené-Mola, Javier Ruiz-Hidalgo, Joan R. Rosell-Polo, Verónica Vilaplana |
|---|---|
| Přispěvatelé: | Universitat Politècnica de Catalunya. Departament de Teoria del Senyal i Comunicacions, Universitat Politècnica de Catalunya. GPI - Grup de Processament d'Imatge i Vídeo |
| Rok vydání: | 2019 |
| Předmět: |
0106 biological sciences
Backscatter Computer science Multi-modal faster R-CNN Horticulture 01 natural sciences Signal Convolutional neural network Yield mapping Robots industrials Robots Industrial Agricultural robotics Radiació -- Mesurament Fruit detection Computer vision Color en la indústria Depth cameras Kinect Precision agriculture Apples Fruit reflectance business.industry RGB-D cameras Attenuation RGB-D Forestry 04 agricultural and veterinary sciences Enginyeria de la telecomunicació [Àrees temàtiques de la UPC] Automation Computer Science Applications visió artificial 040103 agronomy & agriculture Radiation -- Measurement 0401 agriculture forestry and fisheries RGB color model Convolutional neural networks Pomes Artificial intelligence Informàtica::Robòtica [Àrees temàtiques de la UPC] business Agronomy and Crop Science 010606 plant biology & botany |
| Zdroj: | Recercat. Dipósit de la Recerca de Catalunya instname UPCommons. Portal del coneixement obert de la UPC Universitat Politècnica de Catalunya (UPC) Repositorio Abierto de la UdL Universitad de Lleida |
| ISSN: | 0168-1699 |
| DOI: | 10.1016/j.compag.2019.05.016 |
| Popis: | Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection. This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s predoctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Adria Carbó for his assistance in Faster R-CNN implementation. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect