Multi-class fruit detection based on image region selection and improved object proposals

Abstract This paper proposes a novel approach for multi-class fruit detection using effective image region selection and improved object proposals. Five complementary features, namely local binary patterns (LBPs), histograms of oriented gradient (HOGs), LBP based on magnitude of Gabor feature (GaborLBP), global color histograms, and global shape features, are utilized to improve the detection accuracy. An optimal combination of regions (i.e., features) is selected using an image region selection method based on feature similarity and cross-validation accuracy. To combine the strength of the five complementary features, a weighted score-level feature fusion approach based on the average confidence coefficient is used. Moreover, during detection, an object proposal method, “EdgeBoxes,” is improved by calibrating scores considering the image region similarity to generate windows that are likely to contain fruits and to speed up detection. The experimental results show that the image region selection method can select an effective and optimal combination of regions, which exhibits better recognition accuracy than the method without image region selection. This proposed method demonstrates a low miss rate (0.0377) at 0.0682 false positives per image (FPPI) and outperforms some baselines: multi-class fruit detection using the traditional sliding window mechanism, the well-known deformable parts model (DPM) method, convolutional neural networks features (CNN) with support vector machine (SVM) for classification (CNN + SVM), cascade detection framework and faster RCNN in terms of the miss rate vs. FPPI and precision vs. recall curves. The proposed multi-class fruit detection can detect multi-class and multiple fruits in a variety of sizes, backgrounds, angles, locations, and image conditions.