Convex Shape Prior for Deep Neural Convolution Network based Eye Fundus Images Segmentation

Convex Shapes (CS) are common priors for optic disc and cup segmentation in eye fundus images. It is important to design proper techniques to represent convex shapes. So far, it is still a problem to guarantee that the output objects from a Deep Neural Convolution Networks (DCNN) are convex shapes. In this work, we propose a technique which can be easily integrated into the commonly used DCNNs for image segmentation and guarantee that outputs are convex shapes. This method is flexible and it can handle multiple objects and allow some of the objects to be convex. Our method is based on the dual representation of the sigmoid activation function in DCNNs. In the dual space, the convex shape prior can be guaranteed by a simple quadratic constraint on a binary representation of the shapes. Moreover, our method can also integrate spatial regularization and some other shape prior using a soft thresholding dynamics (STD) method. The regularization can make the boundary curves of the segmentation objects to be simultaneously smooth and convex. We design a very stable active set projection algorithm to numerically solve our model. This algorithm can form a new plug-and-play DCNN layer called CS-STD whose outputs must be a nearly binary segmentation of convex objects. In the CS-STD block, the convexity information can be propagated to guide the DCNN in both forward and backward propagation during training and prediction process. As an application example, we apply the convexity prior layer to the retinal fundus images segmentation by taking the popular DeepLabV3+ as a backbone network. Experimental results on several public datasets show that our method is efficient and outperforms the classical DCNN segmentation methods.

[1]  T. P. Dinh,et al.  Convex analysis approach to d.c. programming: Theory, Algorithm and Applications , 1997 .

[2]  Jun Liu,et al.  Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-Shape Priors for Image Segmentation , 2020, Journal of Mathematical Imaging and Vision.

[3]  Antonio J. Plaza,et al.  Image Segmentation Using Deep Learning: A Survey , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Jun Liu,et al.  A Regularized Convolutional Neural Network for Semantic Image Segmentation , 2019, Analysis and Applications.

[5]  Francisco Fumero,et al.  RIM-ONE: An open retinal image database for optic nerve evaluation , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[6]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Yue Zhang,et al.  Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation , 2018, MICCAI.

[8]  Jie Wang,et al.  An Edge-Weighted Centroidal Voronoi Tessellation Model for Image Segmentation , 2009, IEEE Transactions on Image Processing.

[9]  Ulugbek Kamilov,et al.  Image Restoration Using Total Variation Regularized Deep Image Prior , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[11]  D. Pallara,et al.  Short-time heat flow and functions of bounded variation in \mathbf{R}^N , 2007 .

[12]  Xue-Cheng Tai,et al.  A variant of the level set method and applications to image segmentation , 2006, Math. Comput..

[13]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Lena Gorelick,et al.  Multi-object Convexity Shape Prior for Segmentation , 2017, EMMCVPR.

[15]  Carsten Rother,et al.  Convexity Shape Constraints for Image Segmentation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  D. Pallara,et al.  Short-time heat flow and functions of bounded variation in RN , 2005 .

[18]  Chi-Wing Fu,et al.  Patch-Based Output Space Adversarial Learning for Joint Optic Disc and Cup Segmentation , 2019, IEEE Transactions on Medical Imaging.

[19]  Xiaochun Cao,et al.  Joint Optic Disc and Cup Segmentation Based on Multi-Label Deep Network and Polar Transformation , 2018, IEEE Transactions on Medical Imaging.

[20]  Lena Gorelick,et al.  Convexity Shape Prior for Binary Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Yang Wang,et al.  Convex Shape Representation with Binary Labels for Image Segmentation: Models and Fast Algorithms , 2020, ArXiv.

[22]  Daniel Cremers,et al.  A Convex Formulation of Continuous Multi-label Problems , 2008, ECCV.

[23]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[24]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[25]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[26]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[27]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Chunming Li,et al.  A level set method for convexity preserving segmentation of cardiac left ventricle , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[29]  Tolga Tasdizen,et al.  Domain adaptation for biomedical image segmentation using adversarial training , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[30]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Aaron Fenster,et al.  Efficient convex optimization-based curvature dependent contour evolution approach for medical image segmentation , 2013, Medical Imaging.

[32]  Xue-Cheng Tai,et al.  A new continuous max-flow algorithm for multiphase image segmentation using super-level set functions , 2014, J. Vis. Commun. Image Represent..

[33]  Hiroshi Ishikawa,et al.  Exact Optimization for Markov Random Fields with Convex Priors , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[35]  Jun Liu,et al.  Convexity Shape Prior for Level Set-Based Image Segmentation Method , 2018, IEEE Transactions on Image Processing.

[36]  Trevor Darrell,et al.  FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation , 2016, ArXiv.

[37]  Roland Glowinski,et al.  Convex Shape Prior for Multi-Object Segmentation Using a Single Level Set Function , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[38]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.