Inspirational Adversarial Image Generation

The task of image generation started receiving some attention from artists and designers, providing inspiration for new creations. However, exploiting the results of deep generative models such as Generative Adversarial Networks can be long and tedious given the lack of existing tools. In this work, we propose a simple strategy to inspire creators with new generations learned from a dataset of their choice, while providing some control over the output. We design a simple optimization method to find the optimal latent parameters corresponding to the closest generation to any input inspirational image. Specifically, we allow the generation given an inspirational image of the user’s choosing by performing several optimization steps to recover optimal parameters from the model’s latent space. We tested several exploration methods from classical gradient descents to gradient-free optimizers. Many gradient-free optimizers just need comparisons (better/worse than another image), so they can even be used without numerical criterion nor inspirational image, only with human preferences. Thus, by iterating on one’s preferences we can make robust facial composite or fashion generation algorithms. Our results on four datasets of faces, fashion images, and textures show that satisfactory images are effectively retrieved in most cases.

[1]  John H. Holland,et al.  Genetic Algorithms and the Optimal Allocation of Trials , 1973, SIAM J. Comput..

[2]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[3]  Kennetb A. De Genetic Algorithms Are NOT Function Optimizers , 1992 .

[4]  Georg Thierauf,et al.  Evolution strategies for solving discrete optimization problems , 1996 .

[5]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[6]  Charlie D. Frowd,et al.  EvoFIT: A holistic, evolutionary facial imaging technique for creating composites , 2004, TAP.

[7]  Andrew Zisserman,et al.  Automated Flower Classification over a Large Number of Classes , 2008, 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing.

[8]  Christopher J. Solomon,et al.  A New Computational Methodology for the Construction of Forensic, Facial Composites , 2009, IWCF.

[9]  Christopher J. Solomon,et al.  New methodology in facial composite construction: from theory to practice , 2009, Int. J. Electron. Secur. Digit. Forensics.

[10]  Charlie D. Frowd,et al.  Giving Crime the 'evo': Catching Criminals Using EvoFIT Facial Composites , 2010, 2010 International Conference on Emerging Security Technologies.

[11]  P. Hancock,et al.  Whole-face procedures for recovering facial images from memory. , 2013, Science & justice : journal of the Forensic Science Society.

[12]  Iasonas Kokkinos,et al.  Describing Textures in the Wild , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[14]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Xiaogang Wang,et al.  DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Bernt Schiele,et al.  Generative Adversarial Text to Image Synthesis , 2016, ICML.

[18]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[20]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[21]  Thomas Brox,et al.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks , 2016, NIPS.

[22]  Andrea Vedaldi,et al.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images , 2016, ICML.

[23]  Alexei A. Efros,et al.  Generative Visual Manipulation on the Natural Image Manifold , 2016, ECCV.

[24]  Percy Liang,et al.  Understanding Black-box Predictions via Influence Functions , 2017, ICML.

[25]  Trevor Darrell,et al.  Adversarial Feature Learning , 2016, ICLR.

[26]  Andrew Brock,et al.  Neural Photo Editing with Introspective Adversarial Networks , 2016, ICLR.

[27]  Sanja Fidler,et al.  Be Your Own Prada: Fashion Synthesis with Structural Coherence , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  David Pfau,et al.  Unrolled Generative Adversarial Networks , 2016, ICLR.

[29]  Ahmed M. Elgammal,et al.  Imagine it for me: Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts , 2017, ArXiv.

[30]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Guillaume Lample,et al.  Fader Networks: Manipulating Images by Sliding Attributes , 2017, NIPS.

[32]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[33]  Lior Wolf,et al.  Unsupervised Creation of Parameterized Avatars , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[34]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[35]  Ahmed M. Elgammal,et al.  CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms , 2017, ICCC.

[36]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[38]  Jonathon Shlens,et al.  Conditional Image Synthesis with Auxiliary Classifier GANs , 2016, ICML.

[39]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[40]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[41]  Hayit Greenspan,et al.  GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification , 2018, Neurocomputing.

[42]  Xi Peng,et al.  A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[43]  Ying Zhang,et al.  Fashion-Gen: The Generative Fashion Dataset and Challenge , 2018, ArXiv.

[44]  Lin Yang,et al.  Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Sebastian Nowozin,et al.  Which Training Methods for GANs do actually Converge? , 2018, ICML.

[46]  Julian Togelius,et al.  Deep Interactive Evolution , 2018, EvoMUSART.

[47]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[48]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[49]  Cordelia Schmid,et al.  Déjà Vu: an empirical evaluation of the memorization properties of ConvNets , 2018, ArXiv.

[50]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[51]  David Lopez-Paz,et al.  Optimizing the Latent Space of Generative Networks , 2017, ICML.

[52]  Yann LeCun,et al.  DeSIGN: Design Inspiration from Generative Networks , 2018, ECCV Workshops.

[53]  Emily L. Spratt Creation, curation, and classification: Mario Klingemann and Emily L. Spratt in conversation , 2018, XRDS.

[54]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[55]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[56]  Anil A. Bharath,et al.  Inverting the Generator of a Generative Adversarial Network , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[57]  Bolei Zhou,et al.  Semantic photo manipulation with a generative image prior , 2019, ACM Trans. Graph..

[58]  Yi Yu,et al.  Text2FaceGAN: Face Generation from Fine Grained Textual Descriptions , 2019, 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM).

[59]  Taesung Park,et al.  Semantic Image Synthesis With Spatially-Adaptive Normalization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Toby P. Breckon,et al.  Style Augmentation: Data Augmentation via Style Randomization , 2018, CVPR Workshops.

[61]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[62]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Lior Wolf,et al.  Live Face De-Identification in Video , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[64]  In Kyu Park,et al.  Joint Face Super-Resolution and Deblurring Using Generative Adversarial Network , 2020, IEEE Access.

[65]  Rubén García-Zurdo,et al.  Creation of Facial Composites from User Selections using Image Gradient , 2020, Informatica.

[66]  Ling Shao,et al.  Learning Digital Camera Pipeline for Extreme Low-Light Imaging , 2019, Neurocomputing.