Deepfaking it: experiments in generative, adversarial multispectral remote sensing

In this work we utilize generative adversarial networks (GANs) to synthesize realistic transformations for remote sensing imagery in the multispectral domain. Despite the apparent perceptual realism of the transformed images at a first glance, we show that a deep learning classifier can very easily be trained to differentiate between real and GAN-generated images, likely due to subtle but pervasive artifacts introduced by the GAN during the synthesis process. We also show that a very low-amplitude adversarial attack can easily fool the aforementioned deep learning classifier, although these types of attacks can be partially mitigated via adversarial training. Finally, we explore the features utilized by the classifier to differentiate real images from GAN-generated ones, and how adversarial training causes the classifier to focus on different, lower-frequency features.

[1]  Shan Zhong,et al.  SAR Image Colorization Using Multidomain Cycle-Consistency Generative Adversarial Network , 2021, IEEE Geoscience and Remote Sensing Letters.

[2]  B. Tondi,et al.  GAN generation of synthetic multispectral satellite images , 2020, Remote Sensing.

[3]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[4]  James Theiler,et al.  Deep snow: synthesizing remote sensing imagery with generative adversarial nets , 2020, Defense + Commercial Sensing.

[5]  Hanxiang Hao,et al.  Manipulation Detection in Satellite Images Using Deep Belief Networks , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Pascal Sturmfels,et al.  Visualizing the Impact of Feature Attribution Baselines , 2020 .

[7]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  J. Theiler,et al.  Cycle-Consistent Adversarial Networks for Realistic Pervasive Change Generation in Remote Sensing Imagery , 2019, 2020 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI).

[9]  Siwei Lyu,et al.  Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Corentin Henry,et al.  SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks - Optimization, Opportunities and Limits , 2019, Remote. Sens..

[11]  Taesung Park,et al.  GauGAN: semantic image synthesis with spatially adaptive normalization , 2019, SIGGRAPH 2019.

[12]  Itaru Kitahara,et al.  SAR2OPT: Image Alignment Between Multi-Modal Images Using Generative Adversarial Networks , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[13]  Carola-Bibiane Schönlieb,et al.  On the Connection Between Adversarial Robustness and Saliency Map Interpretability , 2019, ICML.

[14]  Larry S. Davis,et al.  Adversarial Training for Free! , 2019, NeurIPS.

[15]  Xiao Xiang Zhu,et al.  The SEN1-2 Dataset for Deep Learning in SAR-Optical Data Fusion , 2018, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[16]  Michael Schmitt,et al.  A Conditional Generative Adversarial Network to Fuse Sar And Multispectral Optical Data For Cloud Removal From Sentinel-2 Images , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[17]  Aleksander Madry,et al.  Robustness May Be at Odds with Accuracy , 2018, ICLR.

[18]  Mark Sandler,et al.  CycleGAN, a Master of Steganography , 2017, ArXiv.

[19]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[20]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[21]  Avanti Shrikumar,et al.  Learning Important Features Through Propagating Activation Differences , 2017, ICML.

[22]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[23]  Samy Bengio,et al.  Adversarial Machine Learning at Scale , 2016, ICLR.

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[26]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[27]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  E. Delp,et al.  Nested Attention U-Net: A Splicing Detection Method for Satellite Images , 2020, ICPR Workshops.