1、Generative Adversarial Nets:Applications and ExtensionsLeCun,NIPS 2016 Reinforcement learning(cherry)Supervised learning(Chocolate)Unsupervised/Predictive learning(Cake)Generative adversarial nets(GAN)For Most Application Tasks For most applications,GANs only serve as theaccessories to theexisting s
2、olutions.GAN How to Make Latte Art(i.e.improve theOther LearningModelstrainability of generator)How to make a perfectLatte Coffee(i.e.incorporate with othermodels for solving realproblems)Content Improve the trainability of GANs:An Application Perspective Theoretical solution Incorporating with othe
3、r learning models Designing generator based on signal/image characteristics Applications Adversarial learning Low level vision Domain adaptation Image translationImprove the trainability of GANsGenerative Adversarial Networks(Goodfellowet al.,NIPS 2014)Update the generator to generate more realistic
4、 image Update the discriminator to discriminate the synthetic images fromreal onesMode Collapse D in inner loop:convergence to correct distribution G in inner loop:place all mass on most likely pointLets first turn to supervised deep learning Unprecedented successes in:Image classification Image den
5、oising,image super-resolution.Can we exploit these achievements to improve GAN training?How to train a good generator(the later half of image restoration?)How to train a good discriminator(classification?)Auto-encoder Auto-encoder Denoising auto-encoderVariational AutoEncoder Variational AutoEncoder
6、 Relaxation of discrete variablesVAE/GAN(Larsen et al.,ICML 2016)VAE GAN VAE/GANClassifier Discriminator Na Lei,Kehua Su,Li Cui,Shing-Tung Yau,David Xianfeng Gu,AGeometric View of Optimal Transportation and Generative Model,Arxiv 2017.Nguyen et al.,NIPS 2016 Optimize the hidden code input(red bar)of
7、 a deep image generatornetwork(DGN)to produce an image that highly activates hInfoGAN(Chen et al.,NIPS 2016)GAN InfoGAN(Chen et al.,NIPS 2016)Input:z,c Interpretable and disentangled representations Easy to trainAC-GAN(Odena et al.,ICML 2017)Class-conditional image synthesis with Auxiliary Classifie
8、r GANs The log-likelihood of the correct source:The log-likelihood of the correct class:Arbitrary Facial Attribute Editing One model for all tasks(He et al.,Arxiv 2018)A Favorable Framework Auto-encoderAttGANExtension for attribute style manipulationSingle taskMulti-taskContinuous attributeAttribute
9、 Style ManipulationTa ke home message Incorporating auto-encoder to improve the trainability of generator;Incorporating deep classification model to improve the trainability ofdiscriminatorLets then turn to the objective of GANs Image generation Whats the characteristics of an image Multi-scale prop
10、erty Manifold property What makes a high quality image Deep image prior Deep image quality assessmentLAPGANs(Denton et al.,NIPS 2015)LAPGANs(Denton et al.,2015)Stack-GAN(Zhang et al.,ICCV 2017)Stack-GAN(Zhang et al.,ICCV 2016)Stage-I GAN Stage-II GANCascaded Refinement Networks(Chen&Koltun,ICCV 2017
11、)CRN:not rely on adversarial trainingManifold property(Benaim&Wolf,NIPS2017)Distance Constraints Self-distance ConstraintsTotal Variation Deep feature visualization Total variation(TV)regularization Better(deep)image prior?Insight from deep image denoising DnCNN for image denoising(Zhang et al.,TIP
12、2017)x y CNN(y;)For noisy image,CNN(y;)y x2CNN(y;)mn 2 For clean image,CNN(y;)2 0 Perceptual regularization(Li et al.,Arxiv 2016)2CNN(y;)Deep image prior(Ulyanov et al.,CVPR 2018)Energy Image restoration A randomly-initialized neural network can be used as a handcraftedprior The structure of a gener
13、ator network is sufficient to capture a greatdeal of low-level image statistics prior to any learningDeep Features as a Perceptual Metric(Zhanget al.,CVPR 2018)Perceptual loss Deep features outperform all previous metrics by huge margins.This result is not restricted to ImageNet-trained VGG features
14、,butholds across different deep architectures and levels of supervision(supervised,self-supervised,or even unsupervised).Deep Non-reference Image Quality Assessment?Ta ke home message Exploiting image property to improve GANs Developing deep models/GANs for better revealing imagepriors/quality Objec
15、t-oriented designApplicationsAdversarial learning(Szegedy et al.,ICLR 2014)Deep neural networks learn input-output mappings that are fairlydiscontinuous to a significant extent.We can cause the network to misclassify an image by applying acertain hardly perceptible perturbation,which is found by max
16、imizingthe networks prediction error.2018-4-22Intriguing properties of neural networks(Szegedy et al.,ICLR 2014)Deep Neural Networks are Easily Fooled(Nguyen et al.,CVPR 2015)2018-4-22 99.6%confidences2018-4-22Adversarial Attacks and Defences Competition(Kurakin et al.,Arxiv 2018)1st place in defens
17、e track:team TsAIL Team members:Yinpeng Dong,Fangzhou Liao,Ming Liang,TianyuPang,Jun Zhu and Xiaolin Hu.Solution:Denoising U-netAdversarially-augmented training(Simon-Gabriel et al.,Arxiv 2018)Adversarially-augmented training Replacing strided by average-pooling layers Increase generalization perfor
18、manceObject detection:A-Fast-RCNN(Wang et al.,CVPR 2017)Visual tracking CVPR 2018 VITAL:VIsual Tracking via Adversarial Learning SINT+:Robust Visual Tracking via Adversarial Hard PositiveGenerationLow level vision SRGAN for super-resolutionDSLR-Quality Photos on Mobile Devices(Ignatov et al.,ICCV 20
19、17)Color loss Texture loss Content loss TV regularizer DiscriminatorWESPE:Weakly Supervised Photo Enhancer(NTIRE 2018)Only require two distinct datasetsImage inpainting:more freedom and non-uniquenessContext-encoders(Pathak et al.,2016)The first key:Auto-encoderProblem with auto-encoder Information
20、bottleneckAdversarial loss is helpful But remains limited.Analyzing U-Net(Ronneberger et al.,2015)Fine-details Unfortunately,also not work for inpaintingReturn to traditional patch-based inpainting Patch processing order PatchMatchCNN and Patch-based Solutions areComplementary CNN-based solutionCont
21、ext-encoders Poor texture Better structure Patch-based solution Better details Poor structure Can we combine them in an end-to-end learning framework?CNN architectureObjective and learning Objective LearningResults Speed MNPS:40mins-40s Ours:82 ms PSNRRandom maskReal imagesGuided face enhancement(Li
22、 et al.,Arxiv2018)Film Restoration,SmartphonesChallenges 1.Blind enhancement:the degradation model is sophisticated andunknown blur,downsampling,noise,compression 2.The guided and degraded images are of different pose,expressionand illuminationChallenge 1 Train on realistic synthetic degraded images
23、,test on real degradedimage The degradation model:Challenge 2:GFRNetModel and losses for WarpNet Landmark loss TV regularizationModel objective Reconstruction loss Adversarial loss ObjectiveAppearance FlowResultsOursMDnCNNMARCNNMDeblurGANMore imagesVideoDomain Adaptation Domain Adaptation:learning f
24、rom a(labeled)source datadistribution a well performing model on a different(but related)(labeled or unlabled)target data distribution(wikipedia)Three categories:Supervised domain adaptation Semi-supervised domain adaptation Unsupervised domain adaptationThe Future of Real-Time SLAM(ICCV 2015Worksho
25、p)Panel discussion:Deep Learning vs SLAM Newcombes Proposal:Use SLAM to fuel Deep Learning Todays SLAM systems are large-scale correspondence engineswhich can be used to generate large-scale datasets Graphics for CNNThe need of domain adaptationSynthetic:DomainTransferReal:Unsupervised domain adapta
26、tion Only the class labels of source samples are known,all class labels ofthe target samples are unknown.Goal:a feature extractor f and a classifier c P(f(x)=P(f(x)st Better classification performance on xs Key issue:Discrepancy metric between two complex distributions D(P(f(x),P(f(x)stWeighted MMD
27、Let Define Weighted MMDOffice-10+Caltech-10Unsupervised Domain Adaptation byBackpropagation(Ganin&Lempitsky,ICML 2015)Simultaneous Deep Transfer Across Domainsand Tasks(Tzeng et al.,ICCV 2015)“maximally confuse”thetwo domains uniform distribution overdomain labelsDomain cocktail network(Xu et al.,CV
28、PR2018)SimGAN(CVPR 2017)Learning from Simulated and Unsupervised Images throughAdversarial Training(Shrivastava,Arxiv 2016)Realism loss Self-regularization is also pixel-level DAUnsupervised PixelLevel Domain Adaptation(CVPR 2017)Image translation(Zhu et al.,CVPR 2017)Pix2pix:supervised image transl
29、ation(Isola etal.,CVPR 2017)Positive pair:(Input,groundtruth)Negative pair:(Input,synthesis)Learning Residual Images(Shen&Liu,CVPR2017)Cycle-Consistent supervision(Zhu et al.,ICCV2017)Cycle consistency lossBicycleGAN:Multimodal Image-to-ImageTranslation(Zhu et al.,NIPS 2017)Suggestion Problem-orient
30、ed Generator Discriminator+References I.Goodfellow,J.Pouget-Abadie,M.Mirza,B.Xu,D.Warde-Farley,S.Ozair,A.Courville,Y.Bengio,Generative adversarial nets,NIPS 2014.D.P.Kingma,M.Welling,Auto-encoding variational bayes,arXiv:1312.6114,2013.N.Lei,K.Su,L.Cui,S.-T.Yau,D.X.Gu,A Geometric View of OptimalTran
31、sportation and Generative Model,Arxiv 2017.A.B.L.Larsen,S.K.Snderby,H.Larochelle,O.Winther,Autoencodingbeyond pixels using a learned similarity metric,ICML 2016.A.Nguyen,A.Dosovitskiy,J.Yosinski,T.Brox,and J.Clune.Synthesizing the preferred inputs for neurons in neural networks viadeep generator net
32、works,NIPS 2016.X.Chen,Y.Duan,R.Houthooft,J.Schulman,I.Sutskever,P.Abbeel,InfoGAN:Interpretable representation learning by informationmaximizing generative adversarial nets,NIPS 2016.A.Odena,C.Olah,J.Shlens,Conditional image synthesis withauxiliary classifier GANs,ICML 2017.Z.He,W.Zuo,M.Kan,S.Shan,X
33、.Chen,Arbitrary Facial AttributeEditing:Only Change What You Want,arXiv:1711.10678,2017.E.L.Denton,S.Chintala,R.Fergus,Deep generative image modelsusing a?laplacian pyramid of adversarial networks,NIPS 2015.H.Zhang,T.Xu,H.Li,S.Zhang,X.Wang,X.Huang,D Metaxas,StackGAN:Text to photo-realistic image syn
34、thesis with stackedgenerative adversarial networks,ICCV 2017.Q.Chen,V.Koltun,Photographic image synthesis with cascadedrefinement networks,ICCV 2017.S.Benaim,and Lior Wolf,One-Sided Unsupervised Domain Mapping,NIPS 2017.K.Zhang,W.Zuo,Y.Chen,D.Meng,L.Zhang,Beyond a Gaussiandenoiser:Residual learning
35、of deep cnn for image denoising,IEEE T-IP2017.M.Li,W.Zuo,D.Zhang,Deep Identity-aware Transfer of FacialAttributes,arXiv:1610.05586.D.Ulyanov,A.Vedaldi,V.Lempitsky,Deep Image Prior,CVPR 2018.R.Zhang,P.Isola,A.A.Efros,E.Shechtman,O.Wang,TheUnreasonable Effectiveness of Deep Features as a Perceptual Me
36、tric,CVPR 2018.C.Szegedy,W.Zaremba,I.Sutskever,J.Bruna,D.Erhan,I.Goodfellow,R.Fergus,Intriguing properties of neural networks,ICLR 2014.A.Nguyen,J.Yosinski,J.Clune,Deep neural networks are easilyfooled:High confidence predictions for unrecognizable images,CVPR2015.C.-J.Simon-Gabriel,Y.Ollivier,B.Sch
37、lkopf,L.Bottou,D.Lopez-Paz,Adversarial Vulnerability of Neural Networks Increases With InputDimension,arXiv:1802.01421.X.Wang,A.Shrivastava,A.Gupta,A-Fast-RCNN:Hard positivegeneration via adversary for object detection,CVPR 2017.C.Ledig,L.Theis,F.Huszar,J.Caballero,A.Cunningham,A.Acosta,A.Aitken,A.T
38、ejani,J.Totz,Z.Wang,W.Shi,Photo-realistic single imagesuper-resolution using a generative adversarial network,CVPR 2017.A.Ignatov,N.Kobyshev,R.Timofte,K.Vanhoey,L.Van Gool,DSLR-quality photos on mobile devices with deep convolutional networks,ICCV 2017.A.Ignatov,N.Kobyshev,R.Timofte,K.Vanhoey,L.Van
39、Gool,WESPE:Weakly supervised photo enhancer for digital cameras,NTIRE 2018.D.Pathak,P.Krahenbuhl,J.Donahue,T.Darrell,A.A.Efros,Contextencoders:Feature learning by inpainting,CVPR 2016.Z.Yan,X.Li,M.Li,W.Zuo,S.Shan,Shift-Net:Image Inpainting viaDeep Feature Rearrangement,arXiv:1801.09392.X.Li,M.Liu,Y.
40、Ye,W.Zuo,L.Lin,R.Yang,Learning Warped Guidancefor Blind Face Restoration,arXiv:1804.04829.H.Yan,Y.Ding,P.Li,Q.Wang,Y.Xu,W.Zuo,Mind the class weightbias:Weighted maximum mean discrepancy for unsupervised domainadaptation,CVPR 2017.Y.Ganin,V.Lempitsky,Unsupervised Domain Adaptation byBackpropagation,I
41、CML 2015.E.Tzeng,J.Hoffman,T.Darrell,Simultaneous Deep Transfer AcrossDomains and Tasks,ICCV 2015.R.Xu,Z.Chen,W.Zuo,J.Yan,L.Lin,Deep Cocktail Network:Multi-source Unsupervised Domain Adaptation with Category Shift,CVPR2017.A.Shrivastava,T.Pfister,O.Tuzel,J.Susskind,W.Wang,R.Webb,Learning from simulated and unsupervised images throughadversarial training,CVPR 2017.K.Bousmalis,N.Silberman,D.Dohan,D.Erhan,D.Krishnan,Unsupervised pixel-level domain adaptation with generativeadversarial networks,CVPR 2017.
