Diverse Image-to-Image Translation via Disentangled RepresentationsHsin-Ying Lee*1 Hung-Yu Tseng*1 Jia-Bin Huang2 Maneesh Singh3 Ming-Hsuan Yang1,4
1University of California, Merced 2Virginia Tech 3Verisk Analytics 4Google Cloud AI
Code available!http://bit.ly/DRIT-ECCV18
Image-to-image translation
Challenges
Contributions
Disentangled representation for image-to-image translation (DRIT)
!"##
$%&
$%'
(%
$%&
$%'
(%
!"##
$)&
$)'
()
$)&
$)'
()
!adv#-./0./ !adv
1-234. !adv1-234.
5
6
7
8
95
96
1. Disentangled representation & cross cycle consistency
2. Diverse translation from unpaired data
3. Competitive performance on domain adaptation
Experimental resultsVisual results
60.876.7
42.3
39.223.3
57.7
25.3 19.6 16.237.5
74.7 80.4 83.862.5
Realism:Preference percentage
Training
1. Lack of aligned training pairs
2. Multiple possible outputs given singe input image
Winter Summer
()
$%'
$)&
()
E(0,1)…
$%'
Example guided translation Randomly sampled translation
Photo Artistic portrait
% domain
) domain
Loss
Prior distribution
Content encoder$'
Attribute encoder$&
Generator(
Diversity: LPIPS scoreDRIT (ours)Cycle/Bicycle
UNITCycleGAN
Real images
Domain adaptation
References[1] Liu et al. Unsupervised Image-to-Image Translation Networks. In NIPS, 2017[2] Zhu et al. Unpaired Image-to-Image Translation using Cycle-Consistent[2] Adversarial Networks. In ICCV, 201[3] Zhu et al. Toward Multimodal Image-to-Image Translation. In NIPS, 2017
Other training loss
(%
$%'
$%&
!KLE(0,1)
!"recon
()
$%'
E(0,1) !"latent
!adv1-234.
$)&
(a) MNIST-M (b) LineMODSource
Targetexample
Randomly sampledtranslation
Paired data Unpaired data
One-to-one Pix2pix [Isola et al.]DiscoGAN [Kim et al.]CycleGAN [Zhu et al.]
UNIT [Liu et al.]
One-to-many BicycleGAN [Zhu et al.]Pix2pixHD [Wang et al.]
DRIT (Ours)1
2
2
Real images