Chinese Typeface Transformation with Hierarchical Adversarial Network

Jie Chang Yujun Gu Ya ZhangCooperative Madianet Innovation Center

Shanghai Jiao Tong University{j chang, yjgu, ya zhang}@sjtu.edu.cn

Abstract

In this paper, we explore automated typeface generationthrough image style transfer which has shown great promisein natural image generation. Existing style transfer meth-ods for natural images generally assume that the source andtarget images share similar high-frequency features. How-ever, this assumption is no longer true in typeface transfor-mation. Inspired by the recent advancement in GenerativeAdversarial Networks (GANs), we propose a HierarchicalAdversarial Network (HAN) for typeface transformation.The proposed HAN consists of two sub-networks: a transfernetwork and a hierarchical adversarial discriminator. Thetransfer network maps characters from one typeface to an-other. A unique characteristic of typefaces is that the sameradicals may have quite different appearances in differentcharacters even under the same typeface. Hence, a stage-decoder is employed by the transfer network to leveragemultiple feature layers, aiming to capture both the globaland local features. The hierarchical adversarial discrim-inator implicitly measures data discrepancy between thegenerated domain and the target domain. To leverage thecomplementary discriminating capability of different fea-ture layers, a hierarchical structure is proposed for the dis-criminator. We have experimentally demonstrated that HANis an effective framework for typeface transfer and charac-ters restoration.

1. IntroductionChinese Typeface design is a very time-consuming task,

requiring considerable efforts on manual design of bench-mark characters. Automated typeface synthesis, i.e. syn-thesizing characters of a certain typeface given few man-ually designed samples, has been explored, usually basedon manually extracted features. For example, each Chi-nese character is treated as a combination of its radicalsand strokes, and shape representation of specified typefacessuch as the contour, orientation and the component size areexplicitly learned [23, 24, 28, 27, 22]. However, these man-

ual features heavily relies on preceding structural segmen-tation of characters, which itself is a non-trivial task andheavily affected by prior knowledge.

In this paper, we model typeface transformation asan image-to-image transformation problem and attempt todirectly learn the transformation end-to-end. Typically,image-to-image transformation involves a transfer networkto map the source images to target images. A set of lossesare proposed in learning the transfer network. The pixelloss is defined as pixel-wise difference between the outputand the corresponding ground-truth [11, 7]. The percep-tual loss [8], perceptual similarity [3] and style&contentloss [1] are proposed to evaluate the differences betweenhidden-level features and all are based on the ideology offeature matching [18]. More recently, several variant ofgenerative adversarial networks (e.g CGAN [14], Cycle-GAN [29]), which introduce a discriminant network in ad-dition to the transfer network for adversarial learning, havebeen successfully applied to image-to-image transforma-tion including in-painting [15], de-noising [25] and super-resolution [9]. While the above methods have shown greatpromise for various applications, they are not directly ap-plicable to typeface transformation due to the following do-main specific characteristics.

• Different from style-transfer between natural imageswhere the source image shares high-frequency fea-tures with the target image, the transformation betweentwo different typefaces usually leads to distortion ofstrokes or radicals (e.g Fig 1), meaning change be-tween different styles leads to change of high-levelrepresentations. Hence, we cannot use a pre-trainednetwork(e.g. VGG [19]) to extract high-level represen-tations as invariant content representation in training orexplicitly define the style representation.• For typeface transformation task, different characters

may share the same radicals. This is a nice peculiar-ity that typeface transformation methods can leverage,i.e. learning the direct mapping of radicals betweensource and target styles. However, sometimes in onecertain typeface, the same radicals may appear quite

1

arX

iv:1

711.

0644

8v1

[cs

.CV

] 1

7 N

ov 2

017

source

target

a b

Figure 1. (a) target style twists strokes in source character, makingthey do not share the invariant high-frequency features though theyare the same character semantically. (b) The components in bluedotted box share the same radicals but their corresponding ones (inred dotted box) with target style are quite different.

differently in different characters. Fig 1(b) presentstwo examples where certain radicals have different ap-pearance in another styles. It will leads to severe over-fitting if we just considering the global property whileignore detailed local information.

To overcome the above problems, we design a hierarchi-cal adversarial network(HAN) for Chinese typeface trans-formation, consisting of a transfer network and a hierar-chical discriminator (Fig. 2), both of which are fully con-volutional neural networks. First, different from existingtransfer network, a staged-decoder is proposed which gen-erates artificial images in multiple decoding layers, which isexpected to help the decoder learn better representations inits hidden layers. Specially, the staged-decoder attempts tomaximally preserve the global topological structure in dif-ferent decoding layers simultaneously considers the localfeatures decoded in hidden layers, thus enabling the trans-fer network to generate close to authentic characters insteadof disordered strokes. Second, inspired by multi-classifierdesign in GoogLeNet [20], which shows that the final fea-ture layer may not provide rich and robust information formeasuring the discrepancy between prediction and ground-truth, we propose a hierarchical discriminator for adversar-ial learning. Specifically, the discriminator introduce ad-ditional adversarial losses, each of which employs featurerepresentations from different hidden layers. The multi-adversarial losses constitute a hierarchical form, enablingthe discriminator to dynamically measure the discrepancyin distribution between the generated domain and target do-main, so that the transfer network is trained to generate out-puts with more similar statistical characteristics to the tar-gets on different level of feature representation. The maincontribution of our work is summarized as follows.

• We introduce a staged-decoder in transfer networkwhich generates multiple sets of characters based ondifferent layers of decoded information, capturing boththe global and local information for transfer.

• We propose a hierarchical discriminator which in-volves a cascade of adversarial losses at different lay-ers of the network, each providing complementary ad-versarial capability. We have experimentally shownthat the hierarchical discriminator leads to faster modelconvergence and generates more realistic samples.• The proposed hierarchical adversarial network(HAN)

is shown to be successful for both typeface transfer andcharacter restoration through extensive experimentalstudies. The impact of proposed hierarchical adversar-ial loss is further investigated from different perspec-tive including gradient propagation and the ideology ofadversarial training.

2. Related Work

Many natural image-to-image transformation tasks aredomain transfer problem that maps images from source do-main to target domain. This transformation can be for-mulated on pixel level(i.e. pixel-wise loss[26, 12]) ormore recently, feature level(i.e. perceptual loss [8], Gram-Matrix [4], VGG loss [9], style loss [1]). Methods basedon feature-level even can be extended to unsupervised sit-uation in the assumption that both the input image and de-sired output image share identical or close high-level rep-resentations. However, this premise of assumption doesnot exist in handwriting transfer since sometimes the high-level representations between source characters and targetones are totally different. Recently, generative adversar-ial networks [5], especially its variants, CGAN [14] andDCGAN [16], have been successfully applied to a widespectrum of image-to-image transformation tasks. Beyondthe transfer network, CGAN-based methods introduce adiscriminator, which involves an adversarial loss for con-straining the distribution of generated domain to be closeto that of target domain. The adversarial loss is employedby all the above GAN-based studies, such as image super-resolution [9], de-noising [25] and in-painting [15]. Sev-eral studies leverage generator or discriminator to extracthidden-level representation and then perform feature match-ing in both domains [3, 21].

In recent years, many image classification, detection orsegmentation problems leverage the information in hiddenlayers of CNN for training. GoogLeNet [20] introducedauxiliary classifiers connected to intermediate layers basedon the conclusion that the features produced by the layersin the middle of the network should also be very discrimi-native. Many other CNN models utilize features producedin different intermediate layers to construct extra loss func-tion( [21, 13]). These auxiliary loss is thought to combatthe gradient-vanish problem while providing regularization.We first applies this thought to discriminator in GAN, mea-suring the similarity of two distributions not only based on

2

Fc

Conv

BN

ELU

ReLU

Concate

C-64

F-2 S-2

C-128-

F-3 S-1

C-128

F-2 S-2

C-256

F-3 S-1

C-256

F-2 S-2

C-512

F-3 S-1

C-64

F-3 S-1Encoder

Generated

Domain

Hierarchical Adversarial Discriminator

Ground-

truth

Shuffle

𝓛𝒑𝒊𝒙𝒆−𝒘𝒊𝒔𝒆

𝓛𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙1

𝓛𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙2𝓛𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙

3

𝓛𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙4

Target Domain

Conv/Deconv

Batch Normalize

Exponential Linear Unit

Rectangle Linear Unit

Concatenate

Skip Connection

Fully Connected

Data Flow

C-512

F-2 S-2

C-512+512

F-2 S-2

C-512

F-3 S-1

C-256

F-3 S-1

C-256+256

F-2 S-2

C-256

F-3 S-1

C-128

F-3 S-1

C-128+128

F-2 S-2 C-128

F-3 S-1

C-64

F-3 S-1

C-64+64

F-2 S-2C-64

F-3 S-1

C , F, S #Channel #Filter size

#Stride Size

C-64

C-128C-256C-512

𝑻𝟑

𝑻𝟐𝑻𝟏

𝑫𝟏𝑫𝟐𝑫𝟑

𝑫𝟒

[ Optional ]

Staged-Decoder

Source

Domain

Figure 2. The proposed Hierarchical Adversarial Network(HAN). HAN consists of an Encoder, a Staged-Decoder and a HierarchicalAdversarial Discriminator. The Encoder follows the Conv-BatchNorm [6]-ELU [2] architecture. The Staged-Decoder follows the Conv-BatchNorm-ReLU while two extra transformed characters are decoded from two intermediate features. The hierarchical adversarial dis-criminator is used to distinguish the transformed characters and the ground-truth from multi-level features.

the high-level features but also relative low-lever features.

3. MethodsIn this section, we present the proposed Hierarchical Ad-

versarial Network (HAN) for typeface transformation task.HAN consists of a transfer network and a hierarchical dis-criminator. The former is further consists of an Encoderand a Staged-Decoder. First, we introduce a transfer net-work T which is responsible for mapping typeface-A char-acters to typeface-B characters. Then we introduce a hier-archical adversarial discriminator which helps the transfernetwork generate more realistic characters especially for thesubtle structure in Chinese characters. Finally, we introducethe details of objective function.

3.1. FCN-Based Transfer Network

Encoder The transfer Network has a similar architectureto that of [17] with some modification. Because any infor-mation of relative-location is critical for Chinese charactersynthesis, we replace pooling operation with strided convo-lution in down-sampling since pooling helps reduce dimen-sion and retains only robust activations in a receptive fields,however leading to the loss of spatial information in somedegree. Additionally, it is a straightforward way to improvethe performance of model by increasing the size of neuralnetwork, especially the depth. So more uniform-sized conv-layers were added in our encoder for extracting more local

features (see Fig 2).

Staged-Decoder Same as encoder, we insert additionaluniform-sized convolution layers before each up-samplingconv-layer in decoder. A deeper decoder helps us model hi-erarchical representations of characters including the globaltopological structure and local topological of complicatedChinese characters. Considering the domain insight of ourtask in Section 1. We further propose a staged-decoder thatleverages the hierarchical representation of decoder. Specif-ically, different intermediate features of decoder are utilizedto generate characters (T1, T2) too. Together with the lastgenerated characters (T3), all of them will be sent to the dis-criminator(see Fig 2). We only measure the pixel-wise dif-ference between the last generated characters (T3) and cor-responding ground-truth. The adversarial loss produced byT1 and T2 helps to refine the transfer network. Meanwhile,the loss produced by the intermediate layers of decoder mayprovide regularization for the parameters in transfer net-work, which will relieves the over-fitting problem in somedegree. In addition, for typeface transformation, the inputcharacter and the desired output are expected to share un-derlying topological structure, but differ in appearance orstyle. Skip connection [17] is utilized to supplement partialinvariant skeleton information of characters with encodedfeatures concatenated on decoded features. Both encoderand staged decoder are fully convolutional networks [12].

3

3.2. Hierarchical Adversarial Discriminator

As mentioned in Section 2, adversarial loss introducedby discriminator is widely used in existing GAN-based im-age transformation task while all of them estimate the dis-tribution consistency of two domain merely based on thefinal extracted features of discriminator. It is actually un-certain whether the learned features in last layers will pro-vide rich and robust representations for discriminator. Ad-ditionally, We know the perceptual loss which penalizesthe discrepancy between representations in different hid-den space of images, is recently used in existing image-relative works. We combine the thought of perceptual lossand GANs, proposing a hierarchical adversarial discrimina-tor which leverage the perceptual representations extractedfrom different intermediate layers of discriminator D andthen distinguishes real/fake distribution between generateddomain Gdomain and target domain Tdomain(See Fig 2).Each adversarial loss is defined as:

Ldi = −Efit∼ptarget(ft)[logDi(f

it )]+

Es∼psource(s)[logDi(fis(T (s))] (1)

Lgi = −Es∼psource(s)[logDi(fis(T (s))] (2)

where f it and f is(T (s)) are ith perceptual representationslearned in Discriminator from target domain and generateddomain respectively. Di is branch discriminator cascadedafter every intermediate layer and i = 1, 2, ..4 which de-pends on the number of convolutional layers in our discrim-inator D. This variation brings a complementary adver-sarial training for our model, which urges discriminator tofind more detailed local discrepancy beyond the global dis-tribution. Assuming Ld4 and its corresponding Lg4 reachnash equilibrium, which means the the perceptual represen-tations f4t and f4s (T (s)) are considered sharing the similardistribution, however other adversarial losses (Ldi , Lgi),i 6= 4 may have not reach nash equilibrium since theselosses produced by shallow losses pay more attention onregional information during training. The still high loss pro-motes the model to be continuously optimized until all per-ceptual representations pairs (f4t , f4s (T (s))), i = 1, 2, ..4are indistinguishable by discriminator. Experiments showsthis strategy makes the discriminator to dynamically and au-tomatically discover the un-optimized space from variousperspectives.

Theoretically, our hierarchical adversarial discriminatoractually plays an implicitly role of fitting distribution fromtwo domains instead of fitting hidden features from pairedimages to be identical compared with existing methods.Thus our HAN model reduces the possibility of over-fittingand does not require pre-trained networks responsible forextracting features adopted by previous methods. Anothermerit our hierarchical adversarial strategy brought is that

these auxiliary discriminators improve the flow of infor-mation and gradients throughout the network. The previ-ous convolutional layers are optimized mainly by its neigh-bour adversarial loss beyond the other posterior adversariallosses so that the parameters existing in every discrimina-tor layer is better optimized and the generator can thus beoptimized better than before.

3.3. Losses

Pixel-level Loss The transfer network can be viewed asthe generator in GANs. It aims to synthesize characters sim-ilar to the specified ground-truth ones. L1- or L2-norm areoften used to measure the pixel distance between paired im-ages. For our typeface transformation task, each pixel incharacter is normalized near 0 or 1 value. So cross entropyfunction is selected as per-pixel loss since this charactergeneration problem can be viewed as a logistic regression.The pixel-wise loss is hence defined as follows:

Lpix−wise(T ) =

E(s,t)[−tλw · (log σ(T (s)))− (1− t) · log(1−σ(T (s)))],(3)

where T denotes the transformation of transfer network,(s, t) is pair-wise samples where s ∼ psource domain(s)and t ∼ ptarget domain(t). σ is sigmoid activation. Par-ticularly, a weighted parameter λw is introduced into pixel-wise loss for balancing the ratio of positive(value 0) tonegative(value 1) pixels in every typeface style. We addthis trade-off parameter based on the observation that sometypefaces are thin (i.e. more negative pixels) while somemay be relatively thick (i.e. more positive pixels). λw is nota parameter determined by cross validation, it is explicitlydefined by:

λw = 1−∑Kk=1

∑Nn=1 1 {tnk ≥ 0.5}∑K

k=1

∑Nn=1 1 {tnk < 0.5}

, (4)

where N the is the resolution of one character image(hereN = 64), K denotes the number of target characters intraining set and tnk denotes the nth pixel value of kth targetcharacter.

Hierarchical Adversarial Loss For our proposed HAN,each adversarial loss is defined by Eq 1 and Eq 2:

Liadversarial(Di, T ) = Ldi + Lgi . (5)

Noted that here we integrate original t ∼ ptarget(t) ands ∼ psource(s) into Eq. 5 for a unified formulation, then thetotal adversarial losses is

Ltotal adversarial(D,T ) =

k∑i=1

λi · Liadversarial(Di, T ),

(6)

4

where λi are weighted parameters to control the effect ofevery branch discriminator. The total loss function is for-mulated as follows:

Ltotal = λpLpix−wise(T ) + λaLtotal adversarial(D,T ),(7)

where λp and λa are the trade-off parameters.We optimize transfer network and hierarchical adversar-

ial discriminator by turns.

4. Experiments4.1. Data Set

There is no public data set available for Chinese charac-ters in different typefaces. We build a data set by download-ing large amount of .ttf scripts denoting different typefacesfrom the the website http://www.founder.com/.After pre-processing, each typeface ends up with 6000+grey-scale images in 64×64.png format. We choose a stan-dard printed typeface named FangSong(FS) as the sourceand the rest typefaces with handwriting styles are used astarget ones. Most of our experiments use 50% characters(3̃000 characters) as training set and the remaining as testset.

4.2. Network Setup

The hyper-parameters relevant to our proposed networkare annotated in Fig 2. The encoder includes 8 conv-layers while the staged-Decoder is more deeper includ-ing 4 transform-conv layers and 8 con-layers. Every convand deconv are followed by Conv-BatchNorm(BN) [6]-ELU [2]/ReLU structure. 4 skip connections are used onmirror layers both in encoder and staged-decoder.

For the trade-off parameters in Section 3.3, λw is deter-mined by Eq 4. The number of adversarial loss of HAN l is4 and weighted parameter {λi}31 is decay from 1 to 0.5 withrate 0.9, λ4 = 1.0. λp and λa are both set to 1.0 to weightthe pixel loss and adversarial loss.

4.3. Performance Comparison

To validate the proposed HAN model, we compare thetransfer performance of HAN with a Chinese calligra-phy synthesis method (AEGG [13]) and two state-of-the-art image-to-image transformation methods(Pix2Pix [7],Cycle-GAN [29]). Our proposed HAN can be trainedin two modes. The first is strong-paired mode whichminimizes pixel-wise discrepancy Lpix−wise obtained bypaired characters as well as hierarchical adversarial lossLtotaladversarial obtained by generated and target domain.The second is soft-paired mode by removing Lpix−wiseand just minimizingLtotaladversarial, which looses the con-strain of pairing source characters with corresponding targetones.

Strong-Paired Learning. Baseline AEGG and Pix2Pixboth need pair the generated images with correspondingground-truths for training so we compare our HAN withthem in strong-paired mode. The transfer network ofPix2Pix shares the identical framework with that in ourHAN(see Fig 2) and the model used in AEGG follows theinstructions of their paper with some tiny adjustment fordimension adaptation. 50%(3̃000) characters randomly se-lected from FS typeface as well as 50% corresponding tar-get style characters selected from other handwriting-styletypeface are used as training set. The remaining 50% of FStypefaces is used for testing. We perform 5 experimentstransferring FS typeface to other Chinese handwriting-style(see Fig 3). All methods can capture general style ofhandwriting however AEGG and Pix2Pix failed to synthe-size recognizable characters because most strokes in gener-ated character are disordered even chaotic. Our HAN sig-nificantly outperforms AEGG and Pix2Pix, especially imi-tating cursive handwriting characters. Experimental resultshows HAN is superior in generating detailed componentof characters. We also observed that both baselines performwell on training set but far worse on test set, which suggeststhe proposed hierarchical adversarial loss makes our modelless prone to over-fitting in some degree.Soft-Paired Learning. Another model Cycle-GAN actu-ally is an unpaired method which does not require ground-truth for training. Nevertheless we experiment unpairedform with Cycle-GAN and proposed HAN, both of theirresults are very bad. So we compare our HAN with Cycle-GAN in soft-paired mode, saving the trouble of tediouspairing but leaving the ground-truths in training set. As il-lustrated in Fig 4, under the condition of soft-paired, ourHAN performs well than Cycle-GAN. Though Cycle-GANcorrectly captures the style of target characters, it cannot re-construct correct location of every stroke and Cycle-GANleads to model collapse. Of course, results of HAN trainedin soft-paired mode is not as good as that strong-pairedmode since the strong supervision information is reducedby removing Lpix−wise.Quantitative Evaluation. Beyond directly illustratingqualitative results of comparison experiments, two quanti-tative measurements: Root Mean Square Error(RMSE) andAverage Pixel Disagreement Ration [10](APDR) are uti-lized as evaluation criterion. As shown in Table 1, our HANleads to the lowest RMSE and APDR value both under themode of strong-paired and soft-paired mode compared withexisting methods.

4.4. Analysis of Hierarchical Adversarial Loss

We analyze each adversarial loss, {Ldi}4i=1 and{Lgi}4i=1, defined in Section 3.2. As shown in Fig 5,the generator loss gen 4 produced by the last conv-layerin hierarchical discriminator fluctuates greatly and then

5

Results on test-set Results on train-set

Source

AEGG

Pix2Pix

Ours

Target

Source

AEGG

Pix2Pix

Ours

Target

Source

AEGG

Pix2Pix

Ours

Target

Source

AEGG

Pix2Pix

Ours

Target

Source

AEGG

Pix2Pix

Ours

Target

Transfer FS-typeface to 5 personal handwriting-styles typefaceFigure 3. Performance of transferring FS typeface to other 5 personal handwriting-style typefaces.

gen 3 produced by the penultimate layer, {gen 2, gen 1}produced by shallower conv-layers are relatively gentlebecause λ4 is set larger than {λi}3i=1 so that networkmainly optimizes gen 4. However for discriminator loss,{dis 4, dis 3, dis 1} derived from D4, D3,D1 are mostlynumerical approach. We further observed that the trend ofincrease or reduction among various discriminator lossesare not always consistent. We experimentally conclude thatadversarial losses produced by intermediate layers can as-

sist training: whenD4 is severely cheated by real/fake char-acters, D3 or D2 or D1 can still give a high confidenceof differentiating, which means True/False discriminationbased on different representations can be compensated eachother(see Fig 5 for more details) during training.

We further explore the influence brought by our hierar-chical adversarial loss. By removing the effect of hierarchi-cal architecture from our HAN model, we run another con-trast experiment, Single Adversarial Network (SAN). The

6

Source Characters

HAN(strong-pair)

CycleG(soft-pair)

HAN(soft-pair)

Target Characters

Figure 4. We compare our HAN with Cycle-GAN by loosing the pairing constraint and HAN performs better than Cycle-GAN.

Model FS→handwriting1 FS→handwriting2 FS→handwriting3 FS→handwriting4RMSE APDR RMSE APDR RMSE APDR RMSE APDR

AEGG [13] 22.671 0.143 28.010 0.211 24.083 0.171 22.110 0.131Pix2Pix [7] 29.731 0.231 27.117 0.225 26.580 0.187 24.135 0.180Cycle-GAN [29] 29.602 0.253 29.145 0.234 28.845 0.241 25.632 0.191HAN(Soft-pair) 20.984 0.125 25.442 0.207 24.741 0.181 20.714 0.134HAN(Strong-pair) 19.498 0.118 23.303 0.181 22.266 0.162 19.528 0.110

Table 1. Quantitative Measurements

700 725 750 775 800 825 850 875 900Iterations of training

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Loss

for G

ener

ator

Loss Curvesgen_4gen_3

gen_2gen_1

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

Loss

for D

iscrim

inat

or

dis_4dis_3

dis_2dis_1

Figure 5. Each generator loss and discriminator loss during ste 700to 900.

detail of network follows Fig 2 and we set trade-off param-eters λ1 = λ2 = λ3 = 0.5 and λ4 = 1 in loss function ofHAN, while we set λ1 = λ2 = λ3 = 0 and λ4 = 1 forSAN in order to remove the influence of extra 3 adversar-ial losses. Considering the value of hierarchical adversarialloss(we accumulate four adversarial losses) is bigger thanthat of single adversarial loss, the gradients in back prop-agation of HAN is hence theoretically bigger than that ofSAN. For demonstrating that our HAN works not for thisreason, we multiply a constant c = λ1+λ2+λ3+λ4

λ4before

the adversarial loss in SAN so that these two adversarialloss respectively existing in HAN and SAN are close prox-imity. Characters generated during different training periodare illustrated in Fig 6 from which we can see qualitative ef-

fect of proposed hierarchical adversarial discriminator. Ourproposed HAN generates more clear characters comparedwith SAN at the same phase of training period, which sug-gests HAN converge greatly faster than SAN. We also run 3parallel typeface-transfer experiments then calculate RMSEalong with the iterations of training on train set. Left loss-curves in Fig 6 demonstrates that hierarchical adversarialarchitecture assists to accelerate convergence and leads tolower RMSE value.

4.5. Character Restoration with HAN

Beyond transferring standard printed typeface to anyhandwriting-style typeface, we also applied our HANmodel to character restoration. We randomly mask 30%region on every handwriting characters in one typeface’straining set. Under strong-paired mode, our HAN learnedto correctly reconstruct the original characters. As illus-trated in Fig 7, our HAN is able to correctly reconstruct themissing part of one character on test set.

4.6. Impact of Training Set Size

Last, we experiment at least how many handwriting char-acters should be given in training to ensure a satisfied trans-fer performance. So we experiment three typeface-transfertasks(type-1, type-2 and type-3) with different proportion oftraining samples and then evaluate on each test set. As thesynthesized characters shown in Fig 8, the performance im-proves along with increase of training samples. We also useRMSE to quantify the performance under different train-

7

S Hepo-1

epo-3

epo-4

epo-5

epo-8

epo-11

epo-24

epo-18

epo-2

HANconverge

Target

Source

S H S H S H

S : SAN structure(without HA framework)H: HAN structure

Figure 6. Contrast experiments for HAN and SAN. Characters generated by HAN are far more better than that by SAN in same trainingepoch. HAN converge row shows characters generated when our HAN model converges. The RMSE evaluation loss along with the trainingiterations under HAN and SAN shows HAN leads to more lower value than SAN.

Incompletecharacters

HAN restoration

Groundtruth

Figure 7. Performance of repairing personal handwriting charac-ters with HAN on test set.

ing samples. All 3 curves suggests when the proportion oftraining size is not less than 35%(2000 samples), the perfor-mance will not be greatly improved.

5. Conclusion and Future WorkIn this paper, we propose a hierarchical adversarial net-

work (HAN) for typeface transformation. The HAN is con-sisted of a transfer network and a hierarchical adversarialdiscriminator. The transfer network consists of an encoderand a staged-decoder which can generate characters basedon different decoded information. The proposed hierarchi-cal discriminator can dynamically estimate the consistencyof two domains from different-level perceptual represen-

Groundtruth

500

100

1500

2500

Figure 8. The RMSE evaluation under different proportion oftraining set. The red and black number denote how many trainsamples we used. We present 3 transferring handwriting charac-ters from FS-typeface to handwriting type-1, type-2 and type-3.

tations, which helps our HAN converge faster and better.Experimental results show our HAN can synthesize mosthandwriting-style typeface compared with existing naturalimage-to-image transformation methods. Additionally, ourHAN can be applied to handwriting character restoration.

References[1] D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua. Stylebank: An

explicit representation for neural image style transfer. arXivpreprint arXiv:1703.09210, 2017.

8

[2] D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast andaccurate deep network learning by exponential linear units(elus). Computer Science, 2015.

[3] A. Dosovitskiy and T. Brox. Generating images with percep-tual similarity metrics based on deep networks. In Advancesin Neural Information Processing Systems, 2016.

[4] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transferusing convolutional neural networks. In Computer Visionand Pattern Recognition, pages 2414–2423, 2016.

[5] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.Generative adversarial nets. In International Conference onNeural Information Processing Systems, pages 2672–2680,2014.

[6] S. Ioffe and C. Szegedy. Batch normalization: Acceleratingdeep network training by reducing internal covariate shift.Computer Science, 2015.

[7] P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks.arXiv preprint arXiv:1611.07004, 2016.

[8] J. Johnson, A. Alahi, and F. F. Li. Perceptual losses for real-time style transfer and super-resolution. In European Con-ference on Computer Vision, pages 694–711, 2016.

[9] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham,A. Acosta, A. Aitken, A. Tejani, J. Totz, and Z. Wang. Photo-realistic single image super-resolution using a generative ad-versarial network. arXiv preprint arXiv:1609.04802, 2016.

[10] M.-Y. Liu and O. Tuzel. Coupled generative adversarial net-works. In Advances in neural information processing sys-tems, pages 469–477, 2016.

[11] J. Long, E. Shelhamer, and T. Darrell. Fully convolutionalnetworks for semantic segmentation. IEEE Transactions onPattern Analysis & Machine Intelligence, 39(4):640, 2014.

[12] J. Long, E. Shelhamer, and T. Darrell. Fully convolutionalnetworks for semantic segmentation. In Computer Visionand Pattern Recognition, pages 3431–3440, 2015.

[13] P. Lyu, X. Bai, C. Yao, Z. Zhu, T. Huang, and W. Liu. Auto-encoder guided gan for chinese calligraphy synthesis. arXivpreprint arXiv:1706.08789, 2017.

[14] M. Mirza and S. Osindero. Conditional generative adversar-ial nets. Computer Science, pages 2672–2680, 2014.

[15] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A.Efros. Context encoders: Feature learning by inpainting.In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, pages 2536–2544, 2016.

[16] A. Radford, L. Metz, and S. Chintala. Unsupervised repre-sentation learning with deep convolutional generative adver-sarial networks. Computer Science, 2015.

[17] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolu-tional networks for biomedical image segmentation. In In-ternational Conference on Medical Image Computing andComputer-Assisted Intervention, volume 9351, pages 234–241, 2015.

[18] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Rad-ford, and X. Chen. Improved techniques for training gans. InAdvances in Neural Information Processing Systems, 2016.

[19] K. Simonyan and A. Zisserman. Very deep convolutionalnetworks for large-scale image recognition. Computer Sci-ence, 2014.

[20] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.Going deeper with convolutions. In Computer Vision andPattern Recognition, pages 1–9, 2015.

[21] C. Wang, C. Xu, C. Wang, and D. Tao. Perceptual adversarialnetworks for image-to-image transformation. arXiv preprintarXiv:1706.09138, 2017.

[22] J. Xiao, J. Xiao, and J. Xiao. Automatic generation of large-scale handwriting fonts via style learning. In SIGGRAPHASIA 2016 Technical Briefs, page 12, 2016.

[23] S. Xu, H. Jiang, T. Jin, F. C. M. Lau, and Y. Pan. Auto-matic generation of chinese calligraphic writings with styleimitation. IEEE Intelligent Systems, 24(2):44–53, 2009.

[24] S. Xu, T. Jin, H. Jiang, and F. C. M. Lau. Automatic genera-tion of personal chinese handwriting by capturing the charac-teristics of personal handwriting. In Conference on Innova-tive Applications of Artificial Intelligence, July 14-16, 2009,Pasadena, California, Usa, 2010.

[25] H. Zhang, V. Sindagi, and V. M. Patel. Image de-rainingusing a conditional generative adversarial network. arXivpreprint arXiv:1701.05957, 2017.

[26] R. Zhang, P. Isola, and A. A. Efros. Colorful image coloriza-tion. In European Conference on Computer Vision, pages649–666, 2016.

[27] X.-Y. Zhang, F. Yin, Y.-M. Zhang, C.-L. Liu, and Y. Bengio.Drawing and recognizing chinese characters with recurrentneural network. IEEE Transactions on Pattern Analysis andMachine Intelligence, 2017.

[28] B. Zhou, W. Wang, and Z. Chen. Easy generation of personalchinese handwritten fonts. In IEEE International Conferenceon Multimedia and Expo, pages 1–6, 2011.

[29] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial net-works. arXiv preprint arXiv:1703.10593, 2017.

9