Ukiyo-e Analysis and Creativity with Attribute and Geometry Annotation

Yingtao TianGoogle BrainTokyo Japan

Tarin ClanuwatROIS-DS Center for

Open Data in the HumanitiesNII

Chikahiko SuzukiROIS-DS Center for

Open Data in the HumanitiesNII

Asanobu KitamotoROIS-DS Center for

Open Data in the HumanitiesNII

Abstract

The study of Ukiyo-e an important genre of pre-modernJapanese art focuses on the object and style like other art-work researches Such study has benefited from the renewedinterest by the machine learning community in culturallyimportant topics leading to interdisciplinary works includ-ing collections of images quantitative approaches and ma-chine learning-based creativities They however have sev-eral drawbacks and it remains challenging to integrate theseworks into a comprehensive view To bridge this gap wepropose a holistic approach1 We first present a large-scaleUkiyo-e dataset with coherent semantic labels and geomet-ric annotations then show its value in a quantitative study ofUkiyo-e paintingsrsquo object using these labels and annotationsWe further demonstrate the machine learning methods couldhelp style study through soft color decomposition of Ukiyo-e and finally provides joint insights into object and style bycomposing sketches and colors using colorization

IntroductionThe Edo period of Japan (16th to 19th century) has seenthe prosper of Ukiyo-e (浮世絵) a genre of pre-modernJapanese artwork that consists of paintings and woodblockprintings Unlike early dominating Emakimono (絵巻物picture scroll) and Ehon (絵本 picture book) that focus onfamous figures and stories in Sinosphere culture and clas-sic Japanese stories the topic of Ukiyo-e extends broadly todaily subjects such as characters like beauties and Kabuki(歌舞伎) landscape arts animals and plants in everydaylife and even contemporary news As an example Fig-ure 1 shows a Ukiyo-e depicting a Kabuki performanceThe popularity of woodblock printing makes it possible toproduce paintings on a larger scale at a lower cost whichcontributes to the flourish of Ukiyo-e and leaves us with avast collection of artworks in this genre (Kobayashi 1994IUS 2008) Such an extensive and varied collection providesa valuable corpus for Japanese artwork research

The subject of such artwork study could be multi-facetedinvolving several aspects of which two crucial are the ob-ject in the painting such as the outline and the shape ofdepicted figures and the style of painting such as texturesand colors For example the former reveals the trend of

1httpsgithubcomrois-codharc-ukiyoe-faces

Attribute Value

Title(画題)

Kizukansuke(「木津勘助」)

Painter(絵師)

Hirosada(広貞)

Format(判種)

Middle-size Nishiki-e(中判錦絵)

Year in AD(西暦) 1849

middot middot middot middot middot middot

Figure 1 An example of Ukiyo-e work in ARC Ukiyo-eCollection (Object Number arcUP2451 ) titled Kizukansukeby painter Hirosada The painting on the left is accompaniedby metadata for this work on the right For example meta-data further indicates this work is a middle-sized Nishiki-e(multi-colored woodblock printing) produced in 1849

objects depicted over time and the latter allows the identi-fication of artists (Suzuki Takagishi and Kitamoto 2018)The renewed interest by the machine learning communityin the culturally essential topics has led to works address-ing the traditional Japanese artworks from an interdisci-plinary perspective Along this line of research buildingopen collections of digitized images has been proposed forEhon (Suzuki Takagishi and Kitamoto 2018) and Ukiyo-e (Art Research Center Ritsumeikan University 2020Pinkney 2020) Further works use quantitative approachesinto the object for artworks such as studying the geometryfeatures of Buddha statues (Renoust et al 2019) and Ukiyo-e faces (Renoust et al 2019) Alternatively inspired by theart nature of painting machine learning-based creativity hasbeen leveraged for studying style such as painting processgeneration (Tian et al 2020) and image synthesis across art-work and photorealistic domains (Pinkney and Adler 2020)These works provide valuable connections between machinelearning and the humanities research of Japanese artwork

We however also notice that these works present sev-eral drawbacks For example collection on digitized im-ages may either comes with no semantic (Pinkney 2020) oris in a format not designed with machine learning-based ap-plications in mindFurthermore quantitative approaches areonly conducted on a small set of artworks (Murakami andUrabe 2007) or require extensive human labor to adapt for

arX

iv2

106

0226

7v1

[cs

CV

] 4

Jun

202

1

Ukiyo-e (Renoust et al 2019) and machine learning-basedcreativity works may deal more with cross-domain art ex-pression (Pinkney and Adler 2020) than the very domain ofartwork on which humanities research focuses Finally theart study into a particular genre requires insights into boththe object and style to acquire a comprehensive understand-ing Current works however only address one of the objector style falling short of the expectation

To overcome the aforementioned drawbacks and to pro-vide deeper insight into the artistic style of Ukiyo-e we pro-pose a new approach that is (1) holistic in both studying theobject and style through the joint use of images labels andannotations and (2) powered by large scale data and state-of-the-art machine learning model than the prior works Tosummarize our main contributions are as follow

bull We present a large-scale (11 000 paintings and 23 000faces) Ukiyo-e dataset with coherent semantic labels andgeometric annotations through augmenting and organiz-ing existing datasets with automatical detection

bull We are the first to conduct a large-scale quantitative studyof Ukiyo-e paintings (on more than 11 000 paintings)providing understanding into object in artworks by jointlyquantifying semantic labels and geometric annotations

bull We show that machine learning-based models could pro-vide insights into style by decomposing finished Ukiyo-e images into color-split woodblocks that reflect howUkiyo-e images were possibly produced

bull We study and show machine learning-based creativitymodel could engage problems that arise jointly studyingobject and style by separating geometry shapes and artis-tic styles in an orthogonal and re-assemblable way

DatasetArt research in traditional paintings often asks questions re-garding the work like the author and production year Onefocus in such research is on faces since they could help an-swer these questions through quantitative analysis In thisdirection Collection of Facial Expressions (Suzuki Takag-ishi and Kitamoto 2018 Tian et al 2020) provides a large-scale (8848 images) set of coarse-grained cropped facesAnother study (Murakami and Urabe 2007) deals with faciallandmarks which are more fine-grained than cropped facesto support quantitative analysis However its manual label-ing process only allows analysis on a small set (around 50images) of Ukiyo-e paintings

To combine both worksrsquo advantage we extend existingdatasets through augmentation and automated annotationresulting in a large-scale Ukiyo-e dataset with a more fine-grained facial feature The rest of this section details theprocess and analysis of our new proposed dataset

Fundamental DatasetsWe build our work based on two foundation datasets Oneof them is ARC Ukiyo-e Collection (Art Research CenterRitsumeikan University 2020) a publicly available servicethat provides access to digitized Ukiyo-e paintings primar-ily in the Edo period plus metadata compiled by domain

Facial Region Landmarks

Left Eye Center Left Right Up DownRight Eye Center Left Right Up DownLeft Eyebrow Left Right UpRight Eyebrow Left Right UpLeft Pupil CenterRight Pupil CenterMouth Left Right Up DownNose Center Left RightJawline Upper Left amp Right Mid Left amp Right Chin Bottom

Figure 2 An exmaple of detected landmarks and the ex-tracted face in Figure 1rsquos Ukiyo-e painting On the left thered dots show detected facial landmarks and the rectangleshows the bounding box inferred from these landmarks Theright image shows the extracted face from the bounding boxThe table lists a summary of all landmark locations

Figure 3 Faces with their landmarks In each row we showsix examples of extracted faces annoated with their corre-sponding landmarks in the same format as Figure 2

experts It has 11 103 entries of painting and the associatedmetadata one example of which is shown in Figure 1 Thisservice allows researchers to dive into curated metadata forcomparative study for art research

Another dataset is Ukiyo-e Faces Dataset (Pinkney 2020)a public available dataset of Ukiyo-e faces extracted fromUkiyo-e images available online With 5 000 high-qualityfaces this dataset plays an essential role in controllable im-age generation across Ukiyo-e faces and photo-realistic hu-man faces (Pinkney and Adler 2020) However as thisdataset focuses on image synthesis it does not include meta-data for Ukiyo-e paintings from which faces are extracted

Geometric Annotation with Facial LandmarkDetectionInspired by Pinkney (2020) we use an face recognitionAPI Amazon Rekognition (link) to detect facial landmarksin in Ukiyo-e Faces Dataset paintings Despite targetingphoto-realistic human face images this API demonstratescompelling accuracy on Ukiyo-e paintings Since the de-tected faces may not be well-aligned we infer the possiblyrotated bounding box of faces for cropping faces from thepainting inspired by the preprocessing in FFHQ (KarrasLaine and Aila 2019) In Figure 2 we show an example of

(a) Distribution of years with respect to authors (b) Distribution of years of all works in the dataset

Painter Examples

Hirosada(広貞)

Kogyo(耕漁)

Kunichika(国周)

Kunisada (1st gen)(国貞初代)

Kunisada (2nd gen)(国貞二代目)

Kunisada (3rd gen)(国貞三代目)

Kuniyoshi(国芳)

Toyokuni (1st gen)(豊国初代)

Toyokuni (3rd gen)(豊国三代目)

Yoshitaki(芳滝)

(c) Example of paintings represented by the extracted faces by authors

Figure 4 Metadata and their associated paintings We jointly show two important metadata year and author We also showshow ten authors with the most painting in the dataset (a) illustrates the year distribution with respect to authors and (b) showsthe overall year distribution Exemplary paintings belonging to these authors are shown in (c)

detected landmarks and the face extraction process

A total of 18 921 faces and their corresponding fa-cial landmarks have been detected from paintings in ARCUkiyo-e Collection Furthermore since Ukiyo-e FacesDataset (Pinkney 2020) also follows the same preprocess-ing as FFHQ its 5 000 faces are comparable to the facesextracted from ARC Ukiyo-e Collection Although faces inUkiyo-e Faces Dataset lack metadata we can still incorpo-rate them for geometry statistics by going through the above-mentioned landmark detecting process In doing so we havea total of around 23 000 Ukiyo-e faces In Figure 3 weshow examples of such faces and their landmarks

Semantic Labels IncorporationAs our dataset is derived from ARC Ukiyo-e Collection wecan also relate faces and the corresponding landmarks withthe original metadata such as the year of creation and the au-thor of the painting In Figure 4 we show these two metadatajointly as well as exemplary paintings belonging to severalauthors For example we can observe Shumei (襲名 namesuccession) system common in traditional Japanese art com-munity where an artist takes hisher teacherrsquos name as thecase of the lineage of Kunisada 1st gen (国貞 初代) Ku-nisada 2nd gen (国貞 二代目) and Kunisada 3rd gen (国貞三代目) Furthermore we can also notice three peaks ofproduction of Ukiyo-e painting occupying the early midand late 19th century The last peak is dominated by Kogyo(耕漁) who painted well into the 20th century and whose

(a) Landmarksdetected auto-matically

(b) Landmakrs ma-nually annotated bydomain experts

(c) Detailed Studyin Jawline land-marks

Facial Region Landmarks (Mean Error in Pixel Distance)

Left Eye Center (102)Right Eye Center (185)Mouth Left (97) Right (134)Nose Center (221) Left (186) Right (165)Left EyeBrow Left (342) Right (430) Up (231)Right EyeBrow Left (174) Right (418) Up (531)Jawline Upper Left (694) Upper Right (574) Mid Left (250)

Mid Right (562) Chin Bottom (578)

(d) Landmarks with the mean error in pixel distance between de-tected and expert labeled postion Landmarks in green are con-

sidered of high-quality and those in red of low-quality

Figure 5 Study of landmark quality by comparing automat-ically detected positions (a) with expert labeled positions(b) As Ukiyo-e faces are mostly towards either left or rightwe normalize all paintings to face left for the geometry pur-pose The study has been conducted for 69 Ukiyo-e paint-ings and the mean error of pixel distances are aggregated in(d) which we use to decide which landmarks are consideredhigh-quality The decision is based on picking landmarkswith a low error of pixel distance (heuristically those lt 20)except for Jawline () that needs special consideration as(c) shows landmarks on the the direction of facing (UpperLeft Mid Left) are useless since they are invisible in mostUkiyo-e The others (Upper Right Mid Right Chin Bot-tom) are valuable since they still lie on the jawline and theyare far from other landmarks allowing larger error margingwhen used for calculating angular features

uniqueness is further shown in his exemplary paintings un-der the influence of modern painting

ExperimentStudy Ukiyo-e Object using Geometry FeaturesRegarding the content art researches may be divided intotwo categories the shape that deals with geometry featuresand the texture that deals with brushes and color featuresTo quantitatively provide insights on attributes such as theauthor and the painting year either category can be used forunsupervised learning like clustering or supervised learn-ing like predicting metadata While the texture featurescould help analyze attributes for a single work (Tian et al2020) the geometry features could also be considered sincethe texture may vary due to Ukiyo-ersquos frequent reprint (Mu-rakami and Urabe 2007) or sculpturersquos preservation condi-tion (Renoust et al 2019) Both works propose to leverage

(a) High quality landmarkscircled in green

(b) Examples of angles formedby high quality landmarks

Figure 6 Extracted geometry features from high qualitylandmarks In (a) we highlight landmarks of high qualityand in (b) we show some a subset of of angles formed byhigh quality landmarks for brevity

facial landmarks to infer geometry features such as anglesand iconometric proportions to quantify artwork

However since both works rely on manually labeled land-marks they either suffer from being too small (only around50 Ukiyo-e paintings are annotated with landmarks) or re-quire extensive human effort if we ever want to apply thetechnique used on sculpture to Ukiyo-e To bridge this gapwe propose to use automatically detected landmarks as ge-ometry features To our best knowledge we are the first toconduct large-scale (more than 10k paintings) quantitativeanalysis of Ukiyo-e painting We hope it could serve as aninvitation for further quantitative study in artworks

Geometry Features from Landmarks Inspired by Mu-rakami and Urabe (2007) we consider the angles formed bylandmarks as they are geometry-invariant under rotation Toattain a clear understanding of the quality of landmarks weconduct a study on 69 Ukiyo-e paintings comparing land-marks that are automatically detected with positions manu-ally annotated by domain experts as detailed in Figure 5We observe that despite the general high-quality of the pre-dicted landmarks on Ukiyo-e painting some landmarks havesystematically worse quality than others we decided not toconsider In the end we calculate 252 angles formed by allpossible triplets of high-quality landmarks as geometry fea-tures for each face as illustrated in Figure 6

Analysis on Authorship To illustrate the information ofgeometry features we conduct unsupervised (PCA T-SNEUMAP) and supervised (LDA) clustering of faces using ge-ometry features in Figure 7 All clusterings show two dis-tinctive authors Kogyo and Hirosada (広貞) are separatedfrom other authors Such separation could be supportedthrough visual inspection into original paintings For ex-ample Figure 4 (c) shows Kogyo and Hirosada has visuallydistinctive styles compared to other painters Furthermoresuch separation could also be cross-verified with analysisleveraging other information sources Figure 4 (a) showsthat Kogyo was active well into the 20th century where incontrast Ukiyo-e paintings are mainly around the middle19th century Furthermore his uniqueness of style is visu-ally illustrated in exemplary paintings We can also observeHirosada forms a unique style related to the geographicalfactors While most of the painters analyzed at that timeworked in Edo (modern Tokyo) Hirosada was active in Os-

PCA T-SNE UMAP LDA (Legend)

Figure 7 Unsupervised (PCA T-SNE UMAP) and supervised (LDA) clustering of facesrsquo geometry features in a two-dimensional plane We show works by six most-frequently appearing authors in the clustering Labels are used for coloringthe authors for illustrative purposes only and are not used in the clustering except for LDA Visually Hirosada and Kogyo areshown with clear separation from other authors which could be cross-verified with explanation using other information

PCA T-SNE UMAP

Figure 8 Unsupervised (PCA T-SNE UMAP) cluster-ing of faces using geometry features for Ukiyo-e paintingsKaokore paintings and photo-realistic human faces Visu-ally we see a separation between faces of different sources

aka Comparison of culture at that time could be made be-tween the around EdoTokyo region the de facto capital ofJapan under Tokugawa shogunate and Kamigata (上方) re-gion encompassing Kyoto and Osaka the de jure capital ofJapan and the cultural center of western Japan The styleof Hirosada and Yoshitaki (芳滝) who were active in west-ern Japan is therefore called Kamigatae (上方絵 Kamigatapainting) and is a subject for comparative study

Comparing Ukiyo-e Ehon and Human Faces As ex-emplary paintings in Figure 4 (c) show Ukiyo-e paintingsare characterized by their particular facial geometry whichcould potentially be different from other art genres or photo-realistic human faces To quantify such observation weconduct unsupervised (PCA T-SNE UMAP) clustering ofUkiyo-e (popular in the 19th century) faces Ehon Emaki-mono (another Japanese artworks genre popular in the 16th

to 17th century) faces and realistic human faces

Concretely we use Kaokore (Tian et al 2020) for Ehon Emakimono faces as well as human face photos collected inFFHQ (Karras Laine and Aila 2019) dataset that are pub-lished under CC BY 20 license In Figure 8 we can ob-serve that the geometry of Ukiyo-e faces is different fromKaokore and only share similarities to a small section of re-alistic human faces This observation confirms the unique-ness of Japanese artworksrsquo way of portraying humans com-pared to the real-world image and shows that the develop-ment of Japanese artworks over time is a drastic one

Study Ukiyo-e Style through Color Separation

Ukiyo-e printings distinguish themselves from other tradi-tional Japanese artworks by the very manner of producingUnlike Ehon which is targeted at a small audience and thuspainted by hand Ukiyo-e is mass-produced using wood-block printing after the painter finishes the master versionAs shown in a modern reproducing process (link) multi-ple woodblocks are carved each for a portion in the imagewith a single color and are printed sequentially with cor-responding inks onto the final canvas Unfortunately sucha process for a given Ukiyo-e painting is not precisely re-producible since the underlying woodblocks are vulnerableeasily worn-out and often discarded after a certain numberof prints Thus from an art research point of view it wouldbe interesting to recover the above-mentioned separated por-tions for a given Ukiyo-e painting with only access to theimage itself

We address this challenge by framing it as a soft colorsegmentation (Aksoy et al 2017) task which decomposesan input image into several RGBA layers of homogeneouscolors The alpha channel (ldquoArdquo in ldquoRGBArdquo) in each layerallows pixels to potentially belong to multiple layers whichcaptures ambiguity unavoidable due to imperfect woodblockcarving and alignment in multi-pass printing In detail weuse state-of-the-art Fast Soft Color Separation (FSCS) (Aki-moto et al 2020) for efficient processing As shown in Fig-ure 9 FSCS decomposes Ukiyo-e paintings into layers ofhomogeneous colors using color palette The inferred lay-ers could be interpreted as woodblocks with correspondingcolors that could be used for making a particular artwork

The decomposition of a painting into multiple layers ofhomogeneous colors allows us to explore further creativ-ity One example in this direction is recoloring where wepick a new color for each of the individual layers and com-pose them into a recolored painting As shown in Figure 10the recoloring could be done either automatically using theinferred color palette from other artworks or manually inAdobe After Effects with alpha add mode for blending Therecoloring here serves as an example to study artworks andopens the door to reinterpret them in a new way

(a) Landmarksdetected auto-matically

(b) Landmakrs ma-nually annotated bydomain experts

(c) Detailed Studyin Jawline land-marks

Facial Region Landmarks (Mean Error in Pixel Distance)

Left Eye Center (102)Right Eye Center (185)Mouth Left (97) Right (134)Nose Center (221) Left (186) Right (165)Left EyeBrow Left (342) Right (430) Up (231)Right EyeBrow Left (174) Right (418) Up (531)Jawline Upper Left (694) Upper Right (574) Mid Left (250)

Mid Right (562) Chin Bottom (578)

(d) Landmarks with the mean error in pixel distance between de-tected and expert labeled postion Landmarks in green are con-

sidered of high-quality and those in red of low-quality

Figure 5 Study of landmark quality by comparing automat-ically detected positions (a) with expert labeled positions(b) As Ukiyo-e faces are mostly towards either left or rightwe normalize all paintings to face left for the geometry pur-pose The study has been conducted for 69 Ukiyo-e paint-ings and the mean error of pixel distances are aggregated in(d) which we use to decide which landmarks are consideredhigh-quality The decision is based on picking landmarkswith a low error of pixel distance (heuristically those lt 20)except for Jawline () that needs special consideration as(c) shows landmarks on the the direction of facing (UpperLeft Mid Left) are useless since they are invisible in mostUkiyo-e The others (Upper Right Mid Right Chin Bot-tom) are valuable since they still lie on the jawline and theyare far from other landmarks allowing larger error margingwhen used for calculating angular features

uniqueness is further shown in his exemplary paintings un-der the influence of modern painting

ExperimentStudy Ukiyo-e Object using Geometry FeaturesRegarding the content art researches may be divided intotwo categories the shape that deals with geometry featuresand the texture that deals with brushes and color featuresTo quantitatively provide insights on attributes such as theauthor and the painting year either category can be used forunsupervised learning like clustering or supervised learn-ing like predicting metadata While the texture featurescould help analyze attributes for a single work (Tian et al2020) the geometry features could also be considered sincethe texture may vary due to Ukiyo-ersquos frequent reprint (Mu-rakami and Urabe 2007) or sculpturersquos preservation condi-tion (Renoust et al 2019) Both works propose to leverage

(a) High quality landmarkscircled in green

(b) Examples of angles formedby high quality landmarks

Figure 6 Extracted geometry features from high qualitylandmarks In (a) we highlight landmarks of high qualityand in (b) we show some a subset of of angles formed byhigh quality landmarks for brevity

facial landmarks to infer geometry features such as anglesand iconometric proportions to quantify artwork

However since both works rely on manually labeled land-marks they either suffer from being too small (only around50 Ukiyo-e paintings are annotated with landmarks) or re-quire extensive human effort if we ever want to apply thetechnique used on sculpture to Ukiyo-e To bridge this gapwe propose to use automatically detected landmarks as ge-ometry features To our best knowledge we are the first toconduct large-scale (more than 10k paintings) quantitativeanalysis of Ukiyo-e painting We hope it could serve as aninvitation for further quantitative study in artworks

Geometry Features from Landmarks Inspired by Mu-rakami and Urabe (2007) we consider the angles formed bylandmarks as they are geometry-invariant under rotation Toattain a clear understanding of the quality of landmarks weconduct a study on 69 Ukiyo-e paintings comparing land-marks that are automatically detected with positions manu-ally annotated by domain experts as detailed in Figure 5We observe that despite the general high-quality of the pre-dicted landmarks on Ukiyo-e painting some landmarks havesystematically worse quality than others we decided not toconsider In the end we calculate 252 angles formed by allpossible triplets of high-quality landmarks as geometry fea-tures for each face as illustrated in Figure 6

Analysis on Authorship To illustrate the information ofgeometry features we conduct unsupervised (PCA T-SNEUMAP) and supervised (LDA) clustering of faces using ge-ometry features in Figure 7 All clusterings show two dis-tinctive authors Kogyo and Hirosada (広貞) are separatedfrom other authors Such separation could be supportedthrough visual inspection into original paintings For ex-ample Figure 4 (c) shows Kogyo and Hirosada has visuallydistinctive styles compared to other painters Furthermoresuch separation could also be cross-verified with analysisleveraging other information sources Figure 4 (a) showsthat Kogyo was active well into the 20th century where incontrast Ukiyo-e paintings are mainly around the middle19th century Furthermore his uniqueness of style is visu-ally illustrated in exemplary paintings We can also observeHirosada forms a unique style related to the geographicalfactors While most of the painters analyzed at that timeworked in Edo (modern Tokyo) Hirosada was active in Os-

PCA T-SNE UMAP LDA (Legend)

Figure 7 Unsupervised (PCA T-SNE UMAP) and supervised (LDA) clustering of facesrsquo geometry features in a two-dimensional plane We show works by six most-frequently appearing authors in the clustering Labels are used for coloringthe authors for illustrative purposes only and are not used in the clustering except for LDA Visually Hirosada and Kogyo areshown with clear separation from other authors which could be cross-verified with explanation using other information

PCA T-SNE UMAP

Figure 8 Unsupervised (PCA T-SNE UMAP) cluster-ing of faces using geometry features for Ukiyo-e paintingsKaokore paintings and photo-realistic human faces Visu-ally we see a separation between faces of different sources

aka Comparison of culture at that time could be made be-tween the around EdoTokyo region the de facto capital ofJapan under Tokugawa shogunate and Kamigata (上方) re-gion encompassing Kyoto and Osaka the de jure capital ofJapan and the cultural center of western Japan The styleof Hirosada and Yoshitaki (芳滝) who were active in west-ern Japan is therefore called Kamigatae (上方絵 Kamigatapainting) and is a subject for comparative study

Comparing Ukiyo-e Ehon and Human Faces As ex-emplary paintings in Figure 4 (c) show Ukiyo-e paintingsare characterized by their particular facial geometry whichcould potentially be different from other art genres or photo-realistic human faces To quantify such observation weconduct unsupervised (PCA T-SNE UMAP) clustering ofUkiyo-e (popular in the 19th century) faces Ehon Emaki-mono (another Japanese artworks genre popular in the 16th

to 17th century) faces and realistic human faces

Concretely we use Kaokore (Tian et al 2020) for Ehon Emakimono faces as well as human face photos collected inFFHQ (Karras Laine and Aila 2019) dataset that are pub-lished under CC BY 20 license In Figure 8 we can ob-serve that the geometry of Ukiyo-e faces is different fromKaokore and only share similarities to a small section of re-alistic human faces This observation confirms the unique-ness of Japanese artworksrsquo way of portraying humans com-pared to the real-world image and shows that the develop-ment of Japanese artworks over time is a drastic one

Study Ukiyo-e Style through Color Separation

Ukiyo-e printings distinguish themselves from other tradi-tional Japanese artworks by the very manner of producingUnlike Ehon which is targeted at a small audience and thuspainted by hand Ukiyo-e is mass-produced using wood-block printing after the painter finishes the master versionAs shown in a modern reproducing process (link) multi-ple woodblocks are carved each for a portion in the imagewith a single color and are printed sequentially with cor-responding inks onto the final canvas Unfortunately sucha process for a given Ukiyo-e painting is not precisely re-producible since the underlying woodblocks are vulnerableeasily worn-out and often discarded after a certain numberof prints Thus from an art research point of view it wouldbe interesting to recover the above-mentioned separated por-tions for a given Ukiyo-e painting with only access to theimage itself

We address this challenge by framing it as a soft colorsegmentation (Aksoy et al 2017) task which decomposesan input image into several RGBA layers of homogeneouscolors The alpha channel (ldquoArdquo in ldquoRGBArdquo) in each layerallows pixels to potentially belong to multiple layers whichcaptures ambiguity unavoidable due to imperfect woodblockcarving and alignment in multi-pass printing In detail weuse state-of-the-art Fast Soft Color Separation (FSCS) (Aki-moto et al 2020) for efficient processing As shown in Fig-ure 9 FSCS decomposes Ukiyo-e paintings into layers ofhomogeneous colors using color palette The inferred lay-ers could be interpreted as woodblocks with correspondingcolors that could be used for making a particular artwork

The decomposition of a painting into multiple layers ofhomogeneous colors allows us to explore further creativ-ity One example in this direction is recoloring where wepick a new color for each of the individual layers and com-pose them into a recolored painting As shown in Figure 10the recoloring could be done either automatically using theinferred color palette from other artworks or manually inAdobe After Effects with alpha add mode for blending Therecoloring here serves as an example to study artworks andopens the door to reinterpret them in a new way

PCA T-SNE UMAP LDA (Legend)

Figure 7 Unsupervised (PCA T-SNE UMAP) and supervised (LDA) clustering of facesrsquo geometry features in a two-dimensional plane We show works by six most-frequently appearing authors in the clustering Labels are used for coloringthe authors for illustrative purposes only and are not used in the clustering except for LDA Visually Hirosada and Kogyo areshown with clear separation from other authors which could be cross-verified with explanation using other information

PCA T-SNE UMAP

Figure 8 Unsupervised (PCA T-SNE UMAP) cluster-ing of faces using geometry features for Ukiyo-e paintingsKaokore paintings and photo-realistic human faces Visu-ally we see a separation between faces of different sources

aka Comparison of culture at that time could be made be-tween the around EdoTokyo region the de facto capital ofJapan under Tokugawa shogunate and Kamigata (上方) re-gion encompassing Kyoto and Osaka the de jure capital ofJapan and the cultural center of western Japan The styleof Hirosada and Yoshitaki (芳滝) who were active in west-ern Japan is therefore called Kamigatae (上方絵 Kamigatapainting) and is a subject for comparative study

Comparing Ukiyo-e Ehon and Human Faces As ex-emplary paintings in Figure 4 (c) show Ukiyo-e paintingsare characterized by their particular facial geometry whichcould potentially be different from other art genres or photo-realistic human faces To quantify such observation weconduct unsupervised (PCA T-SNE UMAP) clustering ofUkiyo-e (popular in the 19th century) faces Ehon Emaki-mono (another Japanese artworks genre popular in the 16th

to 17th century) faces and realistic human faces

Concretely we use Kaokore (Tian et al 2020) for Ehon Emakimono faces as well as human face photos collected inFFHQ (Karras Laine and Aila 2019) dataset that are pub-lished under CC BY 20 license In Figure 8 we can ob-serve that the geometry of Ukiyo-e faces is different fromKaokore and only share similarities to a small section of re-alistic human faces This observation confirms the unique-ness of Japanese artworksrsquo way of portraying humans com-pared to the real-world image and shows that the develop-ment of Japanese artworks over time is a drastic one

Study Ukiyo-e Style through Color Separation

Ukiyo-e printings distinguish themselves from other tradi-tional Japanese artworks by the very manner of producingUnlike Ehon which is targeted at a small audience and thuspainted by hand Ukiyo-e is mass-produced using wood-block printing after the painter finishes the master versionAs shown in a modern reproducing process (link) multi-ple woodblocks are carved each for a portion in the imagewith a single color and are printed sequentially with cor-responding inks onto the final canvas Unfortunately sucha process for a given Ukiyo-e painting is not precisely re-producible since the underlying woodblocks are vulnerableeasily worn-out and often discarded after a certain numberof prints Thus from an art research point of view it wouldbe interesting to recover the above-mentioned separated por-tions for a given Ukiyo-e painting with only access to theimage itself

We address this challenge by framing it as a soft colorsegmentation (Aksoy et al 2017) task which decomposesan input image into several RGBA layers of homogeneouscolors The alpha channel (ldquoArdquo in ldquoRGBArdquo) in each layerallows pixels to potentially belong to multiple layers whichcaptures ambiguity unavoidable due to imperfect woodblockcarving and alignment in multi-pass printing In detail weuse state-of-the-art Fast Soft Color Separation (FSCS) (Aki-moto et al 2020) for efficient processing As shown in Fig-ure 9 FSCS decomposes Ukiyo-e paintings into layers ofhomogeneous colors using color palette The inferred lay-ers could be interpreted as woodblocks with correspondingcolors that could be used for making a particular artwork

The decomposition of a painting into multiple layers ofhomogeneous colors allows us to explore further creativ-ity One example in this direction is recoloring where wepick a new color for each of the individual layers and com-pose them into a recolored painting As shown in Figure 10the recoloring could be done either automatically using theinferred color palette from other artworks or manually inAdobe After Effects with alpha add mode for blending Therecoloring here serves as an example to study artworks andopens the door to reinterpret them in a new way

Input Decomposed Layers and Color Palette Reassembled

Figure 9 Soft color separation takes as input Ukiyo-e paintings (left) and a color palette (middle) and produces decomposedlayers of homogeneous colors (middle) These layers can be used as the inferred woodblocks for corresponding colors andcomposed back to a reassembled painting (right) resembling the original one We infer the color palette by applying K-meansclusttering (Lloyd 1982) on the input paintingrsquos pixels

(a) (b) (c) (d) (e)

Figure 10 Decomposing an Ukiyo-e painting (a) with colorpalette (b) and recoloring which could be done automati-cally (c d) using color palettes inferred from the referenceimages or manuallly with Adobe After Effects (e)

Study Jointly Ukiyo-e Object and Style byComposing Sketch and ColorAs we deals with a dataset focusing on artworks it be-comes natural to ask whether we could engage them withapproaches invoking creativity and artistic expression One

Figure 11 Pairs of original Ukiyo-e faces on the upper rowand corresponding line art sketches on the lower row

direction is to examine whether the recent advances of ma-

(a) Origin (b) Sketch (c) Reference (d) Colorized

Figure 12 Colorization on Ukiyo-e faces For a face paint-ing (a) we extract its line art sketch (b) A colorizationmodel takes both the sketch and a reference painting (c) andproduces a colorized painting (d) reflecting the sketchrsquos ge-ometry and the referencersquos style in colors and textures

chine learning models could create structurally sound oreven artistically impressive results In this direction gener-ative models has been proposed to generate faces in Japanesepainting style (Tian et al 2020) and blend generative modelstrained on data of different domains by swapping layers oftwo image generation neural networks (Pinkney and Adler2020) However the former lacks controllability in the gen-eration as it can only produce images as a whole and thelatter focuses on transferring across separated different do-mains by the nature of its design

Thus we identify an unbridged gap in the in-domain sep-aration of artistically essential aspects In detail we askthe following question what is the (dis)entanglement be-tween the object and style within the Ukiyo-e Answeringthis question reveals the relation between Ukiyo-ersquos objectand style Furthermore it also allows editing one of themwhile keeping another intact for creative expression Oneway to separate the object and style is to represent the for-mer with line art sketches for what personscene is depictedand the latter with color and texture information showing thepainting style They could be composed with a colorizationprocess which blends a sketch as an object reference and animage as a reference for instance-level painting style

Face Images We extract line art sketches from Ukiyo-e

(a) Ukiyo-e faces (b) Whole Ukiyo-e paintings

Figure 13 Matrices of blending line art sketches and painting style for Ukiyo-e faces (a) and whole Ukiyo-e paintings (b)Within a single matrix each row represents an art line sketch each column represents the reference image for style and imagesat an intersection are the blending results of the corresponding row and column

(a) Origin (b) Sketch (c) Reference (d) Colorized

Figure 14 Top Pairs of whole Ukiyo-e painting and line artsketches in the same format as Figure 11 Bottom Coloriza-tion on Ukiyo-e faces in the same format as Figure 12

images using SketchKeras (Lvming Zhang 2017) as illus-trated in Figure 11 We further train image colorization (Leeet al 2020) using a public-available implementation (SouHasegawa 2020) The whole pipeline is illustrated in Fig-ure 12 As shown in Figure 13 Since the model learnsto separate the object indicated in the sketch image and

the style indicated by reference image as two orthogonaland composable semantics it could blend arbitrary combi-nation of sketch and reference style images Such separationcould enable future works to help with humanities researchon combinations of Ukiyo-e color and subject For examplein Ukiyo-e depicting Kabuki the attributes and colors of thecharacters are somewhat correlated semantically Therefore

Figure 15 Colorization in-the-wild woodblock printing us-ing the model trained on the whole Ukiyo-e paintings Eachrow represents a woodblock printing work and each columnrepresents the reference image for style

swapping colors can change the meaning of scenes and peo-ple in the painting We envision that discoveries could bemade by studying how the impression of Ukiyo-e paintingschanges through the process of swapping colors

Whole Painting We go beyond faces and work on wholeUkiyo-e painting images By employing the same pipeline

Figure 16 Comparation of conditional and the uncondi-tional colorization method The former uses style referenceimages while the latter does not Four rows are the groundtruth color image conditional colorization unconditionalcolorization and line art sketches respectively

to the whole painting images as shown in Figure 14 themodel can be further leveraged to colorize in-the-wild wood-block printing images as Figure 15 shows However whilethe resulting colorized images are reasonable they are oflower quality than those of faces Such observation is an-ticipated since the whole Ukiyo-e painting is more complexthan face in many ways like topics and topological config-uration of objects which presents a much more challengingtask for colorization This issue could be further exaggeratedby the discrepancy between the Ukiyo-e domain where themodel is trained and the woodblock painting domain wherethe model is applied We would leave higher quality wholeUkiyo-e painting colorization for future study

Conditional vs Unconditional Colorization While wechoose to use a conditional colorization method which pro-duces results from a sketch and a reference image for colorand styles it is also worth considering a simper uncondi-tional colorization method that directly generates the resultsfrom a sketch such as Pix2PixHD (Wang et al 2018) Thisalternation however suffers from the inability to control thecolor and style of the generated image Moreover as weshow in Figure 16 the unconditional colorization methodproduces worse colorization results than the conditional col-orization method (Lee et al 2020) We argue that this isexpected since the former method has to fall back to safecolors that valid for any Ukiyo-e images while the lattercould make a wiser choice based on the reference images

Discussion We show that Ukiyo-e paintings can be studiedby (1) representing object with line art sketches (2) repre-senting style as a color reference image and (3) composingthem using colorization This pipeline provides a clear sep-aration of two semantics important in the art research andallows further creativity through compositions of both in un-seen ways As it is just one possible way of studying the in-teraction between the object and the style we expect furtherworks could explore different forms of creative expression

ConclusionIn this work we propose to bridge the machine learning andhumanities research on the subject of Ukiyo-e paintings Be-sides the presented dataset with coherent labels and annota-

tions we also show their value in the quantification approachto humanities research Furthermore we demonstrate thatmachine learning models in a creative setting could addressart-style research problems

AcknowledgementWe thank Hanjun Dai David Ha Yujing Tang NeilHoulsby and Huachun Zhu for their comments and helpfulsuggestions

References[Akimoto et al 2020] Akimoto N Zhu H Jin Y andAoki Y 2020 Fast Soft Color Segmentation In IEEECVFConference on Computer Vision and Pattern Recognition

[Aksoy et al 2017] Aksoy Y Aydin T O Smolic A andPollefeys M 2017 Unmixing-based Soft Color sSegmen-tation for Image Manipulation ACM Trans Graph

[Art Research Center Ritsumeikan University 2020] ArtResearch Center Ritsumeikan University 2020 ARCUkiyo-e database (ARC所蔵浮世絵データベース) Infor-matics Research Data Repository National Institute of in-formatics httpsdoiorg1032130rdata21

[IUS 2008] IUS 2008 Encyclopedia of Ukiyo-e (浮世絵大事典) Tokyo Japan Tokyodo Shuppan (東京堂出版)

[Karras Laine and Aila 2019] Karras T Laine S andAila T 2019 A Style-based Generator Architecture forGenerative Adversarial Networks In IEEECVF Conferenceon Computer Vision and Pattern Recognition

[Kobayashi 1994] Kobayashi T 1994 Encyclopedia Nip-ponica sv Ukiyo-e Tokyo Japan Shogakukan

[Lee et al 2020] Lee J Kim E Lee Y Kim D ChangJ and Choo J 2020 Reference-Based Sketch Image Col-orization Using Augmented-Self Reference and Dense Se-mantic Correspondence In IEEECVF Conference on Com-puter Vision and Pattern Recognition

[Lloyd 1982] Lloyd S 1982 Least squares quantization inPCM IEEE transactions on information theory

[Lvming Zhang 2017] Lvming Zhang 2017 SketchKeras[Murakami and Urabe 2007] Murakami M and Urabe J2007 A Quantitative Analysis of Portraits of Kabuki ActorsIn Proceedings of the Institute of Statistical Mathematics

[Pinkney and Adler 2020] Pinkney J N and Adler D2020 Resolution Dependant GAN Interpolation for Con-trollable Image Synthesis Between Domains arXiv preprintarXiv201005334

[Pinkney 2020] Pinkney J N M 2020 Aligned Ukiyo-eFaces Dataset Link

[Renoust et al 2019] Renoust B Franca M Chan J Gar-cia N Le V Uesaka A Nakashima Y Nagahara HWang J and Fujioka Y 2019 Historical and Modern Fea-tures for Buddha Statue Classification In SUMAC 2019

[Sou Hasegawa 2020] Sou Hasegawa 2020 Automatic LineArt Colorization Link

(a) Ukiyo-e faces (b) Whole Ukiyo-e paintings

Figure 13 Matrices of blending line art sketches and painting style for Ukiyo-e faces (a) and whole Ukiyo-e paintings (b)Within a single matrix each row represents an art line sketch each column represents the reference image for style and imagesat an intersection are the blending results of the corresponding row and column

(a) Origin (b) Sketch (c) Reference (d) Colorized

Figure 14 Top Pairs of whole Ukiyo-e painting and line artsketches in the same format as Figure 11 Bottom Coloriza-tion on Ukiyo-e faces in the same format as Figure 12

images using SketchKeras (Lvming Zhang 2017) as illus-trated in Figure 11 We further train image colorization (Leeet al 2020) using a public-available implementation (SouHasegawa 2020) The whole pipeline is illustrated in Fig-ure 12 As shown in Figure 13 Since the model learnsto separate the object indicated in the sketch image and

the style indicated by reference image as two orthogonaland composable semantics it could blend arbitrary combi-nation of sketch and reference style images Such separationcould enable future works to help with humanities researchon combinations of Ukiyo-e color and subject For examplein Ukiyo-e depicting Kabuki the attributes and colors of thecharacters are somewhat correlated semantically Therefore

Figure 15 Colorization in-the-wild woodblock printing us-ing the model trained on the whole Ukiyo-e paintings Eachrow represents a woodblock printing work and each columnrepresents the reference image for style

swapping colors can change the meaning of scenes and peo-ple in the painting We envision that discoveries could bemade by studying how the impression of Ukiyo-e paintingschanges through the process of swapping colors

Whole Painting We go beyond faces and work on wholeUkiyo-e painting images By employing the same pipeline

Figure 16 Comparation of conditional and the uncondi-tional colorization method The former uses style referenceimages while the latter does not Four rows are the groundtruth color image conditional colorization unconditionalcolorization and line art sketches respectively

to the whole painting images as shown in Figure 14 themodel can be further leveraged to colorize in-the-wild wood-block printing images as Figure 15 shows However whilethe resulting colorized images are reasonable they are oflower quality than those of faces Such observation is an-ticipated since the whole Ukiyo-e painting is more complexthan face in many ways like topics and topological config-uration of objects which presents a much more challengingtask for colorization This issue could be further exaggeratedby the discrepancy between the Ukiyo-e domain where themodel is trained and the woodblock painting domain wherethe model is applied We would leave higher quality wholeUkiyo-e painting colorization for future study

Conditional vs Unconditional Colorization While wechoose to use a conditional colorization method which pro-duces results from a sketch and a reference image for colorand styles it is also worth considering a simper uncondi-tional colorization method that directly generates the resultsfrom a sketch such as Pix2PixHD (Wang et al 2018) Thisalternation however suffers from the inability to control thecolor and style of the generated image Moreover as weshow in Figure 16 the unconditional colorization methodproduces worse colorization results than the conditional col-orization method (Lee et al 2020) We argue that this isexpected since the former method has to fall back to safecolors that valid for any Ukiyo-e images while the lattercould make a wiser choice based on the reference images

Discussion We show that Ukiyo-e paintings can be studiedby (1) representing object with line art sketches (2) repre-senting style as a color reference image and (3) composingthem using colorization This pipeline provides a clear sep-aration of two semantics important in the art research andallows further creativity through compositions of both in un-seen ways As it is just one possible way of studying the in-teraction between the object and the style we expect furtherworks could explore different forms of creative expression

ConclusionIn this work we propose to bridge the machine learning andhumanities research on the subject of Ukiyo-e paintings Be-sides the presented dataset with coherent labels and annota-

tions we also show their value in the quantification approachto humanities research Furthermore we demonstrate thatmachine learning models in a creative setting could addressart-style research problems

AcknowledgementWe thank Hanjun Dai David Ha Yujing Tang NeilHoulsby and Huachun Zhu for their comments and helpfulsuggestions

References[Akimoto et al 2020] Akimoto N Zhu H Jin Y andAoki Y 2020 Fast Soft Color Segmentation In IEEECVFConference on Computer Vision and Pattern Recognition

[Aksoy et al 2017] Aksoy Y Aydin T O Smolic A andPollefeys M 2017 Unmixing-based Soft Color sSegmen-tation for Image Manipulation ACM Trans Graph

[Art Research Center Ritsumeikan University 2020] ArtResearch Center Ritsumeikan University 2020 ARCUkiyo-e database (ARC所蔵浮世絵データベース) Infor-matics Research Data Repository National Institute of in-formatics httpsdoiorg1032130rdata21

[IUS 2008] IUS 2008 Encyclopedia of Ukiyo-e (浮世絵大事典) Tokyo Japan Tokyodo Shuppan (東京堂出版)

[Karras Laine and Aila 2019] Karras T Laine S andAila T 2019 A Style-based Generator Architecture forGenerative Adversarial Networks In IEEECVF Conferenceon Computer Vision and Pattern Recognition

[Kobayashi 1994] Kobayashi T 1994 Encyclopedia Nip-ponica sv Ukiyo-e Tokyo Japan Shogakukan

[Lee et al 2020] Lee J Kim E Lee Y Kim D ChangJ and Choo J 2020 Reference-Based Sketch Image Col-orization Using Augmented-Self Reference and Dense Se-mantic Correspondence In IEEECVF Conference on Com-puter Vision and Pattern Recognition

[Lloyd 1982] Lloyd S 1982 Least squares quantization inPCM IEEE transactions on information theory

[Lvming Zhang 2017] Lvming Zhang 2017 SketchKeras[Murakami and Urabe 2007] Murakami M and Urabe J2007 A Quantitative Analysis of Portraits of Kabuki ActorsIn Proceedings of the Institute of Statistical Mathematics

[Pinkney and Adler 2020] Pinkney J N and Adler D2020 Resolution Dependant GAN Interpolation for Con-trollable Image Synthesis Between Domains arXiv preprintarXiv201005334

[Pinkney 2020] Pinkney J N M 2020 Aligned Ukiyo-eFaces Dataset Link

[Renoust et al 2019] Renoust B Franca M Chan J Gar-cia N Le V Uesaka A Nakashima Y Nagahara HWang J and Fujioka Y 2019 Historical and Modern Fea-tures for Buddha Statue Classification In SUMAC 2019

[Sou Hasegawa 2020] Sou Hasegawa 2020 Automatic LineArt Colorization Link

Figure 16 Comparation of conditional and the uncondi-tional colorization method The former uses style referenceimages while the latter does not Four rows are the groundtruth color image conditional colorization unconditionalcolorization and line art sketches respectively

to the whole painting images as shown in Figure 14 themodel can be further leveraged to colorize in-the-wild wood-block printing images as Figure 15 shows However whilethe resulting colorized images are reasonable they are oflower quality than those of faces Such observation is an-ticipated since the whole Ukiyo-e painting is more complexthan face in many ways like topics and topological config-uration of objects which presents a much more challengingtask for colorization This issue could be further exaggeratedby the discrepancy between the Ukiyo-e domain where themodel is trained and the woodblock painting domain wherethe model is applied We would leave higher quality wholeUkiyo-e painting colorization for future study

Conditional vs Unconditional Colorization While wechoose to use a conditional colorization method which pro-duces results from a sketch and a reference image for colorand styles it is also worth considering a simper uncondi-tional colorization method that directly generates the resultsfrom a sketch such as Pix2PixHD (Wang et al 2018) Thisalternation however suffers from the inability to control thecolor and style of the generated image Moreover as weshow in Figure 16 the unconditional colorization methodproduces worse colorization results than the conditional col-orization method (Lee et al 2020) We argue that this isexpected since the former method has to fall back to safecolors that valid for any Ukiyo-e images while the lattercould make a wiser choice based on the reference images

Discussion We show that Ukiyo-e paintings can be studiedby (1) representing object with line art sketches (2) repre-senting style as a color reference image and (3) composingthem using colorization This pipeline provides a clear sep-aration of two semantics important in the art research andallows further creativity through compositions of both in un-seen ways As it is just one possible way of studying the in-teraction between the object and the style we expect furtherworks could explore different forms of creative expression

ConclusionIn this work we propose to bridge the machine learning andhumanities research on the subject of Ukiyo-e paintings Be-sides the presented dataset with coherent labels and annota-

tions we also show their value in the quantification approachto humanities research Furthermore we demonstrate thatmachine learning models in a creative setting could addressart-style research problems

AcknowledgementWe thank Hanjun Dai David Ha Yujing Tang NeilHoulsby and Huachun Zhu for their comments and helpfulsuggestions

References[Akimoto et al 2020] Akimoto N Zhu H Jin Y andAoki Y 2020 Fast Soft Color Segmentation In IEEECVFConference on Computer Vision and Pattern Recognition

[Aksoy et al 2017] Aksoy Y Aydin T O Smolic A andPollefeys M 2017 Unmixing-based Soft Color sSegmen-tation for Image Manipulation ACM Trans Graph

[Art Research Center Ritsumeikan University 2020] ArtResearch Center Ritsumeikan University 2020 ARCUkiyo-e database (ARC所蔵浮世絵データベース) Infor-matics Research Data Repository National Institute of in-formatics httpsdoiorg1032130rdata21

[IUS 2008] IUS 2008 Encyclopedia of Ukiyo-e (浮世絵大事典) Tokyo Japan Tokyodo Shuppan (東京堂出版)

[Karras Laine and Aila 2019] Karras T Laine S andAila T 2019 A Style-based Generator Architecture forGenerative Adversarial Networks In IEEECVF Conferenceon Computer Vision and Pattern Recognition

[Kobayashi 1994] Kobayashi T 1994 Encyclopedia Nip-ponica sv Ukiyo-e Tokyo Japan Shogakukan

[Lee et al 2020] Lee J Kim E Lee Y Kim D ChangJ and Choo J 2020 Reference-Based Sketch Image Col-orization Using Augmented-Self Reference and Dense Se-mantic Correspondence In IEEECVF Conference on Com-puter Vision and Pattern Recognition

[Lloyd 1982] Lloyd S 1982 Least squares quantization inPCM IEEE transactions on information theory

[Lvming Zhang 2017] Lvming Zhang 2017 SketchKeras[Murakami and Urabe 2007] Murakami M and Urabe J2007 A Quantitative Analysis of Portraits of Kabuki ActorsIn Proceedings of the Institute of Statistical Mathematics

[Pinkney and Adler 2020] Pinkney J N and Adler D2020 Resolution Dependant GAN Interpolation for Con-trollable Image Synthesis Between Domains arXiv preprintarXiv201005334

[Pinkney 2020] Pinkney J N M 2020 Aligned Ukiyo-eFaces Dataset Link

[Renoust et al 2019] Renoust B Franca M Chan J Gar-cia N Le V Uesaka A Nakashima Y Nagahara HWang J and Fujioka Y 2019 Historical and Modern Fea-tures for Buddha Statue Classification In SUMAC 2019

[Sou Hasegawa 2020] Sou Hasegawa 2020 Automatic LineArt Colorization Link

[Suzuki Takagishi and Kitamoto 2018] Suzuki C Takag-ishi A and Kitamoto A 2018 rsquoCollection of facial ex-pressionsrsquo with IIIF Curation Platform - Close Reading andDistant Reading for Style Comparative Studies Proceedingsof IPSJ SIG Computers and the Humanities Symposium

[Tian et al 2020] Tian Y Suzuki C Clanuwat T Bober-Irizar M Lamb A and Kitamoto A 2020 KaoKoreA Pre-modern Japanese Art Facial Expression Dataset InInternational Conference on Computational Creativity

[Wang et al 2018] Wang T-C Liu M-Y Zhu J-Y TaoA Kautz J and Catanzaro B 2018 High-Resolution Im-age Synthesis and Semantic Manipulation with ConditionalGANs In IEEECVF Conference on Computer Vision andPattern Recognition

• Introduction
• Dataset
• • Fundamental Datasets
  • Geometric Annotation with Facial Landmark Detection
  • Semantic Labels Incorporation
  • • Experiment
    • • Study Ukiyo-e Object using Geometry Features
      • Study Ukiyo-e Style through Color Separation
      • Study Jointly Ukiyo-e Object and Style by Composing Sketch and Color
      • • Conclusion
        • Acknowledgement