Date post: | 11-Dec-2015 |
Category: |
Documents |
Upload: | maverick-hatch |
View: | 217 times |
Download: | 3 times |
Ying Cao Antoni B. Chan Rynson W.H. Lau
City University of Hong Kong
Automatic StylisticLayout
Background
• Manga layout is crucial for manga production, with unique styles
©AYOYAMA Gosho / Shogakukan Inc.
Manga pages Their layouts
Background
• Effective manga layout can benefit– Storytelling– Attention guidance– Visual attractiveness
• It is a difficult task
Goal
• To create high-quality manga layout with ease
Resulting layout
1
2
2
3
3
3
SemanticsArtworks
Challenge
• Not a well-studied problem
• Our solution: data-driven strategy to learn stylistic aspects from existing manga pages
• No explicit rules
Related Work
• General layout problem: global optimization
[ Yu et al. 2011] [Merrell et al. 2011]
Related Work
• Comic layout: heuristic rules or templates
[Kurlander et al. 1996] [Shamir et al. 2006] [Preu et al. 2007]
Related Work
• Computational Manga
[Qu et al. 2006]
[Qu et al. 2008]
Overview
Overview
Overview
Overview
Manga Database
• 4,000 scanned manga pages from two manga series
Panel annotation Page clustering
One manga series
3-panel pages 10-panel pages4-panel pages …
Overview
Style Models
• Represent stylistic aspects of manga layout
• Learned from manga examples
3) Panel shape
…2) Panel importance (size)
1 2 3
1) Layout structure (i.e., spatial arrangement of panels)
…
• A probabilistic generative model:pdf(gm,[x,y])
• Synthesize novel plausible layout structures
Layout structure Model
Root
©AYOYAMA Gosho / Shogakukan Inc.
Layout structure Model
• Generative process: recursive spatial division
R1 R2 R3
C1C2 C1
R2R1
C3 C2 C1
Layout structure Model
• Parameterization: spatial division instance
: Instance label [Root-R]
: Number of rows [3]
: Splitting configuration [()]
X 1
X 2
X 3
©AYOYAMA Gosho / Shogakukan Inc.
Layout structure Model
: Instance label
: Number of rows
: Splitting configuration
𝓛𝑵
𝐗Probabilistic graphical model
• Parameterization: spatial division instance
Layout structure Model
• Sample splitting configuration
: Instance label
: Number of rows
: Splitting configuration
𝓛𝑵
𝐗Probabilistic graphical model
Layout structure Model
• Sample splitting configuration
: Instance label
: Number of rows
: Splitting configuration
𝓛𝑵
𝐗Probabilistic graphical model
Sample𝐗
Layout structure Model
Layout structures sampled from our modelTraining example
• Sampling: recursive splitting using sampled
• Panel clustering
Width
Hei
ght
Panel Importance
3 12
Size Importance Shape?
A shape-to-importance classifier𝑚= 𝒇 (𝒙 )
Panel Shape Variation Model
• Captures panel shape variability‒ Active Shape Model [Cootes et al. 1995]
…{𝒔2 ,𝐔2}
…{𝒔1 ,𝐔1 }
{𝒔3 ,𝐔3 }…
Overview
Semantic SpecificationSingle-panel semantics
Inter-panel semanticsImage geometry
Group of related panels
3
Importance
Overview
Initial Layout Generation
• A layout structure
• Maximum a posteriori (MAP) inference
Fitness between and Our generative model
Input semantics
Existing ones
matches
resembles
Initial Layout Generation
• Likelihood term
Penalize panel-wise mismatch in aspect ratio & importance
Single-panel Likelihood
𝑃 𝑗
𝐼 𝑗
Image geometry
panel geometry
Initial Layout Generation
• Likelihood termInter-panel Likelihood
Initial Layout Generation
• Likelihood termInter-panel Likelihood
Measure the smoothness of path through panels
Initial Layout Generation
• Likelihood termInter-panel Likelihood
Align group boundary with layout boundary
Initial Layout Generation
• Estimate optimal initial layout‒ Exact MAP inference is computationally expensive
…
pdf(gm,[x,y])
Generative Model Maximum Posteriori
Layout Optimization
Unoptimized
• Fit to and reproduce panel shape irregularity
Layout Optimization
• Energy function
Collinearity constraint
𝐕
Boundary constraint
𝐕
𝑃 𝑖
𝐓𝑖 ∙ 𝐼𝑖
Regularization term
Layout Optimization
• Minimize via an alternating solver
Results
(1)
(2)
(2)
(3)
(2)
(2)
(1)
(2)
(2)
(2)
(3)
(3)
(1)
Comparison with existing manga page
Input Our result Existing manga page
(3)
(3)
(1)
(3)
(3)
(2)
(3)
©AYOYAMA Gosho / Shogakukan Inc.
Layouts of different styles
(1)
(2)
(3)
(1)
(3)
(2)
Input Style of “Fairy Tail” Style of “Detective Conan”
Layouts of Western comic style
User Study
• 10 participants: manual tool + our tool
• 10 Evaluators: pairwise comparison
Manual tool Our tool012345 4.09
1.36Aver
age
time(
min
)
Functionality Visual Appeal Style0%
20%
40%
60%
Manual placement Our toolTie
Summary
• First attempt to computationally reproduce layout styles of manga
• A data-driven approach for automatic generation of stylistic manga layout
• Easy and quick production of professional-looking and stylistically rich manga layouts
Limitations & Future Work
• Story pacing
• Art composition & balloon placement
• Generic framework for other layout problems
Thanks