Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Image Fusion for Context Enhancementand Video Surrealism
Adrian IlieAdrian Ilie
UNC Chapel HillUNC Chapel Hill
Ramesh RaskarRamesh Raskar
Mitsubishi Electric Mitsubishi Electric Research Labs,Research Labs,
(MERL)(MERL)
Jingyi YuJingyi Yu
MITMIT
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Dark Bldgs
Reflections on bldgs
Unknown shapes
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
‘Well-lit’ Bldgs
Reflections in bldgs windows
Tree, Street shapes
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuBackground is captured from day-time scene using the same fixed camera
Night Image
Day Image
Context Enhanced Image
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Mask is automatically computed from scene contrast
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
But, Simple Pixel Blending Creates Ugly Artifacts
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuPixel Blending
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuPixel Blending
Our Method:Integration of
blended Gradients
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
OutlineOutline
• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Surrealism
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Nighttime imageNighttime image
Daytime imageDaytime image
Gradient fieldGradient field
Importance Importance image Wimage W
Fina
l res
ult
Fina
l res
ult
Gradient fieldGradient field
Mixed gradient fieldMixed gradient field
GG11 GG11
GG22 GG22
xx YY
xx YY
II11
I2
GG GGxx YY
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Reconstruction from Gradient Reconstruction from Gradient FieldField
• Problem: minimize errorI’ – G|• Estimate I’ so that
G = I’
• Poisson equation
I’ = div G
• Full multigridsolver
I’I’
GGXX
GGYY
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Why Gradient-based ApproachWhy Gradient-based Approach
• Comparison of intensity values are important• Maintain gradients to capture local variations• Directly solve for desired gradients
• Maintain subtle details• Mix dissimilar images• No need for precise segmentation
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
ComparisonComparison
• Average– Subtle details are lost
• Pixel-wise blending– Sharp transitions
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
IssuesIssues
• Boundary conditions• Color shifts
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Boundary ConditionsBoundary Conditions
• Assumed Neumann condition at borders, I’ · N = 0, • Enforced by haloing image with blacks
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Color ShiftColor Shift• Mixing dissimilar images
• Goal: final image appearance matches input images at corresponding pixels
Ifinal(x,y) = c1Ipoisson(x,y) + c2
• SolveWi(x,y) Ioriginal(x,y) = c1
Ipoisson(x,y) + c2
• Each color channel reconstructed separately
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
OutlineOutline
• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Surrealism
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Enhanced video Note: exit ramp, lane dividers, buildings not visible in
original night video, but clearly seen here.
Overview of ProcessOverview of ProcessDay time image: By averaging
5 seconds of day videoOriginal night time traffic camera
320x240 video
Mask frame (for frame above): Encodes pixel with intensity change
Input
Output
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
AlgorithmAlgorithmFrame Frame NN
Daytime imageDaytime image
TimeAveraged TimeAveraged importance maskimportance mask
Gradient fieldGradient field
Final resultFinal result
Processed binary maskProcessed binary mask
Gra
dien
t fie
ldG
radi
ent f
ield
Frame Frame N-1N-1
Mixed gradient fieldMixed gradient field
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
OutlineOutline• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Related Work
• Surrealism
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Related WorkRelated Work
• Spatio-temporal Composition– Duchamp (Nude descending a staircase)– Freeman 2002– Fels 1999, Klein 2002, Cohen 2003
• Gradient-based Techniques– Multi-spectral: Socolinsky 1999– Shadow removal: Weiss 2001– High dynamic range: Fattal 2002– Image editing: Perez 2003– Some at Siggraph’04
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Rene Magritte, ‘Empire of the Light’
Surrealism
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
OutlineOutline
• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Surrealism
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Time-lapse MosaicsTime-lapse Mosaics
Maggrite StripesMaggrite Stripes
timetime
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Time Lapse MosaicTime Lapse Mosaic
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Time Lapse MosaicTime Lapse Mosaic
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
tt
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Sunrise at NightSunrise at Night
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
BiSolar SystemBiSolar System
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
DiscussionDiscussion• User Experience
– More effective in conveying scene context– ‘Dreamy’ appearance– Nonrealistic : False conditions
• Applications– Tools for artists– Surveillance– Amusement park rides
• Performance– ~1 sec/frame for 320x240– ~ 3 min for 4Mpixel image
Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu
Image Fusion for Context EnhancementImage Fusion for Context Enhancement
• Nonrealistic but comprehensible context• Fusion using multiple images
– Enhancing night images with day bgrnd• Gradient-based fusion• Video surrealism tools
tt