Final Year Project 2004/2005Final Year Project 2004/2005LYU0402LYU0402
Augmented Reality Table for Augmented Reality Table for Interactive Card GamesInteractive Card Games
Supervisor: Prof Michael LyuSupervisor: Prof Michael Lyu
Presented by: Kevin Chow, Albert LamPresented by: Kevin Chow, Albert Lam
OverviewOverview
IntroductionIntroduction ObjectiveObjective Trading Card GameTrading Card Game Architecture of ARTArchitecture of ART Implementation in First TermImplementation in First Term ExperimentExperiment ConclusionConclusion Future WorkFuture Work
IntroductionIntroduction
What is What is Augmented RealityAugmented Reality??
• Mixed Reality combines the content from Mixed Reality combines the content from the real world with virtual imaginarythe real world with virtual imaginary
• Augmented Reality is a subset of this Augmented Reality is a subset of this where virtual content is overlaid into real where virtual content is overlaid into real objects of the worldobjects of the world
Introduction (Con’t)Introduction (Con’t)
• Paul Milgram characterized Mixed Paul Milgram characterized Mixed Reality interfaces on his “Reality-Reality interfaces on his “Reality-Virtuality Continuum” in 1994Virtuality Continuum” in 1994
Mixed Reality (MR)
VirtualEnvironment
Augmented Virtuality (AV)
AugmentedReality (AR)
RealEnvironment
Reality-Virtuality (RV) Continuum
Introduction (Con’t)Introduction (Con’t)
An AR system has the following three An AR system has the following three main characteristics:main characteristics:• Combines real and virtual objects in a Combines real and virtual objects in a
real environmentreal environment• Runs interactively, and in real timeRuns interactively, and in real time• Registers virtual objects onto the real Registers virtual objects onto the real
worldworld
Introduction (Con’t)Introduction (Con’t)
ART stands for ART stands for Augmented Reality TableAugmented Reality Table• User can play Card Games on the ART.User can play Card Games on the ART.
• ART can enhanceART can enhance the visual effect the visual effect during the playingduring the playing of the card game. of the card game.
ObjectiveObjective
Develop a generic Interactive ART Develop a generic Interactive ART Trading Card Game application. Trading Card Game application.
Enhance the traditional card games:Enhance the traditional card games:• Visual effectVisual effect• Sound effectSound effect• Complex calculationComplex calculation
Trading Card GameTrading Card Game
Trading Card GamesTrading Card Games• Two players play a match.Two players play a match.• Players play cards to fight with each Players play cards to fight with each
other. It includes summoning monster, other. It includes summoning monster, casting magic, or setting traps, etc.casting magic, or setting traps, etc.
• We choose We choose “YU-GI-OH”“YU-GI-OH” as our as our implementationimplementation
Trading Card Game (Con’t)Trading Card Game (Con’t)
Trading card game includes five kinds of Trading card game includes five kinds of actionsactions• RestoreRestore• Draw cardsDraw cards• Play cardsPlay cards• Attack/ChallengeAttack/Challenge• Discard cardsDiscard cards
An Example - YU-GI-OHAn Example - YU-GI-OH In YU-GI-OH, Different information are In YU-GI-OH, Different information are
stored on the card.stored on the card.
An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)
Game Mat of YU-GI-OHGame Mat of YU-GI-OH Different kinds of card must be put inside Different kinds of card must be put inside
particular card zonesparticular card zones
An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)
LP : 2000 LP : 2000
ATK : 500 ATK : 1200
1200 – 500= 700-700
LP : 1300
An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)
LP : 1300 LP : 2000
DEF : 1300 ATK : 1200
1300 – 1200= 100-100
LP : 1900
ArchitectureArchitecture
Hardware SetupHardware Setup• Plasma monitorPlasma monitor• Overhead cameraOverhead camera
System architectureSystem architecture• Augmented Reality Perception Augmented Reality Perception • Game CoreGame Core• DatabaseDatabase• Game Enhancement Game Enhancement
Hardware SetupHardware SetupOverhead mounted camera
Analyze the video captured from the camera, and generate output to the screen
The plasma monitor for playing trading card games
Plasma monitor act as a table
Hardware Setup (Con’t)Hardware Setup (Con’t)
ComputerComputer• processes perceiving imageprocesses perceiving image• generates visual and audio enhancementgenerates visual and audio enhancement• controls game flowcontrols game flow
Plasma MonitorPlasma Monitor..• acts as the Game Tableacts as the Game Table• displays computer-generated scenedisplays computer-generated scene
Overhead mounted cameraOverhead mounted camera• captures the cards and the screen of the captures the cards and the screen of the
plasma.plasma.• only input of the systemonly input of the system
System ArchitectureSystem Architecture
Game Core ModuleGame Core Module Perception ModulePerception Module Database ModuleDatabase Module Game Enhancement ModuleGame Enhancement Module
System Architecture (Con’t)System Architecture (Con’t)
ART Card GameGeneric Card Game Database
Perception
CommandDetector
Calibration
Card DetectorGame
Enhancement
Video Decoder
Card Locator
Calibration
Card Recognizer
3D Animation
Input Analyzer
Sound Effect
ART Card GameCore
Game RulesDatabase
DisplayCamera
Card Information
Input Information
Environment Information Output Generator
Sound
Card ImageDatabase
CardsDatabase
System Architecture (Con’t)System Architecture (Con’t)
Microsoft DirectX SDKMicrosoft DirectX SDK• A set of low-level APIs for high-A set of low-level APIs for high-
performance multimedia applicationsperformance multimedia applications• directly access the hardwaredirectly access the hardware• provides device independent through HALprovides device independent through HAL• We will use DirectShow, Direct3D and We will use DirectShow, Direct3D and
DirectSound in our implementationDirectSound in our implementation
ImplementationImplementation
Perception ModulePerception Module• CalibrationCalibration• Search window locatorSearch window locator• Card locatorCard locator• Card RecognizerCard Recognizer
Database ModuleDatabase Module• Identify card typeIdentify card type• Image retrievalImage retrieval• Image matchingImage matching
Perception ModulePerception Module
Read and process the raw video Read and process the raw video perceived from the cameraperceived from the camera
Detect input card and commandDetect input card and command Some assumptionsSome assumptions
• fixed camera, fixed tablefixed camera, fixed table• camera is approximately right above the camera is approximately right above the
tabletable• cards can only be placed in predefined cards can only be placed in predefined
region called card zonesregion called card zones
CalibrationCalibration
Why?Why?• Captured image Captured image
varies with varies with environmental environmental lighting conditionlighting condition
• Take advantage of Take advantage of the assumptions the assumptions mademade
Calibration (Con’t)Calibration (Con’t)
How?How?• Set the card zone positionSet the card zone position• Set the card area threshold for each Set the card area threshold for each
card zonecard zone• Calibrate the colorsCalibrate the colors• Calibrate card type colorsCalibrate card type colors
Search Windows LocatorSearch Windows Locator
Find region to search for cardsFind region to search for cards The screen is partitioned into some fixed The screen is partitioned into some fixed
small search windowssmall search windows Search windows which just stop changing Search windows which just stop changing
will be processedwill be processed
Search Windows Locator Search Windows Locator (Con’t)(Con’t)
Located search windows Located search windows that have changesthat have changes• Image differentImage different
• Pixel changed ifPixel changed if
• A Search window is saidA Search window is said to be changed to be changed ifif
Activate search window fromActivate search window from“change” to “unchange” state“change” to “unchange” state
),(),(),( yxIyxIyxI previouscurrentdiff
1),( yxI diff
2),( yxI diff
Card LocatorCard Locator
Locate four corner card positions Locate four corner card positions given a search windowgiven a search window• Apply Apply Canny AlgorithmCanny Algorithm• Extract ContoursExtract Contours• Search for contours whichSearch for contours which
contain 4 pointscontain 4 points contain nearly right contain nearly right
angle cornersangle corners have area within ahave area within a
certain thresholdcertain threshold
Card Locator (Con’t)Card Locator (Con’t)
Canny AlgorithmCanny Algorithm• Remove noise by Remove noise by
Gaussian FilterGaussian Filter• Compute edge strengthCompute edge strength
and edge direction by and edge direction by Sobel OperatorSobel Operator
• Apply Apply Nonmaximum Nonmaximum Suppression Suppression to trace to trace edges edges
• Use Use Hystersis Hystersis to eliminate streakingto eliminate streaking
Card RecognizerCard Recognizer
Card orientationCard orientation• Compare the lengths of the four edgesCompare the lengths of the four edges• Compare color difference between top Compare color difference between top
and bottom parts of the cardand bottom parts of the card
Card Recognizer (Con’t)Card Recognizer (Con’t)
Get Card ImageGet Card Image• Extract the undistorted card image from Extract the undistorted card image from
a distorted imagea distorted image
Card Recognizer (Con’t)Card Recognizer (Con’t)
• Pixel coordinate transformationPixel coordinate transformation Map a point (x,y) on the distorted image to a Map a point (x,y) on the distorted image to a
point (x’,y’) on the undistorted imagepoint (x’,y’) on the undistorted image
(x0,y0)
(x1,y1)
(x2,y2)
(x3,y3)
. (x,y)h
w
. (x’,y’)
3210
3210
))(1())(()1)(()1)(1(
))(1())(()1)(()1)(1(
yh
y
w
xy
h
y
w
xy
h
y
w
xy
h
y
w
xy
xh
y
w
xx
h
y
w
xx
h
y
w
xx
h
y
w
xx
Card Recognizer (Con’t)Card Recognizer (Con’t)
• Brightness interpolationBrightness interpolation Assign brightness value to theAssign brightness value to the
point (x,y) point (x,y) Apply Apply Bilinear InterpolationBilinear Interpolation
)(),(
)(),(
)1,1())((
)1,())(1(
),1()1)((
),()1)(1(),(
yfloorybyfloork
xfloorxaxfloorh
khIba
khIba
khIba
khIbayxfbilinear
Card Recognizer (Con’t)Card Recognizer (Con’t)
Identify the card uniquelyIdentify the card uniquely• Query the Image Database with the card Query the Image Database with the card
image for unique card Idimage for unique card Id• Query the Card Database with the card Query the Card Database with the card
Id for card information and detailsId for card information and details
Database ModuleDatabase Module
Contains all game informationContains all game information Identify card image by 3 steps:Identify card image by 3 steps:
• Identify card typeIdentify card type Classify card by card typeClassify card by card type
• Image retrievalImage retrieval Retrieve several similar candidate cardsRetrieve several similar candidate cards
• Image matchingImage matching Find the best match cardFind the best match card
Identify Card TypeIdentify Card Type
• Compare the background color of the Compare the background color of the card to the calibrated colorcard to the calibrated color
Image RetrievalImage Retrieval
Retrieve similar images of same card Retrieve similar images of same card type.type.
Use color-based retrieval method due Use color-based retrieval method due to low resolutionto low resolution
Color Histogram method select cards Color Histogram method select cards with largest color histogram intectionwith largest color histogram intection
Image Retrieval (Con’t)Image Retrieval (Con’t)
Color histogram intersectionColor histogram intersection
n
jD
n
jDQ
DQH
jHH
jIHjIH
IID
1
1
),(
)),(),,(min(
),(
Select the best matched Select the best matched card from the candidates card from the candidates
• Split the captured image Split the captured image into 4 channelsinto 4 channels
• Compare only inner imageCompare only inner image• Compare the different Compare the different
channels to the channels to the database separatelydatabase separately
Image MatchingImage Matching
Image MatchingImage Matching
• Find the pixel difference byFind the pixel difference by
• Reject the image if the pixel difference Reject the image if the pixel difference is larger than the thresholdis larger than the threshold
• Accept the image with minimum pixel Accept the image with minimum pixel differencedifference
w
i
h
jDQDQP jiIjiIIID
1 1
2)),(),((),(
ExperimentExperiment
Match 15 cards to a set of different Match 15 cards to a set of different cards with threshold set to 1500cards with threshold set to 1500
Here, we have chosen results of Here, we have chosen results of three cards for discussionthree cards for discussion
Experimental ResultExperimental Result
Overall Pixel Difference of Card 14
0
1000
2000
3000
4000
5000
6000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Card ID
Dif
fere
nce
H value R vaule G vaule B value
Experimental Result (Con’t)Experimental Result (Con’t)Overall Pixel Difference of Card 13
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Card ID
Dif
fere
nce
H value R vaule G vaule B value
Experimental Result (Con’t)Experimental Result (Con’t)Overall Pixel Difference of Card 9
0
1000
2000
3000
4000
5000
6000
7000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Card ID
Dif
fere
nce
H value R vaule G vaule B value
ConclusionConclusion
We have implemented a simplified We have implemented a simplified version of the ART systemversion of the ART system
Concerning low resolution of the Concerning low resolution of the image, we have develop an image, we have develop an algorithm to recognize card with high algorithm to recognize card with high accuracy but less efficiencyaccuracy but less efficiency
Future WorkFuture Work
Color CalibrationColor Calibration Develop an efficient card recognition Develop an efficient card recognition
algorithm while retaining high algorithm while retaining high accuracyaccuracy
Game EnhancementGame Enhancement Interactive input detectorInteractive input detector
End of PresentationEnd of Presentation
Thank you very muchThank you very much
Q & A SessionQ & A Session