Post on 21-Dec-2015
transcript
Video Object Tracking Video Object Tracking and Replacement for and Replacement for
Post TV ProductionPost TV Production
LYU0303 Final Year ProjectLYU0303 Final Year Project
Fall 2003Fall 2003
OutlineOutline
Project IntroductionProject Introduction Basic parts of the purposed systemBasic parts of the purposed system Working principles of individual partsWorking principles of individual parts Future WorkFuture Work Q&AQ&A
IntroductionIntroduction
Post-TV production software changes the Post-TV production software changes the content of the original video clips.content of the original video clips.
Extensively used in video-making Extensively used in video-making industries.industries.
Why changing the content of a video?Why changing the content of a video? Reducing video production costReducing video production cost Performing dangerous actionsPerforming dangerous actions Producing effects those are impossible in realityProducing effects those are impossible in reality
Difficulties to be overcomeDifficulties to be overcome Things in video can be treated individually Things in video can be treated individually
called “video objects”.called “video objects”. Computers cannot perform object detection Computers cannot perform object detection
directly because…directly because… Image is processed byte-by-byteImage is processed byte-by-byte Without prior knowledge about the video objects to be detectedWithout prior knowledge about the video objects to be detected Result is definite, no fuzzy logic.Result is definite, no fuzzy logic.
Though computers cannot perform object Though computers cannot perform object detection directly, it can be programmed to detection directly, it can be programmed to work indirectly.work indirectly.
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
RGB/HSV converterRGB/HSV converter
Human eyes are more sensitive to the Human eyes are more sensitive to the brightness rather than the true color brightness rather than the true color components of an object.components of an object.
More reasonable to convert the More reasonable to convert the representation of colors into HSV (Hue, representation of colors into HSV (Hue, Saturation and Value (brightness)) model.Saturation and Value (brightness)) model.
After processing, convert back to RGB and After processing, convert back to RGB and save to disk.save to disk.
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
Edge detectorEdge detector
Usually, a sharp change in hue, saturation or Usually, a sharp change in hue, saturation or brightness means that there exist a boundary brightness means that there exist a boundary line.line.
HSV: (0,0,0)
HSV: (0,255,255)
Smoother for Edge detectorSmoother for Edge detector
Sometimes noise will affect the edge Sometimes noise will affect the edge detection result of low resolution images.detection result of low resolution images.
Include an image smoother to remove Include an image smoother to remove large noise points in the image.large noise points in the image.
In some cases performing a smoothing will In some cases performing a smoothing will greatly enhance the performance of edge greatly enhance the performance of edge detection due to the decrease in “fake” detection due to the decrease in “fake” edge points.edge points.
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
Edge equation finderEdge equation finder
Derives mathematical facts out of the edge Derives mathematical facts out of the edge points.points.
Works with voting algorithm of Hough Works with voting algorithm of Hough Transform.Transform.
Automatically adjusts tolerance value to Automatically adjusts tolerance value to minimize the effect of noise points.minimize the effect of noise points.
This helps when the edge is not completely This helps when the edge is not completely straight or blurred.straight or blurred.
Edge equation finderEdge equation finder
0
4590
(x1,y1)
Angle in degree Frequency
0 1
45 3
90 1
135
135 1
Desired linear equation in point-slope form:
)1(*45tan1 xxyy
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
Equation processorEquation processor
After finding out the equation, constraints After finding out the equation, constraints can be applied in order to remove can be applied in order to remove redundant equations, get shadows or redundant equations, get shadows or detect occultation.detect occultation.
Find out the corner points for translation Find out the corner points for translation detector and texture mapper.detector and texture mapper.
Equation processorEquation processorBefore edge findingAfter edge and equation findingAfter extra equation removal
Equation processorEquation processor
The following criteria are currently adopted The following criteria are currently adopted in the equation processor:in the equation processor: Distance between the equationsDistance between the equations Angle between the equationsAngle between the equations Whether the equation intersects the object or Whether the equation intersects the object or
not.not. Since equation processor is a potential Since equation processor is a potential
burden to the system, it may be replaced burden to the system, it may be replaced in the future by improving other parts.in the future by improving other parts.
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
Translation detectorTranslation detector
A simple object motion tracker.A simple object motion tracker. Collects the data of the first key frame to Collects the data of the first key frame to
accelerate the process of the remaining accelerate the process of the remaining video frames.video frames.
Can be beneficial if the video segment is Can be beneficial if the video segment is long and the scene seldom changes.long and the scene seldom changes.
Translation detectorTranslation detector
Records the approximate location of the Records the approximate location of the object in the key frame.object in the key frame.
When processing the following video When processing the following video frames, just scan within a certain boundary frames, just scan within a certain boundary near the recorded location.near the recorded location.
Will improve its efficiency later.Will improve its efficiency later.
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detectorEdge detector Edge equation finderEdge equation finder Equation processorEquation processor Texture mapperTexture mapper
Texture MapperTexture Mapper
A graphics design technique used to wrap A graphics design technique used to wrap a surface of a 3-D object with a a surface of a 3-D object with a texture texture mapmap
The 3-D object acquires a The 3-D object acquires a surface texturesurface texture similar to the texture map.similar to the texture map.
ColorsColors, , brightnessbrightness valuesvalues or or altitudesaltitudes
Texture MapperTexture Mapper
Definition of terms:Definition of terms: Image coordinates Image coordinates (r, c)(r, c) : location of pixel in : location of pixel in
the the imageimage Texture coordinates Texture coordinates (u, v)(u, v) : location in : location in texture texture
mapmap which contains color information for which contains color information for image coordinatesimage coordinates
Mapping functionMapping function : determines : determines howhow texture texture coordinates are mapped to image coordinates coordinates are mapped to image coordinates or vice versa. or vice versa.
e.g. linear scan-line interpolatione.g. linear scan-line interpolation
Texture MapperTexture Mapper
Definition of termsDefinition of terms Forward mappingForward mapping maps from the maps from the texture texture
spacespace to to image spaceimage space Inverse mapping Inverse mapping maps from the maps from the image spaceimage space
to to texture spacetexture space Scan-line conversionScan-line conversion: an area-filling technique : an area-filling technique
processing a surface processing a surface line by lineline by line. It can be . It can be applied with forward/inverse mapping.applied with forward/inverse mapping.
triangletriangle//parallelogramparallelogram scanning scanning line by lineline by line, from top to bottom, from top to bottom process process eacheach pixel on every line pixel on every line
Scan-line conversionScan-line conversion
Scanline yk
Scanline yk+1
scanning order
for every line
Scan-line conversionScan-line conversion
For a row scan, maintain a list of scanline / polFor a row scan, maintain a list of scanline / polygon intersections.ygon intersections.
Intersection at scanline r+1 efficiently computeIntersection at scanline r+1 efficiently computed from scanline r.d from scanline r.
Scanline yk
Scanline yk+1
Coordinates differ by Δx and Δy
Scan-line conversionScan-line conversion
quadrilateral quadrilateral trianglestriangles or or parallelogramsparallelograms scan each scan each sub-polygonsub-polygon special case: only special case: only 22 sub-polygons sub-polygons
1
2
3
Scan-line conversion with forward Scan-line conversion with forward mappingmapping
AlgorithmAlgorithm
texture scanning
forward mapping functions
r = R(u,v)r = R(u,v)c = C(u,v)c = C(u,v)
copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
for u = umin to umaxfor u = umin to umax for v = vmin to vmixfor v = vmin to vmix
Scan-line conversion with inverse Scan-line conversion with inverse mappingmapping
texture scanning
inverse mapping functions
u = U(r,c)u = U(r,c)v = V(r,c)v = V(r,c)
copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
for (r,c) = polygon pixelfor (r,c) = polygon pixel
AlgorithmAlgorithm
ComparisonComparison
Possibility ofPossibility of aliasing aliasing
NoNoYesYesCalculation Calculation ofof
fractional area of pixel fractional area of pixel coveragecoverage
a bita bit complicated complicated as it as it involves image scanninginvolves image scanning
easyeasy if the mapping if the mapping function is knownfunction is knownEase Ease of implementationof implementation
image image texture texturetexture texture image imagePrinciplePrinciple
Inverse Inverse mappingmapping
Forward Forward mappingmapping
ComparisonComparison
Yes, but can be avoided Yes, but can be avoided with simple filtering/resawith simple filtering/resa
mplingmplingYesYesPossibility ofPossibility of aliasing aliasing
NoNoYesYesCalculation Calculation ofof
fractional area of pixel fractional area of pixel coveragecoverage
a bita bit complicated complicated as it as it involves image scanninginvolves image scanning
easyeasy if the mapping if the mapping function is knownfunction is knownEase Ease of implementationof implementation
image image texture texturetexture texture image imagePrinciplePrinciple
Inverse Inverse mappingmapping
Forward Forward mappingmapping
We used inverse mapping
Mapping functionsMapping functions
Simple linear transformationsSimple linear transformations Translation, scaling, etcTranslation, scaling, etc parallelogramsparallelograms only only very fastvery fast
Linear scan-line interpolationLinear scan-line interpolation Based on Based on proportionproportion any any quadrilateralsquadrilaterals
not suitable
feasible
Linear scan-line interpolationLinear scan-line interpolation
Idea of Idea of interpolationinterpolation
(x1, y1)
(x2, y2)
a
1-a
(x, y) = a(x2, y2) + (1-a)(x1, y1)
Linear scan-line interpolationLinear scan-line interpolation
(r4, c4)
(r1, c1)
(r3, c3) (r2, c2)
(r5, c5)
(r, c)
image
(u1, v1)(u3, v3)
texture map
(u2, v2)
(u4, v4)
(u5, v5)
For a particular scanline,
(u, v)
Shadow mappingShadow mapping
Mapping of surface Mapping of surface brightnessbrightness Retain the brightness of the Retain the brightness of the original original
surfacesurface
Method:Method: Convert the original surface Convert the original surface HSVHSV Get the Get the VV value value Replace the Replace the VV value of the mapped surface value of the mapped surface
Future WorkFuture Work
Anti-aliasingAnti-aliasing Mapping different shapes like cansMapping different shapes like cans Speed optimizationSpeed optimization Movie manipulationMovie manipulation Use of 3D markersUse of 3D markers