Master’s Project DefenseBill Champlin
University of Colorado at Colorado SpringsDept of Computer Science
Java Quasi-Connected Components (JQCC) Tracking System
April15, 2010Advisor - Dr. Terrance Boult
Agenda• Overview/Scope• Approach• Activities• Analysis & Design• Re-host Effort• Port to Linux• Effort &Performance Metrics• Adapting QCC to a new Domain• Lessons Learned/Issues/Recommendations• JQCC Demonstration
2
Project Overview
• JQCC re-hosts Quasi-Connected Components (QCC) portion of the Lehigh Omni-Directional Tracking System (LOTS) to Java– Object tracking software used to track snipers, ships, UAVs, etc
• Dr. Boult wanted a version to post on the web– Means Java for maximum portability– Need to prove that by running different platforms– Would Java be fast enough? i.e. 30 FPS?
3
Approach• Options: write from scratch or port QCC portions of LOTS QTRACK?
• From scratch, had these obstacles:• Had no track region or “cluster” association algorithm• Legacy had numerous special cases to handle tracking issues - would be missing
those • Porting LOTS QTRACK daunting
• QTRACK baseline ~ 96 KSLOC of C++• “You won’t understand what a lot of what the code does” – Dr. Boult• Complex, undocumented and sparsely commented
– Supports various input sources, architectures i.e. Intel MMX and camera types– A lot of dead code
• Chose to port QTRACK• Knew it worked – just had to translate? • Not quite – a lot of architecture work first
4
Activities1. Obtain LOTS s/w baseline (QTRACK) and get running2. Extract core functionality (QCC)3. Develop requirements and UML design4. Port LOTS core functionality to Java (JQCC):5. Verify execution under Windows and Linux platforms6. Evaluate performance vs. legacy QTRACK7. Adapt to tracking night sky images8. Generate promised “artifacts”
o SRS, Certification Specification, and User Handbooko Project Reporto Project Website: http://jqcc.pbworks.com/
5
Analysis Model
6
• Requirements broken out by JQCC Use Cases:
Design Model
7
Concurrency diagram depicts top level process/thread allocation
QCC Technique (from 20K ft)QCC tracks targets by:• Building a background model
– Blends input frames into primary and secondary “reference” images
ref = (1-alpha)ref + alpha*ref // big alpha blends faster– Primary ref = ref closest to input frame pixel and will be used for
thresholding – means a more accurate threshold
• Thresholding / Connecting Thresholded Pixels– Identifies candidate target pixels exceeding dynamic high threshold
» Sum them into a 4x4 reduced “Parent” image – Identify neighbors that exceed a lower threshold around target pixel
» Sum into into Parent image (form small blobs or “regions”)– Adjust dynamic threshold for next time based on pixel count exceeding
high threshold
8
QCC Technique-Con’t• Parent regions are grown and “connected” to those from last
parent - get target ID (Label) from past parent if they overlap a region from last time
• All pixels of the same region are “labeled” with the same target id• Regions are matched with previous overlapping and non-
overlapping regions• Regions that are too sparse or to big (due to lighting shadows/etc)
are “cleaned” (discarded)• Regions are assigned a “confidence” based on density, shape,
speed, etc and classified “tracks” if they exceed confidence threshold
• Every nth frame, reference and variance frames are updated
9
QCC Performance Boosters
• Thresholding, swapping references done over column segments of each row (cords)
• Reference frames only blended every 16th frame or so (configurable)– Input buffer “absorbs” periodic slow downs
• Parent image reduces pixels to process• Only parent rows with used pixels are
processed
10
Porting to Java Approach
• Keep legacy track class structure where possible• Preprocessor statements conversion
– #defines variables to static final or static variables– Groups of related #defines converted to Java enums– #define macros to functions– #if, #else, #endif converted to if, if-else blocks
• Had to put debug in QTRACK to figure out what defines were “TRUE”
11
Porting to Java Approach – Con’t• Convert unsigned char to byte, etc.
– Save memory over unsigned char to short or int– No unsigned primitive types in Java – use masks
• Example: uchar val=0x8F;int result = 0xFF – val;Convert to: byte val=0x8F;int result = 0xFF – val;• Error: want - 0x70, get - 0x170Java converts byte val to -113, then subtracts – logic error• Instead use: byte val=0x8F;int result=0xFF-val&0xFF;• Error prone – QTRACK uses a lot of compound expressions
– Java right shift “>>” is a “signed” shift, shifts in 1’s if value being shifted has sign bit set
• Use “>>>” instead to shift in 0’s from left to match C++12
Porting to Java Approach – Con’t• No pass by reference “&” in Java – only by value
– Convert reference or “out” arguments to “wrappers”public class ByteWrapper{ public byte val;}
– Caller sets val , method gets val inside, method sets val before returning, caller retrieves val after method returns
– Autobox types i.e. Integer, Double, etc. don’t work for this
• No pointer arithmetic in Java– QTRACK uses pointer arithmetic extensively– Convert pointers to use indexed arrays and do math on the
indexes• Error prone separating ptr math from “content” math
• Convert static function variables to static class fields
13
Code Metrics
14
Summary Information LOC Classes Methods Ported 8139New 2527Reuse 426Totals 11092 32 370
Average 270.5366 9.02439Max 5426 125Min 5 0
Approximate Commented Out 1500
Raw Data LOC Classes MethodsPackage Name
Effort
BaseTracker.java 5426 1 125 qcc Ported
ColorModeChoicesEnum.java 39 0 0 qcc
Ported
Cord.java 11 1 1 qcc Ported
image_transmission_t.java 37 0 0 qcc
Ported
JQTracker.java 647 1 16 qcc Ported
region.java 1570 1 61 qcc Ported
RegionPriorityEnum.java 43 0 0 qcc Ported
RegionStatusEnum.java 45 0 0 qcc Ported
RoiCord.java 14 1 1 qcc Ported
SumImage.java 42 1 4 qcc Ported
Image.java 112 1 20 shared Ported
Performance Comparison
15
Item Measured FPSQTrack with MMX 70QTRACK no MMX 64.5
JQCC 59
Port to Linux
16
• Goal – a portable, ready-to-run solution that doesn’t force the user to download and install a lot of packages (preferably nothing but Java)
• Biggest issue – getting movies played/injected on Linux using Java• JMF pure Java version advertised as runnable under Linux
– Unable to run JMF MPEG Camera Simulator under RedHat Enterprise 5.0
• Examined other options:– Xuggle Xuggler (licensing issues / involved to build source on Linux)– FOBS4JMF (requires FFMPEG, other C libs)
• Finally used Java Advanced Imaging (JAI) to load JPEG image sequences– Created a 2nd version of the Camera Simulator (JPEGCameraSimulator)
• Same JQCC class files now run under Linux– Startup bash scripts only difference – can run same Java class files as MS Windows
• Verified under both Ubuntu 9.10 and RedHat Enterprise 5.0
Adapting JQCC to new Domain
17
• Domain chosen: Tracking objects in night sky images• Difficulty finding sample movies
– Web searches not that productive• NASA JPL Search
– Very short videos of Martian moons Phobos & Deimos– GALEX Space Telescope Debris Movie [5]
• Ran across Kevin Fetter’s web site[3] www.kfetter.com– Movie camera telescope setup http://www.kfetter.com/setup/videosetup.htm– Had no images when I contacted him– Collected several sets of satellites for me moving at various directions and
speeds (GEO, Near Earth, Blinker) (has collected a lot more since then)• Initially, JQCC would track only one of the fast satellites• Modified parameters to get it to work for most images
Lessons Learned• Java weaknesses
– Movie support– No unsigned types– Garbage Coll. causes display updates to “pause”
• Can’t avoid it totally – JVM / open source packages• GC runs longer the more memory there is allocated• Work-arounds
– Use incremental “-Xinc” option to help GC “pauses”– Consider using direct memory allocation to allocate memory outside of
JVM so GC doesn’t check it
– Using signed types with masking is error prone– Predicates with final constants leads to “dead code”
• Promise of Java Portability Holds– Write once, build once, run anywhere (with a JVM) works
18
Issues/Recommendations
19
• Modify JPEG Camera Simulator to “read ahead” instead of loading all frames into memory
• Find and fix possible dynamic thresholding issue• Examine tracking stability• Improve reusability - finish supporting all tuning parameters
in calibration file and on GUI; remove scenario specific processing
• Finish debugging rectangle files for “ignoring” regions and outside of a radius for omni-directional images
• Re-factor into an OO design; add more / better comments• Improve display performance / fix “update” color artifacts• Implement remaining capabilities
Backup Slides
20
Demonstration ScriptSatellite images courtesy of [4], GALEX Debris courtesy of [5]
21
• Low earth satellite– Rate Adjustment / Tracking Parameters
• Vertical Object• MPEG GEO• Blinker• Linux low earth satellite
– Side by side comparison– Frame Rates
• GALEX Debris• Toolbag
Class Diagrams
22
DifferencingFor every “used” pixel i primaryDiff = |inim[i] - refim1[i]| secondaryDiff = |inim[i] - refim2[i]| // use closest difference if secondaryDiff < primaryDiff diff = primaryDiff swap(refim1[i], refim2[i]) // keep bkgrnd closest to last input as primary thresh = dynThresh + varim[i] if diff[i] > thresh // threshold difference frame outim[i] = diff else outim[i] = 0//Build parent image (see next slide for what a parent image is)For pos in every parent pixel For j in every image pixel pos that maps to parent if outim[j] parent[pos]++ if outim[i]-delta > lowThresh parent[pos] += 256 // Increments count above low threshold
23
Differencing - Con’t• A parent is a smaller image that summarizes regions across
the larger FOV• Each parent pixel maps to a 4x4 pixel in main image• Each parent pixel is a short which counts pixels above high
threshold and low threshold at same time– Upper 8 bits count of pixels > High threshold– Lower 8 bits sums pixels > Low threshold
• Smaller image = faster processing for subsequent processing i.e. growing regions
24
High Level Track Processing
25
Track Processing
26
See Differencing slide
Dynamic threshold adjusted up based on count of differenced
pixels > threshold
Parent regions grown and matched to overlapping regions from past parent
Assigns a common label (integer value) to all
parent pixels of a region
Reducing threshold around target
improves tracking
Confidences converted to priority which
dictates next threshold reduction
Input frame blended into ref. frames and variance frame
updated to be slightly bigger than background
Associates region with past targets that overlap in
last update
Associates region with past regions
that have gaps between them
Associates region with regions
older than last update
Discards regions that are too big
or sparse
Regions sorted by confidence for
next update comparison
Discards regions that are too small
Metrics – Con’t
27
Raw Data LOC Classes Methods Package Name Effort
OutputImageEnum.java 40 0 0 sharedPorted
RegionTypeEnum.java 49 0 0 sharedPorted
ScenarioEnum.java 54 0 0 sharedPorted
XPoint.java 5 1 0 sharedPorted
XRectangle.java 5 1 0 sharedPorted
DataLogger.java 96 1 5 qccNew
FrameHeader.java 89 1 12 qccNew
JQCCManager.java 41 1 2 qccNew
LogData.java 42 1 3 qccNew
Receiver.java 152 1 4 qccNew
BooleanWrapper.java 5 1 0 sharedNew
ByteWrapper.java 5 1 0 sharedNew
ColorEnum.java 39 0 0 sharedNew
ImageDimensions.java 83 1 1 sharedNew
InputFrame.java 53 1 9 sharedNew
Metrics – Con’t
28
Raw Data LOC Classes Methods Package Name Effort
IntWrapper.java 5 1 0 shared New
ShortWrapper.java 5 1 0 shared New
TimeHelper.java 24 1 3 shared New
TrackConstants.java 107 1 1 shared New
JPEGCameraSimulator.java 510 1 7 cameraSimulator
New
MPEGCameraSimulator.java 575 1 10 cameraSimulator
New
CannedImageGenerator.java 85 1 3 converter
New
ColorLookupTable.java 65 1 8 ui New
DisplayManager.java 131 1 5 ui New
DisplayWindow.java 235 1 22 ui New
TrackMenu.java 139 1 5 ui New
TrackSpinnerInterface.java 5 0 0 ui New
Xy.java 36 1 7 ui New
ImageDisplay.java 376 1 30 ui Reuse [6]
StopWatch.java 50 1 5 shared Reuse [11]
COCOMO II JQCC Effort EstimateUsing COSTAR[4] Tool
29
More Lessons Learned
• Threading with SMP architectures improves performance
• Forget about design and code “zealousies”– Do what’s simplest when translating code
• Using Java Enum wasted time over constants– Had to create code and convert using code
• Forget about making object oriented initially – Don’t disturb the existing design that works; refactor later
30
References• [1] T.E. Boult, R.J. Micheals, X. Gao, M. Eckmann, “Into the
woods: visual surveillance of non-cooperative and camouflaged targets in complex outdoor settings”, in Proc. Of the IEEE, Oct. 2001
• [2] T.E. Boult, T. Zhang, R.C. Johnson, “Two threshold are better than one”, CVPR, pp.1-8, 2007 IEEE Conference on Computer Vision and Pattern Recognition
• [3] K. Fetter, Astronomical Images, www.kfetter.com• [4] Softstar Systems, Costar Software Estimation Tool,
www.softstarsystems.com• [5] GALEX Science Team - California Institute of Technology,
GALEX Space Telescope Movie, www.galex.caltgech.edu
31
