Multi-Touch Navigation Engine
Presented by Team Extra Touch:Chris Jones
Shuopeng YuanNathan Wiedeback
Multi-Touch Navigation Engine
• Background• System Design• Accomplishments
2
Background• AIS offers single-touch• Multi-touch is the “cool
new thing”– iPhone, MacBook– MS Surface
• Mitch Butzer had good ideas…
3
4
Project Specifications
•Microcontroller-based•Written in C•2 - 6 independent control channels• Defined at compile time
•Recognize defined patterns• e.g., finger but not palm
5
System Flow
Serial Input
6
Serial Stream Parser•74 byte “packets” from the screen•Processed to get 32 sensor values, etc.
•Subtract the lowest value from each
7
(Little-endian)
Average: 0x488B = 18,571First sensor: 0x4886 = 18,566
Interpolator
8
Interpolator
•Red dots: intersections of sensors (16 x 16)•Green dot `P`: interpolated point
•Set a threshold and discretize at the end•Each pixel -> 1 or 0
9Image source: Wikipedia
Phantom Filter
10
Phantom Filter
11
One touch Two touches Phantom touches (unfiltered)
•Intersections -> phantom touches
•Compare touches to last frame for filtering
Pattern Recognizer
12
Pattern Recognizer: Flowchart
13
Data In : Frame of Data
Resolution: 240 * 180
Blob Classification and Checking Engine
14
Input Example: Resolution 60*20
Blob edge finding algorithm
0 0 0 0 0 1 0 0 00 0 0 0 1 1 0 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 0 1 1 0 00 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0
0 0 0 0 0 1 0 0 00 0 0 0 1 1 0 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 0 1 1 0 00 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0
0 0 0 0 0 1 0 0 00 0 0 0 1 1 0 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 0 1 1 0 00 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0
First edge Last edge
16
Blob Classification and Checking EngineInput Example: Resolution 60*20 -- two blobs found
Voronoi Skeleton Algorithm
17
Ideal case Actual case
18
Voronoi Skeleton with Test Data
Slope Algorithm
Linear skeleton should have the same angle at each point.(a1 = a2 = a3 = a4)
19Linear -> accept and find center Nonlinear -> reject
Channel Assignment
20
21
Channel Frames – Overall Look
1 1 1 0 0 0 3 3 3
1 1 1 0 0 0 3 3 3
1 1 1 0 0 0 3 3 3
0 0 0 0 0 0 3 3 3
2 2 2 2 2 0 3 3 3
2 2 2 2 2 0 3 3 3
2 2 2 2 2 0 3 3 3
2 2 2 2 2 0 3 3 3
2 2 2 2 2 0 3 3 3
2 2 2 2 2 0 3 3 3
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
1 1 1 1 0 2 2 2 2
Current Form Potential Form
22
Current Setup Channel Frames
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
1 1 1 1 0 0 0 0 0
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
0 0 0 0 0 2 2 2 2
Channel 1 Frame Channel 2 Frame
Channel Assignment – How it Works
0 0 0 0 0
0 0 1 1 0
1 1 0 1 0
1 1 0 0 0
1 1 0 0 0
Channel 1 Frame0 0 1 1 1
0 0 0 0 1
0 0 0 0 1
0 0 1 1 1
0 0 1 1 1
Channel 2 Frame
Channel 3 Frame1 1 0 0 0
1 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
Output to One Touch per Channel Limiter: Coordinates 3,3 Channel 3
Input: Coordinate Data (3,3) touched
We have a match at Channel 3!
23
24
One Touch per Channel Limiting54 132 1 11
1 2 3 4 5
Channel A – 3 touches5
4 1
3 12 1 1
1
1 2 3 4 5
Channel A -Phase 1 Reduction
Channel A - Phase 2 Reduction5
4 13
2 1 1 1
1
1 2 3 4 5
25
Output Data
X0
Y0
X1
Y1
X2
Y2
X3
Y3
C0
C1
C2
C3
Altered Coordinate Array
Channel Array
Accomplishments
• Translate sensor wires into 16x16 matrix• Interpolate• Recognize finger touches, reject other shapes• Assign to either of two channels
26
Challenges
Challenge• Phantom touches• “[40K] ought to be enough
for anybody!”• Need “tap points” for
intermediate data
Solution• Compare to previous frame• Low resolution (240 x 180 ?)
for proof-of-concept• Make each stage’s data
available at the output
27
Future
• Serial interrupt issues• Phantom filtering implementation
– Would also enable more shapes• Different touchscreen technology?
28
Questions?
Channel Assignment – Text OverviewData In: Center coordinates (X,Y)
Processing:Data gets transformed into a 2d array, and the compared against each channel frame to determine its channel. Channel frames are defined before compile.
Data Out:(X,Y) coordinates with channel data attached.
Details:Aiming for 2 channels initiallyShould be simple to add channels in the final iteration.
Channel Assignment: Flow Chart
31
Summary• Major design: “Proof-of-concept”
– 240 x 180 x 1-bit “color”– Simple phantom filtering– Accept a single finger per channel, reject other cases– Two channels
32