Multi-Touch Navigation Engine

Presented by Team Extra Touch:Chris Jones

Shuopeng YuanNathan Wiedeback

Multi-Touch Navigation Engine

• Background• System Design• Accomplishments

2

Background• AIS offers single-touch• Multi-touch is the “cool

new thing”– iPhone, MacBook– MS Surface

• Mitch Butzer had good ideas…

3

4

Project Specifications

•Microcontroller-based•Written in C•2 - 6 independent control channels• Defined at compile time

•Recognize defined patterns• e.g., finger but not palm

5

System Flow

Serial Input

6

Serial Stream Parser•74 byte “packets” from the screen•Processed to get 32 sensor values, etc.

•Subtract the lowest value from each

7

(Little-endian)

Average: 0x488B = 18,571First sensor: 0x4886 = 18,566

Interpolator

8

Interpolator

•Red dots: intersections of sensors (16 x 16)•Green dot `P`: interpolated point

•Set a threshold and discretize at the end•Each pixel -> 1 or 0

9Image source: Wikipedia

Phantom Filter

10

Phantom Filter

11

One touch Two touches Phantom touches (unfiltered)

•Intersections -> phantom touches

•Compare touches to last frame for filtering

Pattern Recognizer

12

Pattern Recognizer: Flowchart

13

Data In : Frame of Data

Resolution: 240 * 180

Blob Classification and Checking Engine

14

Input Example: Resolution 60*20

Blob edge finding algorithm

0 0 0 0 0 1 0 0 00 0 0 0 1 1 0 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 0 1 1 0 00 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0

0 0 0 0 0 1 0 0 00 0 0 0 1 1 0 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 0 1 1 0 00 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0

0 0 0 0 0 1 0 0 00 0 0 0 1 1 0 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 1 1 1 0 00 0 0 0 0 1 1 0 00 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0

First edge Last edge

16

Blob Classification and Checking EngineInput Example: Resolution 60*20 -- two blobs found

Voronoi Skeleton Algorithm

17

Ideal case Actual case

18

Voronoi Skeleton with Test Data

Slope Algorithm

Linear skeleton should have the same angle at each point.(a1 = a2 = a3 = a4)

19Linear -> accept and find center Nonlinear -> reject

Channel Assignment

20

21

Channel Frames – Overall Look

1 1 1 0 0 0 3 3 3

1 1 1 0 0 0 3 3 3

1 1 1 0 0 0 3 3 3

0 0 0 0 0 0 3 3 3

2 2 2 2 2 0 3 3 3

2 2 2 2 2 0 3 3 3

2 2 2 2 2 0 3 3 3

2 2 2 2 2 0 3 3 3

2 2 2 2 2 0 3 3 3

2 2 2 2 2 0 3 3 3

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

1 1 1 1 0 2 2 2 2

Current Form Potential Form

22

Current Setup Channel Frames

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

1 1 1 1 0 0 0 0 0

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

0 0 0 0 0 2 2 2 2

Channel 1 Frame Channel 2 Frame

Channel Assignment – How it Works

0 0 0 0 0

0 0 1 1 0

1 1 0 1 0

1 1 0 0 0

1 1 0 0 0

Channel 1 Frame0 0 1 1 1

0 0 0 0 1

0 0 0 0 1

0 0 1 1 1

0 0 1 1 1

Channel 2 Frame

Channel 3 Frame1 1 0 0 0

1 1 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

Output to One Touch per Channel Limiter: Coordinates 3,3 Channel 3

Input: Coordinate Data (3,3) touched

We have a match at Channel 3!

23

24

One Touch per Channel Limiting54 132 1 11

1 2 3 4 5

Channel A – 3 touches5

4 1

3 12 1 1

1

1 2 3 4 5

Channel A -Phase 1 Reduction

Channel A - Phase 2 Reduction5

4 13

2 1 1 1

1

1 2 3 4 5

25

Output Data

X0

Y0

X1

Y1

X2

Y2

X3

Y3

C0

C1

C2

C3

Altered Coordinate Array

Channel Array

Accomplishments

• Translate sensor wires into 16x16 matrix• Interpolate• Recognize finger touches, reject other shapes• Assign to either of two channels

26

Challenges

Challenge• Phantom touches• “[40K] ought to be enough

for anybody!”• Need “tap points” for

intermediate data

Solution• Compare to previous frame• Low resolution (240 x 180 ?)

for proof-of-concept• Make each stage’s data

available at the output

27

Future

• Serial interrupt issues• Phantom filtering implementation

– Would also enable more shapes• Different touchscreen technology?

28

Questions?

Channel Assignment – Text OverviewData In: Center coordinates (X,Y)

Processing:Data gets transformed into a 2d array, and the compared against each channel frame to determine its channel. Channel frames are defined before compile.

Data Out:(X,Y) coordinates with channel data attached.

Details:Aiming for 2 channels initiallyShould be simple to add channels in the final iteration.

Channel Assignment: Flow Chart

31

Summary• Major design: “Proof-of-concept”

– 240 x 180 x 1-bit “color”– Simple phantom filtering– Accept a single finger per channel, reject other cases– Two channels

32