Windows Touch Deep Dive
Reed TownsendProgram ManagerMicrosoft Corporation
CL13
Agenda> Introduction> Overview of Windows Touch API
surface> Real-world ISV questions and solutions> Demo projects
> 3D manipulations> Reimplementing Collage’s PhotoStrip
control> Wrap up
Touch - Framing the Opportunity> What is touch?
> It’s an input method> What does it mean to design for
touch?> When designing for inputs, we’re really
talking about user interface design (in the broad sense)
> UI is often synonymous with visuals and the input and interaction half is forgotten
Touch - Framing the Opportunity> 99% of design is done in the platform
> Controls and APIs are the language of UI> “The limits of my language mean the
limits of my world” – Ludwig Wittgenstein> Games (and sometimes web) follow a
different model> What are the common controls for
touch? ?
Windows Touch APIs> Good/Better/Best platform model
> Good:> What you get for free> Legacy, non-touch-aware apps
> Better> Minimal investment> Typically WM_GESTURE, mouse message
inspection> Best
> Enables touch-optimized experiences!> WM_TOUCH, manipulations & inertia, and
more
WM_GESTURE> Provides
notifications when the user performs gestures over your window
> Contains additional information like center of gesture and gesture-specific arguments
Typical sequence:WM_TABLET_QUERYSYSTEMGESTURESTAT
USWM_GESTURENOTIFYWM_GESTURE (GID_BEGIN)
WM_GESTURE (GID_PAN, GF_BEGIN)WM_GESTURE (GID_PAN, no flags)...WM_GESTURE (GID_PAN,
GF_INERTIA)...WM_GESTURE (GID_PAN, GF_END)
WM_GESTURE (GID_END)
Real-world Questions and AnswersWM_GESTUREQ: I registered for WM_TOUCH but I no longer get
WM_GESTURE. What gives?A: Yes, WM_GESTURE and WM_TOUCH are exclusive
in Win7.
Q: I’m implementing gestures in my app, but I stopped getting panning bounce feedback. Why?
A: Two possibilities: Panning messages are being accidentally eaten, or your app is providing custom handling of them but not calling the bounce feedback API.
GetSystemMetrics()> GetSystemMetrics():
> SM_DIGITIZER returns data about available digitizers on the system
> SM_TABLETPC returns data about Tablet functionality
> SM_MAXIMUMTOUCHES yields the largest number of contacts any of the available digitizers supports
Mouse Messages & GetMessageExtraInfo()
> GetMessageExtraInfo() returns the extra info associated with a message
> Mouse up and down messages are tagged with a special signature indicating they came from touch or pen:> Mask extra info against 0xFFFFFF80> 0xFF515780 for touch, 0xFF515700 for
pen
Richer Touch Experiences
demo WM_TOUCH Applications
WM_TOUCH> Provides raw touch
input data, conceptually similar to mouse messages
> Finger painting, custom gestures, feeding higher-level controls (e.g. Manipulations)
Typical sequence:WM_TOUCH (first finger down)WM_TOUCH (first finger moving)...WM_TOUCH (two fingers moving,
third comes down)WM_TOUCH (three fingers moving)...WM_TOUCH (one goes up, two
fingers moving)WM_TOUCH (two fingers moving)...WM_TOUCH (last finger goes up)
Real-world Questions and AnswersWM_TOUCHQ: How can I disable the press and hold gesture in my
window?A: WM_TABLET_QUERYSYSTEMGESTURESTATUS can be
used to disable P&H, flicks, and tap feedback. (WM_TOUCH windows already disable flicks by default. SetGestureConfig() controls gestures available via WM_GESTURE.)
Q: I’m building a UI framework. How can I add touch support?
A: Start with WM_TOUCH. If you want pan/zoom/rotate, use manipulations on top of WM_TOUCH.
Touch-Optimized Experiences
demo WM_TOUCH, Manipulations, and More
Manipulations & Inertia> Manipulations:
> Provides the full range of 2D affine transforms corresponding to user input
> Enables multiple concurrent interactions> Can be fed by any source of raw data> Are a superset of WM_GESTURE support
> E.g. Rotating and translating 3 photos simultaneously
Typical Flow
On contact down:> Hit test against
targets> (Add to map of
contact ids and MPs)
> MP->ProcessDown()
On contact moves:> (Look up mapping)
> MP->ProcessMove()
On contact up:> (Look up mapping)> MP->ProcessUp()> (Kick off inertia)
Get callbacks on event interface> Apply transforms
to targets
Advanced Manipulation Topics> Single-finger rotation
> PivotPoint, PivotRadius properties> Rotation around a point
> Generalization of single-finger rotation> Supporting mouse-based
manipulations
> All covered in the Platform SDK manipulations sample
Inertia> Grab initial state from manipulation
processor in completed event> Set up deceleration properties> Set up borders/margins> Call Process() on a timer
Demo Projects!> Demos:
> 3D manipulations – An approach to mapping 2D onto 3D.
> Reimplementing Collage’s PhotoStrip – Building custom touch controls
> Examples of new ways to use the platform and new user interface options
3D Manipulations> Set up 3D view,
axes, objects> Add WM_TOUCH> Add manipulations> Map manipulations
to 3D navigation> 2 finger camera
operations, zoom> Spherical panning
> Add inertia and other tweaks
3D Manipulations> Classes of note:
> D3DXDriver> CComTouchDriver> Camera
> Inherits from InertiaObj and _IManipulationEvents
> Example data flow:> WM_TOUCH message
arrives> CComTouchDriver::Process
InputEvent()> Camera::ProcessInputEven
t()> InertiaObj::ProcessInputEv
ent()> Manipulation Processor> Camera::ManipulationDelt
a()> Update the camera
Building a PhotoStrip Control> Part of a control
framework> Respond to touch
messages> Add constrained
manipulations for panning
> Add support for dragging in/out of control
> Add API surface> Flesh out overlay and
gallery
Building a PhotoStrip Control> Window hierarchy: > Example data flow:
> WM_TOUCH arrives> Hits overlay, routed to
appropriate control> Photostrip gets
message> Touch pans and
triggers photo removal
> Message to app window
> Message to gallery> Photo is displayed in
gallery
Overlay HWND
PhotoStrip1 HWND
PhotoStrip2 HWND
Gallery HWND
Implementation Notes> Smoothing input and manipulation
data> Mixed input layers
> 3D: manipulations + gestures> PhotoStrip: raw data + manipulations
> Supporting multiple interactions across windows> Overlay + forwarding approach> Capture> Mediating between inputs
Final Thoughts> What are the common controls for
touch?> We don’t know – they need to be defined> The existing controls were optimized for
mouse and keyboard over decades> Help shape the future of touch user
interfaces!
>>FUTURE
Surface HardwareWindows
VistaWindows 7
NativeWin32
Application
Touch Development Roadmap
WPF 3.5 SP1
Surface SDK1.0
Managed Wrapper and
Interop
WinForms Application
.NET 4.0 & Surface Toolkit (Q1 2010)
WPF 4.0 w/ Multi Touch APIs
Multi-Touch ControlsSurface
Multi-Touch Controls
&API
Multi-Touch API
Surface Toolkit for Windows 7
Multi-Touch Controls &
API
WPF Application
Surface Application
Related Content and ResourcesWorkshops and sessions:
> PDC09-WKSP08 – Windows 7 Developer Boot Camp
> PDC09-CL27 – Multitouch on Microsoft Surface and Windows 7 for .NET Developers
Contact us:> Forums:
http://social.msdn.microsoft.com/Forums/en-US/tabletandtouch/threads/
> Email: [email protected]
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Extra Slides
WM_GESTURE> Provides
notifications when the user performs gestures over your window
> Contains additional information like center of gesture and gesture-specific arguments
Typical sequence:WM_TABLET_QUERYSYSTEMGESTURESTAT
USWM_GESTURENOTIFYWM_GESTURE (GID_BEGIN)
WM_GESTURE (GID_PAN, GF_BEGIN)WM_GESTURE (GID_PAN, no flags)...WM_GESTURE (GID_PAN,
GF_INERTIA)...WM_GESTURE (GID_PAN, GF_END)
WM_GESTURE (GID_END)
Controlling Gesture Behavior> WM_GESTURENOTIFY – “Heads up, a
gesture is coming”
> SetGestureConfig()> Control how gestures work in your
window> Examples: 2D panning, explicitly setting
SFP flags> Can set in response to
WM_GESTURENOTIFY, at window creation, or other times
Single-Finger Panning (SFP)Enabling 2D panning
GESTURECONFIG gc = { GID_PAN, GC_PAN & GC_PAN_WITH_SINGLE_FINGER_VERTICALLY &
GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY & GC_PAN_WITH_INERTIA,
0 };
BOOL bResult = SetGestureConfig( hWnd, // window for which configuration is specified 0, // reserved, must be 0 1, // count of GESTURECONFIG structures &gc, // array of GESTURECONFIG structures
sizeof(GESTURECONFIG) // sizeof(GESTURECONFIG) );
Legacy Gesture Handling> If an app doesn’t handle
WM_GESTURE, it is treated as a gesture-unaware legacy app
> Panning and zooming fallback behaviors are provided
> If you want any default handling of gestures, make sure GID_BEGIN, GID_END, and the GID_ for the gesture in question are passed to DefWindowProc()
Real-world Questions and AnswersWM_GESTUREQ: I registered for WM_TOUCH but I no longer get
WM_GESTURE. What gives?A: Yes, WM_GESTURE and WM_TOUCH are exclusive
in Win7.
Q: I’m implementing gestures in my app, but I stopped getting panning bounce feedback. Why?
A: Two possibilities: Panning messages are being accidentally eaten, or your app is providing custom handling of them but not calling the bounce feedback API.
GetSystemMetrics()> GetSystemMetrics():
> SM_DIGITIZER returns data about available digitizers on the system
> SM_TABLETPC returns data about Tablet functionality
> SM_MAXIMUMTOUCHES yields the largest number of contacts any of the available digitizers supports
Mouse Messages & GetMessageExtraInfo()
> GetMessageExtraInfo() returns the extra info associated with a message
> Mouse up and down messages are tagged with a special signature indicating they came from touch or pen:> Mask extra info against 0xFFFFFF80> 0xFF515780 for touch, 0xFF515700 for
pen
Advanced WM_GESTURE Topics> Compound gestures:
> Zoom and Pan simultaneously: both messages contain information about both gestures
> Message bubbling> Unhandled gestures bubble up the parent
window chain> If they aren’t handled, OS legacy support kicks
in> Coalescing
> WM_GESTURE message are coalesced> Capture
> Similar to mouse, implicit capture to window on down
Advanced WM_TOUCH Topics> Primary contact
> First finger down is the “primary contact” and determines the target window for capture
> Capture> Similar to mouse, contacts are implicitly
captured on down and limited to the primary contact’s window
> Coalescing> WM_TOUCH is coalesced. Can disable
coalescing explicitly w/ a flag in RegisterTouchWindow().
Surface HardwareWindows
VistaWindows 7
NativeWin32
Application
Managed Wrapper and Interop
Touch Development Roadmap
WinForms Application
Windows 7 (2009)
Multi-Touch Controls
Multi-Touch API
WPF 3.5 SP1
WPF Application
Surface SDK1.0
Multi-Touch Controls &
API
Surface Application
>>FUTURE
Surface HardwareWindows
VistaWindows 7
NativeWin32
Application
Touch Development Roadmap
WPF 3.5 SP1
Surface SDK1.0
Managed Wrapper and
Interop
WinForms Application
.NET 4.0 & Surface Toolkit (Q1 2010)
WPF 4.0 w/ Multi Touch APIs
Multi-Touch ControlsSurface
Multi-Touch Controls
&API
Multi-Touch API
Surface Toolkit for Windows 7
Multi-Touch Controls &
API
WPF Application
Surface Application
>>FUTURE
Surface HardwareWindows 7
Windows 7
NativeWin32
Application
Touch Development Roadmap
Managed Wrapper and
Interop WPF 4.0 w/ Multi Touch APIs
Surface Toolkit
Surface SDKvNext
WinForms Application
Surface vNext (Future)
Multi-Touch ControlsSurface
Multi-Touch Controls
&API
Multi-Touch API
Surface-specific
Controls & API
WPF Application
Surface Application
YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation
forms online atMicrosoftPDC.com
Learn More On Channel 9> Expand your PDC experience through
Channel 9
> Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses
channel9.msdn.com/learnBuilt by Developers for Developers….
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.