How to Handle International Keyboards for Virtual Desktops in Cloud

Zhenjun Zhuo(zzhuo@vmware.com), Qiang Wan(qwan@vmware.com)

VMware, Product Globalization

2

Introduction

Windows Client Mac Client

Web Client

iOS Client and Android Client

International Keyboard Testing

Agent

Virtual Desktops in Cloud

Desktop VMs

3

International Keyboards

4

International Keyboard Issues

Windows 17%

Linux 5%

Mac 20%

iOS 21%

Android 13%

ChromeOS 8%

Web 16%

INTERNATIONAL KEYBOARD ISSUES ACROSS DESKTOPS/DEVICES

German 20%

French 20%

Japanese 18%

Simplified Chinese

8%

Traditional Chinese

8%

Korean 8%

Spanish 5%

Portuguese 2%

General 6%

Others 5%

INTERNATIONAL KEYBOARD ISSUES ACROSS LANGUAGES

• Above international keyboard issues are filtered from VMware bug tracking system as of July 2016 • General: general international keyboard issues are applying to all languages • Others: others includes 15 other input languages 5

Keyboard Handling Architecture Agent

Client

Keyboard

SendInput

Network

Keyboard Event Filtering

Keyboard Input Steam

Key Packets

Key Packets

6

7

Introduction

Windows Client Mac Client

Web Client

iOS Client and Android Client

International Keyboard Testing

Agent

Windows Keyboard Input

8

* https://msdn.microsoft.com/en-us/library/windows/desktop/ms646267(v=vs.85).aspx

Simulating Input: SendInput

UINT WINAPI SendInput( _In_ UINT nInputs, _In_ LPINPUT pInputs, _In_ int cbSize );

Scan code

Unicode

SendInput OR Keyboard Packets

9

* https://msdn.microsoft.com/en-us/library/windows/desktop/ms646310(v=vs.85).aspx

Language Synchronization: InstallLayoutOrTip

BOOL InstallLayoutOrTip( _In_ LPCWSTR psz, _In_ DWORD dwFlags, );

Layout Id

Lang Id

Keyboard State

Packet AND InstallLayoutO

rTip

Layout Id Lang Id

“0409” (United States) “040C” (French)

10

Remapping Keys - From US 101 to JA 106

US 101 JA 106

@ "

^ &

* (

( )

] [

\ ]

Some mismatches:

11

Remapping Keys - From US 101 to JA 106

• Add 20 key remaps in registry: ID US 101 JA 106

0 tilde Shift + @

1 Shift + tilde Shift + ^

2 Shift + 2 @

3 Shift + 6 ^

4 Shift + 7 Shift + 6

5 Shift + 8 Shift + :

6 Shift + 9 Shift + 8

7 Shift + 0 Shift + 9

8 Shift + - Shift + \

9 = Shift + -

ID US 101 JA 106

10 Shift + = Shift + ;

11 [ [

12 Shift + [ Shift + [

13 ] ]

14 Shift + ] Shift + ]

15 Shift + ; :

16 ' Shift + 7

17 Shift + ' Shift + 2

18 \ Yen/Vertical bar

19 Shift + \ Shift + Yen/Vertical bar 12

13

Introduction

Windows Client Mac Client

Web Client

iOS Client and Android Client

International Keyboard Testing

Agent

Windows Client: General Process /* WM_KEYDOWN, WM_KEYUP */ bool type = get_event_type(e); /* Windows Scan Code */ uint scancode = get_scancode(e); /* Filtering and special handling */ if(is_special(scancode, type) { /* do something for special events*/ } /* Generate and send keyboard package */ if(scancode) { r = send_keyboard_event(scancode, type); }

Following Slides: 1. Handling special events 2. Language synchronization

14

Handling Special Events: Issue

* Cannot catch KEYDOWN messages for VSCAN_HANGUL_EN and VSCAN_HANJA_EN

15

Handling Special Events: Solution

Bool sendExtraDownToGuest = FALSE; if (up) { if (vscancode == VSCAN_HANGUL_EN || vscancode == VSCAN_HANJA_EN) { sendExtraDownToGuest = TRUE; } }

16

Language Synchronization: Client Process

EnumEnabledLayoutOrTip

• langId • layoutId

• langId • layoutId

• langId • layoutId

Send to Agent

* Windows API

17

* https://msdn.microsoft.com/en-us/library/windows/desktop/bb847907(v=vs.85).aspx

18

Introduction

Windows Client Mac Client

Web Client

iOS Client and Android Client

International Keyboard Testing

Agent

Mac Client: General Process /* NSKeyDown, NSKeyUp, NSFlagsChanged */ bool type = get_event_type(e); /* Mac ADB(Apple Desktop Bus) code */ uint adb_code = get_ADBCode(e); /* Translate Mac ADB code to scan code*/ scancode = Mapping_ADBtoScan(adb_code); /* Generate and send keyboard package */ if(scancode) { r = send_keyboard_event(scancode, type); }

19

Map Mac-specific keys

20

21

Introduction

Windows Client Mac Client

Web Client

iOS Client and Android Client

International Keyboard Testing

Agent

Web Client: General Process /* keydown, keyup, keypress */ bool type = get_event_type(e); /* keyCode is used to represent a unique key * Need to handle browser difference here */ var keyCode = get_keyCode(e); /* Translate keyCode to scan code or Unicode*/ var result = Mapping_KeytoSU(keyCode); /* Generate and send keyboard package */ if(result.value) { r = send_keyboard_event(result.value, result.isunicode, type); }

Following Slides: 1. Get keyCode from different browsers 2. Example of special events

22

Keyboard Event Properties

* https://dvcs.w3.org/hg/d4e/raw-file/tip/key-event-test.html

23

Browser Difference: “a” of EN

Keydown Keyup Keypress

keyCode which keyIdentifier keyCode which keyIdentifier keyCode which keyIdentifier

Edge/20 65 65 65 65 97 97

Firefox/47.0 65 65 65 65 0 97

Chrome/51 65 65 U+0041 65 65 U+0041 97 97 U+0041

Safari/9.1.1 65 65 U+0041 65 65 U+0041 97 97

24

Browser Difference: Get keyCode var keyCode = 0, isUnicode = false, property = ‘keyCode’; if (e.keyCode) { keyCode = e.keyCode; } else if (e.which) { keyCode = e.which; property = ‘which’ ; } else if (e.keyIdentifier) { keyCode = e.keyIdentifier; property = ‘keyIdentifier’; isUnicode = true; } else { /* Special events handling */ }

25

Browser Difference: German Characters Demo

26

Browser Difference: Issue(“ü” of DE)

Keydown Keyup Keypress

keyCode which keyIdentifier keyCode which keyIdentifier keyCode which keyIdentifier

Edge/20 186 186 186 186 252 252

Firefox/47.0 0 0 0 0 0 252

Chrome/51 186 186 U+00BA 186 186 U+00BA 252 252 U+00FC

Safari/9.1.1 219 219 U+00FC 219 219 U+00FC 252 252

27

Browser Difference: Solution

if (this.keyboardLayoutId == "de-DE" &&e.keyCode == 0 && WMKS.BROWSER.isFirefox()){ keyCode = Mapping_keytokeyCode(e.key); }

• Check other properties and map to keyCode, for example: ‘key’

e.key keyCode

ü/Ü 252

ö/Ö 246

ä/Ä 228

• Mapping table for German characters:

28

29

Introduction

Windows Client Mac Client

Web Client

iOS Client and Android Client

International Keyboard Testing

Agent

iOS Client and Android Client

30

Onscreen vs. Physical Keyboards

• No modifier keys and functional keys

31

Onscreen vs. Physical Keyboards

• Long tap and popup alternate keys

32

Onscreen vs. Physical Keyboards

• Emoji characters

33

Onscreen vs. Physical Keyboards

• 10-key input for CJK phones

34

Keyboard Controller Workflow

35

Direct Input Mode

Local Input Mode

Accessory Keys

Btn onClick Send Scan

code

Key-Scancode Mapping

Send Unicode

Send Scan code

On Enter or Tap

Send Btn

On Text Change

Send Unicode

Yes No

Use Case 1: Typing Korean Characters • Typical blocks for Korean characters

• For example • Typing “한글” (Hangul)

• ㅎㅏㄴ(한)ㄱㅡㄹ(글) • Typing “가” (ga)

• ㄱㅏ(가) • Typing backspaces for “간” (gan)

• History issues • Jamos typed failed to form a Korean character • Typing backspace key, the last Korean character failed to decompose into jamos

I M I M

I M

M I M

F * From wikipedia

간 가 ㄱ

36

Use Case 1: Typing Korean

37

Use Case 2: Typing French

38

39

Introduction

Windows Client Mac Client

Web Client

iOS Client and Android Client

International Keyboard Testing

Agent

International Keyboard Testing Matrix

For Platform P = Language L = , and browser B=

Matrix M = P * L + P*L* B

Windows Linux Mac iOS Android Chrome OS

German

French

Japanese

Simplified Chinese

Traditional Chinese

Korean

IE - - - - -

Chrome Chrome Chrome - Chrome

Chrome

Firefox Firefox Firefox - - -

- - Safari Safari

- -

40

International Keyboard Test Cases

41

Automate repetitive tasks that testers slowly become less focused

2915 Overall international keyboard test cases

2889 Automated

99% Automation rate

Automation for Intl Keyboard E2E Testing

Tools Sample Function

Windows C# keybd_event

Mac Apple Script tell application "System Events" to key code {1}

Linux Python Virtkey

iOS Instruments UIAButton tap

Android UI Automator device.click(int x, int y)

42

Takeaways

• International keyboard handling process • Use cases and demos • International keyboard testing

43

Questions & Answers

44