Abstract This paper presents a finger joints tracking-basedkeystroke detection and recognition approach for the devel-opment of virtual keyboard. Activities of user’s hands arecaptured in a video sequence for the finger joints localizationand tracking. Averaging background subtraction, connectedcomponent labeling and contour analysis are performed forhands region detection and finger joints localization. Proba-bilistic regional density-based kernel tracking estimates fin-ger joint trajectories. Estimated trajectories are further inter-preted into respective feature vectors based on spatial andpathic information. These feature vectors are fed into logic-based technique and Dynamic Bayesian Network for the clas-sification of feature vectors leading towards the detection andrecognition of keystrokes. Using actual video data set andreal-time implementation, experimentation results demon-strate accurate keystroke detection and recognition for twentyeight (28) keys. Successful research results motivate for theinclusion of non-intrusive vision-based virtual keyboard inmobile devices in addition to existing tiny keypads for facil-itation of the users.
M. H. Yousaf (B) · H. A. HabibVideo and Image Processing Laboratory, Departmentof Computer Engineering, University of Engineering and Technology,Taxila, Pakistane-mail: [email protected]
Technology advancements in the past decade have massivelyincreased the usage of traditional (computers, laptops) andnon-traditional (cell phones, PDAs, smart phones, iPhone,etc.) computing devices. To handle the portability and mobil-ity features, the size of non-traditional devices is reduced. Asa result, data input units of these devices become smaller insize, so textual input to mobile devices turns out to be atedious, hectic and cumbersome task. Whereas, technologyevolution intend to facilitate users with reduced system intri-cacy. QWERTY keyboard for computing devices are easierfor use and became de facto standard for textual input to thesedevices.
To overcome the tiny keypad issues, virtual keyboardsare designed for interaction with new generation of mobiledevices and smart phones. Researchers focused on devel-opment of new interaction and textual input possibilities tothese devices. New sensors and new software methodolo-
123
Arab J Sci Eng
gies are proposed to solve this research problem. Virtualkeyboards are based on either sensor-based designs or vision-based systems. Objective of these systems are to detectkeystrokes and reveal the key pressed. The designs of vir-tual keyboards vary in layouts, number of keys and fingers,finger-to-key mapping, finger-key switch time, user famil-iarity, feedback mechanism, estimated bandwidth and visualappearance/incarnation of keyboard.
Finger Joint Gesture Wearable Keypad [1] used the pha-langes of one hand to be act as the mini virtual keyboard.However, it contained less number of discrete keys and lackof feedback. Chording Glove [2] employs the pressure sen-sor on the fingers in a glove for sensor-based input device.Accelerometers were deployed on fingers to detect the sur-face impact in FingerRing [3]. Chording Glove and Finger-Ring did not catch user attention due to absence of visualincarnation and less number of discrete keys. In VType [4],user wears a data glove. This data glove detects keystrokesfor each finger in the air based upon curvature angle of opti-cal fiber in glove. Samsung Scurry [5] also implemented asensor-based approach for the keystroke recognition usingtiny gyroscopes. VType and Scurry are good at discrete num-ber of keys and operator–key mapping, but are not user-friendly and have high cost. The Senseboard [6] consists oftwo rubber pads that fit on the user’s hands. Muscle move-ments in the palm were detected (with unspecified, non-invasive means) and translated into keystrokes using patternrecognition. Visual panel [7] implemented computer visionalgorithm to find the location of index finger on a sheet ofpaper on which keyboard keys are printed. Visual Panel hasfixed visual incarnation of keyboard and less bandwidth of 20in characters/min. VKey [8] and VKB [9] both consists of atable top unit having a projector. Both projects a virtual key-board on a flat surface using laser technology and interactionbetween the projected image and user’s hand is recognizedby means of an IR camera. But VKey and VKB users cannotrest their hands on the projected keyboard image by keepingthe fingers on it. However, VKey and VKB were not costeffective due to the cost of the projector involved.
In addition to the methods and devices mentioned above,computer vision algorithms were also proposed and exper-imented for the detection and recognition of keystrokes forvirtual keyboard development. Wijewantha et al. [10] pre-sented a cost-effective, user-friendly, portable solution basedon image capturing, character identification and device emu-lation modules named as VistaKey, but VistaKey introducedan intrusive mechanism by using a light pen instead of fin-gers. Du et al. [11] proposed a 3D optical ranging-based vir-tual keyboard; system consists of pattern projector and 3Drange camera detecting the fingertips by depth curves. Du andCharbon [12] further extended the research work presented in[11] by constructing 3D hand shape model. Lee and Woo [13]presented an Augmented Reality Keyboard (ARKB) which
uses fiducially colored markers to augment the virtual key-board through vision-based interface. But solutions availabletill now did not satisfy key principles of design, i.e., learn-ability, flexibility and robustness as well as are not so muchcost-effective and fully integrated with every platform.
Virtual interaction with systems have also been widelyexperimented by deploying human activity recognition-based approaches in modern human–computer interactioninterfaces. A variety of computer vision applications arebased on human activity recognition, e.g., sign languagerecognition [14], human machine interfacing using hand ges-tures [15], camera mouse control [16], controlling the auto-motive’s secondary tasks using gestures [17] and robotic con-trol [18], etc. A brief description of our research work onmono-vision-based virtual keyboard using gesture recogni-tion and fuzzy-logic is presented in [19].
This research work proposes a real-time finger jointstracking-based approach for keystroke detection and recog-nition. A flat surface is used as the workspace for the key-stroke recognition replacing the physical keypads. Figure 1shows the comparison between traditional keypads and fingerjoints tracking-based virtual keyboard. Finger joints move-ments are analyzed using computer vision algorithms todetect and recognize the keystrokes made by the fingers.Proposed research work provides significant amount of newalgorithmic and practical improvization in previous researchwork [19] in all three major parts of system, i.e., fingerjoints detection, trajectory estimation, Keystroke detectionand recognition, which have been indicated in subsequentphases. Detailed architecture of proposed scheme is demon-strated in Sect. 2.
2 Architecture of Proposed Scheme
The architecture of the proposed scheme is modular as theoutput of current phase becomes input to the next phase. TheInitialization phase defines a workspace where movements ofhuman hands are captured using camera. The system definesa naming terminology for finger joints. Also, a finger-to-key mapping structure is proposed to associate fingers tothe keys. These will act as foundation for the subsequentprocesses. Finger joints detection and trajectory estimationphase is proposed for finger joints localization and for estab-lishing correct inter-frame correspondence between fingerjoints. This module results in spatiotemporal trajectories ofall finger joints. Later, in keystroke detection and recognitionphase,feature vectors are computed for these spatiotemporaltrajectories. Keystrokes are detected by classifier for thesefeatures and the key pressed is revealed.
Proposed system has real-time processing capabilitiesusing video data of 30 frames per second. Therefore, theentire sequence of steps with exception of initialization up
123
Arab J Sci Eng
Fig. 1 Comparison betweentraditional keypads and fingerjoints tracking-based virtualkeyboard
to finger joints detection is performed on a frame-by-framebasis. The following subsections describe the algorithmsinvolved in each sequence of operation of the system.
2.1 Initialization
The proposed research work assumes that users type on aflat surface to enter textual data into the mobile devices andsmart phones. The proposed system assumes the absence ofany physical keyboard. Few experimental and environmentalconstraints are considered to ensure hand and finger locationsfor proper finger to key mapping scheme. Localization andnaming of finger joints is carried out using hand skeletonmodel [20]. Joints in the human hands are named accordingto their location on the hand as metacarpophalangeal (MCP)(i.e., joining fingers to the palm), interphalangeal (IP) (i.e.,joining finger segments) and carpometacarpal (CMC) (i.e.,connecting the metacarpal bones to the wrist). As it is clearfrom the skeleton model, keystrokes generation is only con-cerned with interphalangeal (IP), because metacarpal andcarpometacarpal bones do not have major impact in key-stroke generation so MCP and CMC are ignored. Keeping inview the hand skeleton model, hands are modeled in termsof fingers and finger joints as shown in Fig. 2. Fingers arenamed as pinky, ring, middle, and index. Whereas, for con-venience each fingertip is named as JD (distal joint of finger),joint between fingertip and middle of the finger as JM (mid-dle joint of finger) and joint at the middle of the finger as JP
(proximal joint of finger). Summing up the naming of fingersand joints, eight fingers are nominated as operators involvedin the keystrokes generation process
F = {F1, F2, . . . , FL} where L = 8
and three joints associated with each finger resulting in 24joints.
Ji = {JiD, JiM, JiP} where i = 1, 2, 3, . . . , L
Fig. 2 Naming conventions for fingers and joints
Keeping record of trajectories for 24 finger joints insteadof 38 finger joints [19] not only reduces the memory com-plexity, but also impacts on computational complexity in real-time. Total twenty eight (28) keys including twenty six (26)alphabets, space and dot (.) are considered in our proposedvirtual keyboard system. Output of the proposed keyboard isdefined as
C = {C1, C2, . . . , CK } where K = 28
For finger-to-key mapping scheme, standard typing pat-tern is considered for the positioning of hands on the flatsurface as shown in Fig. 3. In QWERTY keyboard user posi-tioned the four fingers of the left hand on A, S, D and F andfour fingers of right hand on J, K, L and ; keys. But for textualinput to mobile devices, right hand fingers are proposed on H,J, K and L keys. A finger-to-key mapping table is proposedas in Table 1 showing the possibility of key pressed by eachfinger.
123
Arab J Sci Eng
Fig. 3 Proposed typing scheme
Table 1 Finger-to-key mapping table
Finger ID Finger Possible keys
F1 L_Pinky Q, A, ZF2 L_Ring W, S, XF3 L_Middle E, D, CF4 L_Index R, F, V, T, GF5 R_Index Y, H, N, BF6 R_Middle U, J, spaceF7 R_Ring I, K, MF8 R_Pinky O, L, P
2.2 Finger Joints Detection and Trajectory Estimation
Prior to developing and deploying a product for mobiledevices and smart phones, the research work is tested overan experimental setup in the laboratory. Video sequence ofmovements of human hands on a flat surface without anyphysical keyboard or layout is captured using simple CCDcamera. Figure 4 describes the detailed algorithm for thedetection of finger joints and estimation of their respectivetrajectories.
Video sequence consists of 30 frames per second; frameextraction is required on regular intervals for analysis ofeach frame. Frame extraction is followed by some pre-processing steps for hand detection and fingers isolation.In previous work [19], pre-processing, i.e., hand and fin-ger joint detection was carried out by implementing tech-niques proposed by Caglar et al. [21] and Kolsch et al. [22].However, it increases the computational and time complex-ity. Hand detection techniques have been used for activityrecognition in [15,16,23,24], which include different skindetection and pattern recognition-based techniques. Thesetechniques are quite useful for complex and dynamic back-grounds. But assuming the simple and static background ofworkspace in our proposed system, these techniques wouldcause to increase the complexity of the system. Therefore,after frame extraction from the video sequence, hand detec-tion is performed using background subtraction proposed byJean and Albu in [25] for detection of feet. Hand detection
Fig. 4 Algorithm for fingers joints detection and trajectories estima-tion
scheme computes the background statistics when it is static,i.e., in the absence of hands in the workspace. To computethe robust background statistics, video of 2–3 s with staticbackground is sufficient. First and second-order backgroundstatistics are computed as fB,μ and fB,σ 2 , respectively,in Eqs. (1) and (2):
123
Arab J Sci Eng
fB,μ(x, y, c) = 1/NN∑
t=1
[ ft (x, y, c)] (1)
fB,σ 2(x, y, c) = 1/N − 1N∑
t=1
[ ft (x, y, c) − fB,μ(x, y, c)]2
(2)
where ft (x, y, c) is the color image (c represents color com-ponents) corresponding to frame at time t and x, y shows thespatial coordinates of the corresponding frame. N is the totalnumber of frames considered for the computation of back-ground statistics. fB,μ(x, y, c) and fB,σ 2(x, y, c) are meanand variance images, respectively. Background statistics inEqs. (1) and (2) are used to calculate the quadratic difference.Quadratic difference is used in the generation of foregroundimage which indicates hands region in the workspace.
fI,t (x, y) ={
1 if f�,t (x, y) ≥ τb
0 otherwise(3)
f�,t (x, y) =∑3
c=1 [ ft (x, y, c) − fB,μ(x, y, c)]2
∑3c=1 fB,σ 2(x, y, c)
(4)
where fI,t (x, y) is the foreground image (hands image)showing hands region and τb is user-defined threshold.f�,t (x, y) is the quadratic difference which tells the differ-ence of intensity at (x, y) from the average background inten-sity. Figure 5a shows a sample frame extracted from the videosequence focusing workspace with hands and Fig. 5b showsthe resultant binary image of hands region detected from thesample frame using Eqs. (3) and (4). To remove the ambi-guity of other dynamic objects in the workspace, regionalfeatures using connected component labeling scheme [26]can be examined for hands detection. Connected componentlabeling also resulted in identification of left and right hand.Hands detection is followed by finger isolation and jointslocalization.
Hand contours are extracted from the binary image. Con-tour tracking is applied to detect the fingertips and valleys(isolation points) between fingers by using the fact that bothsides of the valleys go upwards while both sides of finger-tips go downwards. Contour tracking results in the detectionof eight fingertips {ft1, ft2, ft3, ft4, ft5, ft6, ft7, ft8} and eightisolation points {ip12, ip23, ip34, ip4, ip5, ip56, ip67, ip78}.Other two isolation points {ip1, ip8} are interpolated using{ip12, ip23} and {ip67, ip78}, respectively. Figure 5c showsthe detected fingertips and isolation points for both hands.Fingertips represent the one-end of the fingers, to locatethe other end of the fingers, i.e., metacarpal point (bone),respective isolation points are used. (x, y) coordinates ofmetacarpal points (M) are computed using Eqs. (5) and (6).
xMi = 0.5 ∗ (xipa+ xipb
) − (xfti − 0.5
∗(xipa+ xipb
)) ∗ 0.4 (5)
yMi = 0.5 ∗ (yipa+ yipb
) (6)
Mi represents the metacarpal point for i th finger. Whereasfti , ipa, ipb are respective fingertip and isolation points for i thfinger. For example, (x, y) coordinates of metacarpal point(M6) for middle finger of right hand (F6) can be computedusing Eqs. (5) and (6) as:
xM6 = 0.5 ∗ (xip56+ xip67
)
−(xip6− 0.5 ∗ (xip56
+ xip67)) ∗ 0.4
yM6 = 0.5 ∗ (yip56+ yip67
)
Multiplying factor of 0.4 is used to overcome the infor-mation loss due to conversion of 3D real-world data into2D image data. By applying Eqs. (5) and (6) on fin-gertips {ft1, ft2, ft3, ft4, ft5, ft6, ft7, ft8} and isolation points{ip1, ip12, ip23, ip34, ip4, ip5, ip56, ip67, ip78, ip8}, metacar-pal points for each finger are estimated as {M1, M2, M3, M4,
M5, M6, M7, M8}.Having a set of coordinates for eight fingertips and
metacarpal points, finger representation is carried out bydrawing a straight line between fingertips and metacarpalpoints using two-point form of equation of line where slopem = (y2−y1)/(x2−x1). In other words, straight line betweenfti and Mi represents the finger. Generalized form of fingerrepresentation through straight lines is defined as:
lim(x,y)=(xfti , yfti )→(xMi ,yMi )
(y − yfti =
(yMi − yfti )
(xMi − xfti )(x−xfti )
)
Figure 5d shows the metacarpal point and finger representa-tion through straight line for fingers F4 and F6. Let npi betotal number of pixels representing i th finger. Segmentationof fingers into phalangeals according to hand anatomy resultsin detection of joints. Distal joint JD lies at the upper neigh-borhood of fingertips, middle joint JM at 30 % of npi andproximal joint JP at 65 % of npi as shown in Fig. 5d. Pro-posed joints detection approach returns the joints positionsin spatial coordinates of hands image. Detected set of jointsare represented through matrix J as:
J =
⎡
⎢⎢⎢⎣
J1D J1M J1P
J2D J2M J2P...
......
JLD JLM JLP
⎤
⎥⎥⎥⎦ where L = 8
Each joint is localized through a bounding box having jointslocation as a centroid. Figure 5e shows the joints localizationalong with coordinates and bounding box on the hands. Kine-matic model of hands can also be constructed in terms fingersF1 to F8 and joints Jrm P , JM and JD as shown in Fig. 5f. Pro-posed simplified approach of finger joints detection results
123
Arab J Sci Eng
Fig. 5 a Sample frame showing workspace; b hands detection using Eqs. (3) and (4); c fingertip (ft) and isolation points (ip) detection usingcontour analysis; d finger isolation and joints detection using, respective, ft and ip; e finger joints localization; f kinematic model for sample frame
in reduction of computation as well as time complexity forkeystroke recognition.
Joints detection phase returns the objects to be tracked inthe subsequent frames. Localized kernel tracking based onprobabilistic densities of joints regions [27] is implementedfor trajectories estimation of finger joints. For each finger F ,a set of trajectories comprising three trajectories representingJD, JM and JP are recorded. Recording the positions of i thfinger joints for N number of frames is defined as:
⎡
⎢⎣TiD
TiM
TiP
⎤
⎥⎦ =⎡
⎢⎣TiD(1)TiD(2)TiD(3) · · · TiD(N )
TiM(1)TiM(2)TiM(3) · · · TiM(N )
TiP(1)TiP(2)TiP(3) · · · TiP(N )
⎤
⎥⎦
where i = 1, 2, 3, . . . , 8.
Finger joints trajectories are examined continuously invertical direction except for index fingers (F4 and F5) andright pinky (F8). They also require trajectories estimation inhorizontal direction, because according to Table 1, index fin-
123
Arab J Sci Eng
Fig. 6 Trajectories of indexfinger of right hand (F5) invertical direction JD, JM and JPin red, green and blue,respectively)
gers (F4 and F5) and right pinky (F8) have the probabilityfor keystroke generation in both horizontal and vertical direc-tion. Figure 6 shows the trajectories of index finger of righthand (F5) in vertical direction recorded for twenty five (25)seconds. JD, JM and JP trajectories are represented usingcolors red, green and blue, respectively. Similarly, a com-plete set of trajectories representing trajectories of all jointscan be recorded and represented as follows:
T =
⎡
⎢⎢⎢⎣
T1D T1M T1P
T2D T2M T2P...
......
TLD TLM TLP
⎤
⎥⎥⎥⎦ where L = 8.
2.3 Keystroke Detection and Recognition
Trajectories estimation leads towards the keystroke detectionand recognition module. Keystroke detection phase predictsthe occurrence of any key-press event. Whereas, recognitionphase classifies the possible character revealed in case ofany key-press event. Parallel down-up sequences in trajec-tories of finger joints for particular time duration indicatesthe occurrence of any keystroke. Considering the typing lay-out and finger-to-key mapping table (Table 1), each fingeroperates three number of keys, when moving in the verti-cal direction, thus localizing the movements of three jointsof each finger accordingly. Keeping in mind the above sce-nario, Fig. 7 describes the proposed algorithm for trainingand classification of keystroke detection and recognition.
For keystroke detection and recognition, all trajectories forF1–F8 are interpreted into feature vectors in real-time, whichcomprise the spatial, pathic and probabilistic information oftrajectories. Feature vector of j th joint of i th finger for knumber of samples is defined as V k
i j = [ V 1i j V 2
i j V 3i j ], where
j represents the joint type {JD/JM/JP}. However, three com-ponents of feature vectors are computed using Eqs. (7), (8)and (9):
V 1i j =
k∑
t=1
(Ti j (t) − Ti j (t − 1)) (7)
Fig. 7 Algorithm for keystroke detection and recognition
V 2i j =
k∑
t=1
(δd(t) − δu(t)) (8)
v3i j =
⎧⎪⎪⎪⎨
⎪⎪⎪⎩
1 if∑k/2
t=1 δd(t) > 0 &∑k/2
t=1 δu(t) > 0
−1 if∑k/2
t=1 δd(t) > 0 &∑k/2
t=1 δu(t) > 0
0 otherwise
(9)
where
δd(t) ={
1 if(Ti j (t) − Ti j (t − 1)
)> 0
0 otherwise
where
δu(t) ={
1 if(Ti j (t) − Ti j (t − 1)
)< 0
0 otherwise.
123
Arab J Sci Eng
Fig. 8 Keystroke detection by analyzing trajectory of JP
After applying Eqs. (7), (8) and (9), feature vectors areextracted for all joints over k number of samples as follows:
V k =
⎡
⎢⎢⎢⎣
V k1D V k
1M V k1P
V k2D V k
2M V k2P
......
...
V kLD V k
LM V kLP
⎤
⎥⎥⎥⎦ where L = 8.
Feature vectors are computed for vertical directional changesin trajectories of all finger joints. However, for F4, F5 and F8,horizontal change in trajectories is also monitored to extractthe feature vectors. To reduce the real-time computationalcomplexity, the concept of dominant finger is introduced forkeystroke detection. Dominant finger is decided on basis ofproximal joint JP for each finger. From a human percep-tion, pink diamond signs on the down-up sequences of bluecolor trajectory (JP) in Fig. 8 show the sample key-pressevents. However, yellow line shows the duration for respec-tive key-press events. Feature vectors are tested using logic-based technique over JP of all fingers resulting in possibledetection of keystroke by respective finger.
Identification of dominant finger proceeds to the recogni-tion phase to classify the key pressed by the user. Fuzzy logicclassification of trajectory data set in [19] affects the real-time system performance in terms of robustness and band-width, i.e., characters per minute, which is overcome hereby interpreting the trajectories into feature vectors and clas-sification of feature vectors by Dynamic Bayesian network(DBN) [24,28]. DBN is known as one of the most expressivestate-space models for recognizing various types of sequen-tial data, including speech and gestures. For keystroke recog-nition, a particular form of DBN is proposed in Fig. 9. Eachkey-press event can be generated by any of the respectivefinger, where each finger is represented using joints features.Extracted information of [ v1 v2 v3 ] for JD, JM and JP for adominant finger is tested on the dynamic Bayesian networkto recognize the particular keystroke generated by the user.
In the proposed DBN of Fig. 9, hidden state Ht isrepresented in terms of random variables; each of ran-dom variable denotes a particular key. Three observationsJ 1:t
P , J 1:tM and J 1:t
D are represented in terms of random vari-ables denoting the probabilities of respective finger jointsJP, JM and JD, respectively. Probability distribution of hid-den nodes JP, JM and JD can be formulated as follows:
P(
J t |O1:t1 , O1:t
2 , O1:t3
)= P
(J t , O1:t
1 , O1:t2 , O1:t
3
)
p(O1:t
1 , O1:t2 , O1:t
3
) (10)
where Ot1, Ot
2, Ot3 are observation nodes implying the obser-
vations of features[v1 v2 v3
]of respective finger joint J t .
Given these three observations J 1:tP , J 1:t
M and J 1:tD , the mar-
ginal probability of the hidden state Ht needs to be calcu-lated for the recognition of particular keystroke. Forwardsalgorithm is applied to calculate the probability P(Ht =j |J 1:t
In other words, product of state transition probabilityP(Ht |Ht−1), previous forward variable P(Ht−1|J 1:t−1
P ,
J 1:t−1M , J 1:t−1
D ) and conditional likelihood of the observationsP(J t
P, J tM, J t
D|Ht ) represents the forward variable P(Ht =CK |J 1:t
P , J 1:tM , J 1:t
D ). Whereas, variable γ is formulated as:
γ t (CK ) = P(Ht = CK |J 1:t
P , J 1:tM , J 1:t
D
)[∑K
CK=1P
(Ht = CK |J 1:t
P , J 1:tM , J 1:t
D
)] (12)
where K represents the total number of keys associated withrespective dominant finger as per Table 1. Keystroke asso-ciated with given observations J 1:t
P , J 1:tM and J 1:t
D can berecognized as C∗
K using Eq. (13).
C∗K = max
1≤CK ≤K
[γ t (CK )
]for 1 ≤ t ≤ T (13)
EM (expectation maximization) algorithm is implementedfor parameter estimation involved in the training of dynamicBayesian network. Dividing the keystroke recognition phaseinto two steps, i.e., dominant finger detection and then respec-tive keystroke recognition reduces the computational com-plexity of the recognition phase.
3 Experimentation Results and Discussions
Video dataset is developed for real-time finger tracking-based keystroke detection and recognition. Total thirty (30)videos of thirty (30) different undergraduate students wererecorded in the established laboratory setup. To achieve accu-rate keystroke recognition results on real-time videos, train-ing video data set was gathered using color markers on fingerjoints. Twenty five (25) videos classifiers are trained and theremaining five (5) are used for testing purpose by threefoldcross-validation. Finger joints localization for these train-ing videos was achieved through simple approach of color-based blob segmentation to achieve maximal recognitionaccuracy.
Proposed system is tested on ten new users typing indefined layout scheme on a flat surface in natural way with-out any kind of color markers. During real-time evaluation,it is obvious that accurate detection and localization of fingerjoints is the key to success of the proposed system. Certainconstraints have to be considered for finger joints detectionscheme as explained in Sect. 2.2 during real-time execution.These constraints includes distance between capturing device
and hands, height of the capturing device, angle between cap-turing device and hands, orientation of hands during initialphase of finger joints localization and non-cluttered environ-ment conditions. Hand detection accuracy is affected due tocluttered background and non-uniform illumination. Clut-tered background can cause false detection of hand’s area.Hand detection scheme was simplified to reduce the compu-tational complexity. User with closed palms or merged fin-gers during finger joints detection phase will affect the robust-ness of finger joints detection. Because the proposed algo-rithm completely relied on isolated fingers, non-existence ofisolated fingers causes the failure in the detection of fingerjoints. Therefore, during hands and finger joints detectionphase, a recursive check has been introduced until eight (8)fingers and twenty four (24) joints have been localized asper hand skeleton model of Fig. 2. System will not acquirejoints trajectories until it have all the joints properly detectedand localized. Computational complexity of the finger jointslocalization process is low with respect to time and memory,but system takes almost 3–4 s initially for joints localization.Keeping in view the imposed constraints, hand detection andfinger joints localization was performed accurately.
After joints localization, trajectories of joints are esti-mated and interpreted into feature vectors continuously onreal-time. Figure 10 shows the trajectories generated by themiddle and ring fingers F2, F3, F6 and F7 for almost 25 s.Feature vectors rely mainly on derivative and summation cal-culations iteratively instead of large calculations, so involvesless computational complexity. Trajectories of Fig. 10 areclassified using the classification scheme of Fig. 7 and fre-quency of each key/characters pressed by each of these fin-gers is displayed in Fig. 11.
Keyboard users were categorized into three levels basedon typing in standard layout and speed, i.e., Beginner, Inter-mediate, Expert. During evaluation of proposed scheme,only intermediate and expert users were considered becausebeginners usually made lot of mistakes in typing in stan-dard layout scheme. Proposed virtual keyboard scheme’sperformance evaluation is carried out in two major phases.First, keystroke detection which determines the occurrenceof any key-press event; secondly, recognition of pressed-keyin case of detection of any keystroke. As keystroke detectionmainly relies on the feature vector obtained from JP of allfingers, which does not comprise any complex calculationsand analysis so that keystroke detection results were foundto be 90–95 % accurate for both types of users.
As far as keystroke recognition phase is concerned, per-formance is evaluated with respect to users and fingers. Key-stroke recognition results are found to be accurate in up to92 % for both kinds of users. Keystroke recognition resultsare not affected by the intermediate kind of user as much,because it mainly impacts the bandwidth of the system. Accu-racy of the proposed system is affected due to extra key to
123
Arab J Sci Eng
Fig. 10 Trajectories of middleand ring fingers of both hands(F2, F3, F6 and F7)
finger ratio of index fingers (F4, F5) and right pinky (F8).Because F4, F5 and F8 has key to finger ratio of 5:1, 4:1and 4:1, respectively, which causes problem in determin-ing accurate position of finger along the horizontal direction,thus leading to the false recognition of keystrokes. Remain-
ing fingers have a uniform key to finger ratio of 3:1, whichdoes not create any ambiguity for the recognition phase. Key-stroke recognition rate per finger for the test users is shownin Fig. 12. Recognition rate per finger in Fig. 12 portrays theeffect of extra key to finger ratio of F4, F5 and F8 which
123
Arab J Sci Eng
Fig. 11 Keystrokes recognitionresults for trajectories in Fig. 10
Fig. 12 % Keystrokesrecognition results for eachfinger
reduces the recognition rate to 85, 89 and 87 %, respectively.F1 has the highest recognition rate due to the left-most opera-tor in the field of view bearing less chances of mixing up withother fingers. Remaining (F2, F3, F6 and F7) have almostsame recognition rate.
Experimentation results are also evaluated on the basis ofkey parameters regarding development of virtual keyboard,i.e., number of discrete keys and fingers, estimated band-width, user familiarity and finger-key switch time. Proposedsystem is evaluated using 28 discrete keys, 8 operators andbandwidth of 4 keys/s. Since at a standard typing speed, akeyboard comprises the bandwidth of 250 implies 250 char-acters/min, hence the average typing speed is 4 characters/s.
4 Conclusions and Future Work
Research work proposed a real-time system for the key-strokes detection and recognition using finger joints tracking.Experimentation results showed a reliable, robust and prac-tical system. The system is software centric and is a low-cost
system in comparison to hardware-based systems. Users hav-ing the typing capability in a standard layout scheme wereemployed to prepare the video data set. System addressed theimportant issues in the development of the virtual keyboard,e.g., cost, user-friendliness, finger-key mapping and band-width. System results in a good user acceptance rate in termsof ease, ergonomics, non-intrusiveness and bandwidth.
Finger joints tracking can be used in other human–computer interaction-based application, e.g., gesture recog-nition. In future, extension is proposed in the research workto increase the efficiency using enhancement in the track-ing algorithm and by increasing the number of discrete keysfrom 28 to full size keyboard. The proposed algorithm willbe implemented for recent mobile and smart phone operatingsystems for testing and fine-tuning of algorithm. Similarly,system will be tested for different tones of skin, differenttyping styles and different distance operation for user.
Acknowledgments This research work is supported and funded byDirectorate of Advance Studies, Research and Technological Devel-opment, University of Engineering and Technology, Taxila, Pakistan.Authors are also thankful to postgraduate research fellows affiliated
123
Arab J Sci Eng
with the video and image processing laboratory for their suggestionsand support in this research work. Authors are also thankful to the stu-dents and peers involved in the video dataset collection for the project.
References
1. Goldstein, M.; Chincholle, D.: Finger-joint gesture wearable key-pad.In: 2nd Workshop on Human Computer Interaction withMobile Devices, pp. 9–18 (1999)
2. Rosenberg, R.: Computing without mice and keyboards: text andgraphic input devices for mobile computing. PhD thesis, Depart-ment of Computer Science, University College, London (1998)
3. Fukumoto, M.; Tonomura, Y.: Body coupled FingeRing: wirelesswearable keyboard. In: ACM Conference on Human Factors inComputing Systems (1997)
4. Evans, F.; Skiena, S.; Varshney, A.: VType: entering text in a virtualworld. Submitted to Int. J. Hum.–Comput. Stud.
5. Samsung Scurry: PCWorld article at http://www.pcworld.com/article/70568/virtual_keyboards_let_you_type_in_air.html
6. Senseboard, Method for virtual input to mobile devices: http://www.senseboard.com (2011). Accessed 15 Dec 2011
7. Zhang, Z.; Wu, Y.; Shan, Y.; Shafer, S.: Visual panel: virtual mouse,keyboard and 3D controller with an ordinary piece of paper. In:ACM Workshop on Perceptive User Interfaces, November 2001
9. i-Tech’s Virtual Keyboard at http://www.vkb-support.com10. Nadeeka Samanthi, W.; Chanaka, A.: VISTAKEY: a keyboard
without a keyboard—a type of virtual keyboard. In: IEE EleventhAnnual Conference, Sri Lanka (2004)
11. Huan, D.; Thierry, O.; Felix, L.: A virtual keyboard based on true-3D optical ranging. In: British Machine Vision Conference (2005)
12. Huan, D.; Edoardo, C.: A virtual keyboard system based on multi-level feature matching. In: International Conference on HumanSystems Interaction (2008)
13. Lee, M.; Woo, W.: ARKB: 3D vision-based augmented reality key-board. In: International Conference on Artificial Reality and Telex-isitence, pp. 54–57 (2003)
14. Bowden, R.; Zisserman, A.; Kadir, T.; Brady, M.: Vision basedinterpretation of natural sign languages. In: Proceedings of 3rdInternational Conference Computer Vision Systems, Graz, Austria,pp. 391–401, April (2003)
15. Yang, L.; Yunde, L.: A robust hand tracking and gesture recognitionmethod for wearable virtual interfaces and its applications. In: 3rdInternational Conference on Image and Graphics (ICIG’04) (2004)
16. Luca, L.; Marco, P.: Adding gestures to ordinary mouse use: anew input modality for improved human–computer interaction. In:14th International Conference on Image Analysis and Processing(ICIAP 2007)
17. Micah, A.; Katie, M.: Developing a car gesture interface for useas a secondary task. In: ACM Conference on Computer HumanInteraction, pp. 32–33 (2003)
18. Xiaoming, Y.; Ming, X.: Finger identification and hand posturerecognition for human–robot interaction. Image Vis. Comput. 25,1291–1300 (2007)
19. Habib, H.A.; Mufti, M.: Real time mono vision gesture based vir-tual keyboard system. IEEE Trans. Consum. Electron. 52(4) (2006)
20. Erol, A.; Bebis, G.; Nicolescu, M.; Boyle, R.D.; Twombly, X.:Vision based hand pose estimation: a review. Comput. Vis. ImageUnderst. 108, 52–73 (2007)
21. Caglar, M.B.; Lobo, N.: Open hand detection in a cluttered sin-gle imageusing finger primitives. In: Computer Vision and PatternRecognition Workshop (CVPRW’06) (2006)
22. Kolsch, M.; Turk, M.: Robust hand detection. In: 6th IEEE Inter-national Conference on Automatic Face and Gesture Recognition(FGR’04) (2004)
23. Dente, E.; Bharath, A.A.; Ng, J.; Vrij, A.; Mann, S.; Bull, A.:Tracking hand and finger movements for behaviour analysis. Pat-tern Recognit. Lett. 27, 1797–1808 (2006)
25. Jean, F.; Albu, A.B.: The visual keyboard: real-time feet trackingfor the control of musical meta-instruments. J. Signal Process.:Image Commun. 23, 505– 515 (2008)
26. Chang, F.; Chen, C.-J.; Lu, C.-J.: A linear-time component-labelingalgorithm using contour tracing technique. Comput. Vis. ImageUnderst. 93(2), 206–220 (2004)
