Kinect-based virtual game for motor and cognitiverehabilitation: A pilot study for older adults

[Extended Abstract]∗

Zelai Sáenz-de-UrturiDeustoTech Institute of

TechnologyDeustoTech-Life UnitUniversity of Deusto

zelai.saenz@deusto.es

Begoña García ZapirainDeustoTech Institute of

TechnologyDeustoTech-Life UnitUniversity of Deusto

mbgarciazapi@deusto.es

Amaia Méndez ZorrillaDeustoTech Institute of

TechnologyDeustoTech-Life UnitUniversity of Deusto

amaia.mendez@deusto.es

ABSTRACTPhysical rehabilitation is often necessary for individuals whosuffer an injury or illness which causes a physical impair-ment, in order to restore movement and strength throughsupervised repetitive exercises. Alternatively, physical ac-tivity also improve cognitive performance and reduce cogni-tive decline. This tool focuses on therapeutic aspects of bothcognitive and physical rehabilitation of older adults, that is,it improves the memory by performing mental activities andphysical rehabilitation at the same time.

To achieve this, a Kinect based virtual game intended forWindows which enables users to control and interact intu-itively with the computer without an intermediary controllerhas been developed. Furthermore, all the data generatedduring the session is stored in order to log every rehabilita-tion activity.

Preliminary tests have shown an increase in the users’ moti-vation while using the tool and it assessed the possible reha-bilitation of 14 patients with motor impairments (p < 0.05)and the maintenance of their cognitive impairment avoidingits degradation.

Categories and Subject DescriptorsH.5.1 [Information Systems]: Multimedia InformationSystems—Artificial, augmented, and virtual realities; H.5.2[Information Systems]: User Interfaces—Input devicesand strategies

KeywordsMotor, cognitive, rehabilitation, virtual game, Kinect, olderadults

∗A full version of this paper is available as Author’s Guideto Preparing ACM SIG Proceedings

1. INTRODUCTIONMany people suffer an injury or illness which causes a phys-ical impairment such as structural deviations, mobility ofbone and joint functions, muscle power or movement func-tions. These physical impairments can affect every structureand function of the body (as described by the WHO) andmay hinder an individual’s ability to perform daily self-careactivities. Consequently, this reduces their participation inthe community and can significantly limit employment andeducational activities [1].

Furthermore, physical rehabilitation is often necessary forthese individuals in order to restore movement and strengththrough supervised repetitive exercises. In a standard med-ical practice in Spain, physiotherapists work with one pa-tient at a time and monitors if their physical movementsare reaching a specific standard, until the patient is ableto regain an appropriate functioning. However, the numberof exercises in a therapy session is relatively small [2]. Apossible solution to this issue is the specialized personnel in-tervention, but it entails a high economic cost. Depending ofthe type of injury or condition the patient should follow anindividual therapy regimen but the 33% of the patients donot perform the exercises as recommended. This can be dueto several factors such as the lack of motivation or access toappropriate technological facilities [3].

In addition, individuals who suffer from brain injuries fre-quently acquire cognitive impairments, or thinking prob-lems, that interfere with their safety and independence. Thetreatment method, known as cognitive rehabilitation, is de-signed to reduce cognitive dysfunction and/or assist indi-viduals in compensating for its impact on daily living. Thebenefits of cognitive rehabilitation have been discussed inmore than 700 published research studies [4].

Although larger and more detailed studies are needed, theresearch by Cay Anderson-Hanley et al. [5] suggests thatsimultaneous cognitive and physical exercise has greater po-tential for preventing cognitive decline, i.e., combining phys-ical and mental exercise playing active games could haveeven more beneficial effects on cognition in older adults thannormal exercise alone. Interactive physical and cognitive ex-ercise over traditional exercise may garner added cognitivebenefit, and perhaps prevent decline, all for the same exer-

cise effort. Due to these reasons, the system described inthis article focuses on therapeutic aspects of both cognitiveand physical rehabilitation.

Here preliminary details of the Kineage rehabilitation sys-tem are provided; a Kinect based 3D serious game intendedfor Windows which enables users to control and interact in-tuitively with the computer without an intermediary con-troller has been developed. This virtual game will facilitatethe physical rehabilitation of chronic patients and old ordisabled people using virtual reality. Furthermore, it is spe-cially oriented towards to work the cognitive stimulation inthe language area as a priority and it has been designed to beused in therapies driven by clinicians. In order to achieve therehabilitation part of the system, observations from psychol-ogists and physiotherapists were taken into consideration.Serious games are an option to provide patients’ exercisecombined with entertainment increasing their motivation [6].

2. MATERIALS AND METHODSThe proposed system uses Microsoft’s Kinect motion sensorconnected to a PC, with an integrated database and video in-structions to form the video game. Kinect is a webcam-styleadd-on for the Windows operating system, which providesa natural user interface (NUI) that allows users to interactintuitively and without any intermediary device, such as agame controller. It captures users’ full body movement in3D virtual environment for interaction within game activi-ties, that is, the user’s body is the game controller operatingin 3D virtual space and multiple users can be tracked.

It is a sensor set on a horizontal bar with a small base, to bepositioned below the video display. It has an RGB cameraand a depth sensor, which provide full-body 3D motion cap-ture capabilities. All this scans your body to identify yourheight and your movements. The tracking system of kinecthas been evaluated successfully [7].

In this work, Microsoft Kinect SDK is used; an open-sourcelibrary which can be used for testing and implementations.The Kinect sensor and its SDK provide a 3D virtual skeleton.This virtual skeleton consists of the positions of 20 joints andbody parts (such as the wrists, knees, head and torso), andfrom here a 3D virtual avatar was generated.The joints that are used during the movement are mappeddirectly into the values placed on the avatar puppet withinthe game’s virtual world.

2.1 ParticipantsFourteen users (9 women, 5 men) aged between 65 and 94with an average age of 81.28 years (SD = 8.94) were re-cruited from Misericordia, a low-income nursing home forthis study. This nursing home’s physiotherapists chose theparticipants with the following criteria: sufficient cognitivelevel to understand the game and the instructions from thephysiotherapists and as physical conditions, to have a min-imum movement in at least one of the arms. Two of thempresented a visual deficit (presbyopia- age-associated eyecondition) which presented sensitivity to light and one ofthem had age-related macular degeneration. Furthermore,two of them use wheelchairs for their daily tasks and anotherone had Parkinson’s disease.

2.2 Kineage Rehabilitation SystemFirstly, and in order to do this rehabilitation system moregeneralized, it allows to specify the typology of the user,i.e., with or without any movement in their legs (use of thewheelchair), and giving the player the option to play stand-ing or sitting. Additionally, users may present limit mobilityin either arm (even absence of absolute movement in eitherof the two members), thus being the game configured in sucha way that the user can choose if it wished to play with theleft arm, right arm or both.

At the beginning of the game, the users will be asked to entertheir name, which will be referenced in the scores, in order tostore any information related during the game play. Then,the users have to calibrate the sensor in order to check theirposition in the settings option from the main menu. Fromthis menu the user can select if he wants to play sited or notand which arm is going to play with.

For the skeletal system calibration, users must stand in frontof the Kinect sensor about 1.5-2 meters away with their armsup and hold this position without moving for a few seconds.With this, the sensor gets a correct position of the user mov-ing the device automatically up and down. This calibrationmethod is predetermined for the sensor and the commercialgames, which might not be appropriate for some users sincemight be unable to complete this procedure [8]. In this case,and in order to take the physical limitations of the users intoaccount, a set of configuration options have been designed tomake the game easier to play, such as the option of changingthe angle of the camera (allowing the user to tilt the deviceup or down to get a better image or more complete view ofthe player) only moving one arm up and down. Addition-ally, the choice of a game-mode has been set up, that is, theuser can chose how to control the game, with the left hand,the right hand, or both.

The Kineage system is divided into two sections: one ded-icated to physical rehabilitation and another aimed at thecognitive one.

2.2.1 Physical RehabilitationThe primary goal of this section is to perform physical re-habilitation exercises while playing having fun. This part ofthe game consist of three different levels in which the usershould recognize various objects appearing on the screen bymoving the arms, in order not to let the objects fall, promot-ing this way both the mobility of the user during the trainingand the cognitive process. The duration of each level can beadjusted by the physiotherapists to avoid fatigue in training.In the first level, the objects (cupcakes and bottles of wine)shall follow a vertical path. In the second level the numberof these objects increases and in the level three the objectsfollow a horizontal path. At the end of each one of the levelsthe user shall reach a piece of cake, until achieving as a finalreward a whole cake after finishing the three levels.

This game is played in 3D virtual environment where theobjects bottles of wine and cupcakes are falling to the floor.Every level has a different background picture of the en-vironment. Players can see their individual arousal scoreindicated below, as well as the time remaining. In order tocatch an object the player must do an arm movement to get

Figure 1: A user during the physical rehabilitationsession

it. If the player chooses the right hand mode to play, theobjects only will appear in the right side of the screen andvice versa. Figure 1 shows a user playing with the seriousgame at the 3rd level during the tests.

Additionally, different control parameters have been estab-lished such as the time per level, speed of the objects ap-pearing on the screen, the quantity or the path they follow,in order to do different movements with the arms. Depend-ing on the characteristics of each user these parameters canbe changed in the main menu, turning it into a slow-motiongameplay, helping users during difficult passages in it. Ifthe configured settings do not provide the right amount ofchallenge (e.g. the patient makes faster progress than ex-pected), the physiotherapist can adjust them easily for thenext session. During the rehabilitation sessions, the systemprovides feedback (collected items, scores) as well as visual,auditory and textual feedback on the patient’s performance.After the session physiotherapists and patients can take alook at the data stored in the patient’s profile to determinethe progress.

2.2.2 Cognitive RehabilitationThe main objective of this section is to improve the memoryand psychomotor activity by performing activities as wellas they do physical exercise. The system specifically on thelanguage area of the brain, which is mainly affected by peo-ple who suffer from dementia. Usually details related to re-membering specific pieces of information such as the namesof people or objects are forgotten by the patient. Due to theimportance of the language area has, these mental exercisesfocus on the reasoning from word-classification activities us-ing different semantic fields such as tools, kitchenware orfurniture.

A range of exercises have been developed following the clini-cians recommendations, in which the user must perform vari-ous physical motions in order to solve them. In this activitiesthe user must to find items which can be stored in a super-market, a stationery store or a pharmacy; or relate numberswith their corresponding definition (1-one), amongst others.Dementia is a progressive deterioration in cognitive function,so the application aims to adapt to individuals’ changingcircumstances, if they become more cognitively impaired asthey grow older, for instance.

Figure 2: A user during the cognitive rehabilitationsession

All these exercises have been designed in a graphic way, inwhich the answers are images that appear on the screen. Theobjective is to choose the correct answers (images) movingthe correct arm improving the psychomotor activity of thepatient. See Figure 2.

The application’s objective is to reinforce the mental answersby having them be drilled into the user by doing motioncontrols. The iteration with the system is based on movingtheir arms. As in the physical rehabilitation part, the gamecan be configured depending on the cognitive state of eachuser adjusting the exposure time of the picture to memorizeor the number of activities that will be performed.

2.3 ProcedurePermission to carry out the study was obtained from thehome nursing and the IRB Committee. Participants pro-vided prior informed consent written in Spanish (the par-ticipant’s natural language) and were approved by the IRBto participate in the study. The participants were informedexplicitly, precisely and unequivocally of each step in theprocess, the consequences of obtaining the data and the pur-pose of collecting it. After this, they were asked to sign theconsent form and complete a general survey form. Test-ing took place in the participants’ usual learning setting soperformance in the study would be subject to the same en-vironmental influences and distractions that prevail in thesituation for which the system is intended.

2.3.1 Experimental DesignThis game was used twice daily and a single subject multiplebaseline design was used to evaluate the physical rehabili-tation system effects with an ABAB sequence in which Ais the no-intervention baseline phase and B the interventionphase. The effect of playing this game was examined using14 subjects suffering from upper limbs impairments. Cred-ibility was enhanced through the taping and recording ofeach session and through written observations.

In relation to the cognitive rehabilitation the participants’cognitive impairment was measured before the interventionby the Mini Mental Status Examination (MMSE) ≥ 23 =Normal; 22-20 = Mild; 19-11 = Moderate; ≤ 10 = Severe.The average score for the 14 participants was 23.

Figure 3: Number of correct movements after therehabilitation program

2.3.2 BaselineDuring the baseline the physiotherapist assigned the sameexercise to each patient: raise the left and right arms morethan 130 degrees 10 times. The number of correct move-ments in each session was counted manually by the physio-therapist.

2.3.3 InterventionIn this phase patients completed the third level of the gamein which the pancakes and bottles of wine path is horizontaland objects move in the across the upper part of the screensimulating 130 degrees when raised the arms in order tocollect them. As in the baseline phase, patients performedthe exercise 10 times each one. The physiotherapist did notinterfere during the rehabilitation session with the game andthe number of correct movements was counted by the gamewhen an object was collected.

3. RESULTSFigure 3 illustrates the experimental design and the dataobtained for one subject during the physical rehabilitationsessions. The baseline and intervention phases recorded thecorrect movements. The average in the first baseline was10.4 and in the second baseline 10.3. During the interventionphases the average at the first phase was 21.1 and in thesecond one 21.4. A Kolmogorov-Smirnov test showed thatthe difference of results between de baseline and interventionphases of the correct movements was significant (p < 0.05).

Preliminary results obtained from the cognitive rehabilita-tion part after doing the activities included in the game inrelation with memory and language are encouraging evenwhen a more longitudinal experiment is needed. After per-forming these activities twice daily during 80 days, the MMSEtest showed again a result of 23 score for participants’ aver-age.

4. CONCLUSIONSThis study assessed the effectiveness of this rehabilitationsystem which might facilitate autonomous physical reha-bilitation prescribed by the physiotherapists. The resultsshowed in this paper are related to one of the patients butthe rest of the participants generated similar results. Thispilot study did not show an improvement of the cognitiveimpairment measured by the MMSE, but the impairment ismaintaining, that is, the cognitive performance is not de-clining and its progression is slow.

Preliminary tests has shown that the users motivation isincreasing while doing rehabilitation and as the experts re-ported, individuals felt more relax doing this kind of activi-ties. In addition, this tool can be adapted taken into accountthe mental and physical limitations of the users.

This system was developed in a first iteration to be played ina home nursing, but looking at the results authors think it ispossible to aim this game at people who live independentlyat their home. In a future work researcher will include amonitoring module to assess the state of their patients andtrack their progress using a web platform. In this way, thepatients’ data has not necessarily be accessed locally.

5. REFERENCES[1] A. Sears and M. Young, Physical disabilities and

computing technologies: An analysis of impairments.The human-computer interaction handbook, 2003.

[2] M. J. Lang C.E. and G. C., “Counting repetitions: anobservational study of outpatient therapy for peoplewith hemiparesis post-stroke,” J Neurol Phys Ther.,vol. 31, no. 1, pp. 3–10, 2007.

[3] M. R. Shaughnessy M., Resnick B.M., “Testing a modelof post-stroke exercise behavior.,” Rehabil Nurs.,vol. 31, no. 1, pp. 15–21, 2006.

[4] A. M. O. G. Katz, DI and S. Connors, “Cognitiverehabilitation: the evidence, funding and case foradvocacy in brain injury.,” McLean, VA: Brain InjuryAssociation of America, 2006.

[5] C. Anderson-Hanley, P. J. Arciero, A. M. Brickman,J. P. Nimon, N. Okuma, S. C. Westen, M. E. Merz,B. D. Pence, J. A. Woods, A. F. Kramer, and E. A.Zimmerman, “Exergaming and older adult cognition: Acluster randomized clinical trial.,” American Journal ofPreventive Medicine, vol. 42, no. 2, pp. 109–119, 2012.

[6] L. Derbali and C. Frasson, Players’ Motivation andEEG Waves Patterns in a Serious Game Environment.Intelligent Tutoring Systems, 2010.

[7] C.-Y. Chang, B. Lange, M. Zhang, S. Koenig,P. Requejo, N. Somboon, A. Sawchuk, and A. Rizzo,“Towards pervasive physical rehabilitation usingmicrosoft kinect,” in 6th International Conference onPervasive Computing Technologies for Healthcare(PervasiveHealth), 2012), p. 159–162, 2012.

[8] I. Pastor, H. A. Hayes, and B. SJ., “A feasibility studyof an upper limb rehabilitation system using kinect andcomputer games,” in Engineering in Medicine andBiology Society (EMBC), 2012 Annual InternationalConference of the IEEE), p. 1286–1289, IEEEComputer Society, August 28- September 1, 2012.