Journal of Ambient Intelligence and Humanized Computing(September 2012, Volume 3, Issue 3, pp 205-218)

Smart Homes for the Elderly Dementia Sufferers:Identification and Prediction of Abnormal Behaviour

Ahmad Lotfi · Caroline Langensiepen ·Sawsan M. Mahmoud · M. J. Akhlaghinia

Postprint

Abstract In this paper we have described a solution for supporting independent liv-

ing of the elderly by means of equipping their home with a simple sensor network

to monitor their behaviour. Standard home automation sensors including movement

sensors and door entry point sensors are used. By monitoring the sensor data, im-

portant information regarding any anomalous behaviour will be identified. Different

ways of visualizing large sensor data sets and representing them in a format suitable

for clustering the abnormalities are also investigated. In the latter part of this paper,

recurrent neural networks are used to predict the future values of the activities for

each sensor. The predicted values are used to inform the caregiver in case anomalous

behaviour is predicted in the near future. Data collection, classification and predic-

tion are investigated in real home environments with elderly occupants suffering from

dementia.

Keywords Smart home · Dementia · Alzheimer · Assistive technology · Prediction ·Abnormality detection · Time series · Sensor network · Intelligent environment

Ahmad Lotfi · Caroline Langensiepen · Sawsan M. Mahmoud

School of Science and TechnologyNottingham Trent UniversityClifton Lane, Nottingham, NG11 8NSUnited KingdomTel:+44 115 8488390E-mail: ahmad.lotfi@ntu.ac.uk

Caroline LangensiepenE-mail: caroline.langensiepen@ntu.ac.uk·Sawsan M. MahmoudE-mail: sawsan.mahmoud@ntu.ac.uk·MJ AkhlaghiniaCentre for Innovation and Technology ExploitationNottingham Trent UniversityE-mail: mohamadjavad.akhlaghinia@ntu.ac.uk

2

1 Introduction

The European Union considers dementia to be one of the most important causes of

disability in the elderly. Its figures show that between 1% and 2% of people aged 65-69

suffer dementia, but this proportion more than doubles for people in the age band

70-74, and studies across a number of countries show that its prevalence “increases

almost exponentially with age” [1]. The socio-economic costs of dementia are large and

increasing, and an international study by Anders Wimo of the Karolinska Institute

suggest, that 72.5 billion euros per annum accross the Europe is the cost of the informal

care provided by family and other carers to dementia sufferers [2]. A further study

by Wimo showed that carers have to spend many hours per day assisting dementia

sufferers [3], and any technology that would reduce this would help to ease the costs -

both financial and emotional.

Most patients would prefer to use a non-intrusive technology to help them with

their day-to-day activities. For example usage of surveillance cameras for patient mon-

itoring is not welcomed and in most cases it is ruled out completely [4]. Research and

development have focused mainly in the utilisation of different low-key technological

devices which are readily available [5] [6] [7]. The major players in patient monitoring

systems rely heavily on the use of monitored call centres rather than carers, with stan-

dard telephone lines for logging data and require significant installation. Some other

companies (e.g. JustChecking Ltd. [8]) have realised the importance to patient care of

an individualised, personal system whereby the carers, relatives and others who know

the dementia sufferers can monitor them; only intervening when the information and

their personal knowledge indicates that the situation has changed significantly.

The aim of our research is to find means of improving the lifestyle of older citizens

by integrating intelligence into their existing homes (making their homes smart). The

smart home could contain different sensors (movement sensors, door entry point, taps,

kettles, cookers sensors, etc.) to determine different classes of context which would help

to identify patterns of use and movement, and eventually allow the categorisation of the

user’s behaviour. When the behavioural pattern is learned, any anomalous behaviour

could then be detected. The most important factor in designing a smart environment

for the elderly is that the technology should not interfere with the normal activities of

the patient. Thus all devices should operate autonomously. We intend to use only low

cost and readily available sensors which could be installed by the user themselves or

their informal carers. We believe developing a technological solution easily post-fitted

in existing homes would definitely assist the patients in gaining independence without

altering their lifestyle or losing their personal dignity.

The structure of this paper is as follows: related work in this field are reported

in Section 2 followed by our methodology in anomaly detection in Section 3. Our

data collection system and the way data is represented and visualised are reported in

Sections 3.1, 3.2 and 3.3 respectively. In Section 4 the prediction technique is explained

followed by case studies in Section 5. Final conclusions are drawn in Section 6.

2 Related Work

A comprehensive survey published in [9] reports on state of the art technologies to

support people at the early stages of dementia during the night. Extensive research

3

has been reported on smart homes with a variety of applications including monitoring

systems for elderly independent living, accident and fall detection [10] [11].

Research in independent living is not limited to dementia sufferers. Many published

works address the issue of independent living in a broader sense [12] [13] [14] [15]. The

smart environment can also help to identify and model progression of dementia of

the Alzheimer’s type by evaluating performance in the execution of activities of daily

living [16]. The smart home can either monitor and collect the activities information

of the user by means of sensors, or communicate and control the environment. The

former approach is widely used for monitoring [17], anomalous behaviour detection

[18], behaviour diagnosis and prediction of activities in an ambient intelligence (AmI)

environment [19] [20] [21] [22]. The latter approach is used to intervene and interact

with the user as a means of preventing accidents and reminding the user.

In [23] the activities extracted from sensors are distinguished and the relationships

between them are established by using data mining techniques such as model based

clustering and association rules. Also, temporal relations for the most frequent events

are used in [24] to identify abnormal events. Moreover, many statistical methods are

used to monitor the daily activities of an inhabitant in an intelligent environment. For

example, Naive Bayesian classifiers are used in [25] to classify and detect activities

using a “tape-on and forget” sensor system.

Hidden Markov Model (HMM) is widely used to detect and predict activities in

an AmI environment. For example, the authors in [18] employed a HMM technique to

model behaviour based on the sensor data collected from a smart home environment.

In [26], the HMM technique is employed to model the occupant behaviour after using

unsupervised classification techniques to group his/her daily routine activities. As a

result, the model is able to detect the anomalies behaviour. Hierarchical Hidden Semi-

Markov Models [27] are also used to identify the daily activities of the occupants in an

assisted living community. However, HMMs do not take into account the relationships

between activities happened in sequential or parallel. Also, for each individual activity,

the sequence of sensor events cannot be separated using these models. Furthermore,

processing large data sets generated from low level sensors (i.e. temporal data from

different time scales) using HMM has shown some difficulties [28] [29].

One alternative candidate is to use Artificial Neural Networks (ANNs) to deal with

temporal patterns generated from sensors. In general, different kinds of ANNs are used

to track and predict the daily routine activities of the occupant in an AmI environ-

ments. For example, an approach named One-Pass Neural Network (OPNN) is used in

[29] in intelligent environment embedded agents to detect the user’s activities. In [30] a

competitive neural network called Growing Self Organizing Map (GSOM) is proposed

to cluster the daily behaviour of humans within a smart environment. Recurrent neu-

ral networks are also proven to be useful tools to solve the difficulties of the temporal

relationships of inputs between observations at different time steps, by maintaining

internal states that have memory. There are a few works that have attempted to use

temporal neural network algorithms to detect, recognize and classify human activities

in an intelligent environment. For example, the authors in [31] [32] developed a tempo-

ral neural-network based agent to identify human behaviour according to the temporal

order of their activities.

4

Fig. 1: An overview of the monitoring and interaction system architecture.

3 Anomaly Detection in Smart Homes

Our research aim is to design an unobtrusive “activities of daily living” (ADL) monitor-

ing system to allow identification and prediction of abnormal behaviour. We approach

this by gathering data on the routine activities of the elders e.g., getting out of bed,

going to the bathroom, preparing meals etc. without altering the elders’ normal be-

haviour. Wireless sensors are used, along with a computerised base station to collect

data that are then analysed and transferred to a secure central web site for viewing by

the carer/relative. Therefore, the aim is that adult children of frail elders living alone

and at a distance could be sent reports or alerts daily/weekly in the form of e-mail or

phone calls and even they should also be informed if any abnormality in the near future

is predicted. An overview of the monitoring and interaction system architecture is de-

picted in Fig. 1. By enriching an environment with sensors and devices interconnected

through a network, an AmI can be formed to take decisions to benefit the users of that

environment based on real-time information gathered and historical data accumulated

[33]. This should help the elders retain independence and to remain in their own homes

longer than they might otherwise.

The challenge we face is to understand human behaviour from low level sensory

data. This could be achieved using common-sense knowledge or using computational

5

Fig. 2: Phases in the data handling work flow.

intelligence integrated with sensory data. An individual user model can be learned from

the sensory data which eventually represents the behaviour model of the user.

Fig. 2 represents the phases in the data handling work flow. The initial data capture

results in a large number of data items which, though time ordered, are not evenly

distributed in time, and are initially labelled only by sensor ID, time and boolean state.

The data must then be re-factored so that it can be more easily accessed, enumerated

and represented. The data representation and clustering/identification phases feed from

each other, as it is only through using different data representations that the separate

activities of clustering and abnormal behaviour identification can be easily carried out

and assessed.

By monitoring the sensor data, important information regarding any irregular be-

haviour will be identified. Anomalies are those odd patterns of data which do not match

the normal behaviour. Anomalies can be recognised using different anomaly detection

techniques. In many real life applications, these kinds of pattern are also called outliers,

discordant observations, exceptions, surprises or peculiarities. Amongst all mentioned

terminology, anomalies and outliers are the most frequently used terms within the con-

text of human behaviour detection. For example, Fig. 3 shows anomalies within two

dimensional data sets representing the sleeping pattern (from bed pressure sensor) of

an occupant. Most values of the data are in two regions N1 and N2 representing night

time sleeping and afternoon nap sleeping respectively. These regions are considered as

normal. At the same time the points in region A1 and points A2 and A3 are considered

to be anomalies because these points are at different time of the day and different from

the normal pattern in the regions N1 and N2. In our study anomalies are detected us-

ing clustering techniques [34]. In Section 3.3 clustering techniques are used to identify

any anomalies within collected sensory data.

3.1 Data Collection

As stated earlier, we rely on a data collection system which provides both sensation

and transmission. The data acquired includes the occupancy of different areas, environ-

6

Fig. 3: Anomalies in a simple 2-dimensional data set.

mental attributes, and interactions between occupants and devices. Sensory devices are

responsible for data collection and a variety of sensors are readily available to perform

this task. Typical sensors are as follows:

– Passive Infra-red Sensors (PIR) or motion detectors are sensitive to the movements

of living objects. They are normally used to monitor the occupancy of different

areas.

– Door/Window entry point sensors are on/off switches which can detect the open

and closed status of a door/window.

– Electricity power usage sensors which can monitor the activity of electrical devices

by measuring their electrical current consumption.

– Bed/sofa pressure sensors to measure the presence in and usage of these areas.

– Flood sensors provide early warning of overflows and leaks.

The choice of sensors for different environments vary. Without the loss of generality,

the discussion presented in this paper concentrate on the usage of PIR and door entry

point. PIR motion sensors responds to changes in heat in the form of infra-red radiation.

They are used to identify the movement and then the movement pattern is interpreted

as the occupancy. It is important to place PIR sensors in locations where the most

effective form of movements are captured. On the other hand, door entry point sensors

are relatively reliable as they clearly represent the movement activities.

The data collected from the sensor network are communicated with a base station

and eventually stored in a central database. The communication between the sensor

network and the base station could be in either wired or wireless format. Wireless

technology for sensor communication is a preferred option, as it is easier to fit wire-

less sensors in existing homes. However, we do not rule out the use of X10 technology

or other well established wired sensor networks in which sensory devices can commu-

nicate with the base station via electrical power lines. Wireless sensors networks, in

7

comparison with wired sensors networks, are more flexible in terms of deployment and

the required infrastructure of the network in a smart home environment. However, in

wireless sensor networks power consumption is the most important concern mainly be-

cause all sensory devices are powered by batteries. A system where the occupant was

required to change batteries frequently is not ideal.

Using either of these two technologies should not make any differences in the results

of our research in identification of anomalies in behaviour. For the sake of simplicity

and ease of installation, we have used a wireless sensor network comprising movement

sensors and door contact sensors.

Fig. 4: A sample of activity of daily living data collected by sensory devices.

Fig. 5: A sample of activity of daily living for a single user.

3.2 Sensor Data Representation

Different techniques have been used for activities representation and interpretation. For

instance, in [36] an activity path for a single continuous vertical trajectory is applied

to identify the activities for a period of time. Whereas the authors in [37] propose a

technique based on binary tree structure called a Routine Tree. In [38] activities are

divided into adjacent subsequences of length n called n-grams. These activities are

regarded as a histogram of their event n-gram. As the value of n is increased, the order

of information of events are more accurately captured. However, increasing the value

of n affects the dimensionality of the histogram.

8

Sensory data collected from a smart home environment needs to be represented in

an appropriate format before any further analysis is carried out. The raw sensory data

is often difficult to understand. This becomes even more complicated when sensory

data from multiple sensors are gathered. A snapshot of the binary data collected in the

base station from two occupancy sensors (PIR sensors) is illustrated in Fig. 4. Due to

the fact that only one occupant is present in the monitored environment, no parallel

activity in different areas is detected.

Fig. 6: Sample occupancy chart for 10 days of data for four different sensors.

Fig. 7: Layout of the house and the location of sensors for sample data set.

9

Fig. 8: Combined activity of daily living signal as a time-series.

The data represented in Fig. 4 can be integrated together to form the ADL chart.

A snapshot of the activity of daily living for a single user is depicted in Fig. 5. Similar

patterns can be obtained for a home with multiple occupants. This is achieved by using

RFID or other tagging devices to tag different users in the environment. A detailed

analysis is reported in [35].

To identify any abnormal behaviour of a user, we need to collect sufficient data of

daily activities to be able to establish the correlation between different events and ac-

tivities. Furthermore, a trend analysis of the information could be obtained if sufficient

data are available. It should be noted that sensor data are collected approximately ev-

ery second and when this frequency of data collection is repeated for multiple sensors,

we would be facing the difficult challenge of interpretation of large amounts of sensor

data.

To illustrate the complexity of the sensor data, Fig. 6 shows the occupancy signals

from 4 PIR sensors over a sample 10 day period. The layout of the environment and

the location of sensors are illustrated in Fig. 7.

In this paper, the process of modelling a large binary data set collected from a

sensor network is represented using two different techniques which are proven to be

useful to summarise the data. These techniques are tested with multiple binary sensors

(occupancy, door, windows, and etc. sensors). The data extracted from those sensors

are usually sparse and contain many repeated constant values.

3.2.1 Combined activity of daily living signal

Assuming different levels for each activity, a combined signal can be shaped. Each level

of the combined signal represents one of the sensors. A sample of combined activity

signal is illustrated in Fig. 8. This signal represents a non-stationary time series and

available techniques in time series prediction may be used to predict the future values

of the signal, which could be interpreted as prediction of the activities of the patient.

This method is explained in more detail in [19].

3.2.2 Start time and Duration

In this method, the signal is represented by the start time and the duration of an event.

For example, we use the start time that the person enters a room and duration

that he/she stays in a specific location. Thus, each observation has three parameters

[39]. These are:

1. Sensor ID (position of the sensor e.g. in the bedroom);

2. Start time (the time of day the observations is occurred); and

10

(a) bedroom

(b) corridor

(c) lounge

(d) kitchen

Fig. 9: Plots of 365 days of the sample data for the four sensors.

11

Table 1: Start time and duration of a sample of bedroom sensor.

Start time (hrs.) 13:01 23:23 14:06 23:15 13:45Duration (hrs.) 2:20 10:20 2:30 8:15 2:05

Fig. 10: Start time, duration and time of bedroom PIR activities for one year sample

occupancy chart.

3. Duration (the amount of time the observation is occurred) of specific room is oc-

cupied.

Table 1 demonstrates a sample of observations representing the time that the pa-

tient spent in the bedroom at each day for a duration of three days. From this sample

data, a sleeping pattern is easily identifiable.

The graphical representation of these attributes shows significant information about

the daily movement behaviour of the occupant. For example, consider Fig. 6 and Fig. 9

which show the plots of a sample data set of four sensors for a single occupant. As can

be seen, the behaviour of the occupant is more easily interpreted in Fig. 9 than Fig.

6. For instance, in Fig. 9-a, the bedroom sensor plot shows that the occupant always

goes to bed at midnight for around 7-10 hours, and he/she usually spends about two

hours period of time in nap sleeping. It is almost impossible to achieve this level of

understanding from the raw sensory data represented in Fig. 6.

Consider the bedroom movement sensor shown in Fig. 9-a which represents the

projection of the sensory data collected over a long period of time into a 2D graph,

collapsing out the axis referring to the individual days. In Fig. 10 the same activity is

illustrated in a 3D graph where start time and duration of each activity over one year

period, with the individual days shown.

Using the above two forms of data representations have provided us with useful

information and this provides the basis of our work for classification of activities. We

intent to use the start-time and duration form of representation in the rest of this

paper.

12

(a) bedroom (b) corridor

(c) lounge (d) kitchen

Fig. 11: Clusters of activities for 14 days of the sample data for the four sensors.

3.3 Sensor Data Visualisation and Clustering

Data visualisation has the potential to help in understanding and recognising large

volumes of data and also detect patterns and anomalies that are not obvious using

non-graphical forms of representation. Good data visualisation eases the examination

of large volumes of data, and allows deduction to be made from the relationships within

the data [40]. If the daily routine activities are clustered together, then odd activities

can then be identified as anomalous behaviour. Clustering is an important process for

condensing and summarising information because it can provide an overview of the

stored data [41]. To identify the abnormalities within the sensory data using clustering

based animality detection, the following three categories are identified [34]:

1. The data that reflect the regular or normal data are grouped in clusters, while

the data that do not fit in any clusters are treated as anomalies. In this case, any

clustering technique can be used and any data that do not find in any cluster are

considered to be anomalies.

2. The data that are near their closed cluster centroid are considered as normal data,

while the data that are located far away from their cluster centroid are treated as

anomalies. In this case, the data are first clustered and then the anomaly score,

which is the distance to its closed cluster, is calculated.

3. The data found in large clusters are considered as normal data, while the small or

sparse clusters contain the anomalies. In this case, depending on a threshold value,

anomalies can be detected. If the size of any cluster is below this value then the

stored data is considered to be anomalous.

In this paper, anomaly detection techniques have been applied using different

clustering algorithms [41][45]. Examples of such algorithms are: self organising maps

(SOM), K-means clustering, and fuzzy C-means (FCM) to cluster training data and

13

then use the clusters to classify test data set. For example, in Fig. 11 the clusters for

the same sample data illustrated earlier in Fig. 9 are shown. In Fig. 11-a and Fig. 11-c

there are some instances of data that do not belong to any cluster (see the first cate-

gory above), so these data are considered as abnormal data. The clusters that contain

a large data set represent the most frequent pattern in these clusters, while the clusters

that contain small datasets are treated as anomalies. It can be concluded that large

volume of data can be easily represented and visualised by using clustering techniques.

The number of clusters could vary from one algorithm to another. For example,

unlike self organizing map algorithms, the number of clusters in fuzzy C-means need to

be known in advance. In our experiments the maximum number of clusters was set for

supervised algorithms depending on the duration times that the occupant spent in a

particular area. In fuzzy C-means clustering, objects on the boundaries between several

clusters do not belong to a specific cluster. They belong to more than one cluster with

a certain degree of belonging. Every time a new activity is recorded, the Euclidean

distance between the new data and cluster centre is calculated. The distance matrix

will represent whether if the data should be considered as normal or abnormal.

4 Sensor Data Prediction

Using the above mentioned data interpretation, we would be able to separate the

anomalous behaviours when an activity has happened. However, this would not help the

carer to make necessary arrangements in advance. Data interpretation does, however,

help us to better understand the activities of daily living. This would also be useful

if we want to generate a report to summarise the activities of the patient over a long

period of time. To improve the proposed system, a predictive method will be utilised

to predict the future values of the activities (start time and duration) based on the

historical data available from activities recorded by each sensor.

Activities of daily living presented either in combined (e.g. see Fig. 8) or separate

signals (e.g. see Fig. 6) represent a time series. To predict the future values of time

series, the hidden Markov model (HMM) has been used widely to find the relationships

between the temporal or sequential data extracted from sensors and identify the routine

activities of an occupant. HMM is also used in time series prediction to predict future

values of the series. The disadvantages of using HMM in time series prediction is

that increasing the length of time series needs large volume of time series runs from

HMM [42][43][44]. Many other statistical techniques in time series prediction have been

reported [41].

In this study, we have investigated different techniques for prediction of stationary

time series. The investigated techniques included; Echo State Network (ESN), Back

Propagation Through Time (BPTT) and Real Time Recurrent Learning (RTRL). Re-

current neural networks (RNNs) are widely used to deal with many dynamical and

non-linear problems, such as time series forecasting. They have feedback connections

which address the temporal relationship of inputs by maintaining internal states that

have memory. RNNs are proven to be effective in learning time-dependent signals that

have short-term structure. RNNs are computationally more powerful than feed forward

neural networks and better approximation results have been obtained for prediction

problems. Based on our empirical investigation, it was found that ESN is a suitable

technique for prediction of binary time series and a short summary of the technique is

presented in the following section.

14

Fig. 12: Structure of an Echo State Network approach. Only the output weights Wout

are adapted, all other weights (input, reservoir and feedback) are chosen randomly.

4.1 Echo State Network

In this section, the recurrent neural network, Echo State Network (ESN) is described.

It was developed recently by Jaeger [46]. The basic architecture of ESN is illustrated

in Fig. 12 which consists of three layers. These include input, hidden and output layer.

The input layer is connected to the hidden layer. Both the input and hidden layer are

fully connected to the output layer. On the other hand, the output layer is backward

connected to the hidden layer only. It is a discrete-time, continuous state where the

activation function for all neurons is the sigmoid function [47].

An ESN consists of a reservoir of conventional processing elements, which are re-

currently interconnected with untrained random weights, and a readout (output) layer,

which is trained using linear regression methods. The key advantage of the ESN is its

ability to model systems without the need to train the recurrent weights [48]. For train-

ing an ESN with an input u(n), a reservoir state x(n) with M processing elements,

and an output y(n), the equations are calculated as follows:

x(n+ 1) = tansig(wx.x(n) + win.u(n) + v(n+ 1)) (1)

and

y(n) = w.x(n) (2)

where x(n) denotes the hidden layer or the internal state. The input and output to the

ESNs are denoted by u(n) and y(n) respectively. tansig denotes hyperbolic tangent

sigmoid function which is applied element wise, v(n + 1) is an optional noise vector.

wx, win and w are respectively the internal connection weights of the reservoir, the

input weights to the reservoir and the readout (output) weights from the reservoir[49].

The ESN approach differs from other methods in that a large RNN is used (on

the order of 50 to 1000 neurons) and in that only the synaptic connections from the

RNN to the output neurons are updated i.e. weights coming from the hidden layer

15

Fig. 13: ESN predicted values for bedroom sensor.

(the reservoir) to the output layer are updated in order to achieve the learning task.

As a result, large datasets are learnt in only few minutes or even seconds [46]. Also,

there are neurons in the reservoir connected in loops [see Fig. 12], therefore the past

states ’echo’ in the reservoir. The convergence of training in ESN is much faster than

other RNN [49]. This has made ESN an attractive model for a wide range of signal

processing and control applications (e.g. time series prediction, pattern generation,

event detection and classification and non-linear control). For instance, in prediction

of chaotic time series, ESN has proven to be a very accurate and valuable tool. The

prediction is accomplished using a black box model, i.e. it only depends on past data

since no further information is used. In addition, no explicit model is given in order to

create a new situation [50] [51].

In this paper, ESN is used as a model to predict and extract behavioural patterns

while keeping learning complexity at a low level. In addition, ESN is a very good choice

for the modelling because in the methodology of sensor networks, new data are arriving

at any time, whilst other approaches need all data input at the same time steps in order

to compute the output.

4.2 Predictive Techniques Comparison

Initially ESN was employed to predict the future values based on the sample data

shown earlier in Fig. 9. In our experiments, instead of using a separate ESN for each

sensor, all available sensors are connected at the same time as inputs to the network

and compute the prediction of the sensors. The advantages of using just one ESN for

all sensors are to reduce the amount of memory and computation time. In this case, the

number of input and output units depends on the number of sensors, which is driven

by the actual sensor value at time t. The output unit is the value of the same sensor at

time t+τ with different hidden units (reservoir) sizes. A number of parameters are used

in the ESN learning algorithm including number of neurons in the hidden layer, the

Root Mean Square Error (RMSE) for training and testing datasets, number of epochs

and time required for training. For training the ESN network, 50 hidden neurons are

16

Fig. 14: ESN predicted values for corridor sensor.

used. The results for τ = 6 step ahead (6 hours) prediction for both bedroom and

corridor sensors are shown in Fig. 13 and Fig. 14 respectively. Results shown are only

for a sample of 14 days dataset split into training and testing data. Samples of 10 days

data are used as the training and the samples of 4 days are used for testing.

The bedroom sensor shows a very good match between the predicted and actual

sensor values. This is slightly less accurate for corridor sensor, as the corridor signal is

relatively more chaotic. Our observation is that more chaotic signals are expected to

be more difficult to predict. Different sizes of reservoir (number of hidden neurons) are

also used to test the performance of ESN. In Fig. 15 the training time using different

reservoir sizes (hidden neurons) are shown. As can be seen, ESN is relatively fast

and datasets are trained in only a few minutes or even seconds. In addition, ESN is

compared with other RNNs used in time series prediction. These techniques are back

propagation through time (BPTT) and real time recurrent learning (RTRL). Table 2

compares the results of all sensor datasets using ESN with the other recurrent neural

networks techniques. ESN prediction results are better as compared with BPTT and

RTRL in that the training time is significantly shorter. The other approaches suffer

from slow convergence as the number of neurons are increased.

Table 2: Prediction results of all sensor datasets using ESN, RTRL and BPTT. (⊕:

No. of hidden neurons; ⊗: Training RMSE; : Testing RMSE; and �: Time(Sec.).)

Method ⊕ ⊗ �ESN 10 0.0556 0.0556 0.0116ESN 50 0.0556 0.0556 0.0519RTRL 10 0.0987 0.0964 281.8231RTRL 50 0.0731 0.0740 1541.0427BPTT 10 0.0759 0.0766 12.7315BPTT 50 0.0803 0.0811 218.0986

17

Fig. 15: ESN training time for different reservoir sizes.

5 Case Studies

In this section two separate case studies are reported. In both case studies, the data

collection system consists of an array of wireless motion sensors (kitchen, bedroom,

bathroom, lounge) and door entry point sensors (front door and back door) were used.

For both case studies, JustChecking [8] equipment is used. More details about the

technical characteristics of the hardware equipment is available from [8]. However, we

should emphasise that the results and discussion presented below is not restricted to

the usage of this equipment only. In both cases only one resident who suffered from

dementia living in their own home is monitored. The caregiver has access to the daily

activities of the patient and they are monitored from a secure web interface. The output

of all sensors are discrete values. These discrete values represent presence or absence

from a specific area. When the patient moves from one room to another, the status of

the sensor will vary.

5.1 Case Study 1 - Anomaly Detection

The aim of this case study is to establish whether the system can detect abnormality

within the behaviour of an occupant living in a real environment. In this case study,

the data is representing the occupancy in different areas of a house and for an elder

person where her health deteriorated during the course of our research. To demonstrate

the difference in the activities based on the sensor network measurement only, the

data set are represented into separate groups. Fig. 16-a and Fig. 17-a show the start

time and duration time of occupancy activities for the lounge and bathroom motion

sensors respectively. In both figures, only a sample of three days are depicted. She was

prescribed with some medications earlier and it worsened her health status. She was

wandering around during the early hours of morning and her behaviour was considered

as abnormal. As can be seen from Fig. 16-a, the occupant has often been in the lounge

area many times after midnight. Frequently the occupant stays 20 minutes or less in

the lounge room but on some occasions she spent more than one hour in this area. She

was prescribed with new medications and just after 20 days from when the first data

set is shown, occupant’s health improved. The normal occupancy behaviour for both

lounge and bathroom are shown in Fig. 16-b and 17-b respectively.

18

(a) Abnormal

(b) Normal

Fig. 16: A sample of three days activities for the lounge area. (a) Abnormal (b) Normal.

It is straightforward to see the difference in the pattern for the lounge area (Fig.

16-a and Fig. 16-b). However, this might not be so obvious for other areas of the house.

For example, the duration of bathroom occupancy for the abnormal behaviour period

is just slightly more than the expected normal bathroom occupancy. Therefore, it is

important to make a collective decision based on all available sensory data. All data

points are clustered and clusters are shown with different markers. A threshold value is

used to detect the anomalies. The unexpected clusters for both lounge and bathroom

are shown in dotted lines in Figs. 16-a and 17-a respectively.

As discussed earlier in Section 3, using an appropriate form of signal representation

and ultimately clustering the activities, it is possible to identify the abnormality from

low level sensory signals even if detailed knowledge about the subject is not available.

5.2 Case Study 2 - Anomaly Prediction

In this case study, the data was collected over a one and half year period. The layout

of the house and where sensors are mounted is shown Fig. 18. The house consists

19

(a) Abnormal

(b) Normal

Fig. 17: A sample of three days activities for the bathroom. (a) Abnormal (b) Normal.

of a lounge, corridor, kitchen, and bedroom (upstairs). Therefore, only three motion

sensors and two door contact sensors (front door and back door) are used to monitor

the occupancy and activities of the resident.

The first stage of our research is to identify the normal behaviour and distinguish

any abnormality. Using the data representation method discussed in Section 3.2, the

sensor data are clustered. Fig. 19-a and Fig. 19-b show start time and duration graph

for front door and back door sensors respectively. From the clusters highlighted in these

two graphs, it is evident that for most days the front and back doors were opened for a

short period of time. However, in some instances both doors were left open for a long

period of time.

Using this form of data representation and visualisation, we have managed to look

at each sensor activity independently and identify the abnormal behaviour for that

specific activity. However, all these activities are interdependent and we should be able

to establish the dependency between the activities. To achieve this, an ESN is used to

learn the temporal relationship of all 5 sensors.

Figs. 20-a and 20-b show the ESN training results for back door and kitchen re-

spectively. The prediction is based on a one month data set where 20 days are used for

20

Fig. 18: Layout of the house and location of sensors for our case study.

training and 10 days for testing. 50 hidden neurons and 6 hours ahead prediction are

used. The results shown here is a sample of prediction for one day only.

The root mean square error (RMSE) for training of these sensors shown in Fig.

20 is about 1% and 7% for back door and kitchen sensors respectively. For instance,

consider the back door sensor shown in Fig. 20-a. The back door is opened only once

during this day and it is closed after 3 seconds. As can be seen from this figure, the

predicted duration and start time is very close to the actual data. There is a very small

error between the predicted data and the actual sensor data.

6 Conclusions and Future Work

The results presented in this paper show that the start-time/duration is the most

effective way of representing a large sensor data set. This will also help with the clas-

sification of the activities to identify the abnormal behaviour. Furthermore, we have

investigated different recurrent neural network technique to predict the future activi-

ties. The results presented in this paper show that Echo State Network (ESN) is a very

promising approach for binary datasets collected from smart environments. Datasets

investigated here are based on a single inhabitant environment equipped with appro-

priate motion and door contact sensors. These sensors are used to record the activities

representing the behaviour of the occupant, and allow the caregiver to observe any

changes to patterns. It can be concluded that using large number of hidden neurons

in ESN yielded a good results in terms of the error and time required for training and

testing.

Based on the results shown here, it appears that a home equipped with some low-

level sensors can provide important information about the status of the occupant. The

proposed approach works better for elderly residents when more routine activities are

expected. We cannot suggest whether this approach would work for a young or more

active occupant.

21

(a) front door sensor

(b) back door sensor

Fig. 19: Start time and duration activities.

Our future work aims at multiple occupancy and prediction of abnormal behaviour.

The approach presented in this paper would not be effective in the presence of visitors

or even when the elderly people have a pet which is true for some cases. We are also

aiming to continue our research in semantic modelling of the behaviour where the

predicted values are communicated with the elderly and carer in linguistic terms. For

the work presented in this paper, only a limited number of discrete sensors were used.

However, more research is required when a combination of discrete sensors (occupancy,

door entry point, ...) and continuous sensors (temperature, humidity, ..) are used.

22

(a) back door sensor

(b) kitchen sensor

Fig. 20: Predicted values for start time and duration activities: A sample of one day

activities.

Acknowledgements

This research was partially supported by Nottingham Trent University’s Stimulating

Innovation for Success (SIS) programme. The authors would like to thank Just Check-

ing Ltd. (www.justchecking.co.uk) for their support of this work.

References

1. European Comission,(2010), Europa Public Health [online]. Available: “http://ec.europa.eu/ health/ph information/ dissemination/ diseases/ alzheimer en.htm” [ac-cessed 16 August 2010]

2. A. Gustavsson, L. Jonsson, R. McShane, M. Boada, A. Wimo, AS. Zbrozek, “Willingness-to-pay for reductions in care need: estimating the value of informal care in Alzheimer’sdisease” Int J Geriatr Psychiatry. 2009 Sep 14.

23

3. A. Wimo, B. Winblad, L. Jonsson, “An estimate of the total worldwide societal costs ofdementia in 2005”, Alzheimers Dement. Apr; 3(2):81-91 2007.

4. A. Kenner1, “Securing the Elderly Body: Dementia, Surveillance, and the Politics of Agingin Place,” Surveillance and Society, Vol. 5, no. 3, pp. 252-269, 2008.

5. Marilyn Cash, At Home with AT (Assistive Technology), Research report available from:“http://www.dementia-voice.org.uk/Projects/At Home with AT main.pdf”, 2004.

6. C. Nugent, M. Mulvenna, F. Moelaert, B. Bergvall-Kareborn, F. Meiland, D. Craig, R.Davies, A. Reinersmann, M. Hettinga, A. -. Andersson, R. -. Droes and J. E. Bengtsson,“Home based assistive technologies for people with mild dementia,” in 5th InternationalConference on Smart Homes and Health Telematics,pp. 63-9, 2007.

7. R. Orpwood, C. Gibbs, T. Adlam, R. Faulkner and D. Meeeahawatte, “The design ofsmart homes for people with dementia - user-interface aspects,” Universal Access in theInformation Society, vol. 4, pp. 156-64, 12, 2005.

8. JustChecking, (2010), Supporting Independence People with Dementia online]. Available:http://www.justchecking.co.uk [accessed 16 August 2010]

9. P.J. McCullagh, W. Carswell, J.C. Augusto, et al., “State of the Art on Night-Time Careof People with Dementia,” IET Assisted Living Conference, 24-25 March, 2009, LondonUK.

10. P. Boissy, S. Choquette, M. Hamel and N. Noury, “User-based motion sensing and fuzzylogic for automated fall detection in older adults,” Telemedicine Journal and e-Health, vol.13, pp. 683-93, 12/01. 2007.

11. X. Yu, “Approaches and principles of fall detection for elderly and patient”, in 2008 10thInternational Conference on e-Health Networking, Applications and Services (Healthcom),2008, pp. 42-7.

12. MJ. Akhlaghinia, A. Lotfi, C. Langensiepen and N. Sherkat, “A fuzzy predictor modelfor the occupancy prediction of an intelligent inhabited environment,” in 2008 IEEE 16thInternational Conference on Fuzzy Systems (FUZZ-IEEE), 2008, pp. 939-46.

13. V. Callaghan, G. Clarke, M. Colley, H. Hagras, J. S. Y. Chin and F. Doctor, “Inhabitedintelligent environments,” BT Technology Journal, vol. 22, pp. 233-247, 2004.

14. H. Hagras, “Embedding computational intelligence in pervasive spaces,” IEEE PervasiveComputing, vol. 6, pp. 85-9, 07. 2007.

15. A. Helal, J. King, R. Bose, et al., “Assistive Environments for Successful Aging,” inAdvanced Intelligent Environments, Achilles D. Kameas, Victor Callagan, Hani Hagras,Michael Weber and Wolfgang Minker (Eds.), Springer US, 2009.

16. A. Serna, H. Pigot and V. Rialle, “Modeling the progression of Alzheimer’s disease forcognitive assistance in smart homes,” User Modeling and User-Adapted Interaction, vol.17, pp. 415-38, 2007.

17. H. Medjahed, D. Istrate, J. Boudy and B. Dorizzi, “A fuzzy logic system for home elderlypeople monitoring (EMUTEM),” in 10th WSEAS International Conference on Fuzzy Sys-tems (FS’09), 2009, pp. 69-75.

18. D. N. Monekosso and P. Remagnino, “Anomalous Behavior Detection: Supporting In-dependent Living” in Intelligent Environments, Methods, Algorithms and Applications,Monekosso, Dorothy; Remagnino, Paolo; Kuno, Yoshinori (Eds.), Springer London, 2009.

19. MJ. Akhlaghinia, A. Lotfi, C. Langensiepen and N. Sherkat, “Occupant Behaviour Predic-tion in Ambient Intelligence Computing Environment,” International Journal of UncertainSystems, 2008.

20. D. J. Cook, “Making sense of sensor data,” IEEE Pervasive Computing, vol. 6, pp. 105-8,04. 2007.

21. Sawsan M. Mahmoud, Ahmad Lotfi, N. Sherkat, C. Langensiepen and T. Osman, “Echostate network for occupancy prediction and pattern mining in intelligent environment,”in Proceedings of the 5th International Conference on Intelligent Environments, 2009, pp.474-481.

22. Dante I. Tapia, Ajith Abraham, Juan M. Corchado and Ricardo S. Alonso, “Agents andambient intelligence: case studies,” Journal of Ambient Intelligence and Humanized Com-puting, Vol. 1, No. 2, pp. 85-93, 2010.

23. Tracy Barger, Majd Alwan, Steve Kell and Beverely Turner, Objective Remote Assessmentof Activities Of Daily Living: Analysis Of Meal Preparation Patterns, Medical AutomationResearch Center, University of Virginia Health System, 2002.

24. Vikramaditya Jakkula, Diane J. Cook and Aaron S. Crandall: Temporal Pattern Discoveryfor Anomaly Detection in a Smart Home, 3rd IET International Conference on IntelligentEnvironments, pp. 339 -345, 2007.

24

25. Emmanuel Munguia Tapia, Stephen S. Intille, Kent Larson, Activity Recognition in theHome using Simple and Ubiquitous Sensors, F. (eds.) PERVASIVE, LNCS, Vol. 3001, pp.158-175. Springer, Heidelberg , 2004.

26. Dorothy N. Monekosso and Paolo Remagnino, Anomalous Behavior Detection: Support-ing Independent Living, Intelligent Environments, Advanced Information and KnowledgeProcessing, Springer, 2009.

27. H. Kautz, O. Etziono, D. Fox and D. Weld: Foundations of assisted cognition systems.Technical report, University of Washington, Department of Computer Science and Engi-neering, 2003.

28. Geetika Singla, Diane J. Cook and Maureen Schmitter-Edgecombe: Recognizing inde-pendent and joint activities among multiple residents in smart environments, Journal ofAmbient Intelligent Human Computing, Vol. 1, pp. 57–63, 2010.

29. H. Li, Q. Zhang and P. Duan: A Novel One-Pass Neural Network Approach for Activi-ties Recognition in Intelligent Environments. Proceedings of the 7th World Congress onIntelligent Control and Automation, pp. 50-54, 2008.

30. H. Zheng, H. Wang and N. Black: Human Activity Detection in Smart Home Environ-ment with Self-Adaptive Neural Networks. IEEE International Conference on Networking,Sensing and Control, pp. 1505-1510, 2008.

31. Rivera Illingworth, V. Callagha and H. Hagras: Towards the Detection of Temporal Be-havioural Patterns in Intelligent Environments. 2nd IET International Conference on In-telligent Environments, pp. 119-125, 2006.

32. Rivera Illingworth, V. Callaghan and H. Hagras: A Neural Network Agent Based Approachto Activity Detection in AmI Environments. IEE International Workshop, Intelligent En-vironments (IE05), Colchester, UK, pp. 1-12, 2005.

33. Juan Carlos Augusto, Ambient Intelligence: Basic Concepts and Applications, Softwareand Data Technologies, Communications in Computer and Information Science, Vol. 10,pp. 16-26, 2008.

34. Varun Chandola, Arindam Banerjee and Vipin Kumar, “Anomaly detection: A survey”,ACM Computing Surveys (CSUR), vol. 41, No. 3, 2009.

35. MJ. Akhlaghinia, A. Lotfi, C. Langensiepen, “Localising Agents in Multiple-OccupantIntelligent Environments”, in Proc. WCCI 2010 IEEE World Congress on ComputationalIntelligence, Barcelona, Spain, July, 18-23, pp. 2202-2208, 2010.

36. K. Vrotsou, K. Ellegard and M. Cooper, “Everyday Life Discoveries: Mining and Visu-alizing Activity Patterns in Social Science Diary Data,” Information Visualization, 11thInternational Conference, pp. 130-138, 2007

37. Raza Ali, Mohamed ElHelw, Louis Atallah, Benny Lo and Guang-Zhong Yang, “PatternMining for Routine Behaviour Discovery in Pervasive Healthcare Environments,” 2008.

38. Raffay Hamid, Siddhartha Maddi, Amos Johnson, Aaron Bobick, Irfan Essa and CharlesIsbell: “A Novel Sequence Representation for Unsupervised Analysis of Human Activities,”Elsevier Science, Artificial Intelligence, Vol. 173, Issue 14 pp. 1221-1244, 2009

39. Fabien Cardinaux, Simon Brownsell1 Mark Hawley, and David Bradley, “Modelling ofbehavioural patterns for abnormality detection in the context of lifestyle reassurance”,Proceeding CIARP, Havana, Cuba, 2008.

40. Michael D. Lee, Rachel E. Reilly and Marcus A. Butavicius, “An Empirical Evaluationof Chernoff Faces, Star Glyphs, and Spatial Visualizations for Binary Data”, In CRPITS24:Proceedings of the Australian symposium on Information visualization, pp. 110, 2003

41. T. Warren Liao, “Clustering of time series data a survey”, Pattern Recognition, vol. 38,pp. 1857-1874, 2005.

42. Sang-Ho Park, Ju-Hong Lee, Jae-Won Song and Tae-Su Park, “Forecasting Change Direc-tions for Financial Time Series Using Hidden Markov Model,” vol. 5589, Lecture Notes inComputer Science, Springer, pp. 184-191, 2009

43. Osman N. Yogurtcu, Engin Erzin and Attila Gursoy, “Extracting Gene Regulation In-formation from Microarray Time-Series Data Using Hidden Markov Models,” vol. 4263,Lecture Notes in Computer Science, Springer, pp. 144-153, 2006

44. H. Li, Q. Zhang and P. Duan, “A Novel One-Pass Neural Network Approach for Activi-ties Recognition in Intelligent Environment”. Proceedings of the 7th World Congress onIntelligent Control and Automation, pp. 50-54, 2008

45. Eamonn Keogh Jessica Lin, Michail Vlachos and Dimitrios Gunopulos, “Iterative incre-mental clustering of time series”, In EDBT, pp. 106-122, 2004.

46. Herbert Jaeger and Harald Haas, “Harnessing Nonlinearity: Predicting Chaotic Systemsand Saving Energy in Wireless Communication”, Appril, VOL 304, SCIENCE, (2004).

25

47. Alexandre Devert, Nicolas Bredeche and Marc Schoenauer, “Unsupervised Learning ofEcho State Networks: A Case Study in Artificial Embryogeny”, Artificial Evolution, Vol.4926, (2008), pp. 278–290.

48. Mark D. Skowronski and John G. Harris, “Minimum mean squared error time series clas-sification using an echo state network prediction model”, IEEE International Symposiumon Circuits and Systems, (2006).

49. Zhiwei Shi and Min Han, “Support Vector Echo-State Machine for Chaotic Time-SeriesPrediction”,IEEE Transactions on Neural Networks, March, Vol. 18, Issue: 2, (2007), pp.359–372.

50. Sven Hellbach, Soren Straussl, Julian P. Eggert, Edgar Korner and Horst-Michael Gross1,“Echo State Networks for Online Prediction of Movement Data - Comparing Investiga-tions”, (2008).

51. Obst, O., Wang, X.R. and Prokopenko, M., “Using Echo State Networks for AnomalyDetection in Underground Coal Mines”, Proceedings of the 7th international conferenceon Information processing in sensor networks, (2008).

52. K. Sawai and M. Yoshida, “Algorithm to detect abnormal states of elderly persons forhome monitoring,” Syst. Comput. Jpn., vol. 38, pp. 34-42, 06/15. 2007.