Date post: | 14-Jan-2017 |
Category: |
Presentations & Public Speaking |
Upload: | george-baryannis |
View: | 115 times |
Download: | 0 times |
Rule-based Real-Time Activity Recognition in a Smart Home Environment
Przemyslaw Woznowski Grigoris Antoniou
10th International Web Rule Symposium (RuleML) 2016, Stony Brook, New York, USA
George Baryannis
Outline
2
Introduction
• Activity Recognition and the Internet of Things• The SPHERE Project• Related Work
Rule-based ADL Recognition
• Offline Version• Online Version• Experimental Evaluation
Conclusions & Future Work
Activity Recognition and the Internet of Things• Sensors have become cheaper, small, widely available• Interconnected within an Internet of Things (IoT)
setting, benefitting from– Distribution of resources– Support for common naming schemas and ontologies– Common access strategies– Availability of computational resources
• Automated Activity Recognition (AR) requires a fusion of multiple sensor-related low-level events
• Challenge: to locate and fuse the right pieces of information from an IoT instance (e.g. sensor network) in order to realise AR at the best quality of information possible
3
Approaches for sensor-based AR
• Data-driven – exploiting machine learning techniquesNoise and uncertainty are handled wellRequire large, annotated training datasetsData conflicts are not handled well
• Knowledge-driven – leveraging logical modelling and reasoning
No training data neededNot as robust against noise and uncertaintyRequire carefully crafted rules
4
Activity Recognition in a multi-modalsmart home environment• Focuses on the so-called Activities of Daily Living
(ADL), with the purpose of supporting Ambient Assisting Living (AAL) efforts– Long-term monitoring of health-related features– Direct assistance
• Main requirements– Increased need for robustness against noise (due to multiple
sensors)– Support for complex, uncertain and non-sequential scenarios– Support for user localization within the smart home, with
minimal user involvement– Inference of real-time, continuous streams of meaningful and
actionable events– Less reliance on training data, since they are difficult to acquire
due to them being environment-dependent 5
The SPHERE Project
Woznowski et al. (2015)6
• SPHERE: a Sensor Platform for Healthcare in a Residential Environment– Common platform
of non-medical networked sensors
– Deployed on a home environment testbed, the SPHERE house
– Impact a range of healthcare needs simultaneously
• Chen et al.: equivalence and subsumption reasoning on ontologies modelling both sensors and activities
Both offline and real-time modes, incrementally-specific recognitionRequires activities to be performed in a predefined, strictly sequential order and fixed time intervals
• MetaQ: SPARQL-based reasoning on sensor data represented as RDF graphs
Recognition building from atomic gestures to complex activitiesWorks only offline, does not take into account missing activities
Related Work (1)
7
• Skarlatidis et al.: hybrid approach, combining event calculus reasoning with Markov Logic Networks
High recognition rates, robustness against missing dataOnly focuses on posture and movement-related activities, as opposed to complex ADL scenarios
• Helaoui et al.: hybrid approach, employing a probabilistic DL reasoner
Recognition building from atomic gestures to complex activitiesRequires training data, works only offline, no support for temporal features
Related Work (2)
8
Outline
9
Introduction
• Activity Recognition• The SPHERE Project• Related Work
Rule-based ADL Recognition
• Offline Version• Online Version• Experimental Evaluation
Conclusions & Future Work
10
• Rule base– rules defined by examining collected sensor data from scripted experiments
• Fact base– derived from sensor data
• The system operates in two modes– Offline: precollected sensor data are stored as individual facts
• Can provide activity reports for past periods (e.g. hourly or daily)– Real-time: facts represent each deployed sensor node and store
its current state/value (as well as its previous one)• Recognises activities as soon as the associated sensor events
happen
RuleBase
Fact Base
Inference Engine (JESS)
“Expert” knowledge Sensor data
Rule-based System Overview
11
• Environmental Sensors– Door contact, electricity meters, water flow meters, PIR– Ambient light useful only when the effect of sunlight is minimal
(i.e. the sun is below the horizon)– Scripted experiment data do not yield patterns from ambient
noise, dust, humidity and temperature
• Video Sensors– 2D bounding box coordinates– Depth coordinates of 3D bounding box
Fact Base
12
PIR-basedLocation
Camera-based
Location
FusedLocation
Second Level Rules
Higher Level Rules
DoorInteraction
ElectricalDevices
WaterFlow
WaterFlow
Clean-upAtomic Activity Rules
Rule Base
rules assume single inhabitant scenarios
13
• Detect changes in sensor values within their reporting windows– From >0 to 0: OpenDoor / SwitchOff– From 0 to >0: CloseDoor / SwitchOn
Doors and Electrical Devices
14
• Water meters do not have a reporting period, only report instantaneously– Positive flow value: OpenTap– Zero flow value: CloseTap
• “Clean up” rules follow to keep only the earliest events for each distinct opening or closing occurrence– If there is no close tap activity between two consecutive open
tap activities, remove the latest one
Water Flow
15
• Rules so far recognise atomic activities• Higher-level rules progressively combine recognised
activities to infer activities of increasingly higher complexity– SwitchOn(device,t1) and SwitchOff(device,t2) Use(device, t1,
t2)– SwitchOn(tv,t1) and SwitchOff(tv,t2) WatchingTV(t1, t2)– Use of taps in kitchen or bathroom WashHands or WashFace– Use of taps in bathroom BrushTeeth or Bathing/Showering– Use kettle and close tap in kitchen PreparingDrink– Open fridge and use toaster PreparingSnack– PreparingDrink or PreparingSnack and use of taps in kitchen
WashDishes
Complex Activities
16
• Basic PIR rule places user in a specific room, from the time PIR is activated till it’s deactivated– Sequences in the same room merged if temporally close or user
not in a different room in between• Basic video rule places user in a specific room, for as
long as the associated camera reports bounding box coordinates
• Detect ghost sequences since they severely compromise validity– Length of less than 30 frames– Stuck in the same coordinates for more than 30 frames– Width and/or height of box consistently and unjustifiably small,
in correlation with depth
Localisation Rules
17
• Assign confidence values to PIR and camera location reports– PIR: confidence inversely proportional to the number of PIR sensors
simultaneously reporting motion– Video: confidence depends on the probability of being a ghost, based on
the detection heuristics• If only a single source reports a location, it is assumed to hold
(with the associated confidence value)• If PIR and video report the same location, it is assumed to hold
(with confidence values summed)• If PIR and video disagree, the correct location is the one
associated with a recognised atomic activity• If both or neither disagreeing reports are supported by an
activity, we assume the one with the higher confidence holds– If confidences are equal, we trust PIR
Fused Location
18
• Facts now represent the state of each distinct sensor– Instead of the history of sensor events– To detect state change, previous state is also stored
• Changes are necessary only for rules at the lowest level– Second and higher-level rules remain unchanged
• Transparent to the way sensor events are generated• Any state change event is linked to a related atomic
activity– Holds for DC sensors, electricity and water flow meters– Rules fire only once when sensor values change – no need for
“clean up” rules
From Offline to Real-time
19
• Each consecutive activation/deactivation of a PIR sensor corresponds to the user being in the associated room
• Subsequent activations extend the user’s stay when– Activation directly follows the last deactivation– The elapsed time between them does not exceed a threshold– No activation has taken place in a different room in the
meantime• State-based approach is not applicable for video sensors
– Video cameras do not broadcast a single value• Each reported bounding box is stored briefly
– Combined to create facts that represent a period of stay in a room
– The same heuristics used for ghost detection
Online Localisation (1)
20
• Each time a PIR sensor is activated, the system fuses available information to decide on its validity– If there is no active video sequence and no activity detected, we
assume PIR is valid– If the active video sequence with the highest confidence agrees
with PIR, we conclude the user is in the room, summing confidence values
– If video reports a different room, we assume the user is in the room where the most recently recognised atomic activity was performed
Online Localisation (2): Fusion
21
• Offline and real-time versions implemented in Java, using Jess as a rule engine– Implemented rules designed to accommodate variable reporting
periods• Real-time version built as an MQTT client
– Sensor messages are broadcast in separate threads• 10 participants executed an ADL script of half-hour
duration, twice, in the SPHERE house– Ground-truth data acquired through annotation of video images
collected using a head-mounted camera– Subset of performed activities that are recognised: interaction
with doors, electrical devices and water taps, preparing a snack/drink, washing hands/dishes, brushing teeth, bathing/showering
Implementation and Data Collection
22
• TP (true positive): activity performed and recognised• FP (false positive): activity not performed but recognised• FN (false negative): activity performed but not recognised• ,
Evaluation Results
Outline
23
Introduction
• Activity Recognition• The SPHERE Project• Related Work
Rule-based ADL Recognition
• Offline Version• Online Version• Experimental Evaluation
Conclusions & Future Work
Concluding remarks
• Rule-based system capable of operating on both historical and real-time, multi-modal sensor data acquired in a smart home– Bottom-up, multi-level rules to support complex ADL scenarios– Non-deterministic patterns to account for missing activities
• Sensor fusion and heuristics to achieve robustness against noise– 95% recall and 88% precision on average for a significant subset
of activities– 93% room-level localisation accuracy due to effective ghost
detection and location fusion rules
24
25
• Integrate wearable sensor data– Infer activities unidentifiable with only the other sensors– Improve localisation accuracy or provide an alternative to video
cameras when they are not available/allowed• Explore multi-inhabitant scenarios
– Use localisation results to pin down activities to the person performing them
– For some activities, localisation needs to be more fine-grained than room-level
• Explore hybrid approach with Machine Learning research within SPHERE– Incorporate rules as features in ML algorithms– Use rules that act on the results of ML algorithms– Devise ML techniques to learn rules
Current and Future Work
27
• Chen, L., Nugent, C.D., Wang, H.: A knowledge-driven approach to activity recognition in smart homes. IEEE Trans. Knowl. Data Eng. 24(6), 961–974 (2012)
• Filippaki, C., Antoniou, G., Tsamardinos, I.: Using constraint optimization for conflict resolution and detail control in activity recognition. In: Keyson, D.V., Maher, M.L., Streitz, N., Cheok, A., Augusto, J.C., Wichert, R., Englebienne, G., Aghajan, H., Krose, B.J.A. (eds.) AmI 2011. LNCS, vol. 7040, pp. 51–60. Springer, Heidelberg (2011)
• Helaoui, R., Riboni, D., Stuckenschmidt, H.: A probabilistic ontological framework for the recognition of multilevel human activities. In: Mattern, F., Santini, S., Canny, J.F., Langheinrich, M., Rekimoto, J. (eds.) UbiComp 2013, pp. 345–354. ACM (2013)
• Meditskos, G., Dasiopoulou, S., Kompatsiaris, I.: MetaQ: a knowledge-driven framework for context-aware activity recognition combining SPARQL and OWL 2 activity patterns. Pervasive Mob. Comput. 25, 104–124 (2016)
• Skarlatidis, A., Paliouras, G., Artikis, A., Vouros, G.A.: Probabilistic event calculus for event recognition. ACM Trans. Comput. Log. 16(2), 11:1–11:37 (2015)
• Woznowski, P., Fafoutis, X., Song, T., Hannuna, S., Camplani, M., Tao, L., Paiement, A., Mellios, E., Haghighi, M., Zhu, N., et al.: A multi-modal sensor infrastructure for healthcare in a residential environment. In: 2015 IEEE International Conference on Communication Workshop, pp. 271–277. IEEE (2015)
References