Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition

Seminar Paper: Emotion Recognition from Physiological signals using Bio-sensors

Revathi priya Muthusamy,

University of Fribourg, Switzerland,

Revathipriya.muthusamy@unifr.ch

Abstract: This paper presents an introduction to various bio-signals used in Emotion recognition such as Electromyography (EMG) electrocardiogram (ECG), electrodermal response (EDR), Skin temperature, and Blood volume pulse (BVP) and respiration changes. This paper also discusses about various kinds of bio-sensors used to detect the Physiological signals. In the remaining of the paper we describe the Multimodal emotion elicitation and Acquisition of emotion-specific physiological signal database. It also deals with feature extraction technique based on the Hilbert-Huang Transform (HHT) method and classification techniques such as Support Vector Machine. the highest recognition accuracy is obtained by using the fission based HHT features method with 28 features, 76% of test cases are correctly classified. Keywords: Bio-sensors, Multimodal emotion elicitation, Hilbert-Huang Transform, Support Vector Machine

1. Introduction to Biomedical Signals Most physiological processes produce signals of several types: Biochemical -in the form of hormones. Electrical - in the form of current, and Physical - in the form of pressure and temperature. [1] For example when we get afraid of something our heart races, our muscles tense, our mouth becomes dry. These changes are controlled by automatic nervous system, which manages heart muscles, smooth muscle and various glands in our body. [7] These bodily reactions can be measured and monitored. These signals are called as bio-signals. We can monitor the body reactions from outside by use of Bio sensors and classify the bio signals corresponding for each emotion.

Fig 1: The Autonomic nervous system and the organs controlled by it. We study the following Bio-signals for Emotion recognition:

Electrocardiogram (ECG). An electrocardiogram (ECG) is a test which measures the electrical activity of the heart. ECG is used to calculate the rate and regularity of heartbeats. Can be recorded from above the chest or limbs. ECG measurements from the limbs is less inconvenient but more vulnerable to artifacts. [1]

We can measure heart rate (HR) and inter-beat intervals (IBI) and thus we can determine the heart rate variability (HRV). A low HRV can indicate a state of relaxation, whereas an increased HRV can indicate frustration or state of mental stress.

Fig 2: (a) heart with Nervous system intervention (b) Heart rate Variability HRV Electromyography (EMG) is a technique used to record electrical potential created by muscle membranes when there is an electrical or neurological triggering. For example a high muscle tension is created when there is stress or frustration. These signals can be measured by placing the bio sensors over face or hands. [3]

Fig 3. Anatomy of hand (left), typical EMG signal (right) Electrodermal response (EDR)-also called as skin conductivity (SC) measures the conductivity of the skin. For example EDR increases if the skin is sweaty. This signal is a sensitive indicator of stress and for other stimuli. It is also used to classify between conflicts-no conflict situations or classify between anger and fear. The main disadvantage of this signal is that it is influenced by external factors like outside temperature. Hence reference measurements and calibrations are recommended. [3]

Figure 4: Anatomy of skin (left), typical EDR signal (right) Skin temperature can be measured by determining the temperature on the surface of the skin. For example under strain, muscles are tensed, the blood vessels will be contracted and hence the temperature will decrease. Similar to the Electro dermal

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition

response the skin temperature also depends on external factors. Furthermore it is a relatively slow indicator of changes in emotional state. Blood volume pulse (BVP) is a measured by determining the amount of blood currently running though the vessels, e.g. in the finger of a test subject. A photoplethysmograph (PPG) is used to measure BVP, it consists of a light source and photo sensor. It is attached to the skin and the amount of reflected light, which depends on the amount of blood, is measured. [1] Respiration sensors measure how deep and fast a person is breathing. This is measured by applying a rubber band around the chest. Fast and deep breathing can indicate excitement such as anger or fear but sometimes also joy. Rapid shallow breathing can indicate tense anticipation including panic, fear or concentration. Slow and deep breathing indicates a relaxed resting state while slow and shallow breathing can indicate states of withdrawal, passive like depression or calm happiness.

2. The Bio-Sensors A Bio Sensor is an analytical device used to convert a biological response into electrical signal.

Figure 5. Main components of Biosensor

Figure 5. Examples of Bio-sensors. Figure [5.1] shows the sensors used for measuring EMG. We measure the muscle activity of the masseter muscle, [7] the muscle movement is described to be reliable in that part of body. Figure [5.2] shows a standard ECG sensor. In Figure [5.3] the respiration sensor is shown applied to the chest while in Figure [5.4] the skin conductivity, BVP and temperature sensor can be seen applied to the fingers of the left hand (the non-dominant hand should be used for measuring).[7]

3. Data Collection Experiments Acquisition of emotion-specific physiological signal database: The physiological signals were acquired using the MP100 system. The sampling rate was fixed at 256 samples s 1 for all the channels. Appropriate amplification and bandpass filtering were performed. The subjects were requested to be as relaxed as possible during this period. Subsequently, emotional stimulus was applied and debriefing and recovery followed. [3] Emotion induction protocol: We study that visual stimulation using still images was not sufficient for effective emotion induction, and we did not study the international affective picture system (IAPS) developed by LANG et al. (1988), despite its being

adopted for many psychophysiological studies involving emotion induction.

Fig.6 Illustration of example of emotion induction protocols'. Its" purpose is to induce status of 'sadness'[3] We study the protocol utilizing a multimodal (audio, visual and cognitive) approach to evoke specific targeted emotional statuses, and it was developed in collaboration with specialists from the field of cognitive and physiological psychology (YANG et al., 2000). The emotion induction protocols are summarized in Table. A preliminary test of the protocols was performed for 80 subjects aged from seven to eight years. [3]

4. Feature Extraction

4.1 Preprocessing, waveform detection and feature extraction: The first necessary step was the detection of the characteristic waveform and extraction of useful information-bearing features for pattern classification. As shown in Fig. 7, the baseline values of each component of the feature vectors were subtracted before they were given to the classifier. Here, the baseline values mean the components of feature vectors extracted from 50 s segments of signals that were acquired without stimulus. [3]

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition

Fig. 7 Overall structure of emotion recognition system [3] [4] 4.1.1. From ECG signals - RR interval and heart rate variability:

Fig.8 Block diagram of information extraction module from ECG Fig.8 above shows the block diagram of the feature extraction module for the heart rate. Heart rate variability (HRV) contains abundant information on the status of the autonomic nervous system and can be derived from ECG or PPG. [3] Frequency-domain features of HRV have also been considered to be significant for the exploration of the autonomic nervous system in many previous studies for cardiac function assessment and psycho physiological investigation (DRUMMOND and QUAH, 2001; MCCRATY et al., 1995).[3] 4.1.2 EMG Recordings and Data Reduction

Fig 9 supercilii and the zygomaticus muscles for EMG signal acquisition. [5] Facial EMG was measured by bipolar attached electrodes placed over the corrugator supercilii and the zygomaticus major on the left side of the face, according to the guidelines given by Fridlund and Cacioppo (Figure 9). [5]

Fig 10. Feature extraction from EMG signal [4] 4.1.3 Electrodermal activity:

Fig 11 (a) Typical waveform of EDA under emotional stimulation. (b) Output signal from detection module in (a). (c) Block diagram of SCR detection module. Sampling rate of traces in (a) and (b) is 21.3 samples s z.[3] Electrodermal activity (EDA) was obtained by measurement of the voltage between two electrodes across which a low-level current was applied. Fig. 11 a shows a typical waveform of EDA under emotional stimulation (after subtraction of mean EDA level). Important features of EDA include the DC level and the distinctive short waveforms that are indicated by arrows in Fig. 11. [3] 4.1.4 Skin temperature variation: No special signal processing was necessary for the feature extraction from the skin temperature (SKT). Although frequency-domain analysis of the time-varying SKT has been reported (SHUSTE~MAN and BARNEA, 1995), here the mean and maximum values within 50 s intervals were used as the features of SKT.[3]

5. Classification Experiments After extracting the features as described in the previous section, statistical classifier is used for learning the corresponding emotion for a set of features with which it is presented. There are different options for building such a classifier. Hilbert Haung transform, wavelet transform, neural network classifier are some of the widely used Classification methods. 5.1 Hilbert Haung transform Huang et al. have proposed a new tool for the analysis of nonlinear and non-stationary data called as the Hilbert-Huang transform method (HHT)[2]. Fission process- Empirical Mode Decomposition (EMD) decomposes a signal into basis functions which are finite called the intrinsic mode functions. Fussion process- After this fission process, the intrinsic mode functions (IMFs) of interest are combined in an ad hoc or automated fashion in order to provide a greater knowledge. [2].

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition

Fig 12. HHT Fission and Fusion methods

Fission process- Each signal is decomposed into a finite set of AM-FM mono components by using the Empirical Mode Decomposition (EMD) which is the important part of the HHT. Fusion process- For the next classification stage, the information components of interest are then combined to create feature vectors [2].

6. Results

Table 2. SVM Classification rates of four emotions for four bio signals. (TD : Time Domain ; FD : Frequency Domain)[2] The features from all methods ( the state-of-the-art method, the fission process and the fusion process based HHT features method) for four bio signals are combines and classified using Support Vector Machine (SVM). The classification results are as listed below

Table 3. Classification results [2].

7. Conclusion In this paper we have surveyed research works using Physiological sensors to recognize human emotions. In this article, we discus about characteristic waveform detection, feature extraction and pattern classification stages. The HHT Hilbert-Huang transform method classified signals by 71 % for Baseline features and 76% for fission method and 62% for fusion method. [2]. Future Research scope: There are clearly more steps to take in this research area. Research has to be done to improve the accuracy of the network. This could be done either by adding some meaningful features (the full feature set) that are computed from the existing sensor signals or by using different bio-signals such as EEG data. Also research has to be done to for incorporation of different sources for emotion recognition such as

video analysis, motion detection or emotion recognition from speech signals to bring a real emotional dialog system to work.

References: 1. Rangaraj M. Rangayyan; Biomedical Signal Analysis – A Case-Study Approach; IEEE Press 2002 2. C. Zong and M. Chetouani, "Hilbert-Huang transform based physiological signals analysis for emotion recognition", in proc. of IEEE International Symposium on Signal Processing and Information Technology, pp. 334-339, 2009. 3. K. H. Kim I S.W. Bang 2 S.R. Kim 2Emotion recognition system using short-term monitoring of physiological signals, 1 Department of Biomedical Engineering, College of Health Science, Yonsei University, South Korea. 2 Human-computer Interaction Laboratory, Samsung Advanced Institute of Technology, South Korea 4. Cheng-Ning Huang Chun-Han Chen Hung-Yuan Chung, The Review of Applications and Measurements in Facial Electromyography,Department of Electrical Engineering, National Central University, Chung-Li, Tiwan, 320 ROC 5. Anton van Boxtel,Facial EMG as a Tool for Inferring Affective States, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands, a.vanboxtel@uvt.nl, Tilburg University, Tilburg, The Netherlands, 6. Marco Tamietto , Emotional Contagion for Unseen Bodily Expressions: Evidence from Facial EMG, Cognitive and Affective Neurosciences Laboratory, Department of Psychology, University of Torino, Torino, Italy, M.Tamietto@uvt.nl 7. Andreas Haag, Silke Goronzy, Peter Schaich, “Emotion Recognition Using Bio-Sensors: First Steps Towards an Automatic System” Jason Williams,Sony Corporate Laboratories Europe, Hedelfinger Str. 61,D-70327 Stuttgart, Germany {Haag, Goronzy, Schaich, Williams@Sony.de} 8. FATMA NASOZ, KAYE ALVAREZ, “Emotion Recognition from Physiological Signals for Presence Technologies”, In International Journal of Cognition, Technology, and Work - Special Issue on Presence, Vol.6(1), 2003. 9.Lang, P.J.,Bradley, M.M., & Cuthbert, B.N. (2001). International affective picture system (IAPS):Instruction Manual and Affective Ratings. Technical Report A-5. Gainesville, FL. The Research in Psychophysiology, University of Florida. 10. Center for the Study of Emotion and Attention [CSEA-NIMH] (2001). The international affective picture system: Digitized photographs. Gainesville, FL: The Center for Research in Psychphysiology, Univerity of Florida 11. Barreto A., Heimer M., and Garcia M., “Characterization of Photoplehtysmograpic Blood Volume Pulse Waveforms for Exercise Evalution,” Proceedings 14th Southern Biomedical Engineering Conference, Shreveport, Louisiana, April, 1995, pp. 220-223 12. Chrisite, “Multivariate Discrimination of Emotion-specific Autonomic Nervous System Activity”, Master Thesis in Science of Psychology, Virginia 13. Healy J. and Picard R., “Digital processing of Affective Signals”, ICASSP 98