School of Electrical and Electronic EngineeringNanyang Technological University Singapore
XU SHIHAO
Xu Shihao, ZixuYang, Debsubhra Chakraborty, Yasir Tahir, Tomasz Maszczyk Chua Yi Han Victoria, Justin Dauwels, Daniel Thalmann, NadiaMagnenat Thalmann, Bhing-Leet Tan and Jimmy Lee Chee Keong
Introduction
Design of Experiment
System Overview
Results
Conclusion
2
One in four people suffer from mental illness at some point in their lives1
Nearly two-thirds of them never seek treatment1
Current clinical assessments and treatments require highly trained clinicians
Methods are subjective and time-consuming Can we move towards automated objective assessments?
31. 2001 WHO Press Release on Mental Illness
A chronic, mental disorder
Heterogeneous symptoms1: ➢ Positive symptoms➢ Negative symptoms➢ Cognitive symptoms
Schizophrenia patients display unusual ways of talking or disorganized speech2
Negative symptoms: Reduction of social behavior, such as speech
Negative symptoms contribute to poor function and quality of life for patients3
4
1. Demily, C., & Franck, N. (2008). Cognitive remediation: a promising tool for the treatment of schizophrenia. Expert Review of Neurotherapeutics, 8(7), 1029-1036.
2. DeLisi, L.E., 2001. Speech disorder in schizophrenia: review of the literature and exploration of its relation to the uniquely human capacity for language. Schizophrenia Bulletin, 27(3), pp.481-496..
3. Kirkpatrick, Brian, et al. "The NIMH-MATRICS consensus statement on negative symptoms." Schizophrenia bulletin 32.2 (2006): 214-219.
Current diagnosic methods➢ Physical exam➢ Tests and screenings➢ Psychiatric evaluation (NSA-16)➢ Based on diagnostic criteria
Treatments➢ Lifelong treatment ➢ Hospitalization may be needed➢ Few clinical treatments for negative symptoms
5
Aim:Automatically identify objective cognitive biomarkers: Linguistic/Lexical Features
1. Can the automated objective measurements be used to predict the subjective ratings (NSA-16)?
This would enable automated long-term assessments of negative symptoms
2. Can the automated objective measurements differentiatebetween patients and healthy individuals?
This would enable more objective diagnosis of mental illnesses
6
Introduction
Design of Experiment
System Overview
Results
Conclusion
7
Study in collaboration with Institute of Mental Health in Singapore 2 types of participants
▪ Schizophrenia patients (Patients)(47)
▪ Healthy (Controls) (24)
8
Patients (47)
Controls (24)
AgeMean (years) 30.4 29.8
Range (years) 20-49 19-47
GenderMale 22 10
Female 25 14
Ethnicity
Chinese 39 20
Malay 5 3
Indian 3 1
Education
University 6 4
Diploma/ Vocational 26 14
High School 15 6
Each participant underwent a semi-structured interview Each Interview was conducted by a trained psychologist No pre-determined time-limit for the interview No role-playing during interview On average, interview lasts for 25 minutes Each interview audio is analyzed in its entirety
9
10
• The psychometric scale used is Negative Symptoms Assessment-16 (NSA-16)
• One of few rating instruments specifically designed for negative symptoms
• 16 item scale with scores for speech behaviour, emotional behaviour, affect, movement and daily activity etc.
• Ratings are on a scale of 1-6
Behaviour Score
Fully normal behaviour 1
Behaviour minimally reduced 2
Behaviour mildly reduced, but noticeable by trained rater
3
Behaviour moderately reduced, noticeable even by untrained rater
4
Behaviour is markedly reduced, hampering social function
5
Behaviour severely reduced or absent 6
Introduction
Design of Experiment
System Overview
Results
Conclusion
11
12
Audio RecordingSpeaker Diarization and
Speech recognition Lexical Feature
Extraction
Classification and Prediction
?
Two lapel microphones record the audio of the participant and psychologists
System pipeline
Speaker Diarization (HMM) Segment out the participants’ speech and filter the interferences
13
Speaker Diarization (HMM) Segment out the participants’ speech and filter the interferences
14
Google Cloud Speech API (Cloud Speech-to-Text) Powered by deep learning and NLP Returns text transcription in Real-Time Handles noisy audio Filters inappropriate content
15
Speech APIParticipant
mean STD1 2 3 4 5 6 7 8 9
Google Cloud 0.87 0.79 0.84 0.84 0.66 0.82 0.78 0.83 0.93 0.82 0.07
stream
text
Transcribe accuracy
16
Linguistic Inquiry and Word Count 1 (LIWC2015) Extract number of linguistic words related to thoughts, feelings, personality,
and motivations. 78 output categories in LIWC2015
1. https://liwc.wpengine.com/
Each category we normalized by the duration of the interview
Category name Abbrev Examples Words in category
Common Adverbs adverb very, really, basically 140
Netspeak netspeak coz, thx, yeah 209
Female references female Mom, sister, her, she 124
Feel feel Feel, touch, warm 128
Binary supervised machine learning (Scikit-learn1)
Features: normalized lexical features
Target labels: Healthy vs. Schizophrenic Patient
Leave-one-out cross-validation with feature selection
17
Patient or Healthy?
1. http://scikit-learn.org/stable/
Features: normalized lexical features
Target labels: Unobservable vs. Observable
➢ Unobservable: NSA score 1-2
➢ Observable: NSA score 3-6
Leave-one-out cross-validation with feature selection
18
NSA-16 indices
Introduction & Literature Review
Design of Experiment
System Overview
Results
Discussion & Conclusion
19
20
Confusion Matrix
Precision Recall F-score AUC Accuracy Baseline ClassifierFeature
evaluatorPredicted class
Patient Healthy
True class
Patient 41 6 0.89 0.87 0.88 0.8384.5% 66.2%
Logistic Regression
Logistic RegressionHealthy 5 19 0.76 0.79 0.78 0.83
True class
Patient 40 7 0.91 0.85 0.88 0.8284.5% 66.2% SVM SVM
Healthy 4 20 0.74 0.83 0.78 0.82
True class
Patient 40 7 0.89 0.85 0.87 0.8183.1% 66.2%
Logistic Regression
SVMHealthy 5 19 0.73 0.79 0.76 0.81
Classification Metrics ( 47 Patients v/s 24 healthy controls)
Automated objective measurements (lexical features) differentiate between patients and healthy individuals
21
Correlation matrix between NSA-16 scores and Lexical features
NSA items
Confusion Matrix
AUC Accuracy Baseline ClassifierFeature
EvaluatorPredicted class
low high
NSA2: Restricted Speech Quantity
True class
low 24 5 0.7778.7% 61.7% Decision tree
Logistic Regressionhigh 5 13 0.77
NSA6: Reduced modulation of
intensity
True class
low 22 3 0.7168.1% 53.2%
K-NeighborsClassifier
Gradient Boostinghigh 12 10 0.71
NSA items can be predicted by Lexical features
NSA
Scores
words / min
22
Kruskal–Wallis Test (p-value < 0.01)
Words patients used less Words patients used more
Category example P-value Category example P-value
Informal netspeak, swear words 0.0029 Female lady, sister, mom 0.0024
Netspeak coz, thx, lol 0.0046 Feel hate, kill, annoyed 0.0602
Assent agree, ok, yes 0.0056 Family mom, father, wife 0.0935
Adverbs very, really, quickly 0.009
23
‘Female’ words (per min) ‘Male’ words (per min)
Patient
Mean STD
Patient
Mean STD
Female 0.39 0.37 Female 0.22 0.14
Male 0.17 0.11 Male 0.15 0.08
Healthy
Mean STD
Healthy
Mean STD
Female 0.1 0.13 Female 0.22 0.32
Male 0.16 0.16 Male 0.21 0.1
Female patients used more female words during interview
Patients may have greater access and higher proximity to a social support network of female caretakers
(Patients vs. Healthy), (Female vs. Male), (Female words vs. Male words)
24
Pearson correlation test
Patients used more ‘feeling’ words related to themselves
Verified that due to impairment in social cognition, schizophrenic patients tend to focus on themselves more
feeling & I feeling & family
Correlation p-value Correlation p-value
Patients 0.648 8.66E-07 0.287 0.05
Healthy 0.5 0.0129 0.026 0.905
Introduction & Literature Review
Design of Experiment
System Overview
Results
Conclusion
25
Objective Lexical features can be utilized to detect the differences of behavior of schizophrenic patients
Increase our dataset Apply more advanced Natural Language Processing method
We have just started a similar study related to depression
26
Mental Illness
•Schizophrenia
•Depression
Symptoms
•Negative symptoms
•Neurocognitive impairments
•Social cognitive deficits
Objective cues
•Speech
•Motor
•Facial expressions
• Xu Shihao• Debsubhra Chakraborty• Dr. Yasir Tahir• Dr. Tomasz Maszczyk• Dr. Victoria Chua Yi Han
• Dr. Jimmy Lee• Dr. Bhing-Leet Tan• Dr. Zixu Yang
▪ External Collaborators (Institute of Mental Health, Singapore)
27
▪ Lab Members
▪ NTU Collaborators (IMI)• Prof. Nadia Magnenat-Thalmann• Prof. Daniel Thalmann
▪ Supervisor: Justin Dauwels
Supervisor: Justin DauwelsSchool of Electrical and Electronic EngineeringNanyang Technological University Singapore
XU SHIHAO
