Date post: | 17-Dec-2015 |
Category: |
Documents |
Upload: | suzan-french |
View: | 213 times |
Download: | 0 times |
Mining Minds
Presenter
12-July-2014
KHU
Information Curation LayerLow Level Context-awareness
/Information Curation Layer 2
/Low Level Context-awareness
• Introduction• Motivation• Related Works• Architecture• Tools and Technologies• Development Timeline• Current Status
3
/Introduction 4
Physical Activities
Emotional States
Social Interaction
• Heterogeneous information source around people• Daily physical activities• Social interactions• Psychological states
• Automatically collect and provide basic information to the system
/Introduction 5
Physical Activities
Emotional States
Social Interaction
Knowledge
Information
Data
• Extract information directly from raw data
• Play an important role to get useful knowledge from people• Daily habit• Behavior
/Motivation 6
• Collect, process different kind of data• Sensory data• Social data
• Analyze and extract useful information at low-level• Physical activities• Social activities• Emotional states
• Provide the interface to interact with higher level and database manager
/Related Works 7
• Activity Recognizer• [WiiRemote] The Wii Remote™ Plus controller is the heart of the
motion gaming experience on your Wii console.• [Han2012] tried to overcome the limitation of accelerometer based
activity recognition. Accelerometer is used to recognize walking, running, and stay, and audio, GPS and wifi are used to recognize bus and subway.
• There lots of segmentation works such as graph-cut based segmentation by [Pourjam2013], and mean-shift algorithm by [Atefian2013] have been proposed for human body segmentation.
/Low Level Context-awareness 8
• Emotion Recognizer• [MITMindReader] MIT’s Mind Reader software can scan faces in a
crowd to determine audience mood, a tool that may replace opinion polls and help public speakers tailor their words for maximum impact.
• Various types of classifiers have been used for the task of speech emotion recognition such as HMM, GMM, SVM, etc. [Ayadi2011].
• Several emotion research works tried to separate the original complex multiple emotion classification problem by applying hierarchical approach with combination of different classifiers [Lee2011].
/Architecture 9
High Level Context-awareness
HDFS Data Access Interface
Low Level Context-awareness
Raw Data
Personal Information
SNSInteractio
nAnalyzer
ActivityRecognize
r
EmotionRecognize
r
/Architecture 10
High Level Context-awareness
HDFS Data Access Interface
Low Level Context-awareness
Social Data(Twitter)
Attribute The-
saurus
Thesaurus Manager
System Data
Morpheme Manager
Analysis
Syntax Analyzer
Morpheme Analyzer
Attribute Extractor
Emotion Extractor
Attribute-Emotion Mapping Module
Positive, Negative Analyzer
Compilation
DMBS Connector
ArchiveListener
Remote Control Request Module
Twitter Analyzer
Sentiment
Activity Recognizer
Wearable Sensor based AR
Data Acquisition
FeatureExtraction
Training Models
Classifying
Smartphone based AR
PreprocessingFeature
Extraction
GPS Validation
Decision Maker
Video based AR
Data Acquisition
SegmentationFeature
Extraction
Classifying
Emotion Recognizer
Audio based ER
Preprocessing Classification Tree Construction
FeatureExtraction
Classifying
Video based ER
Face Detection HMM Training
FeatureExtraction
HMM Testing
Physiological sensor based ER
Statistical Feature Extraction
Non-Param Cumulative Sum Auto Associate
Neural Network
Deci
sion
Fusi
on
Synch
roniz
ati
on
Pro
babili
ty C
om
pu
tati
on
Sensory Data(Acc, GPS, Video)
Sensory Data(Heart rate, Video,
Audio)
Personal Information(behavior, interest)
Activity Label(standing, sitting, running,
…)
Emotion Label(happy, angry, boredom,
…)
/Architecture 11
High Level Context-awareness
HDFS Data Access Interface
Low Level Context-awareness
Social Data(Twitter)
Attribute The-
saurus
Thesaurus Manager
System Data
Morpheme Manager
Analysis
Syntax Analyzer
Morpheme Analyzer
Attribute Extractor
Emotion Extractor
Attribute-Emotion Mapping Module
Positive, Negative Analyzer
Compilation
DMBS Connector
ArchiveListener
Remote Control Request Module
Twitter Analyzer
Sentiment
Activity Recognizer
Wearable Sensor based AR
Data Acquisition
FeatureExtraction
Training Models
Classifying
Smartphone based AR
PreprocessingFeature
Extraction
GPS Validation
Decision Maker
Video based AR
Data Acquisition
SegmentationFeature
Extraction
Classifying
Emotion Recognizer
Audio based ER
Preprocessing Classification Tree Construction
FeatureExtraction
Classifying
Video based ER
Face Detection HMM Training
FeatureExtraction
HMM Testing
Physiological sensor based ER
Statistical Feature Extraction
Non-Param Cumulative Sum Auto Associate
Neural Network
Deci
sion
Fusi
on
Synch
roniz
ati
on
Pro
babili
ty C
om
pu
tati
on
Sensory Data(Acc, GPS, Video)
Sensory Data(Heart rate, Video,
Audio)
Personal Information(behavior, interest)
Activity Label(standing, sitting, running,
…)
Emotion Label(happy, angry, boredom,
…)
• SNS Analyzer - Twitter• Take input from Twitter
API in schema format• Analyze Twitter data in
different contexts• Activity
• Emotion
• Behavior
• Provide the output based on keyword
/Architecture 12
High Level Context-awareness
HDFS Data Access Interface
Low Level Context-awareness
Social Data(Twitter)
Attribute The-
saurus
Thesaurus Manager
System Data
Morpheme Manager
Analysis
Syntax Analyzer
Morpheme Analyzer
Attribute Extractor
Emotion Extractor
Attribute-Emotion Mapping Module
Positive, Negative Analyzer
Compilation
DMBS Connector
ArchiveListener
Remote Control Request Module
Twitter Analyzer
Sentiment
Activity Recognizer
Wearable Sensor based AR
Data Acquisition
FeatureExtraction
Training Models
Classifying
Smartphone based AR
PreprocessingFeature
Extraction
GPS Validation
Decision Maker
Video based AR
Data Acquisition
SegmentationFeature
Extraction
Classifying
Emotion Recognizer
Audio based ER
Preprocessing Classification Tree Construction
FeatureExtraction
Classifying
Video based ER
Face Detection HMM Training
FeatureExtraction
HMM Testing
Physiological sensor based ER
Statistical Feature Extraction
Non-Param Cumulative Sum Auto Associate
Neural Network
Deci
sion
Fusi
on
Synch
roniz
ati
on
Pro
babili
ty C
om
pu
tati
on
Sensory Data(Acc, GPS, Video)
Sensory Data(Heart rate, Video,
Audio)
Personal Information(behavior, interest)
Activity Label(standing, sitting, running,
…)
Emotion Label(happy, angry, boredom,
…)
• Activity Recognizer• Take input from
different sensors• Wearable sensors
• Smartphone’s sensors
• Video sensors
• Recognize activities based on specific machine learning algorithms for each kind of data
• Provide output as activity label in text format
/Architecture 13
High Level Context-awareness
HDFS Data Access Interface
Low Level Context-awareness
Social Data(Twitter)
Attribute The-
saurus
Thesaurus Manager
System Data
Morpheme Manager
Analysis
Syntax Analyzer
Morpheme Analyzer
Attribute Extractor
Emotion Extractor
Attribute-Emotion Mapping Module
Positive, Negative Analyzer
Compilation
DMBS Connector
ArchiveListener
Remote Control Request Module
Twitter Analyzer
Sentiment
Activity Recognizer
Wearable Sensor based AR
Data Acquisition
FeatureExtraction
Training Models
Classifying
Smartphone based AR
PreprocessingFeature
Extraction
GPS Validation
Decision Maker
Video based AR
Data Acquisition
SegmentationFeature
Extraction
Classifying
Emotion Recognizer
Audio based ER
Preprocessing Classification Tree Construction
FeatureExtraction
Classifying
Video based ER
Face Detection HMM Training
FeatureExtraction
HMM Testing
Physiological sensor based ER
Statistical Feature Extraction
Non-Param Cumulative Sum Auto Associate
Neural Network
Deci
sion
Fusi
on
Synch
roniz
ati
on
Pro
babili
ty C
om
pu
tati
on
Sensory Data(Acc, GPS, Video)
Sensory Data(Heart rate, Video,
Audio)
Personal Information(behavior, interest)
Activity Label(standing, sitting, running,
…)
Emotion Label(happy, angry, boredom,
…)
• Emotion Recognizer• Take input from different
sensors• Audio sensor
• Video sensors
• Physiological sensors
• Recognize emotions based on specific machine learning algorithms for each kind of data• Apply Fusion technique to
increase confident of predict output from different decisions.
• Provide output as emotion label in text format
/Tools and Technologies 14
• Tools for development• MATLAB• Eclipse• Android SDK
• Technologies• Machine Learning
• Platforms• Microsoft Windows• Android OS
/Development Timeline 15
1st year
2nd year
Interface Definition
Adapter Development
Component Modification
Component Validation
Evaluate Components based on collected data
Component Adjustment
Output
Modified Compo-nents
Validated Compo-nents
Evaluation Report
1st Integration Phase
2nd Evaluation Phase
Final module
Interface Report
/Current Status 16
• Social Interaction Analyzer• Need to define the input and output to interact with Tapacross’s
engine
• Activity and Emotion Recognizer• Each individual module is available and ready for integration
/References 17
[WiiRemote] https://www.nintendo.com/wii/what-is-wii/#/controls[Han2012] Manhyung Han, La The Vinh, Young-Koo Lee and Sungyoung Lee, "Comprehensive Context Recognizer Based on Multimodal Sensors in a Smartphone", Journal of Sensors, vol. 12, no. 9, pp. 12588-12605, 2012[Pourjam2013] Pourjam, E., Ide, I., Deguchi, D., & Murase, H. Segmentation of Human Instances Using Grab-cut and Active Shape Model Feedback. In proceddings of MVA2013 IAPR International Conference on Machine Vision Applications, pp. 77–80, May 20–23, 2013.[Atefian2013] Atefian, M., & Mahdavi-Nasab, H. (2013). A Robust Mean-Shift Tracking Using GMM Background Subtraction, J. Basic. Appl. Sci. Res., vol. 3, no. 4, 596–607, 2013.[MITMindReader] http://trac.media.mit.edu/mindreader/[Ayadi2011] Ayadi, M.E., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44 (3), 572 - 587 (2011).[Lee2011] C.-C. Lee, E. Mower, C. Busso, S. Lee, and S. Narayanan. Emotion recognition using a hierarchical binary decision tree approach. Speech Commun., 53(9-10):1162-1171, Nov. 2011.