Date post: | 08-Jan-2017 |
Category: |
Data & Analytics |
Upload: | euroiota |
View: | 118 times |
Download: | 0 times |
Outline
• Introduction
• Hybrid Neural Network (HNN)
• HNN for Real
• Preliminaries
• TreNet: a HNN for learning the local trend of time series
• Experiment results
• Conclusion
2
Introduction
• Time series data: a sequence of data points consisting of
successive measurements made over time.
• Internet of Things
• Sensor networks
• Mobile phones
• And more…
4 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Time series analytics in a variety of applications
• Classification
• Prediction
• Anomaly detection
• Pattern discovery
• And more…
5
Pattern 1 Pattern 2
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Time series analytics tools
• Statistics
• Hidden Markov Model (HMM)
• State Space Model
• ARIMA
• Machine learning
• Random Forest
• SVM
• Gaussian Process
•
6
Random Forest
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Neural Network and Deep Learning
• Language translation
• E.g., Google’s Multilingual Neural Machine Translation System
• Computer vision
• E.g., Microsoft Research’s PReLU network outperforms Human-Level
performance on ImageNet Classification
• Speech recognition
• E.g., Amazon Echo, Apple Siri
• Time series classification
• E.g. recognize patterns in multivariate
time series of clinical measurements
7
Z. Lipton, et al. “Learning to Diagnose with LSTM Recurrent Neural Networks”. ICLR, 2016.
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Fundamental network architectures
• Convolutional neural network: input two-dimensional data, e.g.,
image
• Recurrent neural network: input
8
Unfolded recurrent connections in a RNN
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Convolutional Neural Network (CNN)
• Feature learning for images
• To extract high-level features from raw data
• Such high-level features are further used for classification or
regression.
9
K. He, et al. “Delving deep into rectifiers: Surpassing Human-Level Performance on ImageNet Classification”. arxiv.org, 2015
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Convolutional Neural Network (CNN)
• CNN in the Human Activity Recognition Problem
• Multichannel time series acquired from a set of body-worn sensors
• To predict human activities by training a CNN over time series
10
J. Yang, et al. “Deep Convolutional Neural Networks On Multichannel Time Series For Human Activity Recognition”. IJCAI 2015
Value
Time
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Recurrent Neural Network (RNN)
• A powerful tool to model sequence data
• To capture dependency in sequence data
• Long-short term memory network (LSTM)
• A widely used variant of RNNs
• Equipped with memory and gate mechanism
• To overcome gradient vanishing and explosion
11
S. Hochreiter, et al. “Long short-term memory’’. Neural computation, 1997.
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• CNN or RNN
• Work well for respective data, i.e. images and sequence data
13 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
• Electroencephalogram (EEG) : multiple time series corresponding to
measurements across different spatial locations over the cortex.
• Mental load classification task:
measures the working memory
responsible for transient retention
of information in the brain.
14
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
• A key challenge in correctly recognizing mental states
• EEG data often contains translation
and deformation of signal in space,
frequency, and time, due to inter-
and intra-subject differences
15
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
16
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• A cascade of CNN and RNN
• EEG data classification
17
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• Time Series
• Noisy
• Non-stationary
• Hidden information: states, dynamics
• Auto-correlated on the temporal dimension
• Manual feature engineering
• Preprocessing: de-trending, outlier removal, etc.
• Dimension reduction: Fourier Transformation
• Piecewise approximation: PAA, PCA, PLA, etc.
• Application-specific, domain knowledge
18
Why do we need hybrid architectures?
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• Hybrid architectures: end-to-end learning framework
• Loss function driven training
• Learning representative features
• Capturing sequential dependency in data
19
Why do we need hybrid architectures?
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• A cascade of CNN and RNN
20
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
HNN for Real:
TreNet for Learning the Local Trend
21
T. Guo, et al. TreNet: Hybrid Neural Networks for Learning the Local Trend in Temporal Data. In submission to ICLR, 2017
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Preliminaries
• Conventional trend analysis in time series
is the seasonal component at time t
is the trend component at t
is the remainder
22 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Preliminaries
• Local trend
• Measure the intermediate local behaviour, i.e. upward or downward
pattern of time series
• For instance, the time series of household power consumption and
the local trends are shown as follows:
• Time series
• Extracted local trend ,
is the duration and is the slope
23 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Preliminaries
• Learning and forecasting the local trend
• Predict ,
• Useful in many applications
• Smart energy
• Stock market
• And more …
24 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Preliminaries
• Learning and forecasting the local trend
• Local raw data
• Global contextual information
in the historical sequence of trend
• To learn a function
is either or
25 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• Overview of TreNet
• is derived by training the LSTM over sequence to
capture the dependency in the trend evolving.
• corresponds to local features extracted by CNN from
26 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
is derived by training the LSTM over sequence to capture
the dependency in the trend evolving.
corresponds to local features extracted by CNN from
TreNet
• Overview
•
27 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• LSTM
• Feed the sequence of
• Output
29 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• Feature Fusion and output layers
31 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• Learning
• Gradient descent
32 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Experiments
• Datasets
• Daily Household Power Consumption (HousePC)
• Gas Sensor (GasSensor)
• Stock Transaction (Stock)
33
E Keogh, et al. “An online algorithm for segmenting time series”. ICDM, 2001
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Experiments
• Baselines
• CNN
• LSTM
• Support Vector Regression (SVR)
• Radial Basis kernel (SVRBF)
• Polynomial kernel (SVPOLY)
• Sigmoid kernel (SVSIG)
• Pattern-based Hidden Markov Model (pHMM)
34 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
P. Wang, et al. “Finding Semantics in Time Series”, SIGMOD 2011
Experiments
• Results: overall accuracy
35 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Experiments
• Results: prediction visualization
36 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Conclusion
38
• Hybrid neural networks
• TreNet for the Local Trend Learning
• Future work: a generic idea
• Social media streams
• Heterogeneous data
• Influence analysis
• And more…
Thanks! Q & A
This work is supported by EU OpenIoT Project (Open Source Solution for the Internet of Things)
http://www.openiot.eu/
Feel free to contact: [email protected], [email protected]