Date post: | 22-Jan-2018 |
Category: |
Technology |
Upload: | setu-chokshi |
View: | 256 times |
Download: | 4 times |
Time Series Predictions using Long Short-Term
Memory
Setu Chokshi
IoT Asia 2017 – 30th March
LSTMs are mainstream
What are Neural Network
Y1
Y2
X1
X2
X3
h1
h2
h3
h4
Information Transfer
Input Values
Calculator – Activations
Output (Activation)
1
2
3
4
Input Hidden Output
X1
X2
X3
W11
W21
W31
h1
Example of = REctified Linear Unit = max(0,input)
Strengthen weak signals; Leave strong signals alone
Weight is the strength of the connection between nodes
1
Challenges
Only fixed sized inputs & outputs
Performs mapping of features from input to output
No memory and hence difficult to model time series
Lets add some memory
Input Hidden Output
Input + prevInput Hidden Output Input+ prevHidden Hidden Output
Input + prevInput Hidden Output
Input + prevInput Hidden Output
Input + prevInput Hidden Output
Input + prevInput Hidden Output
Input + prevHidden Hidden Output
Input + prevHidden Hidden Output
Input + prevHidden Hidden Output
Input + prevHidden Hidden Output
Approach 1: Add previous inputs Approach 2: Add previous hidden
Lets do 4 time series steps Lets do 4 time series steps
Lets add some memory…and color
Input Hidden Output
Input + prevInput Hidden Output Input+ prevHidden Hidden Output
Input + prevInput Hidden Output
Input + prevInput Hidden Output
Input + prevInput Hidden Output
Input + prevInput Hidden Output
Input + prevHidden Hidden Output
Input + prevHidden Hidden Output
Input + prevHidden Hidden Output
Input + prevHidden Hidden Output
Approach 1: Add previous inputs Approach 2: Add previous hidden
Lets do 4 time series steps Lets do 4 time series steps
Lets build an LSTM
Xt
0 1 2 3
ht-1
Ct-1 Ct✖
✖
✖σ σ σtanh
tanh
ht
ht
Ct-1
Now lets build an LSTM
Element-wiseSummation /Concatenation
Element-wisemultiplication
Xt
ht-
1
Ct-
1
ht
Ct
0
σ
✖
tanh
Inputs: Outputs:
Input vector
Memory from previous blockOutput of previous block
Memory from current blockOutput of current block
Nonlinearities:
Sigmoid
Hyperbolic tangent
Vector operations:
Bias:
Lets skip the math, ok?
Element-wisesummation
Element-wisemultiplication
✖ = =
Memory Pipeline
Xt
0 1 2 3
ht-1
Ct-1 Ct
ht
✖
✖
✖σ σ σtanh
tanh
ht
✖
Forget Layer
Xt
0 1 2 3
ht-1
Ct-1 Ct
ht
✖
✖
✖σ σ σtanh
tanh
ht
Generate new memories: Input
Xt
+ 0 1 2 3
ht-1
Ct-1 Ct
ht
✖
✖
✖σ σ σtanh
tanh
ht
2
tanh
Generate new memories: Candidate
Xt
0 1 2 3
ht-1
Ct-1 Ct
ht
✖
✖
✖σ σ σtanh
tanh
ht
Memory Pipeline
Xt
0 1 2 3
ht-1
Ct-1 Ct
ht
✖
✖
✖σ σ σtanh
tanh
ht
Generate the output
Xt
0 1 2 3
ht-1
Ct-1 Ct
ht
✖
✖
✖σ σ σtanh
tanh
ht
EXAMPLES
Sin wave predictor
Generate sin curve
Load 5000 X 50 sequences
90:10 split on train/test sets
Power Consumption Dataset
Power Consumption Dataset
47 Months of data
2075259 measurements
Active energy consumed per min
Load 4567 X 50 sequences
90:10 split on train/test sets
References
Understanding LSTM Networks
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
General Sequence Learning using Recurrent Neural Networks
https://www.youtube.com/watch?v=VINCQghQRuM
Recurrent Neural Networks Part 1: Theory
https://www.slideshare.net/gakhov
Facebook Prophet
https://github.com/facebookincubator/prophet
Images adapted from Shi Yan
https://medium.com/@shiyan/understanding-lstm-and-its-diagrams-37e2f46f1714
Anyone Can Learn To Code
https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/
THANK YOU
@setuc
www.linkedin.com/in/setuchokshi/
github.com/setuc/iotAsia2017