LEARNING AND LEARNING TO SOLVE PDES Bin Dong (董彬)• Beijing International Center for Mathematical Research, Peking University
• Laboratory for Biomedical Image Analysis & Laboratory of Deep Learning Research, Beijing Institute of Big Data Research
OUTLINE
Overview and Motivations
Deep Neural Networks (DNNs) and PDEs
Learning PDEs: PDE-Net for Inverse Problems
Learning to solve PDEs: A Reinforcement Learning Framework
2
Learning Learning to solve
OVERVIEW
Motivations and Intuitions
DEEP LEARNING REVOLUTION
4
Biomedicine
Computer Vision
DEEP LEARNING FROM MATHEMATICS PERSPECTIVE
What are still challenging
Theoretical guidance
Transparency, interpretability, robustness
Idea: find links of DL with mathematics
Our perspective: control
5
DEEP LEARNING FROM MATHEMATICS PERSPECTIVE
Control Perspective:
6
• E, CMS, 5(1):1–11, 2017.• Li, Chen, Tai, E, JMLR, 18(1), 2017.• Haber, Ruthotto, IP, 34(1), 2017.• Lu, Zhong, Li, Dong, ICML 2018. • Gregor, LeCun, ICML 2010. • Yang, Sun, Li, Xu, NIPS 2016. • Li, Tai, E, ICML 2017.• Liu, Theodorou, arXiv:1908.10920.
Fokker-Planck equation:
Dynamics:
Training loss:
Training loss:
• Chen, Wei, Pock, CVPR 2015.• Long, Lu, Ma, Dong, ICML 2018. • Long, Lu, Dong, JCP, 2019.• Ray, Hesthaven, JCP, 367, 2018• Han, Jentzen, E, PNAS, 115(34), 2018.• Wang, Shen, Long, Dong, arXiv: 1905.11079.• Li, Shi, arXiv:1708.05115.• Sun, Tao, Du, arXiv:1812.00174.
Credit: Pengfei Jin @ PKUSee also: Haber & Ruthotto, arXiv:1705.03341
DEEP LEARNING FROM MATHEMATICS PERSPECTIVE
Control Perspective: Supervised Learning (SL)、Reinforcement Learning (RL)、Meta Learning (Meta)
7
Dynamics:
Training loss:
Typical SL-control
Typical RL-control
Typical Meta-control
Solver:
• BP/PMP
• DP
Common:Uncommon:
OTHER RELATED WORKS
Architecture design Wang B, Yuan B, Shi Z, et al. EnResNet: ResNet Ensemble via the
Feynman-Kac Formalism. arXiv:1811.10745, 2018. Tao Y, Sun Q, Du Q, et al. Nonlocal Neural Networks, Nonlocal
Diffusion and Nonlocal Modeling. NeurIPS 2018. Zhu M, Chang B, Fu C. Convolutional Neural Networks combined
with Runge-Kutta Methods. arXiv:1802.08831, 2018. Zhang L, Schaeffer H. Forward Stability of ResNet and Its Variants.
arXiv:1811.09885, 2018. Sun Q, Tao Y, Du Q. Stochastic Training of Residual Networks: a
Differential Equation Viewpoint. arXiv:1812.00174, 2018. Li H, Yang Y, Chen D and Lin Z. Optimization Algorithm Inspired
Deep Neural Network Structure Design. ACML 2018. He J, Xu J. MgNet: A Unified Framework of Multigrid and
Convolutional Neural Network. arXiv:1901.10415, 2019. Theory
E W., Han J, Li Q. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, vol. 6, no. 10, pp. 1–41, 2019.
Thorpe M, van Gennip Y. Deep Limits of Residual Neural Networks. arXiv:1810.11741, 2018.
8
OTHER RELATED WORKS
Optimization Li Q, Hao S. An optimal control approach to deep learning and applications to
discrete-weight neural networks. ICML 2018. Chen T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations.
NeurIPS 2018. (Best paper) Parpas P, Muir C. Predict Globally, Correct Locally: Parallel-in-Time Optimal
Control of Neural Networks. arXiv:1902.02542. Zhang D, Zhang T, Lu Y, Zhu Z, Dong B, You Only Propagate Once: Accelerating
Adversarial Training via Maximal Principle, NeurIPS 2019.
Applications Chen Y, Yu W, and Pock T. On learning optimized reaction diffusion processes
for effective image restoration. CVPR 2015. Sun, Li, and Xu. Deep ADMM-net for compressive sensing MRI. NIPS 2016. Zhang S, Lu Y, Liu J, Dong B. Dynamically Unfolding Recurrent Restorer: A
Moving Endpoint Control Method for Image Restoration. ICLR 2019 (arXiv:1805.07709).
Zhang H, Dong B and Liu B, JSR-Net: A Deep Network for Joint Spatial-Radon Domain CT Reconstruction from incomplete data, IEEE-ICASSP, 2019
Lu Y, He D, Li Z, Sun Z, Dong B, Qin T, Wang L, Liu T-Y. Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View. arXiv: 1906.02762, 2019.
9
BRIDGING DIFFERENTIAL EQUATIONS WITH DEEP NETWORKS
DNNs and numerical PDEs: Learning PDEs• Zichao Long, Yiping Lu, Xianzhong Ma and Bin Dong, PDE-Net:
Learning PDEs from Data, ICML 2018. (arXiv:1710.09668)• Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs
from Data with A Numeric-Symbolic Hybrid Deep Network, accepted by Journal of Computational Physics, 2019 (arXiv:1812.04426).
PDE-NET: LEARNING PDES FROM DATA
Can we learn principles (e.g. PDEs) from data?
11Biology
Meteorology
Computer Graphics
PDE-NET: LEARNING PDES FROM DATA
12
Earlier work
Other earlier attempts:• Bongard & Lipson,
PNAS, 2007• Liu, Lin, Zhang & Su.
ECCV 2010.
PDE-NET: LEARNING PDES FROM DATA
• S. Brunton, J. L. Proctor and J. N. Kutz Proceedings of the National Academy of Sciences, 2016• Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Science Advances, 3(4), 2017.• Hayden Schaeffer. Proc. R. Soc. A, volume 473, The Royal Society, 2017.
13
PDE-NET: LEARNING PDES FROM DATA
Room for improvements
Can we go beyond sparse coding framework
(linear dictionary)?
—— Bigger model class with less prior knowledge
Can we learn discrete forms of differential
operators and does it help?
——More accurate estimation of the PDE and
prediction 14
PDE-NET: LEARNING PDES FROM DATA
Assuming：
15
PDE-NET: LEARNING PDES FROM DATA
PDE-Net 2.0
16
Assuming：
PDE-NET: LEARNING PDES FROM DATA
PDE-Net 2.0
17
identity identity
More Constraints:• Pseudo-upwind• Sparsity on moment matrices• Sparsity on the symbolic network
PDE-NET: LEARNING PDES FROM DATA
1st orderlearnable
2st orderlearnable
1st orderfrozen
18
• J.F. Cai, B. Dong, S. Osher and Z. Shen, Journal of the American Mathematical Society, 2012.• B. Dong, Q. Jiang and Z. Shen, Multiscale Modeling & Simulation, 2017
PDE-NET: LEARNING PDES FROM DATA
19Prediction
Model recovery
PDE-NET: LEARNING PDES FROM DATA
20
Model Recovery
BRIDGING DIFFERENTIAL EQUATIONS WITH DEEP NETWORKS
DNNs and numerical PDEs: Learning to solve PDEs• Yufei Wang, Ziju Shen, Zichao Long and Bin Dong, Learning to
Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning, arXiv: 1905.11079, 2019.
NEURAL NETWORKS (NNS) AND NUMERICAL PDES – SOME RECENT DEVELOPMENTS
NNs as a new ansatz: M. Raissi, P. Perdikaris, G. E. Karniadakis. arXiv:1711.10561 & arXiv:1711.10566,
2017. J. Magiera, D. Ray, J. S Hesthaven, Rohde. arXiv:1904.12794, 2019. C. Michoski, M. Milosavljevic, T. Oliver, D. Hatch. arXiv:1905.04351, 2019. G.-J. Both, S. Choudhury, P. Sens, R. Kusters. arXiv:1904.09406, 2019.
NNs to approximate complex solution mappings Y. Khoo, J. Lu, L. Ying. arXiv:1707.03351, 2017. Y. Khoo, L. Ying. arXiv:1810.09675, 2018. Y. Li, J. Lu, A. Mao. arXiv:1905.02789, 2019. Y. Fan, L. Lin, L. Ying, L. Zepeda-Núnez. arXiv:1807.01883, 2018.
NNs to solve very high dimensional PDEs C. Beck, W. E, A. Jentzen. JNS, 1–57, 2017. W. E and B. Yu, CMS, 6(1), 1-12, 2018. J. Han, A. Jentzen, W. E. PNAS, 115(34):8505–8510, 2018. D. Pfau, J.S. Spencer, A.G. Matthews, W. M. Foulkes, arXiv:1909.02487. W. Cai and Z. Xu, arXiv:1910.11710
NNs integrated with classical solvers N. Discacciati, J. S Hesthaven, D. Ray. Technical report, 2019. D. Ray, J. S Hesthaven. JCP, 367:166–191, 2018.
LEARNING TO DISCRETIZE
Roe speed:
23
LEARNING TO DISCRETIZE
Reinforcement learning (RL)
• RL is to learn to make sequential decisions by interacting with the environments (learn from rewards).
• Can be modeled as a finite Markov Decision Process (MDP):
24
LEARNING TO DISCRETIZE
CL solver revisited
25
Action:
State transition
LEARNING TO DISCRETIZE
RL-WENO
26
LEARNING TO DISCRETIZE
RL-WENO v.s. WENO-5: Overall accuracy:
Comparable with WENO-5; Generalize well to other mesh sizes, flux, temporal
discretization, and terminal time.
Near singularities
27
LEARNING TO DISCRETIZE
28
Action
LEARNING TO DISCRETIZE
CONCLUSIONS
30
THANKS FOR YOUR ATTENTION!
MY WEBPAGE:HTTP://BICMR.PKU.EDU.CN/~DONGBIN
Jointly with
Quanzheng Li@ Harvard MGH
Jiaying Liu@ PKU ICST
Xiaoshuai Zhang@ PKU ICST
Zichao Long@ PKU SMS
Xianzhong Ma@ PKU SMS
Aoxiao Zhong@ Harvard SEAS
Zhanxing Zhu@ PKU SMS
Jikai Hou (SMS)Tianyuan Zhang (ICST)Dinghuai Zhang (SMS)
@ PKU
Yufei Wang@ PKU Yuanpei College
Ziju Shen@ PKU AAIS
Liwei Wang@ PKU CS
Tie-Yan Liu@ Microsoft
