PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping...

transcript

LEARNING AND LEARNING TO SOLVE PDES Bin Dong (董彬)• Beijing International Center for Mathematical Research, Peking University

• Laboratory for Biomedical Image Analysis & Laboratory of Deep Learning Research, Beijing Institute of Big Data Research

OUTLINE

Overview and Motivations

Deep Neural Networks (DNNs) and PDEs

Learning PDEs: PDE-Net for Inverse Problems

Learning to solve PDEs: A Reinforcement Learning Framework

Learning Learning to solve

OVERVIEW

Motivations and Intuitions

DEEP LEARNING REVOLUTION

Biomedicine

Computer Vision

DEEP LEARNING FROM MATHEMATICS PERSPECTIVE

What are still challenging

Theoretical guidance

Transparency, interpretability, robustness

Idea: find links of DL with mathematics

Our perspective: control

Control Perspective:

• E, CMS, 5(1):1–11, 2017.• Li, Chen, Tai, E, JMLR, 18(1), 2017.• Haber, Ruthotto, IP, 34(1), 2017.• Lu, Zhong, Li, Dong, ICML 2018. • Gregor, LeCun, ICML 2010. • Yang, Sun, Li, Xu, NIPS 2016. • Li, Tai, E, ICML 2017.• Liu, Theodorou, arXiv:1908.10920.

Fokker-Planck equation:

Dynamics:

Training loss:

• Chen, Wei, Pock, CVPR 2015.• Long, Lu, Ma, Dong, ICML 2018. • Long, Lu, Dong, JCP, 2019.• Ray, Hesthaven, JCP, 367, 2018• Han, Jentzen, E, PNAS, 115(34), 2018.• Wang, Shen, Long, Dong, arXiv: 1905.11079.• Li, Shi, arXiv:1708.05115.• Sun, Tao, Du, arXiv:1812.00174.

Credit: Pengfei Jin @ PKUSee also: Haber & Ruthotto, arXiv:1705.03341

Control Perspective: Supervised Learning (SL)、Reinforcement Learning (RL)、Meta Learning (Meta)

Dynamics:

Training loss:

Typical SL-control

Typical RL-control

Typical Meta-control

Solver:

• BP/PMP

• DP

Common:Uncommon:

OTHER RELATED WORKS

Architecture design Wang B, Yuan B, Shi Z, et al. EnResNet: ResNet Ensemble via the

Feynman-Kac Formalism. arXiv:1811.10745, 2018. Tao Y, Sun Q, Du Q, et al. Nonlocal Neural Networks, Nonlocal

Diffusion and Nonlocal Modeling. NeurIPS 2018. Zhu M, Chang B, Fu C. Convolutional Neural Networks combined

with Runge-Kutta Methods. arXiv:1802.08831, 2018. Zhang L, Schaeffer H. Forward Stability of ResNet and Its Variants.

arXiv:1811.09885, 2018. Sun Q, Tao Y, Du Q. Stochastic Training of Residual Networks: a

Differential Equation Viewpoint. arXiv:1812.00174, 2018. Li H, Yang Y, Chen D and Lin Z. Optimization Algorithm Inspired

Deep Neural Network Structure Design. ACML 2018. He J, Xu J. MgNet: A Unified Framework of Multigrid and

Convolutional Neural Network. arXiv:1901.10415, 2019. Theory

E W., Han J, Li Q. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, vol. 6, no. 10, pp. 1–41, 2019.

Thorpe M, van Gennip Y. Deep Limits of Residual Neural Networks. arXiv:1810.11741, 2018.

OTHER RELATED WORKS

Optimization Li Q, Hao S. An optimal control approach to deep learning and applications to

discrete-weight neural networks. ICML 2018. Chen T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations.

NeurIPS 2018. (Best paper) Parpas P, Muir C. Predict Globally, Correct Locally: Parallel-in-Time Optimal

Control of Neural Networks. arXiv:1902.02542. Zhang D, Zhang T, Lu Y, Zhu Z, Dong B, You Only Propagate Once: Accelerating

Adversarial Training via Maximal Principle, NeurIPS 2019.

Applications Chen Y, Yu W, and Pock T. On learning optimized reaction diffusion processes

for effective image restoration. CVPR 2015. Sun, Li, and Xu. Deep ADMM-net for compressive sensing MRI. NIPS 2016. Zhang S, Lu Y, Liu J, Dong B. Dynamically Unfolding Recurrent Restorer: A

Moving Endpoint Control Method for Image Restoration. ICLR 2019 (arXiv:1805.07709).

Zhang H, Dong B and Liu B, JSR-Net: A Deep Network for Joint Spatial-Radon Domain CT Reconstruction from incomplete data, IEEE-ICASSP, 2019

Lu Y, He D, Li Z, Sun Z, Dong B, Qin T, Wang L, Liu T-Y. Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View. arXiv: 1906.02762, 2019.

BRIDGING DIFFERENTIAL EQUATIONS WITH DEEP NETWORKS

DNNs and numerical PDEs: Learning PDEs• Zichao Long, Yiping Lu, Xianzhong Ma and Bin Dong, PDE-Net:

Learning PDEs from Data, ICML 2018. (arXiv:1710.09668)• Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs

from Data with A Numeric-Symbolic Hybrid Deep Network, accepted by Journal of Computational Physics, 2019 (arXiv:1812.04426).

PDE-NET: LEARNING PDES FROM DATA

Can we learn principles (e.g. PDEs) from data?

11Biology

Meteorology

Computer Graphics

Earlier work

Other earlier attempts:• Bongard & Lipson,

PNAS, 2007• Liu, Lin, Zhang & Su.

ECCV 2010.

• S. Brunton, J. L. Proctor and J. N. Kutz Proceedings of the National Academy of Sciences, 2016• Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Science Advances, 3(4), 2017.• Hayden Schaeffer. Proc. R. Soc. A, volume 473, The Royal Society, 2017.

Room for improvements

Can we go beyond sparse coding framework

(linear dictionary)?

—— Bigger model class with less prior knowledge

Can we learn discrete forms of differential

operators and does it help?

——More accurate estimation of the PDE and

prediction 14

Assuming：

PDE-Net 2.0

Assuming：

PDE-Net 2.0

identity identity

More Constraints:• Pseudo-upwind• Sparsity on moment matrices• Sparsity on the symbolic network

1st orderlearnable

2st orderlearnable

1st orderfrozen

• J.F. Cai, B. Dong, S. Osher and Z. Shen, Journal of the American Mathematical Society, 2012.• B. Dong, Q. Jiang and Z. Shen, Multiscale Modeling & Simulation, 2017

19Prediction

Model recovery

Model Recovery

BRIDGING DIFFERENTIAL EQUATIONS WITH DEEP NETWORKS

DNNs and numerical PDEs: Learning to solve PDEs• Yufei Wang, Ziju Shen, Zichao Long and Bin Dong, Learning to

Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning, arXiv: 1905.11079, 2019.

NEURAL NETWORKS (NNS) AND NUMERICAL PDES – SOME RECENT DEVELOPMENTS

NNs as a new ansatz: M. Raissi, P. Perdikaris, G. E. Karniadakis. arXiv:1711.10561 & arXiv:1711.10566,

2017. J. Magiera, D. Ray, J. S Hesthaven, Rohde. arXiv:1904.12794, 2019. C. Michoski, M. Milosavljevic, T. Oliver, D. Hatch. arXiv:1905.04351, 2019. G.-J. Both, S. Choudhury, P. Sens, R. Kusters. arXiv:1904.09406, 2019.

NNs to approximate complex solution mappings Y. Khoo, J. Lu, L. Ying. arXiv:1707.03351, 2017. Y. Khoo, L. Ying. arXiv:1810.09675, 2018. Y. Li, J. Lu, A. Mao. arXiv:1905.02789, 2019. Y. Fan, L. Lin, L. Ying, L. Zepeda-Núnez. arXiv:1807.01883, 2018.

NNs to solve very high dimensional PDEs C. Beck, W. E, A. Jentzen. JNS, 1–57, 2017. W. E and B. Yu, CMS, 6(1), 1-12, 2018. J. Han, A. Jentzen, W. E. PNAS, 115(34):8505–8510, 2018. D. Pfau, J.S. Spencer, A.G. Matthews, W. M. Foulkes, arXiv:1909.02487. W. Cai and Z. Xu, arXiv:1910.11710

NNs integrated with classical solvers N. Discacciati, J. S Hesthaven, D. Ray. Technical report, 2019. D. Ray, J. S Hesthaven. JCP, 367:166–191, 2018.

LEARNING TO DISCRETIZE

Roe speed:

Reinforcement learning (RL)

• RL is to learn to make sequential decisions by interacting with the environments (learn from rewards).

• Can be modeled as a finite Markov Decision Process (MDP):

CL solver revisited

Action:

State transition

RL-WENO

RL-WENO v.s. WENO-5: Overall accuracy:

Comparable with WENO-5; Generalize well to other mesh sizes, flux, temporal

discretization, and terminal time.

Near singularities

Action

CONCLUSIONS

THANKS FOR YOUR ATTENTION!

MY WEBPAGE:HTTP://BICMR.PKU.EDU.CN/~DONGBIN

Jointly with

Quanzheng Li@ Harvard MGH

Jiaying Liu@ PKU ICST

Xiaoshuai Zhang@ PKU ICST

Zichao Long@ PKU SMS

Xianzhong Ma@ PKU SMS

Aoxiao Zhong@ Harvard SEAS

Zhanxing Zhu@ PKU SMS

Jikai Hou (SMS)Tianyuan Zhang (ICST)Dinghuai Zhang (SMS)

Yufei Wang@ PKU Yuanpei College

Ziju Shen@ PKU AAIS

Liwei Wang@ PKU CS

Tie-Yan Liu@ Microsoft

PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping...

Documents