82
Forecasting technologies Johan Hartnack
Forecasting technologiesJohan Hartnack
Forecasting
© DHI #2
Roles
• Decision makers
• Operator/system maintenance
• Public
© DHI #3
© DHI #4
Needs
• Overview with quick assessment of
operational choices and impacts
• Configuration and maintenance.
Advanced operations of system
• Clear messages to be understood
without specialist knowledge
Decision maker
• Evaluate choices quickly
• Impact assessment
• Issue warnings
• Initiate mitigation measures
© DHI #5
Operator/system maintenance
• Model set-up
• Uncertainty assessment
• Scenario evaluations
• Configuration
• Data hook up
© DHI #6
Dissemination
• GIS
• Time series
• Temporal development
• Levels of data access
© DHI #7
Configuration and operating system
© DHI #8
• Model adaptors configuration through XML files
• Workflow - Jobs manager
• Create automatic reports
Openness as you need it – empowering you
© DHI #9
Spreadsheets
• Well known spread
sheet functionality
• Access live data
• Build models
• Create reports
© DHI
Time
seriesGIS Spreadsheet Scenario Jobs Indicator
Analysis
MCA/CBADocument Metadata
Scripting
Scripting
• An Iron Python
environment
• API to all modules
• Functionality to store
and manage scripts
Code your own Tools
• Develop your own
• API Access to data
• Customization
Open Software Architecture
© DHI
MIKE Operations
MIKE Workbench
Desktop WEB
Data base
GIS Time-series Scripting Spreadsheets Jobs
Workbench
MIKE
MODELS
MIKE
INFOMIKE
PLANNING
MIKE
OPERATIONS
MIKE Technologies for all Water Environments
• Packaged as standard products
• Configurable
• Open (e.g. to data)
• Extendable (scripting, API)
© DHI
Aarhus – Danish for Progress
… Architecture
and art
… Business and
industries…Aarhus
University
…Recreational
and active use of
the environment
#16
© DHI
Expected project outcome
…Improved water
quality/partly bathing water
in River Aarhus
… Bathing water in the
Harbour (hygienic)
…Bathing water in Lake
Brabrand (hygienic)
#17
© Aarhus Water © Aarhus Water © Aarhus Water
© DHI #18
© DHI
MIKE 11MIKE URBAN
MIKE 3MIKE SHE
Data flow
Model preparation
Model execution
Real time control
Issues of warning
Distributed
rainfall
from Radar
Automated Integrated Modelling
© DHI
Layer 3
PLC/SCADA
Sensors/Actuators
WISYS
Short term rainfall
forecast (MAR) Dynamic Overflow
Risk Analysis
(DORA)
WWWTP max.
hydraulic load
Predicted run-off
(MIKE URBAN)
Data validation and
filtration
Software sensors:
Flow, Elevations,
tank filling, etc.
PID (flows) at each
storage tank
PID (elevations) at
each storage tank
PID output: 0-100%
distributed to
setpoints for pumps,
weirs/gates at each
storage tank
Layer 2
Layer 1
Levels, flows and weir/gate positions Set-points
DIMS.CORE
Real-Time Integrated Control
DHI bathing water forecast service
© DHI #21
• App and web based public warning system
Public information on bathing water quality
© DHI #22
Internet
23
Environmental Section
Aarhus Municipality
Aarhus Water
Utility company
Waterforecast
Operated by DHI
One warning system -Integrating data from multiple organizations and authorities
© DHI
© DHI
Project Status
…Improved water quality in
River Aarhus
… Bathing water in the
Harbour (hygienic)
…Bathing water in Lake
Brabrand (hygienic)
#24
© Stiften ©Politiken © Stiften
Saving in investment
© DHI
• Ordinary and larger retention basins 79 million EUR
• Controllable and smaller retention basins 45,6 million EUR
• Automation and control system 1,7 million EUR
• Total 47,3 million EUR
• Saving 32 million EUR
40 %
#25
Simple and effective web applications for
real time monitoring and early warning
systems
© DHI #26
Greve – Flood Warning System
Web solution
MIKE Models
Rainfall forecast
Greve – Flood Warning System
MIKE21 FM
MIKE URBAN CS
MIKE OPERATIONS
Automatic Operation
Results every 4 hours
Forecast 24 hours
Web solution based
on MIKE products
Greve – Flood Warning System
MIKE Web technology
Time-series
GIS
Spreadsheets
…
MIKE Web API
Website
Polymer/Web
Components
Configurable MIKE product Project specific implementation
MINERWA- Minimising Non-Revenue Water in distribution networks
© DHI
“NRW is produced water, which
is not paid for. It can be due to
real losses (like leakages) or
apparent losses (such as theft or
metering inaccuracies). In many
parts of the world, water
resources are limited and thus
NRW can have significant
consequences on the level of
service as well as on revenue
loss
LONG-TERM SOLUTION TO SUSTAINABLE
WATER NETWORK MANAGEMENT Our solution to Minimise Non-Revenue Water (MINERWA) works with the data you
have, follows international recommendations and uses a well-proven methodology
– and it pays off from the start. Only a minimum of input data is required to
establish an overview and understanding. MINERWA offers:
• a well-structured data repository
• analytical engines
• an efficient, yet customisable user-interface with key performance indicators as
well as in-depth reporting
© DHI
“MINERWA is a solution
delivered jointly by DHI and
EnviDan International. It is
offered as a hosted solution
where we take full responsibility
of running and maintaining the
MINERWA and making it
available to you
MINERWA benefits • reduced water losses
• reduced pipe burst risks
• reduced energy consumption
• Documentation of effects
• Facilities for for rehabilitation and emergency planning, water quality risk
analysis and much more.
Minerwa web solution
© DHI
How to build your own integrated water
management system
© DHI #34
Building blocks – an example
Data management system
•Data integration/ validation
•Processing
•Reporting
•SCADA
Model analysis
•Numerical / empirical models of the system
Operationalzation
•Scheduling and coupling of data management system and models
Control / optimization
•Control
•Optimization
•Feedback to SCADA
•Manually
•Automatic
Warning
•Operational people
•Public
Our solution builds on generalised
software components to provide
standard products as well as
custom solutions
DHI Software frame work for integrated solutions
• Integration of data and models
− MIKE OPERATIONS
• Data management and integration to SCADA
− DIMS.CORE
• Numerical models
− MIKE URBAN (collection system)
− MIKE 11 (River)
− MIKE SHE (Catchment + river)
− MIKE 3 (Harbour and ocean)
Looking ahead
• Data
− Assimilation techniques
• Faster model execution
− Hardware
− Smart systems
− Surrogate models
© DHI 38
Data
Assimilation to internal measurements
© DHI #39
Data Assimilation Framework
• Kalman filter state updating procedure
• Introduction of boundary errors in each simulation
of the ensemble
• Ensemble statistics
• Facilitates state updating on a wide range of
dependent variables
• Uncertainty analyses
Traditional MIKE model
MIKE model
Time series Topography Model parameters
Water level,
Discharge,
Velocities,
etc.
Future
Statistics:
Mean values,
Covariance,
Confidence intervals
etc.
Main:
Input
Results
Simulation
Uncertainty Uncertainty Uncertainty
Data assimilation
Measurements
Data assimilation
Mathematical setting
Model description in the form of a model operator
Where
xk The state variables of the system (H- ,Q- , depth integrated velocities)
F The model operator (a time step in MIKE HYDRO River)
k the time step
uk The forcing terms of the system (boundary conditions)
),(1 kkk uxx F
Model errors
− Governing equations
− Discretization
− Conceptualization
− Sub-optimal model parameters
− Initial conditions
− Temporal forcing terms
Difficult to quantify
• River model as stochastic process
Stochastic setting
ek The model errors of the system
Updating scheme Measurements included
K Kalman gain matrix Depends on the covariance matrix of state variables
Weighting matrix for the system
The evaluation of the covariance matrix is the main bottleneck
),(1 kkkk εuxx F
KΔxx 11 k
updated
k
Ensemble Kalman filter
x1k x2
k xMk
x2k+1 xMk+1x1
k+1
Construct
Filter
x1k+1 x2
k+1 xMk+1
Next time step
Simulation
Data
assimilation
Ensemble size M
Output
Measurements
Model errors
Model refinement using an auto regressive process of
first order for model errors
I.e. Updating can be carried out on the model
error
Test example
L = 23000 mI = 0.025 %
L = 11000 mI = 0.025 %
Q/H relation
Q
t
Effectiveness of updating algorithm
Internal point
+ reference runooerroneous runD updated
Effectiveness of updating algorithm
Downstream boundary internal point
+ reference runerroneous run
D updated
Effectiveness of updating algorithm
Upstream boundary condition updating
+ reference runerroneous run
D updated
Uncertainty assessment
Method uses an ensemble of simulation thus as an added bonus this ensemble may be used to estimate
• Confidence intervals (50% , 80 % etc.)• Standard deviations (in all points)
Valuable tool for sensitivity analysis of boundary conditions
Hardware
Brute force for speed
© DHI #52
© DHI
Parallelization – A case study
• Christchurch, New Zealand Catchment area approx. 420 km2
including three river systems in
the model domain:
Avon River
Styx River
Heathcote River
2D model domain:
4.2 million elements
10 m x 10 m resolution flexible
mesh (rectangular elements)
Distributed rainfall-runoff with
no losses (rain-on-grid)
- 1% AEP event
- 21 hour storm
© DHI#54
Hybrid Parallelization – A case study
• Christchurch, New Zealand Run time on desktop PC (MPI)
is 8.9 hours:
16 core Dell Workstation
2 x Intel® Xeon® CPU ES-
2687W v2 (8 core, 3.40 GHZ)
32 GB of RAM
Windows 7 operating system
Run time with 1 x GeForce GTX
TITAN GPU card is 3.1 hours
Run time with 2 x GeForce GTX
TITAN GPU card is 1.7 hours
© DHI#55
Hybrid Parallelization• Christchurch, New Zealand
1024; 0.17768; 0.23
512; 0.26
256; 0.45
128; 0.81
64; 1.31
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
0 200 400 600 800 1000 1200
Sim
ula
tio
n t
ime [
ho
urs
]
Number of HPC Cores
Christchurch MIKE 21 FM modelHPC Cluster simulation perfomance
(Simulations executed by HPC Wales)
Below 10 minutes
Klimaspring – next generation smartsolution
© DHI
“Smart real-time control
of water systems ”
(Smart realtidsstyring af
vandsystemer)
© DHI
Klimaspring Scope
© DHI
• Scope:
− Developing an scalable IT-supported system for the real-time monitoring, modelling, warning and management of rainwater in both drainage systems and on the ground.
• Aim:
− Reduce the need to invest in enlarging and upgrading the existing drainage system
− Make managing rainfall less expensive.
− Open up new possibilities for the use of water on the ground
Danish R&D project financed by RealDania
• Collaboration project with university, utility and DHI
© DHI
Improved rainfall forcasting and
handling of forcast uncertainties
▼
Knowledge of the amount of
rainfall provides the basis of
improving the system
beforehand
Development of computer
models for calculation of where
and how much rainfall you will
have in the system
▼
Possibility of calculating the best
optimization of the system
Real time control of controllable
elements in the sewer system
▼
When rainfall starts the system
can automatically lead the water
to places where it will cause
least damage
© DHI
User surfaces for display of
operation and system status
▼
Management, planning and
operational personnel can see
how the system is running or
look into control of previous
rainfall events
© DHI
The system will be able to send
information to e.g. web/mobile
(awaiting final clarification)
▼
Information to citizens
Klimaspring system architecture
• MIKE Powered by DHI software
Key DHI tasks in Klimaspring
© DHI
• Model engine optimization
− Deterministic models Surrogate models
• Radar data processing
− Improve data image correction and forecasting
• Control algoritm optimization
− Standardaizing and modulazation
• Integration of data and models
• Visualisation
Web front-end
© DHI
Surrogate models
Model Predictive Control
© DHI #68
Why think of a new optimisation approach?
© DHI
Current optimisation system
• Uses simplified optimisation model (few decision variables)
• Includes execution of many hundreds of simulations
• Takes time
New optimisation system
• Uses detailed optimisation model (thousands of decision variables)
• Model dynamics described by a simplified (surrogate) model
• Takes few minutes on today’s laptop
Local control
System-wide Model Predictive Control (MPC)
TOF
Optimisation
Model prediction
Forecast data
Real-time
data
Implement optimal control
until next optimisation
TimeNext TOF
Optimisation
Model prediction
Updated real-time
and forecast data
Data
assimilation
Operational
Workflow
Optimisation/control problem
Physical system
HiFi model (MIKE model)
Optimiser
Mathematical formulation of (simplified)
optimisation model
Optimisation/control problem
Physical system
HiFi model (MIKE model)
Optimiser
Mathematical formulation of
optimisation model
Surrogate model (Linear)
MPC optimisation framework
What is a surrogate model?
• Derived from the HiFi model (MIKE model)
• Sufficiently accurate for modelling the most important characteristics relevant for the
problem at hand
• Computationally fast
© DHI
Real-time Control Framework
Optimal combination of HiFi and surrogate models
MPC optimisation with surrogate model
Implementation of
optimal control
Update surrogate model
from HiFi model
MPC optimisation with surrogate model
HiFi model simulation
with implemented
control and assimilation
of system observations
Handling uncertainties
• Uncertainty in model forcing is
described by an ensemble
forecast
• MPC model extended to use
ensemble model forcing (Multiple
MPC)
• Probability assigned to each
ensemble member
• Will provide optimal control that is
robust to forecast uncertainty
The steps
MIKE
model
Surrogate
model1
2
Formulation of optimisation model• Constraints
• Operation targets
• Objective function
3Behind the scenes• Automatic setup of MPC optimisation model
• Efficient optimisation solver
CARM, Australia
Optimisation of irrigation system
© DHI #77
Irrigation Water Delivery Infrastructure
22 June, 2016© DHI #78
CARM operational goals
• Supply ordered water to the users
The right amount at the right time
• Keep the river in a lean state
Minimize losses due to evapotranspiration
Leave room for accommodating natural inflow
• Keep environmental flow requirements at end-of-system
• Minimise surplus flows at end-of-system
© DHI
Benefits
• New technology enables solving large system-wide optimisation and control problems
in real-time
− Problems that we cannot solve today without brute force
• May be applied within several business areas
© DHI
Reduced flooding Environmental
protectionOptimized hydropower
Conclusions
• Forecast system must adapt to roles
• Openness – that suits you
• Smart water solutions
© DHI 81
