Extending FEWS with External functionalities Zhengtao Cui NOAA National Weather Services Office of...

1

Extending FEWS with External functionalities

Zhengtao CuiNOAA National Weather ServicesOffice of Hydrologic Development

2

Outline

• Day 11. FEWS Workflow (Page 3-7)2. The FEWS General Adapter Concept(Pages 8 –

19)3. OHD FEWS adapter framework (Pages 20 – 60)4. Examples (Pages 61 - 90)

• Day 2– Hands on Examples

3

FEWS workflow, module and model

• A workflow specifies a sequence of other workflows or modules to execute– Usually executed to generate a forecast

• A module is a unit of execution– Can be a FEWS provided internal module, for example, the import module

• Flow of data/files is hidden from the user• http://publicwiki.deltares.nl/display/FEWSDOC/05+Configuring+the+available+DELFT-FEWS+modules

– A General Adapter is a reference to an external model• A FEWS General Adapter connects workflow with external models • Flow of data/files is visible to the user• Can have multiple execution units

• Model/External model– Numerical code (source, binary, scripts, etc.) that does some kinds of calculations based a particular

physical process,– Rainfall-runoff, routing, snow melt, etc.– Java model

• NWSRFS models written in Java

– Legacy model• NWSRFS models written in C++/C/Fortran

– http://publicwiki.deltares.nl/display/FEWSDOC/Models+linked+to+Delft-Fews

http://publicwiki.deltares.nl/display/FEWSDOC/05+Configuring+the+available+DELFT-FEWS+modules

http://publicwiki.deltares.nl/display/FEWSDOC/05+Configuring+the+available+DELFT-FEWS+modules

http://publicwiki.deltares.nl/display/FEWSDOC/Models+linked+to+Delft-Fews

http://publicwiki.deltares.nl/display/FEWSDOC/Models+linked+to+Delft-Fews

4

Workflow example

5

Module example

6

Model example

• SAC-SMA• SNOW-17• ……

7

Quiz

1. Describe the relationship of FEWS workflow and module.

Answer: A module is a part of a workflow.2. Which of the following could be listed inside a

workflow?1. A FEWS internal module2. Another workflow3. A general adapter module4. A binary executable of Snow-17

Answer: 1,2, and 3.

8

The FEWS General Adapter Concept• The general adapter invokes an

external model allowing for FEWS workflow to execute that model• preAdapter, postAdapter,

Model• The user must provide a

model adapter to run an external model

• The general adapter communicates with external model using the Published Interface (PI)• PI is an XML based data

exchange schema• Pre/Post adapter converts PI

to/from native format

FEWS

Model Adapter/External Model

9

The FEWS Published Interface (PI)• FEWS has no knowledge of any external models– Initiate by the FEWS General Adapter

• An external model has no knowledge about FEWS or any other external models– Read/Write PI XML data

• Including diagnostic information and model states

– physical model calculations• Communication between FEWS and external

models is done through PI– XML data files on disk

10

FEWS Configuration Files1. Module Configuration

– Provides unique information about the module (name, input/output data, properties, parameter filename, external model, etc.)

2. Module Instance Descriptors– Defines each module ( e.g. module name, type). Type can be

FEWSTransformation or GeneralAdapter• FEWSTransformation are modules that come with FEWS• GeneralAdapter are modules developed by FEWS users

3. Workflows– Controls and defines the order in which to execute a series of workflows or

modules

4. Workflow Descriptors– Defines each workflow’s characteristics– Such as a descriptive name for a workflow

11

…<workflowDescriptor … id=“workflow_file”</workflowDescriptor> …

WorkflowDescriptor.xml

…<activity> <moduleInstanceId>moduleConfigFile</moduleInstanceId></activity> …

…<moduleInstanceDescriptor id = “moduleConfigFile”> <moduleId>GeneralAdapter</moduleId></ModuleInstanceDescriptor> …

…< executeActivity> <command>cmd</command></executeActivity> …

Workflow_file.xml

ModuleInstanceDescriptors.xml

moduleConfigFile.xml

FEWSAdapter

12

‘general’ section defines several some commontags that will be used by other sections.

‘startUpActivity’ defines activities that are runprior to a module run and export any data.

‘exportActivities’ defines all items to be exportedthrough the PI XML to the external module.

‘executeActivities’ defines external executablesor Java classes to be run.

‘importActivities’ defines all items to be importedFollowing successful completion of the module run.

Structure of General Adapter Module Configuration File

13

Example: Basic Dummy AdapterDummyAdapter

• Has only one “main” method.• Prints “Hello World!” to the screen.• No input/output, parameters, properties, etc.• No external model is executed.

package dummy;

public class DummyAdapter{ public static void main(final String[] args) { System.out.println("Hello World"); }}

14

General Adapter Module Configuration for the DummyAdapter

15

Example: Basic Dummy AdapterDummyAdapter2

• Has only one “main” method.• Copy the output file in a property file to

another file named “rootDir/output/outputs.xml”.

• No external model is executed.

rootDirectory=Modules/dummy_module/dummy_basin2outputFile=output_benchmark.xml

args.file

16

try { prop.load(bis); } catch(final IOException e) { e.printStackTrace(); }//get property identifying root directory final String rootDir = prop.getProperty("rootDirectory");

//get property identifying output file final String outputFile = prop.getProperty("outputFile");

//copy output file to output directory under root dir

try { FileUtils.copyFile(new File(rootDir, outputFile), new File(rootDir + "/output/outputs.xml")); } catch(final IOException e) { e.printStackTrace(); } }

package dummy;

import java.io.BufferedInputStream;import java.io.File;import java.io.FileInputStream;import java.io.FileNotFoundException;import java.io.IOException;import java.util.Properties;

import org.apache.commons.io.FileUtils;

public class DummyAdapter2{ public static void main(final String[] args) { final File propertyFile = new File(args[0]);

BufferedInputStream bis = null; try { bis = new BufferedInputStream(new FileInputStream(propertyFile)); } catch(final FileNotFoundException e) { e.printStackTrace(); } final Properties prop = new Properties();

DummyAdapter2.java

1

2

17

General adapter configuration for DummyAdapter2

18

• The <command> option can be any executable, for example,

19

Quiz

1. How does FEWS exchange information with external modules?

Answer: Published Interface XML.2. Which configuration file specifies the model to run?Answer: The module configuration file.3. FEWS must have some knowledge about external

modules, true or false?Answer: False.4. What method a Java model adapter must implement?Answer: The main method.

20

OHD FEWS Adapter Framework• An instance of the FEWS General Adapter concept.• Conforms the latest version of FEWS PI XML schema• Abstracts some common tasks for all OHD models.

– Encourage code reuse– Leveraging the effort of creating an adapter for each of the models.

• Can be used for both Java based model or legacy C/FORTRAN based model (a prebuilt binary executable).

• Configured through the general adapter configuration – Need a run info file to pass information to the OHD model, such as

input/output file locations, diagnostics files, etc.– The run info file is created by the <exportRunFileActivity> activity in the

module configuration file.• Has an executable jar in the <binDir> folder.

21

OHD NWSRFS Java and non-Java (legacy) models

Java models• SACSMA• UNITHG• SNOW17• gridded SNOW17 for MARFC's

FFG application• TATUN• RSNWELEV• MUSKROUT• GLACIER• LAYCOEF• LAGK

Non-Java (Legacy) models• CHANLOSS(FORTRAN)• BASEFLOW(FORTRAN)• SARROUTE(FORTRAN)• CONSUSE(FORTRAN)• APICONT(FORTRAN)• gridded APICONT for MARFC's

FFG application(PCRASTER)• RESSNGL(FORTRAN)• RESJ(C++)• SSARRESV(FORTRAN)

22

FEWS

params.xml

input.xml

mods.xml

states.xml

stateI.txt

output.xml

diag.xmlO

HD

Few

sAda

pter

ModelDriver

Extends

xxxModelDriver

Model output filesare PI XML files

Data are exchanged in memory.

Model specific filesLoaded into OHDFewsAdapter

Data Flow (Java model)PI

PI

PI

PI

PI

PI

23

Model Parameter File Example

1. Follow the pi_moduleParameter.xsd schema2. XML tags are used to identify the parameter3. Use basic data types, string, double, int,

table

24

Model State files

• states.xml

• statesI.txt/statesO.txt

25

Important Java Classes (1)

• OHDFewsAdapter– Entry point of OHD models– Executed by the FEWS general adapter.– Invoke OHD model execution• Read in model configuration, timeseries, parameters,

and states• Calls the ModelDriver.executeDriver() method

– Can be executed independent of FEWS• Given all necessary information

26


• The ModelDriver Abstract Class– Abstracts all NWSRFS models– All models must implement the ‘execute()’

method• Executed by the ModelDriver.executeDriver() method• Output results and the diagnostics file in PI XML format • Execute the model using common data structures

(RegularTimeSeries, ModelState, ModelParameters)

27


• Common xml handling classes for– Parsing FEWS pi files • Input time series• Input states meta data • Input model parameters

– Writing FEWS pi files• Output time series

28

How is it implemented?

class OHDFewsAdapter{ … void main() { … //instantiate modelDriver= Class.forName( “ohd.model.foo”); modelDriver.executeDriver(); … } …}

class ModelDriver{ … void executeDriver() { … //the physical model //starts execute(); … } …}

<executeActivity><className>ohd.model.util.fews.OHDFewsAdapter</className></executeActivity><exportRunFileActivity> <properties> <string key=“model” value=“ohd.model.foo”/> </properties></exportRunFileActivity>

XML

Java

Java

29

OHDFewsAdapter

IDriver

Driver

ModelDriver

XXModelDriver

Implements

Extends

ExtendsInstantiate

Interface

Abstract class

User class

Class Hierarchy

OHD class

30

The ModelDriver Abstract Class (1)

31

The ModelDriver Abstract Class(2)

32

The ModelDriver Abstract Class(3)

33

OHD Java External Model Structure (1)

34

OHD Java Model Structure (2)

35

If rootDir is defined as the%TEMP_DIR%, the directoryis temporary; will beremoved after the adapter run. This behavior can be modified by the F12 menu.

36

How to configure OHDFewsAdapter (1)

• Header Info

Can be missing. Set <ignoreDiagnostics> in <executeAcitvity>

Every external module is a generalAdapterRun

37


• Startup activity

Most are standard directories thatmust be cleaned out prior to modelexecution.

Create working directories if they don’t exist

38


• Export activity: Export States

In order to use relative dating,states must be exported. States can be empty.

39


• Export activity: Input Time SeriesId of module that generated the needed data

Relative dating requires states be exported

40


• Export activity: Parameters

If not specified, current module instance id is used.

41


• Pass general information to the Adapter• Read by the OHDFewsAdapter

• Export activity: run info file

42

Example of OHDFewsAdapter Runinfo File

This is the <properites> entry in the model configure file.

43

Runinfo properties• “model” entry

– The name of the Java class that implements the ModelDriver interface• “printDebugInfo” entry

– Controls how detailed messages are printed to the diagnostic file.– Five levels (0 to 4).– Higher values mean more detailed information– Do not to be confused with the “Error Levels” in diag.xml

• 0 – fatal level, when an exception occurs• 1 – error level, similar to the fatal level• 2 - warning level, warning messages are printed• 3 – the regular level• 4 – debug level, the most detailed information about the execution of the

model.

• All other entries are model specific.

44


• Execute activity

The runinfo file is passed as an argument to the adapter.

45


• Import Activity: Import model output time series

The module instance ID of this module.

46

Example: Basin Dummy ModelDummy Model

• Use the OHDFewsAdapter to drive the dummy model.– The dummy model is written in Java– Extends the OHD ModelDriver abstract class– Input a time series, add a value, and then subtract a

value from the time series– Configure the OHDFewsAdapter in FEWS

• Set input time series• The value to be added• The value to be subtracted

47

package dummy.models;import java.util.List;import ohd.hseb.measurement.Measurement;import ohd.hseb.measurement.MeasuringUnit;import ohd.hseb.measurement.RegularTimeSeries;import ohd.hseb.ohdmodels.ModelDriver;import ohd.hseb.util.fews.FewsRegularTimeSeries;

public class DummyModelDriver extends ModelDriver{ public DummyModelDriver() { super._state = new ModelState(); super._parameters = new ModelParameters(); } @Override protected void execute() throws Exception { // get list of input time Series final List<RegularTimeSeries> inputTsList = getTsList();

_drivingTs = inputTsList.get(0); super.runModelDriverValidation();

final long startTime = _drivingTs.getStartTime(); final long endTime = _drivingTs.getEndTime(); final int interval = _drivingTs.getIntervalInHours(); final MeasuringUnit measuringUnit = _drivingTs.getMeasuringUnit();

// get measurement list; just a copy of input values final List<Measurement> measurementList = _drivingTs.getMeasurementList();// create an output time series using properties of input time series final FewsRegularTimeSeries tempTimeseries = new FewsRegularTimeSeries(startTime, endTime, nterval, measuringUnit);

// copy values in input time series to output time series final FewsRegularTimeSeries outputTimeSeries = new FewsRegularTimeSeries(tempTimeseries, _drivingTs);

// get run info properties final Properties runInfoProperties = getDriverProperties();

// get some properties in run_info.xml file final int valueToAdd = Integer.parseInt(runInfoProperties.getProperty("valueToAdd")); final int valueToSubtract = Integer.parseInt(runInfoProperties.getProperty("valueToSubtract"));

for(int index = 0; index < measurementList.size(); index++) { final Measurement measurement = easurementList.get(index); final double newValue = measurement.getValue() + valueToAdd - valueToSubtract; measurementList.set(index, new Measurement(newValue, measurement.getUnit())); }

outputTimeSeries.setMeasurementList(measurementList); super.addTStoResultMap(outputTimeSeries);

// each model driver must set the end state time in the states.xml file // this is important for validating that the state time is updated for the end of the run super._state.setDateTime(endTime);

}

}

DummModelDriver.Java

48

…<exportRunFileActivity> <exportFile>%ROOT_DIR%/run_info.xml</exportFile> <properties> <string key="model" value="dummy.models.DummyModelDriver"/> <string key=“outputLocationId" value=“SELWE"/> <int key=“valueToAdd” value=8/> <int key=“valueToSubstract” value=8/> <int key="printDebugInfo" value="1"/> </properties></exportRunFileActivity>…<executeActivities> <executeActivity> <command> <className>ohd.hseb.ohdfewsadapter.OHDFewsAdapter</className> <binDir>$OHDBINDIR$</binDir> </command> <arguments> <argument>%ROOT_DIR%/run_info.xml</argument> </arguments> <timeOut>300000</timeOut> </executeActivity> </executeActivities>

Dummy Model Configuration

Set name of the Java classthat implements the model.

Explicitly set the location ID for model output time series

49

OHD Legacy Models

• Uses the OHDFewsAdapter• Extends the ModelDriver• A framework has been developed to alleviate

the effort– Uses the LegacyModelAdapter to execute the

legacy model• A separate System Process (called from Java)

– LegacyModelAdapter has many helper methods such as writeStandAloneFiles().

50

51

Legacy Model Parameter PI XML File

1. All parameters are defined in one XML parameter element.

2. The OHD legacy adapter framework will export the parameters in native format and pass the filename to the model.

OHDFewsAdapter

IDriver

Driver

ModelDriver

XXXLegacyModelDriver

Implements

Extends

Extends

Instantiate

Interface

Abstract class

User class

LegacyAdapter

LegacyModelAdapter

Extends

Instantiate

OHD class

Class Hierarchy

53

How is it implemented?

class OHDFewsAdapter{ … void main() { … //instantiate modelDriver= Class.forName( “ohd.model.foo”); modelDriver.executeDriver(); … } …}

class ModelDriver{ … void executeDriver() { LegacyModelAdapter adapter = new LegacyModelAdapter(); adapter.executeModel( “executable”, args ); } …}

className>ohd.model.util.fews.OHDFewsAdapter</className></executeActivity><exportRunFileActivity> <properties> <string key=“model” value=“ohd.model.foo”/> <string key=“legacyLocation”, value=“executable”/> </properties></exportRunFileActivity>

XML

Java

Java

54

These are model specific functions.

The framework helps wrap the existing models that are C or Fortran subroutines.

55

56

57

• The example legacy model– Is a shell script– Uses the OHD legacy framework

• Configured as a OHDFewsAdapter• Executed by the OHD LegacyModelAdapter.executeModel()

– Takes an input time series and copies it to the output directory.

– Write diagnostic messages to the text file, diag.txt• The OHD LegacyAdapter will convert the text file to PI XML file,

diag.txt for FEWS to ingest.• ‘diag.txt’ has to be in a specific format.

– For example, 4| this is a log message

Example: Dummy Legacy Model

58

public class DummyLegacyModelDriver extends ModelDriver{ @Override protected void execute() throws Exception { /** * Every legacy model needs to identify a driving time series from the time series read in the inputs.xml file. * This time series usually controls the run length and time step for executions In this example there is only * "1" input time series. Another approach is to search by time series type (e.g. parameterId) */ _drivingTs = getTsList().get(0);

final LegacyModelAdapter dummyAdapter = new LegacyModelAdapter(this, getWorkDir(), _logger);

dummyAdapter.writeStandAloneFiles();

/** * The line below is used to execute a stand alone program (separate executable) if there are any model specific * arguments to be sent, fill in a SortedProperties object and send to executeModel as last argument. Make sure the * executable returns a '0' exit code. The arguments will be inserted into a text file named arguments.txt and passed * as an argument to the executable. */ dummyAdapter.executeModel(getDriverProperties().getProperty(OHDConstants.LEGACY_LOCATION_DIR) + "/" + "dummy_executable", null);

getState().setDateTime(getComputationEndTime());

final List<RegularTimeSeries> resultsList = dummyAdapter.readFromFlatFileToRegularTimeSeriesList(); for(int i = 0; i < resultsList.size(); i++) { addTStoResultMap(resultsList.get(i)); }

}

}

Arguments will be written to a text file,arguments.txt. The file is passed to theExecutable as an argument.

59

…<exportRunFileActivity> <exportFile>%ROOT_DIR%/run_info.xml</exportFile> <properties> <string key="model" value="dummy.models.DummyLegacyModelDriver"/> <string key=“legacyLocation" value=“Modules/bin"/> <string key=“outputLocationId" value=“SELWE"/> <int key="printDebugInfo" value="1"/> </properties></exportRunFileActivity>…<executeActivities> <executeActivity> <command> <className>ohd.hseb.ohdfewsadapter.OHDFewsAdapter</className> <binDir>$OHDBINDIR$</binDir> </command> <arguments> <argument>%ROOT_DIR%/run_info.xml</argument> </arguments> <timeOut>300000</timeOut> </executeActivity> </executeActivities>

Dummy Legacy Model Configuration

Set the location of the binary executable.

Explicitly set the location ID for model output time series

60

Quiz1. OHDFewsAdapter is an executable Java class, true or false?Answer: True2. What ModelDriver method does the OHDFewsAdapter invoke?Answer: executeDriver()3. List at least three differences between an OHD model and an OHD legacy model.Answer: One possible answer 1) Java vs. C/Fortran binary; 2) Legacy model reads/writes text files; 3) Legacy model uses the LegacyModelAdapter 4. What argument is passed to the OHDFewsAdapter?Answer: run_info.xml5. Which property in the runinfo file specifies the class name for the model?Answer: The “model” property.6. The runinfo file shall be created by the user, true or false?Answer: False.7. Which ModelDriver method must be implemented by a modelAnswer: The “execute” method.

61

NWSRFS Models: SACSMA• Rewrite in Java

OHDFewsAdapter

ModelDriver

SacSmaModelDriver

Extends

Used by

Abstract class

User class

SacSmaRainfallRunoffModelAdapterSacSmaRainfallRunoffModel

Instantiate

Used by

62

SACSMA Java Code (1)

63

SACSMA Java Code (2)

64

Example SACSMA Configuration 1

65


66


67


SACSMA can accept state modifications as separate input time series.

68


69


70

SACSMA runinfo example

71

SACSMA parameter PI XML

72

SACSMA state files

• states.xml


73

NWSRFS Models: LAGK• A native binary executable

OHDFewsAdapter

ModelDriver

LagKModelDriver

Extends

Used by

Abstract class

User class

LegacyModelAdapter

Instantiate LegacyAdapter

Extends

Binaryexecutable

Executed by

74

LagK Java Code (1)

75

LagK Java Code (2)

76

LagK Configuration(1)

77


78


79


80

LagK runinfo

81

LagK parameter PI XML files

82

LagK state files• states.xml


83

LagK directorieslagk`-- ctwc1 |-- input | |-- inputs.xml | `-- params.xml |-- output | |-- diag.xml | |-- outputs.xml | `-- statesO.txt |-- run_info.xml |-- states | |-- states.xml | `-- statesI.txt `-- work |-- arguments.txt |-- diag.txt |-- nwsrfs_datatype_mapping_file.txt |-- nwsrfs_dataunit_file.txt |-- outputs.txt |-- params.txt |-- run_info.xml |-- states.xml |-- statesI.txt |-- statesO.txt `-- ts.txt

84

HL-RDHM

• A software package for NWS distributed hydrologic models– Stand-alone tool for distributed hydrologic modeling research and

development.– A prototype for validating technique in NWS field offices prior to

operational software development• Written in C++/C/Fortran• Distributed models

– SACSMA, SAC-HT, SACHTET, Snow-17, and Hill slope/channel routing, DHMTF

– Auto-calibration– Models can be run in sequence

• Incorporated HL-RDHM functionalities into CHPS

85

Approaches– Develop a FEWS distributed model general adapter– Develop a HL-RDHM model driver – Port HL-RDHM components as FEWS external modules• If a feature exists in FEWS, use FEWS otherwise use HL-

RDHM component• Use Java Native Interface (JNI) to link HL-RDHM

component with the model driver– Develop an external FEWS module that converts NetCDF

gridded time series to XMRG files– Reuse OHD code if possible

86

Data flow in FEWS environment

87

Software Design

88

OHDDistFewsAdapter.javapublic class OHDDistFewsAdapter{ …… public static void main( final String[] args ) { if( all necessary data are not present or checks are not OK )

{ throw exception

}else{

//start logger defineModelRunInfoAndSetupModels() Call the ModelDriver's executeDriver() method // execute the model

//write logger

} } ……}

89

DistModelDriverpublic class DistModelDriver extends ModelDriver{ …… public void executeDriver( ) throw Exception { …… execute() …… } ……}

public class RDHMDriver extends DistModelDriver{ …… public void execute( ) throw Exception { …… getRunInfo() modelFactory registerModels setupDomain call hlrms_algo() …… } ……}

90

Configuration

Date post:	01-Jan-2016
Category:	Documents
Upload:	egbert-webb
View:	217 times
Download:	0 times

Extending FEWS with External functionalities Zhengtao Cui NOAA National Weather Services Office of...

Documents