Developing applications that harness the Agricultural and Natural Resource Grid
(and linking your own data)
Matthew Laurenson and Seishi NinomiyaNational Agricultural Research Center
TsukubaJapan
Contents• Concepts
–Remote Procedure Calls (RPC)–Brokers
• Using a broker–Metadata requests–Data requests–Issues–Linking your own data
• SOAP• New brokers and future enhancements
Overall Goal: To help you make “portable” software applications (DSS).
Current Situation: Web-based but…
• Applications closely coupled to databases• Limited range of applications at each site• Duplicated development at each site
Soils
Ag and NR Grid – a “Hotel Room” for Agricultural and Natural Resource DSS
Room
Weather Elevation Maps
Connectors for local data
DSSDSS DSSDSS
User interface in national language
New ZealandNew Zealand
Soils
Weather Elevation
Maps
地
気象 地面の高さ
地図
日本
中国中国
土壤气象 海拔
交通图
Goal of Ag and NR Grid:To provide this “hotel room” in your
country, so you can use models developed by others (and they can use your models).
Distributed Computing Concepts
• Remote Procedure Calls• Brokers to handle heterogeneous databases
Remote Procedure Calls (RPC)
• RPCs let a program execute a procedure on a remote computer as if it were local.
Remote Procedure Call
My Computer
remote=findRemote()A=5B=6C=remote.add(A,B)Print C...
Remote Computer
procedure add(x,y) return x+y}...
Remote Procedure Calls (RPC)
• RPCs let a program execute a procedure on a remote computer as if it were local.
• Newer RPC mechanisms let programs send and receive data objects.
• RPC mechanisms: CORBA, .NET, Java RMI, SOAP...
• RMI – only Java programs can use• SOAP – used by other languages as well
RPC vs Local Calls
• Latency (delay)– RPCs much slower than local calls– difficult to predict how long call will take
• May fail– network problems– problems with other computer (“server”)
• Security issues– firewalls...
But...
• Can distribute processing and data over multiple machines
• Great when multiple users must share the same dynamic data item or communicate with each other.
Mediated Architecture – “Brokers”
Broker
DomainDatabases
(Heterogeneous) Applications
Consistent data access(using RPC mechanism)
MetBroker
MetBroker
Korea
Japan
New Zealand
WeatherDatabases
Data Retrieval
Applications
South Africa
USA
Data RetrievalServlet
Weather Risk
Crop Growth
Disease Risk
Web Browsers
Contents• Concepts
–Remote Procedure Calls (RPC)–Brokers
• Using a broker–Metadata requests–Data requests–Issues–Linking your own data
• SOAP• New brokers and future enhancements
Data and Metadata Requests
Metadatarequest
Metadataresult
Broker Database(various)
Application
Standardrequest
DB-specificquery
DB-specificresult
Standardresult
Metadata Data
Interaction between applications and brokers uses objects – data packages
Example - MetBroker Services
• Weather Data– Single station– All stations in a region
• Metadata (data about data)– Database details (coverage of globe, login req.)– Regions within databases– Station details (latitude, longitude, start date, end
date, elements recorded, resolution...)
Elements of a Station Data RequestWeather station
Required weather elements
Time interval
Username/password
Resolution
Allow summarizing
Send Request to MetBroker and Receive Result
Data ResultTsuchiura
Operated 1976-present1999/1/3 06:00-1999/2/1 00:00
10.1, -0.2, 012.6, 3.8, 0
11.3, -1.5, 0.3...
To MetBroker Data Request
amedas40341
rain, air temperature1999/1/1-1999/2/1
dailyOK to summarize
Query Results - Single Station
Station& query details
rain air temp. solar rad.
Station details anddata sequencesare encapsulatedin a singleresult object
BrokerTemperature
Rainfall
Radiation
Client
Spatial Request
WakayamaRainfall DB AMeDAS
Daily max/min temperatures & rainfall for all stations in region
Spatial Request Background map dynamically retrieved from http://www.mapblast.com
Query results - multiple stations
Using Brokers – Implementation Issues
RMI Blocked by Firewalls
ClientApplication
Java Broker
Server MachineClientApplication
Java
Wrapper Servlet
HTTP
RMI
RMI
Client-side firewalls
Applet Using Multiple Brokers
Spatial Risk Applet
MetBroker
DEMBroker
ChizuBroker
Background maps
Elevation data
Meteorological data
Resource Server
Application text
Without Multiple Threads - Sequential
MetBroker
DEMBroker
ChizuBroker
Background maps
Elevation data
Meteorological data
Resource Server
Application text 12 seconds
Spatial Risk Applet
With Multiple Threads – Parallel
MetBroker
DEMBroker
ChizuBroker
Background maps
Elevation data
Meteorological data
Resource Server
Application text 5 seconds
Spatial Risk Applet
JavaBeans for Java GUI programs
• For rapid GUI development in Java can use existing MetBeans
• Transparently handle firewalls, latency...• Making a simple data retrieval application
takes about 30 minutes• Used in many MetBroker demo applications
Easy Development of New Applications
Databases Linked Using “Drivers”
DatabasesApplications
Broker
Linking a Database to a Broker• Need to write one Java class (a “driver”) with two
or three main methods• Typically use Java Database Connectivity (JDBC)
drivers to query relational databases• Can reuse existing classes for Oracle, Postgres, and
MySQL JDBC.• Drivers can read data from files via HTTP, or parse
HTML tables.• “Test harness” available for driver development
How much work?
• Add a new database to MetBroker– roughly working - 3 days– tested more thoroughly – 1 week– (assuming that database is readily accessible)
• Make a MetBroker applet– working prototype in one day– (assuming basic science is “done”)
Developer Resources
• Demos, documentation, downloads, pictures– http://www.agmodel.net
• Mailing lists – subscribe at web site• Brokers are open source under the
GNU Lesser General Public License.– Can be used in commercial applications– Any improvements must be released under same
license
• Linking your data to the grid lets you use grid tools developed by others.
• It doesn’t mean you must share your data with everyone (password protection).
• The key to interoperability is the interface
Reminders
Contents• Concepts
–Remote Procedure Calls (RPC)–Brokers
• Using a broker–Metadata requests–Data requests–Issues–Linking your own data
• SOAP• New brokers and future enhancements
SOAP
• An XML-based RPC mechanism• Independent of programming language• SOAP toolkits for C++, Visual Basic, Delphi, Java,
ASP, PHP, Perl, Python…• Firewall-friendly• Basis of Microsoft .NET architecture• A SOAP Web service is described in standard format
using a .WSDL file(Web Services Description Language)
Visual Basic client application
Applicationcode
MS SOAPToolkit
VB function calls
VB objects
MetSOAP
SOAP over HTTP
MetBroker
Delphi client application
Applicationcode
Borland SOAPToolkit
Delphi function calls
Delphi objects
SOAP over HTTP
RMI
Example - MetSOAP
SOAP Summary
• Positives:– Opens brokers to a range of programming languages– Already the standard for Web-based RPC– Avoids most firewall issues
• Negatives– Can’t reuse Java classes - must “mirror” in each language– Interoperability is still imperfect (but improving fast)– Inefficient for large data sets
Typical Array in SOAP
• ... • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">23.3</item> • <item xsi:type="xsd:float">21.11</item> • <item xsi:type="xsd:float">19.49</item> • <item xsi:type="xsd:float">18.48</item> • <item xsi:type="xsd:float">16.52</item> • <item xsi:type="xsd:float">14.73</item> • <item xsi:type="xsd:float">14.73</item> • ...
Model Deployment Options - Summary
• Java applet– flexible, responsive once loaded but requires plug-in
• Java servlet– simple browsers, rapid load, but clunky interface
• SOAP– direct access for Delphi, VB…
• Existing models (FORTAN,C,BASIC)– Use simple Java app. to retrieve data and write to a file
Contents
•Concepts–Brokers–Remote Procedure Calls (RPC)
•Using a broker–Metadata requests–Data requests–Issues–Linking your own data
•SOAP•New brokers and future enhancements
New Brokers
• DEM Broker - Digital Elevation Model data– Japan 50m resolution– World 1km resolution
• ResourceServer– Internet-based service for software localization
• ChizuBroker – Web map services– www.mapfan.com (Japan)– www.mapzone.co.nz (New Zealand)
• SoilBroker – soils data
Future enhancements
• Broker-side interpolation (using DEMBroker)– temporal– spatial
ServerTemperature
Rainfall
Radiation
Client
“Virtual station”Broker-side interpolation at single site
Future enhancements
• Broker-side interpolation (using DEMBroker)– temporal– spatial
• Other brokers– Place names– Crops and crop models (prototyped)– Agrichemicals
Final Points
• The Grid provides a stable, powerful foundation for Ag and NR software
• You can now develop grid applications in your favourite programming language.
• NARO is keen to help you to both develop new applications and link new databases.
• Database owners around the world will be keen to evaluate your applications.
Thank you