<Insert Picture Here>
NIEM Team, Oracle Public Sector
Technology Introduction – September 2011
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
XML Validation Test Suiteswith CAMV
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
The following is not intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Disclaimer Notice
Slide 2
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Test Suites Introduction
• A key need in working with XML information exchanges, is developing examples and matching templates including rules, context, and roles for correctly managing information exchange content between partners. The W3C Schema by themselves are not sufficient.
• Learn how to use open source tooling, XPath rules, and the OASIS CAM templates standard to create automated testing tools.
• Integrating SQL table lookup validation rules• The CAMV validation engine can also be used in
production environments to manage live information exchanges.
Slide 3
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Contents
• Part 1• The XML Validation Framework• XPath Rules Techniques• Code lists and SQL lookups• Rules use cases
• Part 2– Example CAM templates – CAMV validation examples
– NIEM example test suite– Elections example test suite
– SQL table database lookups– Running validations from visual CAM Editor – Results handling techniques– Summary
* National Information Exchange Model (NIEM) – see http://www.niem.gov
Slide 4
<Insert Picture Here>
XML Validation Framework
Architecture and DeploymentCAMVTest Model
Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
XML Validation Framework
CAMVXML instance
Template (compiled)
Database Lookup
(optional)
Results XML
Java Handler Errors and Warnings
Structure
Rules
Code Lists
SQL Query
Slide 6
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAMV Engine
CAMV Java (OneJar)
JDOM XML
JDBC Spring
XPath
API Error Handler
Slide 7
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAMV Validation Engine
• Standalone XML validator using XPath rules
• Context aware and declarative rules approach
• Integrated with CAM editor menus for instant testing
• External code list and SQL table lookup support
• Callable via Spring framework or Java API; thread safe
and server container deployable
• Configurable error levels – error, warning• Java error results object returned• Embedded validation results attributes in XML results
Slide 8
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM Editor / CAMV Environment
CAM Editor CAMV
CAM Editor
XSLT Reports
CAM template
Single XML Test Instance
XML Validation Result
Slide 9
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAMV / ANT Script Environment
XML TestInstances
ANT
CAMV XML ValidationResults
XSLT
Reports
CAM template(s)
CAMV Control Script XML
22
11
Slide 10
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAMV Test Control Script XML (single)
<camvTestSuite><testCase>
<Example-Test-v19-3><cxxfile>Test-Template-v19.cxx</cxxfile><xmlfile>Test-Instance-v19-3.xml</xmlfile><verbose>true</verbose><inlineErrors>true</inlineErrors><parameters>
<abc>Param1</abc><cde>Param2</cde>
</parameters></Example-Test-v19-3>
</testCase></camvTestSuite>
Validation Template
Test XML instance
Test case name
Singleton Test Case Example
Optional items
Slide 11
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAMV Test Control Script XML (multi-instance)
<camvTestSuite> <testCase> <Example-Test-With-Folder>
<cxxfile>../samples/templates/EML-330-list-v7.cxx</cxxfile><testdir>../samples/XMLsamples/UOCAVA/330</testdir>
<verbose>true</verbose><inlineErrors>true</inlineErrors><parameters>
<abc>Param1</abc><cde>Param2</cde>
</parameters> </Example-Test-With-Folder> </testCase></camvTestSuite>
Validation Template
Test XML instances folder
Optional items
Test case nameMulti-Test Folder Example
Slide 12
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Linux GTK ANT Configuration
• Linux distributions (e.g. Ubuntu) has ANT pre-installed in /usr/share/ant/lib
• Download Ant-contrib from • http://ant-contrib.sourceforge.net/
• Copy ant-contrib jar file to /lib directory inside ANT installation
Slide 13
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Mac OS ANT Configuration
• Mac OS X 1.7+ “Lion” comes pre-installed - use command : ant –version and it will self-configure
• Mac OS X 1.6 and earlier – need to install “Xcode” development software first
• Download Ant-contrib from • http://ant-contrib.sourceforge.net/
• Copy ant-contrib jar file to /lib directory inside ANT installation
Slide 14
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
ANT – Windows configuration
• Download ANT from • http://ant.apache.org
• Download Ant-contrib from • http://ant-contrib.sourceforge.net/
• Copy ant-contrib jar file to /lib directory inside ANT installation
• Set the ANT_HOME environment variable to the directory where you installed ANT
• Add %ANT_HOME%/bin folder to executable path
Slide 15
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Windows System Path Settings
22
11
33
Slide 16
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Running CAMV with ANT
• Go to CAMV location on Sourceforge
• http://www.cameditor.org/#CAMV_Testing
• Download CAMV release JAR and Test example ZIP
• Unpack test example from ZIP and put CAMV jar into
folder
• Go to the command line (run cmd)
• Switch to folder where CAMV test suite is located
• ANT runAll
• Output is automatically placed in folder called CAMVTestResults
located within the test instance location folder(s)
Slide 17
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Reviewing Validation Results HTML
Slide 18
<Insert Picture Here>
XPath Rules Techniques
Quick Syntax PrimerControlling Scope, Context, EvaluationDynamic Structure ComponentsUseful XPath Resources
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
XPath Syntax Primer
• XPath is a simple assertion language where expressions reference components hierarchically within an XML instance structure – e.g.• /animals/canines/domestic/dogs/poodle[1]• /animals/canines/domestic/*
• CAMV uses the XPath v2.0 Java library• In CAM templates XPath is used extensively to
provide rules and content details for XML structure components
Slide 20
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
XPath Functions
XPath Function sampler logical
concat(string,string,…)
substring(string,start,len)
string-length(string)
normalize-space(string)
upper-case(string)
contains(string1,string2) Yes
starts-with(string1,string2) Yes
not(arg) Yes
exists(item,item,...) Yes
count((item,item,...))
/bookstore/book[price>35]/title
Slide 21
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM XPath Extensions (most used)
Structure Functions Content Functions
makeOptional(xpath…) setNumberMask(xpath,arg)
makeRepeatable(xpath…) setStringMask(xpath,arg)
makeMandatory(xpath…) restrictValues(xpath,args)
excludeTree(xpath…) setDateMask(xpath,arg)
makeNillable(xpath…) datatype(xpath,arg)
setChoice(xpath…) setLength(xpath,arg)
setLimit(xpath…) lookup(xpath,args)
setValue(xpath,arg)
printMessage(xpath,arg) setNumberRange(xpath,args)
setDefault (xpath,arg)
Slide 22
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM templates and XPath expressions
• Within a CAM template XPath expressions are found in the <as:BusinessUseContext> section of the template.
Slide 23
<Insert Picture Here>
Rules Use Cases
Typical Business NeedsQuick Examples
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Standard Pattern for Business Rules
• Claim – An English statement of a business rule that conveys its meaning.
Example: A Chapter MUST have a Title.• Test – A formal statement of a business rule
that can be parsed and processed by machine.Example: <Assert test="count(para) >=1">
• Diagnostic – An English statement that identifies and explains what happened or was found when a Test fails.
Example: Chapter "X" does not have a Title.• Outcome – Error, Warning, Pass
Slide 25
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Controlling Scope and Context
Condition controls if and when the content rule action applies
Condition can control structure as well; optional, mandatory, exclude
Condition can reference different component(s) for contextual rule actions
Slide 26
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Some more examples
Business Rule: Items with color attributes don't need a tray number—the color is sufficient for picking the item:
<as:constraint condition="exists(//Item/@color)" action="makeOptional(//Item/TrayNumber)" />
Business Rule: If order weight exceeds 25 kg, purchase order must specify a freight carrier:
<as:constraint condition="//Item/@weight > 25" action="makeMandatory(//Item/FreightHandler)">
Using XPath axis referencing (when things are not adjacent in hierarchy):
<as:constraint condition="exists(ancestor::Item/@color)" action="makeOptional(//Item/TrayNumber)" />
* Excerpted from Michael Sorens DEVX CAM article examples
Slide 27
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Useful XPath Resources
• Michael Sorens article on using CAM from DEVX.com• http://www.devx.com/xml/Article/41150
• Martin Roberts – tutorial on writing CAM rules• http://merryflame.co.uk/jcam/site/tutorials/CAMTemplateBasics.htm
• XPath tutorial resources• Search on XPath tutorial• Books available
• Tools – XPath expression constructors
Slide 28
<Insert Picture Here>
Code Lists and SQL Lookups
Introduction to CAM code listsAdvantagesTools and Hints
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Code lists
• CAM lookup() function allows referencing to external XML lists of values
• Can replace use of restrictValues() static lists• Database - CAM lookup() function can reference SQL table
for dynamic value checking• Where to get code lists?
• Import XSD enumeration schema into CAM lookup lists• Convert UBL Genericode into CAM lookup lists• Dump from database into lookup list XML format• Direct via SQL table lookup
Slide 30
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM Code list XML format
Textual value
Code value
Code list name
Slide 31
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Combining Rules and Code lists
• Real power of code lists and CAM – ability to select code list values based on XPath rule criteria to handle different partner configurations, versioning and cross-table validations • e.g. Ford / Mustang | Chevy / Corvette• Message version id = 1.6 – use code list XYZ-1-6
Slide 32
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Code lists examples
• Load lookup list XML into CAM editor• Assign lookup() rule to XML component
• Using SQL lookups – need to define connectivity to database and the SQL query to perform – we will do this part in the next section
Slide 33
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Code Lists Summary
• Lookups can be to external XML file instances• Simple XML format for optimized performance and
readability• Dynamic runtime control via XPath rules to select lists
to be applied• Lookup can reference SQL statement for database
content checking
Slide 34
<Insert Picture Here>
Part 2
Example CAM templates Running CAMV validationsResults handling techniquesSummary
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM templates
• Here we introduce CAM templates and understanding their component sections
• Template is abstraction layer over schema structure and rules
• Template can be automatically built from existing XSD schema
• Add additional XPath validation rules• Compiled templates provide the validation instructions
for the CAMV validation engine• Using CAMV validations and handling results
Slide 36
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM Templates Overview
• Consists of 4 functional sections:
•Header declares properties and parameters for the CAM process to reference
•AssemblyStructure captures the business information exchange structures
•BusinessUseContext provides the rules to apply during validation
•Extension points to the annotations and external lookup tables
Slide 37
<Insert Picture Here>
Example CAM templates
Examples ReviewedQuick walk through of important details
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Available Test Suite Examples
• Two test example templates and XML instances:• Baseline SAR with just structure and content usage rules,
schema parlance “cardinality, facets and enumerations”• Baseline SAR advanced – logical business rules added to
check consistency of information and structure components
• Each test set contains examples that pass and those that fail depending on the template validation applied
* SAR – Suspicious Activity Report NIEM exchange
Experimental SAR Examples
OASIS EML V7 – UOCAVA Testing• Set of test examples for Election Management:
• Both individual test cases and folder based multiple examples• Each test uses different template and validation scenario
* EML – Election Markup Language Standard
Slide 39
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Experimental SAR Overview
* SAR – Suspicious Activity Report
Areas of Interest:
- Drugs
- Fraud
- Gangs
- Internet
- Pornography
- Missing Persons
- Property
- Vehicles
- Community
- Weapons/Chemicals
- Prescriptions
- Travel/Immigration
- Terrorism
Slide 40
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Visual Template Rules Wizard Tools
Slide 41
<Insert Picture Here>
Running CAMV Validation
Illustrative examples – pass, fail, warningsDefault structure handlingData type and Content checkingCross-field validation rulesSQL database lookup
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Example Advanced Rules
<as:constraintcondition="//sbl:PrivacyDetails/sbl:PrivacyPurgeDate < //sbl:PrivacyDetails/sbl:PrivacyPurgeReviewDate"action="printMessage(//sbl:PrivacyDetails/sbl:PrivacyPurgeReviewDate,Warning: Review date not before purge date)"><as:annotation/></as:constraint>
Business Rule: Providing warning if PurgeDate is earlier than PurgeReviewDate
<as:constraintcondition="not(./nc:LocationAddress) and(not(./nc:LocationHighway)) and(not(./nc:LocationTwoDimensionalGeographicCoordinate))"action="printMessage(//j:TargetLocation/sbl:Location,Empty Location information)"><as:annotation><as:documentation type="documentation">Example of advanced rule</as:documentation></as:annotation></as:constraint>
Business Rule: Location must contain an address or a highway or coordinate set
Slide 43
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Default Structure Handling
By default all structure components are required and string content
Content Masks
Structure control
Slide 44
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Running CAMV details
Batch Command Line Execution Window Viewing Same Template in Visual Editor
From within the Test Suite folder on the command line execute > Ant runALL
Slide 45
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Sample SAR Validation Results HTML
Slide 46
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Installing and using SQL lookup exampleDB lookup sample install folder contents
Contains test sample and template
Run start and stop database scripts
Simply download and unpack the ZIP archive and drop the DatasourceExample package into the samples Test Suite folder
Then add the camv-test-DbList.xml to the testsuites folder
Slide 47
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
SQL DB configuration details
camv-test-DbList.xml contains data source reference and connection configuration
Modify to match your local directory structure
Modify inside dbProps.props file to match your local directory structure
Slide 48
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Sample SQLDB Test Run
• Start the database instance (slide 47)
• From within the Test Suite folder on the command line execute > Ant runALL
Slide 49
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
DB lookup test results rendered as HTML
SQL DB lookup result
Slide 50
<Insert Picture Here>
CAM Editor Visual Tools
SQL lookup entry wizardRun CAMV validation wizardVisual entry of SQL connection detailsVisual review of validation results
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM template SQL lookup rule definition
This is the myLists.cam example template
Slide 52
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAM template SQL lookup rule wizard
Slide 53
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Test from CAM Editor validation Run Wizard
jdbc:derby://localhost:1527/firstdb
11
22
33
44
Pick test case XML file
Choose JDBC driver
Confirm driver name
Replace default connection with actual JDBC string
NOTE: this is the same information as contained in the dbProp.props file
Slide 54
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Sample Visual Test Results
SQL lookup result displayed
Slide 55
<Insert Picture Here>
More Results Handling
XSLT script post-processingXML Diff considerationsRegression Testing with CAMVMiddleware Integration
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Default XSLT rendering of results
HTML output
Finds CAM validation results attributes
Formats and outputs error details
Slide 57
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Post-Processing Tips and Tricks
• XSLT post-processing not only for HTML reporting
• Create response error messaging directly from CAMV output
• Create receipt notification directly from CAMV output
• Can use simple DIFF to compare generic HTML report
output to detect regression test changes
• Rule based checking is superior to simple result content
change testing
• Java API and Spring supports direct integration of CAMV
and returning validation results directly to middleware
Slide 58
<Insert Picture Here>
Summary
ReviewNext StepsFuture enhancementsResources and Links
CAMVTest Model Data
Deploy Requirements
Build Exchange
GenerateDictionary
Exchange Development
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Review and Value Proposition
• Testing and verification of your exchange XML examples
• SQL database lookups and flexible code lists handling
• Business rule checking
• Post-processing options and reporting with XSLT scripts
• Regression testing with batch processing
• Flexible deployment of XML validation framework
• Integration via Spring API to middleware
Slide 60
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
What are we working on?
• Improvements• “CAM for web services” – currently exchange-centric• Enhancing ANT capabilities and soapUI integration
• New features• Dictionary integration with NIEM Domain Update Tool• Evaluation tools for dictionaries and components• Enhancing XML example generation with synthetic data
Slide 61
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
CAMV Summary
• Open source, open public standards based, cross-platform deployment written using Java
• Uses Java APIs, Spring API, JDOM, etc.• Validation features:
• XPath V2.0 support• Looks ups• 20 built-in functions
• Reporting• Embedded CAMERROR, CAMWARN tags• Spring API and results set handler• Optional sysout test result log
Slide 62
Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation
Links and References
DOWNLOADS - http://cameditor.org/ #CAMV_Testing
NIEM MATERIALS -Oracle NIEM website
http://www.oracle.com/goto/niem Oracle XMLOrb blog site
http://blogs.oracle.com/xmlorb
Slide 63