8/31/2016Nlets Implementers Workshop

NIEM UpdateMike Hulme, NIEM Technical Architecture Committee Co-chair andUnisys Solution Architect

2

Topics

• NIEM 3.2• NIEM 4.0• niem.github.io• Enhanced code list capabilities• Enhanced business rule specification capabilities• NIEM Java binding tool• Use of JSON with NIEM• The NTAC needs you!

3

NIEM 3.2

• NIEM 3.2 was released in June 2016• A minor release that incorporates content identified since

NIEM 3.1 was released in May 2015• Highlights of NIEM 3.2 include:

– The onboarding of a new Surface Transportation domain by the US Department of Transportation

– Substantial updates to the NIEM Emergency Management domain– Additional content updates to the Biometrics, Military Operations,

Justice, and Human Services domains

• No changes to Naming and Design Rules (NDR)• Keep in mind Nlets uses NIEM 2.0, but NIEM versions last

forever!

4

NIEM 4.0

• Development starts in October 2016 for a mid-2017 release• Harmonization of content• Potential technical improvements

– Relax some NDR constraints– Relax some extension schema rules– Enable object URIs other than document-relative IDs (to refer to

external content)– Allow for a “lightweight” IEPD– Allow code lists to be relaxed or more stringent (as appropriate)– Complete Schematron in XSD specification– Formalize NIEM-conformant JSON IEP– Your suggestions?

5

niem.github.io

• niem.github.io is a collaborative website with content managed using the git open source distributed version control system

• Allows the NTAC and NBAC to more directly manage content and accept contributions from the community

• Still a work in progress

6

Enhanced code list capabilities

• NTAC developed a specification for using Genericode code lists with NIEM (released March 2016)

• Generalizes the use of code lists to incorporate code list tables, with additional columns beyond code value and definition

• Allows for advanced functionality such as keys composed of multiple columns

• Includes artifacts that support use of code lists at run time and at schema definition time

• Includes identifiers for well-known columns that have semantics needed across the NIEM community

7

Enhanced code list capabilities (cont.)

• Uses Genericode (an OASIS specification)• Code List Representation (Genericode) Version 1.0• Available at http://docs.oasis-open.org/codelist/cs-

genericode-1.0/doc/oasis-code-list-representation-genericode.html

• Use is optional• Available at

http://reference.niem.gov/niem/specification/code-lists/1.0

8

Enhanced business rule capabilities

• NTAC is developing a specification for embedding Schematron in XML schema documents (in alpha 4 as of August 2016)

• Uses Schematron (an ISO specification)• ISO/IEC STANDARD 19757-3: Information Technology —

Document Schema Definition Languages (DSDL) Part 3: Rule-Based Validation — Schematron

• See http://www.schematron.com/• A language for making assertions about the presence or

absence of patterns in XML documents

9

Enhanced business rule capabilities (cont.)

• It is very simple (only five important elements), very powerful (it can express many kinds of constraints impossible in other schema languages)

• It captures constraints in human language assertions and generates appropriate human-language diagnostics

• Use is optional• Draft available at

https://reference.niem.gov/niem/specification/sch-in-xsd/

10

NIEM Java Binding tool

• An open source library of functions that uses Java Architecture for XML Binding (JAXB) to implement NIEM-based information exchanges

• Can generate consistent bindings to Java objects for NIEM schema subsets and extension schema documents

• Has the potential to facilitate and simplify ongoing development by automatically generating bindings from a new or updated IEPD

• Contains example Representational State Transfer (REST) services with XML

• Beta released in August 2016:https://github.com/NIEM/IEPD-Java-bindings

11

Using JSON with NIEM: Why consider JSON?

• Use of JSON has become widespread and is increasing• The NIEM PMO is receiving questions about how to create

NIEM-conforming JSON• As of today there is no such thing as “NIEM-conforming

JSON”• NTAC would like to support people who want to do what

NIEM does, only using JSON instead of XML• The challenges:

– A solution that doesn’t dilute the useful characteristics of JSON– A solution that doesn’t dilute NIEM information sharing principles– No accepted standard for a JSON equivalent to XML schema

• The NTAC needs your input!

12

Using JSON with NIEM: NTAC actions to date

• Released an executive summary on NTAC approach:http://niem.github.io/technical/json/using_json_with_niem.pdf

• Released draft guidance on using NIEM with JSON:http://niem.github.io/technical/json/guidance/

• Uses JSON-LD, a JSON-based format to serialize Linked Data: http://www.w3.org/TR/json-ld/

• The near-term goal is a specification for an IEP (an actual information exchange instance) that uses JSON instead of XML; but still specified by a NIEM IEPD using XML schemas to support contract-first development

• Also examining ad-hoc JSON IEP using up and down transformations to support code-first development

13

The NTAC needs you!

• The NIEM Technical Architecture Committee (NTAC) defines and supports the technical architecture that governs NIEM– https://www.niem.gov/meet-us/ntac/Pages/default.aspx– Responsible for NIEM technical specifications such as the Naming

and Design Rules (NDR) and Model Package Description (MPD) specs

– Works in conjunction with the NIEM “lead developer”, Georgia Tech Research Institute (GTRI)

– Composed of federal, state, local, and industry representatives

• Consider becoming an active NTAC member!https://www.niem.gov/news/Pages/Drive-NIEM-Tech-Join-the-NTAC.aspx

14

NIEM/NTAC contacts

• NIEM:– https://www.niem.gov– http://niem.github.io/– https://reference.niem.gov/– LinkedIn NIEM Group: https://www.linkedin.com/groups/1903175

• NTAC:– https://www.niem.gov/meet-us/ntac/Pages/default.aspx– Mike Hulme, co-chair, mike.hulme@unisys.com– Dr. Scott Renner, co-chair, sar@mitre.org

15

Backup JSON slides

16

Using JSON with NIEM:What NIEM does

• Successful machine-to-machine data exchange requires the developers to have a compatible understanding of the exchanged data’s meaning

• NIEM helps arrange that compatible understanding, through– A documentation framework– Community-agreed data components

ApplicationApplication

CompatibleunderstandingDevelopers

Users

Graphics developed by Dr. Scott Renner

17

Using JSON with NIEM:Doing NIEM: Contract-based data exchange

System /Application

Data ConsumersInformation Exchange Package (IEP)the data exchanged at runtime

IEPD

defines

Information Exchange Package Documentation (IEPD)build-time description of the data to be exchanged

Developers

System / Application IEP

In NIEM, the exchange specification is defined firstSoftware for the interface is written to that specification afterwards

18

Using JSON with NIEM:Doing JSON: Often code-first, not contract-based

Data ConsumersData Producers

Developers

JSON

In this approach the application code comes firstThe interface software is generated from the application codeThere is no explicit contract (apart from the application code)

1. Design their programming-language data structure2. Generate the interface software from the data structure

tool

12

19

Using JSON with NIEM:NTAC Approach

• Identify potential actions to accommodate JSON in NIEM• Decide which of the identified actions we think are feasible

and should be pursued, and develop a roadmap• Explore each accepted action in detail

– Validate the feasibility and appropriateness– Determine how to do it– Write the appropriate product (guidance, specification, etc.)– Consider and address tool support

• Document a rationale for each rejected action

20

Using JSON with NIEM:Potential actions to accommodate JSON in NIEM

1. Provide non-normative guidance on a way to generate a JSON instance from a NIEM-conformant XML schema and/or XML instance (based on an existing standard, specification, or practice), but the JSON instance would not be NIEM-conformant.

2. Provide a normative specification on a way to generate a JSON instance from a NIEM-conformant XML schema and/or XML instance, and such a JSON instance could be NIEM-conformant.

3. If a JSON instance could be NIEM-conformant, provide a normative specification on verifying NIEM-conformance of a JSON instance to a NIEM-conformant XML schema.

4. Provide a normative specification for a NIEM-conformant JSON schema (based on an existing JSON Schema specification).

5. If a JSON schema could be NIEM-conformant, provide a normative specification on verifying NIEM-conformance of a JSON instance to a NIEM-conformant JSON schema.

6. Provide the NIEM reference schemas as JSON schemas.

7. Develop a separate NDR for JSON (as opposed to XML).

21

Using JSON with NIEM:NTAC planned and completed actions

• Develop an executive summary and roadmap of planned NTAC actions and solutions related to JSON (actual 2Q2016)

• Develop and document a rationale for why NTAC is not pursuing #4 through #7 (actual 2Q2016)

• Do #1: Develop and document non-normative guidance on generating a JSON instance from an XML schema and/or XML instance, but the JSON instance would not be considered NIEM-conformant (actual 3Q2016)

• Do #2 and #3: Develop and document a draft normative specification (building on the initial guidance) for generating a NIEM-conformant JSON instance from an XML schema and/or XML instance, and for verifying NIEM conformance of a JSON instance to a NIEM-conformant XML schema (planned 4Q2016)

• Changes to NIEM Conformance Specification to support JSON instance conformance target

• Solicit feed back from the NIEM technical community as we go along