Accelerating the data-sharing process Copyright © 2007, Sypherlink, Inc. All Rights Reserved. Rehan...

Accelerating the data-sharing process

Copyright © 2007, Sypherlink, Inc. All Rights Reserved.

Rehan Chawdry

Integrated Justice Practice Leader

614-652-6200

[email protected]

NIEM In Practice

• Launched on Feb 28, 2005 • Partnership between U.S. DOJ and U.S. DHS• NIEM v1.0 released 11/11/06• NIEM v2.0 released 7/31/07

• Framework for a common XML data vocabulary• Provides standard naming conventions, type definitions• Standardizes field/element relationships• Describes cardinality, domain space of fields/elements• Evolved from GJXDM from justice world• Uses best practices from other standards (ebXML, etc)

• Technical features• Set of XSDs distributed with documentation• XML Elements built around a common core• Namespaces allow domains to evolve independently of

each other, but still build on common elements

• Usage is growing rapidly• Future DOJ State/Local funding tied to NIEM

compliancy• Florida, Texas, other state-wide data sharing efforts• FBI N-DEx program• DoD UCore support (Aug 7, 2008)

• Governance• NIEM Program Management Office• Comprised of government and industry stakeholders

NIEM Overview

NIEM Usage (1 of 2)

• What NIEM is and is not• NIEM IS not a database model

• NIEM does not define a set of web services

• NIEM IS an exchange standard

• NIEM is intended to be extended before it is used• Intended to be a template that defines common data elements

• Initiatives should only pick only those elements they need to use• If data elements do not exist, users are encouraged to extend using NIEM

naming and design rules (NDR)• There is a standard process to “pick”/extend NIEM for your use (IEPD process)

• NIEM needs to be coupled with other standards to be effective• SOAP and Web Services for SOA functionality

NIEM Usage (2 of 2)

• Using NIEM typically involves three(3) steps1. Requirements gathering

2. Design and documentation

3. Implementation

• Information Exchange Package Documentation (IEPD) • Standard process that guides how to do #1 and #2 above• Most current NIEM training is about IEPD development

• Implementation (#3) is not defined by NIEM and is rarely covered

• Why do this?• The NIEM process is essentially used to define an API• Insures that an API uses common nomenclature and structure

Example NIEM instance

<?xml version="1.0" encoding="UTF-8"?><Person personNameCode="BirthName">

<PersonName><PersonGivenName>Bart</PersonGivenName><PersonMiddleName>Jojo</PersonMiddleName><PersonSurName>Simpson</PersonSurName>

</PersonName></Person>

• Post 9-11 accelerated need to share data between state, local, and federal law enforcement agencies

• Niche industry (over 300+ vendors) provide operational software to law enforcement agencies all over the country.

• CAD, RMS, JMS, Court

• Each vendor has a unique system with their own platforms, schemas, features, and naming standards (i.e, subject vs offender vs suspect)

• Agencies with the same vendor could usually data share. Agencies with different vendors often could not

• Jurisdictional boundaries made data sharing difficult (State can’t directly connect into local police system and “pull” data out)

• Most CAD, RMS, JMS, Court systems focus on operational activities -> not great analytic capabilities, especially dealing with large volumes of data

• Different vendors emerged to provide analytic software and warehouses to consolidate data from different law enforcement systems (LInX, CopLink, etc)

• Data standards were still evolving (GJXDM), but not supported by the majority of industry initially.

Public Safety Domain Background

Florida Data Sharing Initiatives Background

• Florida created seven (7) domestic security regions within the State to coordinate homeland security functions

• Each of the regions consisted of between 30-90 agencies

• Each region independently pursued data sharing initiatives with its member agencies

• Four (4) regions each selected different analytic vendors.

• Regional implementation involved getting data out of each operational agency systems and into selected analytical system (some warehouse, some federated)

• Three (3) regions were looking to start data sharing initiatives

• Florida Department of Law Enforcement wanted to align all regions and have strategic approach to data sharing

RLEX(3 regions)

RLEX(3 regions)

Florida Department of Law EnforcementData Sharing Initiatives

TallahasseeRegion

Ft. MyersRegion

Miami Region

PensacolaRegion

JacksonvilleRegion

TampaRegion

Orlando Region

SmartCopSmartCop LInXLInX

FINDERFINDER

CopLinkCopLink

Florida Law Enforcement Exchange (FLEX) would link all seven Regional Project nodes together

PROBLEMEach region also had their own data sharing solution

~30 agencies

~30 agencies

~90 agencies

Regional Law Enforcement Exchange (RLEX) needed to integrate data from 3 regions and over 150 law enforcement agencies

GOALS1.Provide data sharing solution for three (3) regions

2. Provide data sharing solution for entire State

Requirements• Provide practical framework for data sharing• Neutral, vendor-agnostic approach• Leverage NIEM and other national standards when possible• Support warehouse (RLEX) and federated (FLEX) approaches

Challenges• Where and how to leverage NIEM?• At an RLEX level, multiple operational systems• At a FLEX level, multiple analytical systems• For both projects, multiple, separate law enforcement agencies

NIEM in Florida

Where to use NIEM

• Develop canonical data model as basis for data sharing• Use NIEM to develop canonical relational database model• Determine how data from all regional systems map to NIEM

• For RLEX, standardize on way to get data to State• Publish an API that can be used to accept data into regional warehouse• Work with each agency to convert their data on an ongoing basis• Warehouse data from agencies in RLEX region

• For FLEX, standardize on way to query data between regional systems• Work with analytic vendors to have them implement a common query API• Each regional system continues to own their data• Federated query allows access to all regional systems

Standard Canonical Model

Florida NIEMData Model

Florida NIEMData Model

• Common relational structure for warehoused data

• Data fields identified by analyzing other analytic systems in FL -> shareable fields

• Used NIEM naming and design rules

• Provided target data model that other systems could map to

Sample Tables

Next Task – Standard submission API

• Analytic vendor chosen by RLEX used a warehouse model• Had their own database schema and submission

API

• All agencies would need to implement it -> Once implemented, agency is locked to vendor

• Agency implementation is most time consuming in data sharing projects

Vendor Application

Current Warehouse Models

Agency 1

Law EnforcementDatabase

Application Database

Agency 2


Agency 3


Vendor specific

Vendor specific

Vendor specific

• Florida wanted to avoid lock-in

• Insure data integration could be leveraged with different vendors down the road

Enter LEXS

• Logical Entity eXchange Specification (LEXS)

• General purpose sharing API based on NIEM first developed in 2005

• Developed by DOJ initially to support data sharing between its federal agencies (FBI, DEA, ATF, BOP)

• What problem does it address that no other current NIEM-based standard addresses?

• Defines a common set of NIEM elements all systems should be able to understand (who, what, where, and how things are related)

• Allows extensions to be written in such a way that allows an interpreting system to ignore parts it doesn’t understand

• A single IEPD as opposed to several IEPDs that cover different areas

Enter LEXS

• LEXS has two parts• Publish/Describe (LEXS-PD)

• Framework for laying out NIEM-based data• Describes actions that systems should take with data (insert, update,

delete)

• Search/Retrieval (LEXS-SR)• Describes search API based on NIEM• Allows wildcard searches, entity-specific searches• Describes search results format

• Several LEXS-based APIs have been developed• FBI’s NDEX initiative• DOJ’s Suspicious Activity Report (SAR)

• Anyone that uses a LEXS framework will be able to interoperate with other LEXS-based systems without changes

• Law enforcement system that describes an Arrest report can be sent and interpreted by DoD even if it doesn’t understand what an Arrest report is

Understanding NIEM / LEXS

NIEM

LEXS

SRPD

SA

R

IEP

D #

2

IEP

D #

3

IEP

D #

1

breadth of applicability

fun

ctio

na

lity

• NIEM: data definitions, element library → design-time compatibility• LEXS: specific structure, constrained entities → run-time interoperability• LEXS-SR: search & retrieval → federated query capability• LEXS-PD: publication & discovery → data replication capability

N-D

Ex

FL

EX

LEXS and Florida

• LEXS-PD implemented as standard API for agencies to push data to RLEX

• Work with vendors to get data into LEXS format• Training existing law enforcement vendors on LEXS

• Help produce LEXS from their software using internal development team or using data integration vendors (Sypherlink)

• Deploy software at law enforcement agencies to provide continual data feeds

• Solved tactical needs but also allowed agencies to position themselves strategically

• Provide data sharing in ways that they couldn’t before

• Support for emerging national systems (NDEX) without having to re-code

Vendor Application

Current Warehouse Models

Agency 1



Agency 2


Agency 3


Vendor specific

Vendor specific

Vendor specific

• Florida wanted to avoid lock-in

• Insure data integration could be leveraged with different vendors down the road

Agency 2

Agency 1

Vendor Application

Best Practice - Warehouse Models


LEXS-PD

Vendor specificLEXS-PD


NIEM Appliance


NIEM Appliance

NIEM Appliance

Agency 3

LEXS-PD


NIEM Appliance

Agency 2

Agency 1

Vendor Application

Best Practice - Warehouse Models


LEXS-PD

Vendor specificLEXS-PD

Agency 3


NIEM Appliance


NIEM Appliance


NIEM Appliance

NIEM Appliance

Vendor Application 2


Vendor 2 specific

LEXS-PD

Support for multiple, simultaneous analytic vendors from same agency feed

Next Task – Standard query API

RLEX(3 regions)

RLEX(3 regions)

TallahasseeRegion

Ft. MyersRegion

Miami Region

PensacolaRegion

JacksonvilleRegion

TampaRegion

Orlando Region

SmartCopSmartCop LInXLInX

FINDERFINDER

CopLinkCopLink

Florida Law Enforcement Exchange (FLEX) would link all seven Regional Project nodes together

Each regional system had it’s own platform and schema

Florida wanted to allow distributed approach between regions (i.e. federated system)

Region 4

Classic Federated Model

Region 1


Region 2


Region 3


Region specific query



Each region would have to understand the search APIs of the other regions in order to perform a federated query across all regions

LEXS and Florida

• LEXS-SR implemented as standard API for agencies to search data between regions

• Each region consumes a LEXS-SR message and figures out how to map it into a local query within the regional system.

• Results are formatted using LEXS-SR and sent back to external originator

• Several vendors were already implementing LEXS-SR for other initiatives.

• Standardizing on LEXS-SR at a State level allows external entities to query the State at some point (i.e. NDEX and Florida)

NIEM Appliance Vendor Application

Best Practice - Federated Models

Agency 1


NIEMView LEXS-SR

Agency 2


NIEM

View

LEXS-SR

Agency 3


NIEM

View

LEXS-SR

Application Query

FBI

NDEX

LEXS-

SR

Conclusion

• NIEM can be used effectively in real-world applications, as long as it is augmented with other related standards

• LEXS• Justice Reference Architecture (JRA) – if dealing in this domain

• Other examples of NIEM use• NCIC project• Workflow exchanges• Many other Federal, State, local law enforcement sharing systems

• NIEM branching out into other areas• April 2008 – Support for DoD UCore standard• Commercial organization exposing their data as NIEM• NIEM Project Management Office is soliciting other federal agencies to

help expand NIEM into other functional areas.

Review

Q&ARehan Chawdry

Integrated Justice Practice LeaderSypherlink, Inc.

[email protected]

Date post:	27-Mar-2015
Category:	Documents
Upload:	kylie-little
View:	218 times
Download:	4 times

Accelerating the data-sharing process Copyright © 2007, Sypherlink, Inc. All Rights Reserved. Rehan...

Documents