EML Alchemy

NIST Voting Data Formats Workshop

GaithersburgOctober, 2009

David RR WebberTechnical EditorOASIS Election Services TCE-mail: drrwebber@acm.org

slide 2

Overview

Change is constant (even in the conservative voting world) - how do we make trusted, cost effective information technology solutions, that improve democratic processes?

How do we leverage and align standards based mechanisms and tools within our voting processes to make them more consistent, predictable, interoperable and verifiable?

How does this fit into modern software engineering techniques, familiar best practices and popular development environments?

Provide voting management staff the ability to vet and operate standard open processes without constant hand holding from technical staff.

slide 3

What makes something interoperable?

Clearly defined open processes and exchanges Based on open standards and approach; not proprietary

Adaptive approach with consistent methods This allows tailoring to specific local profiles and use patterns

When requirements change can be adjusted consistently

Fault tolerant and not brittle Ability to ignore non-critical interchange items and especially

not to fail for trivial technical reasons not related to actual information details

Leveraging XML capabilities to make self-describing transactions and exchanges possible

Support new uses without extensive reprogramming Usage patterns set via external configuration allowing broad but

controlled uses

slide 4

Agenda

EML Technical Overview Introduction to transaction sets Typical simple illustrative use path OASIS interoperability event

Recent US related EML work California Election Results Reporting Pew/Google VIP Project Geo-districting Massachusetts State Elections, IRV and EML

Rapid EML localization tools Using OASIS CAM toolkit and EML templates Quick look – EML templates and NIEM IEPD process The Importance of Domain Dictionaries

Summary / Review

EML Technical Overview

Introduction to XML transaction setsTypical simple illustrative use pathOASIS interoperability event

slide 6

The Voting Process Overview

slide 7

Why XML?

XML is multi-faceted – it can represent: data (information exchanges)

rules

workflows and processes

forms and reports

configuration and control

semantics (metadata)

documents and contracts

multimedia and presentations

XML pervasive across software systems

slide 8

EML V6.0 What’s in the box?

OASIS EML transactions – XSD Schemas

EML XML dictionary – Excel spreadsheet

Formal specification document

EML templates – OASIS CAM templates*

We will discuss EML transactions first…

* CAM = Content Assembly Mechanism, OASIS standard template specification

slide 9

Category Overview of EML transactions

Pre-election Election details (100) Candidates (200) Voters (300) Ballot Options (600)

Election Voting (400)

Post election Results (500) Analysis (500) Audit (350, 400)

One or more series of XML schemas are provided to support each general process area.

slide 10

EML V6.0 Election Transactions FamilyStandard record structures to hold election information, ballot details, voting records and services

Communicating specific result details on candidates and electionsEML 520 – results reporting

Results of election contest(s) and countsEML 510 - count

Documents access to voting records and reasonEML 480 – audit log

Group of votes being transferred for countingEML 460 – votes group

Actual record of vote castEML 440 – cast vote

Used for voter authentication during a voting processEML 420 – voter authentication

Used to register voters for an electionEML 310 – voter registration

Details of actual voters for an electionEML 330 – voter election list

Notification to voter of an election, their eligibility and how to voteEML 340 – polling information

Describes the actual ballot to be used for an electionEML 410 – ballot

Contest and candidates detailsEML 230 – candidate list

Used to nominate candidates or parties, consenting or withdrawingEML 210 – candidate nomination

Information about an election or set of elections.EML 110 – election event

PurposeDocument Name

EML 150 – election boundaries Boundary geo-districting information and polling places.

EML 530 – statistics reporting Statistics relating to the election results and votes cast

slide 11

EML V6.0 – Supporting / Control transactionsUsed to manage security, audit control, access, system, setup and event tracking.

List of accepted option proposals for referendum ballot formEML 630 – list of proposal options

Confirm receipt of option proposal nomination detailsEML 620 – options receipt confirm

Access and audit control to vote records, reason, eventEML 445 – retrieve vote

Acknowledges receipt of vote from voting channelEML 450 – vote confirmation

Voting token digital certificate channel use log EML 470 – voting token log

Nomination of option proposals for referendumEML 610 – options nomination

Voting channel device selection and useEML 360 – channel in/out detail

General system level of messages for operational controlEML 350 – series of messages

Confirm receipt and processing of EML transaction.EML 130 – response

PurposeDocument Name

EML 220 – response Candidate nomination response and confirmation.

slide 12

Example - EML 230 Candidate List<EML Id="230" SchemaVersion="6.0">

<TransactionId>OK-2007-09-1</TransactionId> <CandidateList> <Election>  <ElectionIdentifier Id=“Oklahoma 2007" /> <Contest>   <ContestIdentifier Id="State Governor" />   <Candidate>Brad Henry</Candidate>   <Candidate>John Wayne</Candidate>   <Candidate>Bill Okapi</Candidate>   <Candidate>Jane Smith</Candidate> </Contest></Election></CandidateList></EML>

Example of an EML artefact in XML

slide 13

Typical EML Process Path EML 230 candidate list EML 330 voter registration EML 410 ballot layout EML 440 cast votes EML 460 vote group EML 480 audit log activity EML 510 vote counts EML 520 election resultsAlso consider EML 360 voting channel use EML 470 voting channel digital token log EML 530 election statistics reporting

Pre-Election configuration

Polling activity

Tallying and Reporting

slide 14

OASIS Demonstration – London, 2007 EML 330 voter registration records

Voter selects voting channel preference

EML 440 retained by voting channels EML 460 sent to IBM tabulator system in

Brussels Reconciled with EML 330 totals per channel EML 510 count results EML 520 voting results

Illustrates the reconciliation of the number of authorised voters with votes cast and votes counted by channel and in total.

White Paper available describing event:http://www.oasis-open.org/committees/download.php/25992/EML%20Interop%20Demo

%20Report.pdf

slide 15

System interoperability testing

Testing,Agreement /Alignment.

Validate

Templates

Report

SystemUses &Creates

Results

Test

Pass / Fail

XMLhtml

32

4

Test Cases

XML

Package

StructureRulesContextExamplesDocumentationDictionary

Publish

1

Verify Result Outcomes

XML

LocalizationContent

Hints

slide 16

Technology Considerations Summary Use of XML in software systems is

pervasive All platforms and all major developer

environments support XML Everything from FaceBook™ to Microsoft

Office™ to Oracle™ database is XML capable

The OASIS EML transaction Schemas can be directly used by developer software tools to handle the associated XML transactions

The OASIS EML templates provide support for generating additional XML artifacts to suit localization needs

Recent US related EML work

California Election Results ReportingPew/Google VIP Project Geo-districtingUse of EML in State Elections with IRV

slide 18

California Election Results Reporting

Audience – news media channels

Developed in summer of 2007

Enhanced EML 510/520

New EML 530 statistics transaction Joint effort between CA SoS staff and OASIS EML TC

slide 19

Outcomes

Extend EML logical geographic constructs to handle USA district / ward structures

Results for California 2006, 2007 and 2008 elections reported

Validated ability of EML to report US election results

Added extensible statistics reporting – same EML 530 now being used in Belgium as well

slide 20

Sample of California Election Statistics

EML 530 XML

EML 530 in web browser

slide 21

EML 150 - Pew / Google VIP Project Audience – voters District boundary,

ballot and polling place information

Mapping tool support

EML 150

slide 22

EML 150 – Lessons Learned Re-usable components from EML allow rapid

development of new transactions EML V6.0 dictionary provides core enabler EML 150 provides capabilities surpassing the

original VIP XML layouts Includes GML standard mapping capabilities Polling place location, layout, access and facilities Aligned to OASIS EDXL for emergency management support

OASIS EML technical committee able to rapidly respond to emerging needs and provide solutions

VIP team is able to transition their existing data to new EML 150 format and gain benefits of broader information access and reuse (currently each state is new project).

slide 23

Massachusetts State Elections, IRV, EML Feasibility study on use of EML to support IRV for

state-wide elections Determine whether EML would be an appropriate

way to convey partially aggregated ballots from an IRV (or other preferential ranking system) from a lower jurisdictional level up to a higher jurisdictional level where the ballots would be counted and theresult of the election would be determined

Currently changes being considered forMassachusetts law to use IRV for state-wide offices

OASIS EML TC members supporting investigation into using EML v6.0

Possible enhancements envisioned during EML v6.0 review period to precisely cover identified needs

Rapid EML Localization tools

Using OASIS CAM toolkit and EML templatesQuick look – Templates + NIEM IEPD processImportance of Domain Dictionaries

slide 25

Localization Key Components

XML Schema

2

1

3

4

DocumentationReadable information use details (HTML), EML cross-reference Spreadsheet

XML artifacts The localized XSD Schema and picklist.xml

Examples & Test CasesRealistic test XML instances for conformance and interoperability testing

Key components(aka IEPD*)

EML Schema XSD aligns voting information model, processes and domain dictionary

*IEPD – Information Exchange Package Documentation

slide 26

National Information Exchange Model

Support for NIEM approach Builds

components for IEPD (Information Exchange Package Documentation)

Best practices - template evaluation for NDR compliance

http://www.niem.gov

OASIS at NIEM training event:

http://events.oasis-open.org/home/ei-summit/2009

slide 27

NIEM and OASIS work OASIS Emergency Management TC - joint MoU

between NIEM and OASIS Development of NIEM “adaptor” for handling

OASIS XSD schema and domain dictionary definitions within NIEM Core

Integrated into NIEM 2.1 (latest release) Successful formula / election services overlaps Opportunity to follow same path for OASIS EML

localization for USA use Potential benefits: NIEM is Federal, State, Local

and Tribal facing - developers already using NIEM

Broad community resources for training/facilitation

slide 28

Localization Approach StepsEML

XMLXSDXMLXSD

VotingVoting

ExchangeExchange

ManageManage

developdevelop

11

ingestingest

22BaseCAM

Template

BaseCAM

Templatelocalizelocalize

33 RevisedCAM

Template

RevisedCAM

Template

GenerateGenerate

44 Local XMLXSD

Local XMLXSD

CompareCompare

55

EMLDictionary

EMLDictionary

Steps:1. OASIS develops standard international XSD schema

Implements the voting process, management and information exchange needs

2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template

Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed

4. Generate new localized XSD schema that conform to EML standard5. Build dictionary cross-reference of localization usage6. Publish documentation and XML examples

PublishPublish

66Documentation

+ Examples

Documentation+

Examples

slide 29

EML V6.0 provides base CAM templates* Templates come pre-packaged in the EML V6

download

* Open source Eclipse desktop tool: http://www.sourceforge.com/projects/camprocessor

slide 30

How it works in a nutshell

EML xmlEML xml

EML xsdEML xsd

LocalizationLocalization

Commonxsd

Namespace

XML Schema

ComponentsComponents CIQ xsdCIQ xsd

INGESTINGEST

Structure

Rules

Documentation

XSLT toolsprocess

entire EML XSDcollection

XSLT toolsprocess

entire EML XSDcollection

CAM = Content Assembly Mechanism

OASIS EML XSD Collection

CAM Template

slide 31

Example – Eclipse XML Structure Editor

EML template is shown in the structure navigator panel for review and editing

Template contains all the default content model and structure rules ingested from the XSD.

All annotations and documentation from XSD also ingested (show as “paperclip” symbol).

Code lists and typical content values inserted for easy visual reference.

slide 32

Using Editor to mark Exclude items

Structure Editor Tools

Select focus on item

Invoke action menu (right mouse click)

Pick action

Tip: exclude rule display is context sensitive and only available on optional items

Tip: use “Add New Rule” mode to specify different XPath for exclude (quick mode assumes “current path”)

slide 33

Extending the base EML schema In addition to excluding existing optional

components you may need to add new pieces

New pieces should be assigned their own extension namespace prefix – to identify them from the base EML components

Typically the OASIS EML committee will review new pieces as candidates for adding to future versions of the standard

• Send comments and suggestions to election-comment@lists.oasis-open.org

slide 34

Adding New Domain Elements

Select root element node in structure; right mouse click for context menu; select “Add Namespace”; then enter prefix and URI

1

Select element node in structure; right mouse click for context menu; select “Add Child Attribute / Element”; then enter prefix and name

2

slide 35

Generating your EML use schema

Once your localization details are completed you are ready to complete the remainder of the process

This starts with generating your own new subset schema

Then once schema details are OK – creating test cases, cross-reference spreadsheet and documentation

slide 36

Generating subset use schema

Select Export CAM as XSD menu option

1

File Menu Option

Confirm the location and filename, and namespace mode.

Select ‘default’ for Mode to write a regular import set of XSD schema for each namespace

2

slide 37

Schema subset generated

Set of XSD files with filename and namespace suffix

Each namespace file is import for those specific type definitions

Reviewing XSD results in a schema visual editor tool

slide 38

Information Restriction Considerations

The CAM template allows full use of XPath conditional expressions and a rich set of over 30 functions including: setNumberRange(), setLength(), setLimit(),

setDateMask(), setNumberMask(), restrictValues(), lookup()

Those that are compatible with XSD constraints will cause constraint schema assertions to be written out when exporting to schema

Also option to set cross field validations using XPath conditional rules

slide 39

Dictionary and Documentation Tools Dictionary Tools

Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible

Documentation Tools Web interactive layout Business review tabular layout Developer raw template view

Dictionary importance across process Discovery, reuse, consistency New development - library of components XML Tool support

slide 40

EML v6.0 spreadsheet dictionary

slide 41

Test Case Generator Feature Summary

Make both Pass / Fail testing examples

Content hinting so examples use real not fake data

Test optional item logic with: all / random / none

Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern

Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value)

Make hundreds of test cases without manual editing

Can link test case to XSD schema for structure tests

You can modify XSLT tool for own testing needs

slide 42

Run CAM Rules Check on Examples

Pick Test Case Example to VALIDATE; click Finish to run validation rules

Review validation results

Run Menu Option

1

3

2

slide 43

Localization Approach Recapitulation

EML

XMLXSDXMLXSD

VotingVoting

ExchangeExchange

ManageManage

developdevelop

11

ingestingest

22BaseCAM

Template

BaseCAM

Templatelocalizelocalize

33 RevisedCAM

Template

RevisedCAM

Template

GenerateGenerate

44 Local XMLXSD

Local XMLXSD

CompareCompare

55

EMLDictionary

EMLDictionary

Steps:1. OASIS develops standard international XSD schema

Implements the voting process, management and information exchange needs

2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template

Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed

4. Generate new localized XSD schema that conform to EML standard5. Build dictionary cross-reference of localization usage6. Publish documentation and XML examples

PublishPublish

66Documentation

+ Examples

Documentation+

Examples

Summary

Review and take-aways

slide 45

EML V6.0 Technical Highlights

Set of standard transactions for core processes in elections

Extended control, event and audit tracking transactions

Domain dictionary of common components used in EML v6.0

OASIS CAM templates allow rapid localizations

Set of IEPD artifacts generation Compatible with NIEM.gov approach

slide 46

Questions?

Localization SupportTools

Quick Feature ReviewSample ScreenshotsTutorial Links

slide 48

Localization Support Tools

Provides automated generation of need XML artifacts: Pick list development from template Schema subsets generation Dictionary cross-reference spreadsheet Use and rule documentation reports XML example instance and test cases

Quick summary screenshots here Tutorial resources:

http://oasis-open.org/committees/download.php/34489/EML%20and%20jCAM%20tutorial%20and%20primer.pdf

slide 49

Test Case Generator Feature Summary

Make both Pass / Fail testing examples

Content hinting so examples use real not fake data

Test optional item logic with: all / random / none

Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern

Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value)

Make hundreds of test cases without manual editing

Can link test case to XSD schema for structure tests

You can modify XSLT tool for own testing needs

slide 50

Test Case Generation Quick Start

File Menu OptionDefault directory to write examples into

Name to be used for the examples

How many examples to create

Repeating elements count

for Quick Test – just click “OK” to use default settings

slide 51

Test Case Results

Active links to view the generated examples

slide 52

Run CAM Rules Check on Examples

Pick Test Case Example to VALIDATE; click Finish to run validation rules

Review validation results

Run Menu Option

1

3

2

slide 53

Dictionary and Documentation Tools

Dictionary Tools Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible

Documentation Tools Web interactive layout Business review tabular layout Developer raw template view

slide 54

Dictionary Compare Tool

Select compare tool menu option

Browse and choose EML dictionary

slide 55

Drag and Drop to Excel Spreadsheet

ClickOK

slide 56

Tabular DocumentationDocumentation Menu Option

Select Format, resulting HTML opens in browser viewer

slide 57

Tabular HTML Content Details

EnhancedData typeLogic

Extended Code listhandling

Clear use pattern

XPath referencesandfunctions

Annotations

Resources

Links to sites and examples

slide 59

IEPD- Information Exchange Package Documentation

NIEM- National Information Exchange Model

XSD- XML Schema Definition

XSLT- XML style sheet language transformations

Glossary

slide 60

Simple EML 440 Example

Example using the EML 440 cast vote schema Simplified to minimum requirements Example can be downloaded here:

http://markmail.org/message/qyrpfq75mtctblyq

slide 61

www.oasis-open.org/committees/election

www.oasis-open.org/committees/cam

www.oasis-open.org/committees/emergency www.niem.gov

www.sourceforge.net/projects/camprocessor

Resources: