Software Group | IBM Israel Software Laboratories SOA ...

transcript

Software Group | IBM Israel Software Laboratories SOA Advanced Technologies

Joshua FoxRegulatory Compliance through Metadata Mining

Software Group | Israel Software Labs

What Does My IT System Mean?

Real World

Metadata

Use Case: Security Marking A simplified example Security labeling has many drivers Focusing here on the semantics

Weaponization-related

Use Case: Security Marking

Not Weaponization-

Biotech Lab

A lab takes its first DoD contract Needs DIACAP approval; cannot risk non-compliance Needs to apply security markings for access control in the

Information Sharing Environment

The Metadata

Metadata for structured (machine-read) data Database schemas Web service WSDLs COBOL copybooks UML & DoDAF Models

Security Markings: Find Subject Find all info services in semantic

area of, e.g. “weaponization” Metadata Repository holds service

descriptions, database schemas, other metadata

Repository also holds standard categories from data dictionary

Tool proposes categorization Analyst uses this as input, saving

valuable manual-analysis time

Semantics

Metadata

<>…<>

Historical MD Situation MD in small quantities Scattered in

DBA teams Development teams

Background

Trends in leading-edge enterprises

Large,

cross-organization,

metadata repositories

The Promise:

Governance across the organization,

but…

Mess of Metadata

……

Heterogeneity in Metadata Different technologies: XML,

RDB, UML Different structures and

terminologies

<xsd>…

……

<xsd>…

……

Confused Semantics in Metadata

“Secure” NSA: No eavesdropping Air Force: Buy it Army: Guard the perimeter Marines: Storm it Navy: Lock the door, turn

off the lights

Huge Quantities of Metadata

<xsd>…

……

<xsd>…

……

Older Approaches

Build taxonomy/ontology Map it to the metadata

Metadata (e.g., XSD)

Ontology

Older Approaches Don’t Work

Older Approaches Don’t Work Painstaking human labor

Older Approaches Don’t Work Painstaking human labor High-cost labor: IT+ business

knowledge

Older Approaches Don’t Work Painstaking human labor High-cost labor: IT+ business

knowledge: Consultants!

Older Approaches Don’t Work Painstaking human labor High-cost labor with IT+ business

knowledge: Consultants! Beyond human limits

New Opportunities Created By:

Moore’s Law Great progress in Data Mining

Searching, classifying and organizing

Recent innovative uses: Terrorist Threat Analysis Security, Web 2.0, Google

The Time is Right

Well-known search and information-management techniques

Now, apply them to metadata

Compliance

MetadataRepository

Functional Architecture

Persistence

Semi-automation ofmapping

Engine

BusinessFunctionality Access Reporting

Real-LifeMeaning

Ontology(AKA taxonomy, dictionary,

glossary, logical model, categories)

Mapping(ontology <->metadata)

Methodology

1) Prepare Metadata2) Set up Categories3) Machine Learning 4) Suggest Category

(1) Prepare Metadata

a) Load metadata into repository

b) Pre-process metadata into Text: e.g., “Deployment”, “Location” Structure: e.g., “Deployment:Location” to

represent Table and Column

(2) Set up Categories

(AKA taxonomy, ontology, glossary, data dictionary, business model, domain model)

a) Follow Security Classification Guide

b) May use Community-of-Interest (CoI) vocabulary

c) Defense Discovery Metadata Standard for categories

d) Keep it simple!

(3) Machine Learning

a) Training on a sample of metadata samples

b) Provide semantic category mappings for this sample

c) Standard Bayesian classification algorithms learn common or uncommon words in a category

(4) Suggest Category for Metadata Item

a) Preprocess metadata

b) Submit to classification engine

c) Receive suggested category

d) Proceed with analysis

ssificatio

Metadata

Analyst

Humans and machines complementing each other

Understand Your IT: Use Cases Legacy Transformation: What business services are

hiding in your legacy applications? Reuse: Where is a service with this business

functionality? Fast Start for Community of Interest

Non-Financial

Non-FinancialNon-Financial

Non-Financial

Financial

Non-Financial

Use Case: SOX Reporting

SOX ComplianceReal World

Metadata

A Telco needs to comply with SOX to avoid penalties

Build reports from all info services with “financial” information

Metadata repository holds services, DB schemas, etc.

Tool proposes categorization Analyst can find relevant data sources

more quickly, then build report

Why Mine the Metadata

Services: Invocation-level data is transient

Metadata already expresses semantics of the data

Metadata uncoupled from ever-changing data

Table: Troop_ Deployment

Column: Total

Troop_Deployment

… … … Total

154,650

25,390

Mining the Metadata: More Secure Tool & human analyst do

not access actual data Human analyst can avoid

accessing even the metadata

Table: Troop_ Deployment

Column: Total

Troop_Deployment

… … … Total

154,650

25,390

Data Mining Complements metadata

mining Build metadata from data Differentiate on the

resource level

Table: Deployment

Column: Location

Deployment

… … … Location

“DC LAN 1”

“Baghdad LAN 2”

SimplicityMetadata Data

Structured Data Documents

Coarse-Grained Fine-Grained

Classification Search, metadata-internal relationships, transformation-building

Schema-to-Semantics Schema-to-Schema

Feasible Long-term Research

Reusable Functionality Specialized Functionality

Business Value Technical Value

Our focus

SummaryReal World

Metadata

Too much metadata: humans need help Use your metadata repository Understand your metadata Identify relevant metadata Comply with regulations using IT

metadata Metadata mining: The time is right

Joshua Fox

Metadata Analytics

Israel Software Labs

joshuaf@il.ibm.com

http://www.joshuafox.com

Thank you