Post on 11-Jun-2015
transcript
© 2013 IBM Corporation
IBM InfoSphere Optim Technical Overview
Tom Love – Optim Senior IT Client Technical Professional
April 2013
© 2012 IBM Corporation2 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Value Proposition
Enterprise Data Management
Production Databases Test and Development Databases
• Segregate Data & Move to
Archive
• Deploy Tiered Storage
Strategies
• Retain Data According to
Value
• Simplify Infrastructure
ManageApplication
Data Growth
• Decommission Redundant or
Obsolete Apps
• Gain Control of Application
Portfolio
• Retain Access to Legacy
Data
• Retire Apps and Repurpose
IT Assets
• Migrate Apps from High to
Low Cost Platforms
• Preserve Historical Data
Enable Portfolio
Optimization
• Protect PII Data
• Apply Single Data
Masking Solution
• Use Range of Masking
Techniques
• Maintain Referential
Integrity
• Maintain Contextual
Look and Feel
Ensure Data
Privacy
• Right size Test Apps
• Repeatable Process
• Quickly Deploy New Apps
• Future proof Apps
Speed Application Deployment
© 2012 IBM Corporation3 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information ManagementInformation Management
3
Supporting Enterprise Environments
Single, scalable, interoperable data management solution provides a central point to deploy policies
to extract, store, port, and protect application data records from creation to deletion
Data Growth Application Retirement Data Privacy Test Data Management
InfoSphere Discovery
OPTIM™
DATACOM
© 2012 IBM Corporation4 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Check Test
Results
Create/ Modify
Application
Create Test
Environment
Inspect and Add
Data for Testing
Refresh Test DataMove to
Production
Test
Fail Pass
A Typical Software Development Lifecycle
Information Management
Product Overview : Optim Test Database Management
Subsetting
Subsetting
Discovery
© 2012 IBM Corporation5 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Optim™ Data Privacy Solution
Production
Contextual,
Application- Aware,
Persistent Data
Masking
Contextual,
Application- Aware,
Persistent Data
Masking
EBS / Oracle
Custom /
Sybase
Siebel / DB2
Test
EBS / Oracle
Custom /
Sybase
Siebel / DB2
� Substitute confidential information with fictionalized data
� Deploy multiple masking algorithms
� Provide consistency across environments and iterations
� Enable off-shore testing
� Protect private data in non-production environments
© 2012 IBM Corporation6 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Current
Production
Historical
Selective Restore
Restored
Archives
Reporting
Data
Historical
DataReference
Data
Archive
� Complete Business Object provides historical reference snapshot of business activity
� Storage device independence enables ILM
� Immutable file format enables data retention compliance
IBM® Optim™ Data Growth Solution: Archiving
Universal Access to Application Data
Application Application XML ODBC / JDBC
IBM MashUPIBM MashUP
Universal Access to Application Data
Application XML ODBC / JDBC
© 2012 IBM Corporation7 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Optim z/OS Implementation
© 2012 IBM Corporation8 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
� Optim Directory
– Supplements information stored in the database
– Maintains product definitions and tracks processing
– Stores database connection information (DB Aliases)
– Stores user-defined relationships
RelationshipsOptimDIRECTORY
Tables
Stored in Database- Catalog- System Tables- Data Dictionary
ReferentialIntegrityRules
AccessDefinitions
DB Aliases
Maps
The Optim Directory
© 2012 IBM Corporation9 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Database Alias
Establishing the Database Connection
� Optim view of a database connection– High-level qualifier for database object names
• DBalias.creatorid.objectname– Enables cross-Database access– Saved in Optim Directory
Relationships
DB Aliases
Column Maps
Table Maps
ACCTS DB
CUSTINFO DB
SALES DB
OptimDIRECTORY
Tables
© 2012 IBM Corporation10 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
OPTIM Concepts: Architecture
Storage Profile
DB Source/
Destination
Server Name• Server address or name •DB Alias• Connectivity via DB Client softwareWork Directory• Server File SystemStorage Profile• Storage and retention policy
Server Name• Server address or name •DB Alias• Connectivity via DB Client softwareWork Directory• Server File SystemStorage Profile• Storage and retention policy
TemporaryData…
Server Name
WorkDirectory
Archive
Architecture
Offline Storage.( Tivoli TSM )
Enterprise Storage( IBM DR550 )
Document Storage
( Tivoli )
Optim Directory
© 2012 IBM Corporation11 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
OPTIM Concepts: Architecture with Extended Data Sources
DB Source/
Destination
Server Name
Architecture
Teradata DatabaseServer
z/OS IMS or VSAMDatacom DB
IDMSAny other Structured d
datasource
IBM Federation Server
Optim Directory
© 2012 IBM Corporation12 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
� DB Relationships are automatically derived from database RI rules
� Application Specified Relationships
• Can be defined individually to Optim
• Can be imported into Optim from DDL
� Shared by all Optim components
OPTIM
RelationshipsOptimDIRECTORY
Tables
ReferentialIntegrityRules
AccessDefinitions
DB Aliases
Maps
Stored in Database- Catalog- System Tables- Data Dictionary
A Word About Relationships...
© 2012 IBM Corporation13 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Extended Relationships
SE00582SE012
SC00234SC005
NW00564NW012
NC00123(NC)003
MANAGER_ID
Char (7)
SALESMAN_ID
Char (5)
Sales Table District Table
Example 1
Using Substr Function
Parent Table Sales
Substr(SALESMAN_ID,1,2)
Child Table District
DISTRICT_CD
00582NW
00234SE
00564SC
00123NC
MANAGER_NO
Char (5)
DISTRICT_CD
Char (2)
Example 2
Using Concat Function
Parent Table Sales
MANAGER_ID
Child Table District
DISTRICT_CD || MANAGER_NO
© 2012 IBM Corporation14 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Extended Relationships
M35
M18
F56
F45
SEX
Char (1)
AGE
Integer
250
145
135
132
Rate
Numeric (5,0)
Age
Integer
Sales Table Female_Rates Table
Parent Table Child Table
Sales Male_Rates
Sex “M”
Age Age
Sales Female_Rates
Sex “F”
Age Age 250
145
135
318
Rate
Numeric (5,0)
Age
Integer
Male_Rates TableExample 3 Data Driven Relationships
© 2012 IBM Corporation15 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Our Unique Capability: Complete Business Object
Example:General Ledger
© 2012 IBM Corporation16 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Extract - Federated Data Support
Retek / Oracle Other apps / any
DBMS
Custom Inventory
Management / DB2
Complete Business
Object Captures
End to End
Business Process
© 2013 IBM Corporation
IBM Optim TDM
© 2012 IBM Corporation18 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
OPTIM TDM Relational Extract Facility
� Creating and maintaining test data bases
� Migrating data
� The data and/or the object metadata can be extracted
ExtractFile
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
CUSTOMERS
ORDERS
DETAILS
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
TESTDB
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
QADB
PRODDB
LOAD
EXTRACT
INSERT/UPDATE
LoadFiles
Create -- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
New_DB
ADTM
CM
© 2012 IBM Corporation19 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Optim’s Relational Compare Facility
� Single-table or multi-table compare
� Creates compare file and/or compare Report of results
� For application testing, QA, and to verify database contents
� Enhances productivity by finding unexpected changes in the data
SOURCE 1
SOURCE 2
COMPAREPROCESS
Optim
COMPAREFILE
Optim
Compare
REPORT
........................................................................................................................
Interactive Browse
Verify Test
Results
© 2013 IBM Corporation
IBM Optim Data Privacy
© 2012 IBM Corporation21 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
During Extract Process
Or
Standalone Convert Process
Or
During Insert/Load Process
Transform or Replace sensitive data using
� Standard mapping rules: Literals, Special Registers,
Expressions, Default Values,Look-up tables
� Complex mapping rules: User exits
De-Identify test data
Production Data
Extract and
Convert
Masked Test Data
© 2012 IBM Corporation22 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
ExtractFile
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
Transform / mask sensitive data
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
TESTDB
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
QADB
LOAD
INSERT/UPDATE
LoadFiles
Data Privacy in Application Testing
Extract a relationally intact subset from production database(s)
• Extract data and/or object definitions
• Define a new set of test tables
• Apply masking during population process
• Extract file may be reused but contains un-Masked data
• Good process for testing masks
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
CUSTOMERS
ORDERS
DETAILS
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
CUSTOMERS
ORDERS
DETAILS
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
NewDB
Create
LegacyFiles
BKORDER
SelectionCriteria
© 2012 IBM Corporation23 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
ExtractFile
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
TESTDB
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
QADB
LOAD
INSERT/UPDATE
LoadFiles
Transform / mask sensitive data
Data Privacy in Application Testing
Extract a relationally intact subset from production database(s)
• Extract data and/or object definitions in pre-masked file
• Use pre-masked Extract file to create new set of tables
• Convert Pre-masked extract file data into second masked extract file
•Share masked extract file to be reused for population step
• Good practice for testing masks using COMPARE
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
CUSTOMERS
ORDERS
DETAILS
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
CUSTOMERS
ORDERS
DETAILS
Masked
ExtractFile
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
NewDB
Create
LegacyFiles
BKORDER
SelectionCriteria
© 2012 IBM Corporation24 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
ExtractFile
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
Transform / mask sensitive data
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
TESTDB
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
QADB
LOAD
INSERT/UPDATE
LoadFiles
Data Privacy in Application Testing
Extract a relationally intact subset from production database(s)
• Most Secure Approach
• Extract data only
• Convert during extract
•Extract file already contains masked data
•Can be shared with testers to reuse
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
CUSTOMERS
ORDERS
DETAILS
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
CUSTOMERS
ORDERS
DETAILSLegacy
Files
BKORDER
SelectionCriteria
© 2012 IBM Corporation25 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
ExtractFile
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
Transform / mask sensitive data
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
TESTDB
-- ---- ---- ---- ------- ----CUST
-- ---- ---- ---- ------- ----ORD
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETL
QADB
LOAD
INSERT/UPDATE
LoadFiles
Data Privacy in Application Testing
Extract a relationally intact subset from production database(s)
• Most Secure Approach
• Extract data only
• Convert during extract
•Extract file already contains masked data
•Can be shared with testers to reuse
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
CUSTOMERS
ORDERS
DETAILS
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
CUSTOMERS
ORDERS
DETAILS
Only Users authorized to see Private data
© 2012 IBM Corporation26 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Consistent Masking across the Enterprise
Masked fields
are consistent
Data is masked
132009824
157342266
SS#s
132009824
157342266
SS#s
DB2
323457245
134235489
SSN#s
323457245
134235489
SSN#s
Client Billing Application
© 2012 IBM Corporation27 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Example 2Example 2Example 1Example 1
PersNbr FstNEvtOwn LstNEvtOwn
27645 Elliot Flynn
27645 Elliot Flynn
Event TableEvent Table
PersNbr FstNEvtOwn LstNEvtOwn
10002 Pablo Picasso
10002 Pablo Picasso
Event TableEvent Table
Personal Info TablePersonal Info Table
PersNbr FirstName LastName
08054 Alice Bennett
19101 Carl Davis
27645 Elliot Flynn
Personal Info TablePersonal Info Table
PersNbr FirstName LastName
10000 Jeanne Renoir
10001 Claude Monet
10002 Pablo Picasso
Referential integrity is maintained with key propagation
Patient InformationPatient InformationPatient Information
Patient No. SSN
Name
Address
City State Zip
Patient No. SSN
Name
Address
City State Zip
112233 123-45-6789
Amanda Winters
40 Bayberry Drive
Elgin IL 60123
123456 333-22-4444
Erica Schafer
12 Murray Court
Austin TX 78704
Data is masked with contextually correct data to preserve integrity of test data
Data Privacy – Key Propagation
© 2012 IBM Corporation28 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Cust ID Item # Order Date
27645 80-2382 20 June 2004
27645 86-4538 10 October 2005
Customers Table
Orders Table
Cust ID Name Street
08054 Alice Bennett 2 Park Blvd
19101 Carl Davis 258 Main
27645 Elliot Flynn 96 Avenue
Original Data
Cust ID Item # Order Date
27645 80-2382 20 June 2004
27645 86-4538 10 October 2005
Customers Table
Orders Table
Cust ID Name Street
10000 Auguste Renoir Mars23
10001 Claude Monet Venus24
10002 Pablo Picasso Saturn25
Without Propagation
Now these
are Orphans!
Why Propagation is Important
© 2012 IBM Corporation29 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Masking Functions
� Column Map � Map unlike column names
� Transform/mask sensitive data
� Datatype conversions
� Column-level semantic date aging
� Literals
� Registers
� Calculations
� Default values
� Substring
� Exits
� Currency conversion
�Social Security
�Credit Card
�Hash Lookup
�Lookup
�Random Lookup
�NAME tables
�ADDRESS table
�Shuffle
�String manipulation
� PROP…
�…
© 2013 IBM Corporation
Optim Archive
Managing Data Growth Retirement and Compliance .
© 2012 IBM Corporation31 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Current
Production
Historical
Selective Restore
Restored
Archives
Reporting
Data
Historical
DataReference
Data
Archive
� Complete Business Object provides historical reference snapshot of business activity
� Storage device independence enables ILM
� Immutable file format enables data retention compliance
IBM® Optim™ Data Growth Solution: Archiving
Universal Access to Application Data
Application Application XML ODBC / JDBC
IBM MashUP
Universal Access to Application Data
Application XML ODBC / JDBC
© 2012 IBM Corporation32 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Current Data
0-2 yrs.
Production
Database
Information Lifecycle
Offline Archive
7+ yrs.
Off-line Retention Platform
CD,Tape,Optical, WORM
HP StorageWorks™,
NetApp NearStore® SnapLock™,
IBM Total Storage® solutions
(including the DR550)
EMC Centera™.
Archive File Management
Compressed
Archive File
Archive
Reporting/Query Toolsusing
ODMCognos, Discoverer,
SQLPLUS, Etc.
Non DBMS
Retention Platform
ATA File Server
Centera
DR550
Etc.
On/Near-Line Archive
5-6 yrs.
Active Historical On-Line
3-4 yrs.
Resto
re
Archive
Database
Native Access
Optional
© 2012 IBM Corporation33 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Define the data to be deleted
� Archive all data
� Delete orders and details after they are safely archived
� Preserve semantic intelligence
CUSTOMERS
ORDERS
DETAILS
RETAIN
DELETE
© 2012 IBM Corporation34 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Create the archive
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
CUSTOMERS
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
ORDERS
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETAILS
ProductionDatabase
ArchiveFile
DELETE
Optim Directory( Tables)
© 2012 IBM Corporation35 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Applications accessing the Archive Files
Optim DirectoryTables
Archive
Library
RESEARCH/BROWSE
Use the OPTIM Archive ODM Option
�Direct Access within Your Application using standard SQL
�Defines data-sources for any ODBC or JDBC application
�Joins between multiple data-sources
�archive files and database tables
End-UserQuery / Reporting
ODM
© 2012 IBM Corporation36 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Archive Collections
Collection A
Archive 1,2
Collection A
Archive 1,2
Collection B
Archive 1,2,3,4
Collection B
Archive 1,2,3,4
Archive 1Archive 1
CustomerCustomer
CustomerCustomer
Order(Y01)
Order(Y01)
Detail(Y01)
Detail(Y01)
CustomerCustomer
Order(Y01, Y02)
Order(Y01, Y02)
Detail(Y01, Y02)
Detail(Y01, Y02)
Credit CardCredit Card
ODBC/
JDBC
ODBC/
JDBC
Archive 2Archive 2
Order(Y2001)
Order(Y2001)
Detail(Y2001)
Detail(Y2001)
Archive 3Archive 3
Order(Y2002)
Order(Y2002)
Detail(Y2002)
Detail(Y2002)
Archive 4Archive 4
Credit CardCredit Card
PhysicalLogical
Tech OverviewTech Overview
© 2012 IBM Corporation37 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Archive Archive
Archive Archive
Index Index
Index Index
• Indexes are used to quickly locate specific column values in an archive file.
• Indexes greatly increase the performance when searching large archives
• Indexes provide quick access to data during an ediscovery, audit or other types of request
• Without indexes access to archive data is extremely slow because the entire file(s), which can be very large, need to be searched for the specific data that is required.
Archive Collections
= Virtual Database
Archive Indexes Tech OverviewTech Overview
© 2012 IBM Corporation38 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Restoring Archived Data
Repository
ArchiveFiles
RESEARCH/
BROWSE
Production/Staging Database
Metadata
Mapping
RESTORE
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
CUSTOMERS
-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----
ORDERS
-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----
DETAILS
Data to Restore
© 2013 IBM Corporation
Optim Archive
Portfolio Optimization/Decommissioning
© 2012 IBM Corporation40 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Portfolio Optimization/Decommissioning
� Remove a system from active service but…
� Retain access to its business-critical data
– Retire an application
– Consolidate redundant systems into single “enterprise
standard”
– Migrate portfolio to lower-cost platform
– Consolidate and eliminate unsupported databases and
versions
© 2012 IBM Corporation41 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
ArchiveArchive
Optim™ value for Application Retirement
Oracle ERP
DB2 Order Entry
Legacy
WMS
Oracle ERPOracle ERP
DB2 Order EntryDB2 Order Entry
Legacy
WMS
Legacy
WMS
Complete Business Object
Year 2006
Year 2005
Year 2004
Year ….
….….
<XML><Name>John</Name><Zip>08540</Zip>
</XML>
<XML><Name>John</Name><Zip>08540</Zip>
</XML>
AccessAccessExpire
Reta
in
Store
DecommissionDecommission
OPTIM™ Solution
© 2012 IBM Corporation42 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
OPTIM™ Technical Advantages
� Archives are immutable–Data + Metadata +
related documents
� Enables compression–Smaller footprint–Simplifies operations,
backup, restore
� Device independent–Direct access–SAN, NAS, CAS, WORM–Retrieve & restore to any
enterprise class database
� Database independent–“Future proofed”
� Supports HSM–As the data ages it can move
to less expensive storage and still be accessed
� Designed with Data Aging policies in mind
–Retention–Expiration
� Part of consistent Optim strategy for all databases and apps
–Decommissioning–Archiving–TDM
OPTIM™ Solution
© 2012 IBM Corporation43 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Access to Archived dataOpen Data Manager (ODM)
� Provides access to data in Archive Files / Archive Collections for applications that use ODBC and JDBC APIs
Optim Archive Files
ODM Primary Server
(Optim Directory & defined
ODM Data Sources)
JDBC or ODBC
accessQuery /
ReportingTools
MS-Excel, MS-Access,
Crystal Reports, etc., that
Use ODBC/JDBC
© 2012 IBM Corporation44 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Archive File Collections
� An Archive Collection is a list of Archive Files that can be logically unioned together as a single data source for Open Data Manager (ODM) access.
� The Archive File Collection Editor is used to create an Archive File Collection
� Tables with matching creator IDs and names in separate Archive Files will be unioned
� Using the Archive File Collection to access data in multiple Archive Files:
� Even if all files do not include a specific table
� Even if all files do not contain a specific column
� Even if the attributes of data in a column vary from file to file
� ODM processes files in the order they are listed in the Archive File
� Collection Editor
© 2012 IBM Corporation45 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Technical Features and Benefits of ODM
� ODM features and benefits include:
–Access archived data, with the ability to join tables, group data, perform unions, etc. under full SQL 92 via
ODBC/JDBC
–Output data in archive files as XML –Retain archive security for archive files and rows/columns.
–Leverage ODM to access archived DB2 data—converts Optim z/OS archive files to Optim (open system) archive
files.
–Perform queries and reports that confederate production and archived data as though data was never relocated
(supports connectors to a data source other than the
archive file).
–Use standard reporting and query tools like Excel, Crystal
reports and SQL utilities against archived data.
© 2012 IBM Corporation46 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
• Application access
− User requests data with application function
• Consolidated view
Application-Based Access to Archives
Application
OLTP
Archive
Archive
© 2012 IBM Corporation47 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Self-Help Access to Archives
� Open & independent
� No training of end users or audit staff
� Leverage existing tools and skills
� OLTP not required
� Excel
� Access
� Report Writers –
• Cognos,
• Business Objects,
• Discoverer
� Any SQL-based tool set
� Browser
� Java ApplicationArchive
Archive
© 2012 IBM Corporation48 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Application-Independent Access to Archives
� Application not required– Enables decommissioning of obsolete or redundant applications and
versions– No reliance on application vendor
� OLTP not required– Offline access available– Archive access does not impede production processing
XML, SQL,
ODBC / JDBC
Delivered Browser
Indexed & Secured
Company
Contact
Activity
Archive
© 2012 IBM Corporation49 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Non Production EnvironmentsSubset & Mask
Production EnvironmentsArchive
� Single, scalable, interoperable EDM solution provides a central point to deploy policies to extract, store, port, and protect application
data records from creation to deletion
NAS SAN ATA CAS Optical Tape
AIX Linux z/OS Windows XP/2000 Solaris HP/UX
DB2 LUW DB2 z/OS DB2 iSeries Informix Sybase SQL Server IMS VSAM Adabas Oracle More ….
Optim ™
Data Growth, Data Privacy, Test Data Management, Application Upgrades, Application Retirement
Enterprise Architecture
© 2012 IBM Corporation50 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
© 2012 IBM Corporation51 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Using Discovery + Optim for Data Privacy
© 2012 IBM Corporation52 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
The Problem: Securing Information
� Locate: Confidential information is dispersed throughout the organization– Hard to locate and easily transmitted, copied and printed
� Monitor: Severe financial, business and regulatory penalties for loss of sensitive and regulated data
– Business challenges - Loss of reputation and customer confidence– Regulatory challenges - GLBA, SOX, HIPAA
� Protect: Difficult to set policies without the capability to monitor and enforce– Lack of automated enforcement mechanisms
© 2012 IBM Corporation53 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Sensitive Data Discovery: Where is all the Sensitive Data? How Long Will it Take to Find It?
DiscoverSensitiveData
Poorly Documented Applications:
… ? …
© 2012 IBM Corporation54 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Sensitive Data Discovery: Structured Data
• Finding Sensitive Data Elements (SDE) in each system can take days• Whole and partial SDE’s can be found in hundreds of tables and fields
R o w M e m b e r S S # A g e P h o n e S e x
1 5 9 5 8 4 6 2 2 6 1 2 3 - 4 5 - 6 7 8 9 1 5 ( 1 2 3 ) 4 5 6 - 7 8 9 0 M
2 5 6 7 4 7 2 5 9 6 1 3 8 - 2 7 - 1 6 0 4 8 ( 1 3 8 ) 2 7 1 - 6 0 3 7 F
3 5 4 0 4 5 0 0 9 1 1 5 4 - 8 6 - 4 1 9 6 2 2 ( 1 5 4 ) 8 6 4 - 1 9 6 1 M
4 5 1 4 7 1 4 3 7 2 1 7 3 - 4 4 - 7 9 0 0 5 5 ( 1 7 3 ) 4 4 7 - 8 9 9 6 F
5 4 9 0 2 0 4 1 6 4 1 9 4 - 2 6 - 1 6 4 8 4 ( 1 9 4 ) 2 6 1 - 6 4 7 6 F
6 4 6 6 8 6 1 1 0 9 2 1 7 - 5 7 - 3 0 4 6 6 6 ( 2 1 7 ) 5 7 3 - 0 4 5 3 M
9 8 7 ,6 2 3 4 4 4 6 2 9 6 2 8 2 4 3 - 6 8 - 1 8 1 2 2 5 ( 2 4 3 ) 6 8 1 - 8 1 0 7 F
9 8 7 ,6 2 4 4 2 3 4 5 6 7 8 9 2 7 2 - 9 2 - 3 6 2 9 8 7 ( 2 7 2 ) 9 2 3 - 6 2 8 0 M
K n o w n S e n s i t iv e D a t aSensitive Data Repository
© 2012 IBM Corporation55 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Why is Sensitive Data Discovery in Structured Data Difficult?
Patient Result Test
3802468 N 53
4182715 N 53
4600986 N 32
5061085 N 53
5567193 N 72
6123913 Y 47
6736304 N 34
7409934 N 34
8150928 N 47
8966020 N 34
System A Table 15
Sensitive Relationship Discovery � Sensitive data can’t be found just by a simple data scan.
– Must connect tables and lookup tables
– Hidden within larger fields (substring)
– Hidden across fields (concatenations)
– Represented differently (lookup tables and case statements)
� “Corporate memory” is poor
– Documentation is incomplete
– SME’s and Data Analysts are only knowledgeable of one or
two systems
� Hundreds of tables and millions of rows:
– Complex
– Difficult to verify
� Data quality problems make discovery even more difficult
Test Name
53 Streptococcus pyogenes
72 Pregnancy
32 Alzheimer Disease
47 Hemorrhoids
34 Dermatamycoses
System Z Table 25
Number Name
4600986 AlexFulltheim
8150928 BarneySolo
6736304 BillAlexander
3802468 BobSmith
5567193 EileenKratchman
7409934 FredSimpson
6123913 GregLougainis
5061085 JamieSlattery
4182715 JimJohnson
8966020 MartinAston
System A Table 1
Number Name
3544600986 AlexFulltheim
5728150928 BarneySolo
3786736304 BillAlexander
6783802468 BobSmith
4035567193 EileenKratchman
8037409934 FredSimpson
4306123913 George Brett
9525061085 JamieSlattery
4594182715 JimJohnson
1288966020 MartinAston
System A Table 1
© 2012 IBM Corporation56 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
InfoSphere Basic Discovery For Sensitive Data
� Analyze up to 20 data sources
simultaneously
� Discover sensitive data by comparing
known sensitive data with data in a wide
variety of systems at the push of a
button
� Identified sensitive data elements
(SDEs) are exported to Optim for
masking
© 2012 IBM Corporation57 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Transformation Analyzer Summary for Hidden Sensitive Data
� Automates discovery of complex business rules between data sources
– Finds sensitive data hidden within longer fields (e.g. SSN hidden in a 46 digit routing number)
– Finds sensitive data that has been divided up across multiple columns (e.g. SSN divided into three separate columns)
– Finds sensitive data that has been transformed (i.e. items converted into codes)
© 2012 IBM Corporation58 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving
Information Management
Automated Sensitive Data Advantages
� Better ROI – faster time to deployment at lower cost – 5-10x time savings for business object discovery– 5-10x time savings for identifying sensitive information
Automated Sensitive Data
Discovery
Customer Success
Optim Deployment