Dealing with Dark Data

1

Agenda

What is dark data? Josef ElliottUsing HP ControlPoint to find, analyse & classify unstructured enterprise data Dominic Johnstone

Using AIO to analyse and manage database and application content Mark Gower

What do I do with the data I find? Dominic Johnstone

Securing and managing data with HP Records Manager Craig Adams

Hosting your data in the secure HP Cloud Bharat MistryPutting it all together - DEMO of ControlPoint and Records Manager Dominic Johnstone

Arthur Leclerc-Chalvet

Q&A and Prize Draw for HP Tablet Device Josef Elliott

Today’s Information Reality

2

EXPLOSION OFDATA

Growing IT Burden, Business Inefficiency and Risk Exposure driving Information Governance

REAL-TIME ACCESS

DATA EVERYWHERE

INCREASED REGULATIONS

• Mobile, virtual,

cloud joins physical and on-premise

• New Data Security considerations

• 67% of users have three or more computing platforms

• Always-on applications and users

• Users have highup time expectations

• Business continuity moves to forefront

• Escalating regulatory and higher volumes

• Find the “needle in haystack”

• Utilize corporate information assets

• Data doubling every 12-18 months

• New, unstructured data types to manage

• Storage growth is outpacing IT budget

• Growth of ungoverned structured data

Overload or opportunity?

3

Enterprise Information Environment

“Missed Opportunity” “Increased Risk” “Cost & Complexity”

10%

Social Media Video

Audio

Email

Texts Messages

Word, Excel

Images

Clickstream Data

Transactional Data Logs

ERP CRM

HRMS ProcurementSupply Chain

Management/Inventory

Mgmt

Organisations need to handle 100% of this information

90%

Human Information

Structured Information

Lots of Systems

4

And it’s growing…

5

Why is it growing…..?

6

We are running out of capacity Let‘s add more disks

Applications are slowing down Upgrade infrastructure

Backup takes longer and longer Change backup infrastructure

We need to retain information... Keep tapes

…For a certain period of time We keep everything forever

We need to be compliant Implement archive, DMS, RM,...We need to retrieve information...

Look into different sources

…Historical information Recover tapes

The myth of cheap storage, coupled with lack of clear policy creates confusion

Current behaviour leads to over-retention

7

Dark Data

“like our own physical universe, the digital universe is rapidly expanding and incredibly diverse, with vast regions that are unexplored and some that are, frankly, scary”. (IDC, Digital Universe, 2013)

Dark Matter

8

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=Y_0blQonRMQ3iM&tbnid=GotUVRqNGpxPzM:&ved=0CAUQjRw&url=http://ircamera.as.arizona.edu/NatSci102/NatSci102/lectures/darkmatter.htm&ei=SiJxUpSyFYWl0QXx5oDAAQ&bvm=bv.55617003,d.d2k&psig=AFQjCNFpqHclX2cqgUV26ok3kkS23WNzsQ&ust=1383232434309384

9

Defining Dark Data

Characteristics of dark data

What is it?• Human readable• Unstructured • Not indexed• Unmanaged• Inactive• Orphaned

Where is it?• File Servers• SharePoint• Email Servers

10

Opportunities and Risks of Dark Data

What lies hidden in your dark data?

Understanding your dark data can have significant benefits:• Cost savings through the reduction of storage and management overhead by allowing

the defensible destruction of non-required data• Prepares and organizes valuable legacy information to provide insight into current and

future business processes• Allows you to recognize information types and structures and develop policies to

properly govern information in future

Data that remains dark has the potential risk of:• Containing sensitive information that is unprotected from data leakage and misuse• Information being used out of context• Decisions are based on outdated versions of information• Duplicate effort is spent in producing the same information again

11

The Information Paradox

Information

Conf

usio

n, C

ost,

Risk

Gartner Inc 2012

Document Production

12

Document Production RequirementApplicable for Litigation, Regulatory Submission or Internal enquiry purposes

Major financial benefit accrues when “document production” becomes a repeatable, defensible process

It’s actually all about Information Management

13

14

Six Steps to Redemption

1. Understand the logical information environment

2. Identify existing information repositories

3. Index to understand your existing data

4. Deep dive analysis and advanced content analytics to extract meaning

5. Archive, Protect, Dispose

6. Secure the benefits

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15

A stepwise approach to making sense of your dark data

Applying Information Governance to Dark Data

1. Identify your dark data sources2. Create a light metadata index3. Reduce obvious ROT4. Perform deeper analysis, incorporating advanced data

analytics to add meaning and context to the data (data becomes information)

5. Categorize the information6. Define and apply auditable policy to the information

Auditable Policy

Analyse

Sample

TagApprove

Execute


Reducing Redundant, Obsolete and Trivial Data

Redundant:• Identify duplicate

documents• Identify the master

(assess against obvious master sources)

• Get rid of the copies

Obsolete:• Identify documents that

haven’t been accessed for a long time

• Compare against your retention schedules

• Get rid of the ones that are obviously past their use by date

Trivial:• Identify documents that

have no valuable content• System Files, Crash Dumps,

Thumbnails etc.• Assess on the basis of file

types• Get rid of the obviously

trivial ones

Freeing up space and reducing volumes in preparation for in-depth analysis


Develop an understanding of the business context of your dark data

Use Advanced Content Analytics to Extract Meaning

Clustering• Visualize common content patterns in your dark data• Identify groupings for policy development and application

Trained Categories• Leverage your investment in records management• Use known documents to train categories• Identify category matches in your dark data

Eduction• Identify sensitive information in documents e.g. Credit Card Numbers, Social Security Numbers etc.• Secure documents appropriately


Dealing with the categorized information

Policy Application and Execution

Defensible Destruction• Seek approval from identified owners• Provide audit reports on all stages of your decision process

(identification, sampling, tagging etc.)• Maintain audit logs of deletion process

Migration• Migrate to records management, archives or secondary storage

system according to the business value of the information• Migrate metadata and tags if possible • Maintain audit logs of the migration process

In-Place Management• Use a policy engine to enforce future application of the policy you

have selected now• Quality of information and policy application depends on the

capabilities of the system under management• Maintain audit trails of any actions applied


Keeping everything under control

Ongoing Information Governance

Identify how business processes use and generate information• Adjust ongoing information governance policies according to business and

regulatory requirements• Cover both structured and unstructured information

Keep indexes up to date• Leverage for enterprise searching• Keeps data accessible• Allows ongoing categorisation

Automate as much as possible:• Implement performance and compliance archives• Apply ongoing policies based on indexed categories • Auto-capture records• Manage records in-place


Summary

Leverage the investment in Legacy Data Cleanup for ongoing

Information Governance – Don’t let new information get dark!

Information Governance is an ongoing business


Design structure

Stages of Legacy Data Cleanup

1. Identify and Index

2. Analyse 4. Reduce3. Organize 5. Manage/Migrat

e


Getting started

Identify and Index

Software Deployment• Legacy data cleanup volume licensing• Packaged deployment tool now includes Exchange Connector

Repository registration• Repositories are automatically registered based on IDOL databases• File system and SharePoint repositories can be added and configured for indexing

through ControlPoint UI

Indexing• Different levels of indexing are available• Preconfigured eduction task to find personally identifiable information


Identify and Index


Understanding what is there

Analyse

Visualise the statistics compiled by the analytics as a summary report:• Based on file level metadata and hashes:

– Redundant data: statistics on duplicates– Trivial data: based on file types with no content value (e.g *.exe, system files,

thumbnails etc)– Obsolete data: based on date created, modified, accessed

• Based on advanced content analysis:– Category matches– Personally identifiable information (eduction)

Provide detail graphs and linked document grid for:• Cluster visualisation• Duplicates• Analytical data: By size, type, age, user, categories and custom fields• Applied Tags


Analyse


Analyse


Preparing for policy assignment

Organise

Supporting a structured approach consisting of:• Filtering – to combine main selection criteria with other facets• Sampling – to randomly select fixed size or percentage samples• Document inspection – to view properties, advanced properties or native content• Tagging – to group the documents into actionable categories

Assign policies to tagged categories• Use the standard ControlPoint policy phases, including workflow policies that can route

the data through an approval process


Organise


Organise


Cut down on the data volume

Reduce

Provide defensible deletion• Apply deletion and deduplication based on ControlPoint policy• Standard ControlPoint policy review step is available• Use ControlPoint workflow policy actions for extended review processes (requires APA)


Taking the step from cleanup to ongoing information governance

Manage/Migrate

Merge legacy data into current information governance policies• Declare move, secure move, hold, manage in place through ControlPoint policies• Migration of organised legacy data between repositories(e.g. File System to HP Records

Manager SharePoint, Exchange to ACA etc.)• Declare legacy data as records (in place or migration)• Ongoing application of ControlPoint policies to new data identified based on trained

categories


Summary

Legacy Data Cleanup

Legacy DataInactive

UnknownOrphaned

Dealing with the past Continuous Information Governance


Summary

Legacy Data Cleanup

Legacy DataInactive

UnknownOrphaned

Dealing with the past Continuous Information Governance

ControlPoint 4.1

Case study: auto records classification

Products: HP TRIM, ControlPoint (5000 users)

Defense Contract Management Agency (DCMA)

Use case: • Working with suppliers to align purchasing of products and services

with policies and requirements. • Seeking to improve management of SharePoint and file shares; also

to improve legal hold programs

Why we won:• Made classification and records management transparent to users• File share and SharePoint content under strict policy management• ControlPoint and ALH for identifying content for legal hold• APA to improve program workflows

Case study: SharePoint governanceProduct: ControlPointUse case: • Uses ControlPoint to lower operating costs of

SharePoint by:• Migrating content from SharePoint farms to

Autonomy cloud storage• Using ControlPoint to identify and move content

from operations to Autonomy cloud

Benefit:• Achieved significant cost reductions in SharePoint TCO

Product: Control Point 4.0

Use case: • Selected Control Point 4.0 for dark data identification • Initial project is to identify, analyse and transform 100 terabytes

- full project to include 2 petabytes• Project sold in conjunction with ES Application

Optimisation program• Will identify unstructured content with high business

context and records value• Goal is information footprint reduction AND identification

of unknown high-risk content

Case study: Legacy Data Cleanup

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

HPApplication Information OptimizerMark Gower31st October 2013


Application Information Optimizer

Addressing the problems of dark structured data

Exploding Data Growth

Lack of Policy Management

Performance Issues

Governance and eDiscovery


Classifying Application Data

• Define a broad classification of the data in the app database• Eg. Trading Data & Audit information

• Identify structure within classifications• Analyse by date quantity of data for each classification• Propose data management policy

Audit Data76%

Oracle System

2%

Other1%

Trading Data22%

Data by Classification


With Application Information Optimizer

Managing Applications

• Relocate Data to a second datastore• Relocate Data to external archive / governance platform• Improve Performance & Reduce Risk

Primary Datastore

Reporting/rendering tools

Retired records

HP RM8 Defensible Disposal

Business Application

Inactive Data


Enterprise Structured Data Management

HP IT’s ongoing database archiving strategypowered by HP AIO since November 2009

•Query time reduced by 89%•Storage reduced by 48%•Full backup window reduced by 37%

By July 2012 - 122 source databases, 836 db-db jobs, 57+billion rows archived


What do I do with the data I find?

AutonomyControlPoint

Structured Data Repositories

Unstructured Data Repositories

Know what information you have

Know where the

information is located

Know your information is

secure

HP Application Information Optimizer

AutonomyConsolidated

Archive

HP Records Manager

Additional Active

Repositories

• Analyse• Classify• Take action

• Archive• Protect• Dispose

Know the information is managed appropriatel

y


HP AutonomyRecords Management StrategyCraig Adams, EMEA Information Governance Manager31st October 2013


HP Records Management - Strategy

HP Autonomy Records Manager

Microsoft SharePoint


HP Records Manager 8.0


• Next generation, scalable electronic document and records management solution

• Designed to meet the needs of Government and regulated industry• Combining the best of HP TRIM, ARM & Meridio• Standards Compliant platform• A cornerstone of HP’s Information Governance framework

Transforming Records Management for Information Governance


HP Records Manager use cases supporting information governance

Transforming records management

Interactive Document Management

Custom Solutions

HP Records Manager

SharePoint Governance

ControlPoint Auto-Declare In-Place

Management

Unstructured

Repositories

SAP ArchiveLi

nk

AIO Structured Records

Physical Records

COM/.NET SDK Services API


HP’s Own Records Management Journey


HP RM Programme - ERMS Purpose

ERMS will facilitate and enhance HP’s compliance activities, because we• Are required by law to manage records

• Need to find the right records at the right time

• Need to produce records swiftly and cost-effectively for tax and other government audits, regulatory compliance, and legal proceedings

• Need to delete records we do not need to keep to reduce storage and litigation production costs

• Need to meet our global needs across:• 330,000 employees

• 16,000 Records Management / Co-Ordinators

• 173 countries

• Multiple languages & jurisdictions

• Stakeholders in Tax, Audit, Legal, IT


HP RM Programme - ERMS Scope


FY08 Q4 –FY09 Q1 FY09-Q2

Phase I FY09 Q3 - FY10 Q1

Phase II FY10 Q3 -

FY13

Phase III FY11 Q4 -

FY14 Beyond Phase

IIIFY14+

HP RM Programme - ERMS Schedule


HP RM Programme - FY13 Registered Records

Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct0

5,000,000

10,000,000

15,000,000

20,000,000

25,000,000

30,000,000

35,000,000

40,000,000

45,000,000

50,000,000

38,203,252

23,373,470

33,817,366

24,946,47524,370,349

43,515,280

26,573,487

Details from registered records report.


Hosting your data in the secure HP Cloud

Bharat MistryCTO Office – Strategy Group,HP Enterprise Security ServicesOctober 31, 2013


53

HP Converged Cloud

INFORMATIONAPPLICATIONS

INFRASTRUCTURE

Public CloudTraditional

Private Cloud Managed Cloud

Choice • Open…standards-based across all delivery models• Heterogeneous…hypervisors, development, infrastructure• Extensible…partner ecosystem

Consistency• Common architecture…across all delivery models• Portability…for flexibility & optimization• Consumption experience…one simple model

Confidence • Security…across info, apps, infrastructure, delivery models• Management …end to end• Automation…for cloud based architectures & processes

Hybrid delivery based on common architecture across traditional & all cloud models


54

A major concern for the CISOSecurity Risks with the Cloud

Uncertain ability to audit provider

Uncertain provider regulatory compliance

Uncertain ability to recover data

Inadequate training and IT auditing

Uncertain ability to enforce provider site security policies

-5% 0% 5% 10% 15% 20% 25% 30% 35%Source: 2012 Global State of Information Security Survey, PricewaterhouseCoopers, CIO magazine, CSO magazine, September 2011


55

Cloud Security Challenges

PaaS

IaaS

SaaS

Application

User Management

Consumer responsible

Provider responsible

Cloud Service Provider Perspective

Application

Application Platform

O/S

Network

Physical

O/Simage

ApplicationPlatform

Application

• Increasing security responsibilities at the information, application & user layers

• Abstraction and automation of O/S, Network, and Physical layers

• Examples:

• IaaS: HP, Amazon EC2

• PaaS: Heroku, Google App. Engine

• SaaS: SalesForce.com

IaaS PaaS SaaS


HP Cloud File Management


Secure access, caching, and storage capabilities for your business files

Cloud File Management

File sync and store

Online workspaces

Storage controllers

Data encryption

Device security

SharePoint integration

File, sync, and share anytime, anywhere, from any device

Base

OptionalAdditional user storage


Secure access, caching, and storage capabilities for your business files.


File sync and store

Online workspaces

Storage controllers

Data encryption

Device security


Additional user storage

Store files in the secure HP cloud and sync automatically to your device when connected.

Setup private and public folders in the cloud with password protection.

Secure data files with 256-bit encryption both in transit and at rest in the cloud.

Segregated data storage within the HP cloud and on-premise to meet data governance rules.

Allows users to access SharePoint files from within the Cloud File Management environment with read/write document access.

Order and configure additional storage in 100 GB blocks, shard across all users.

Set up multiple security configurations with flexible access or more secure modes to provide options.


Flexible new access across multiple devices


Limited mobile access

ECS–Mobility

Email Database Server

Mobile Desktop

Files are shared/stored across systems and deviceswith restricted access.

Files secured in the cloud and synced across deviceswith flexible access.

Notebook

SharePoint


Store files in the secure HP cloud and sync automatically

File sync and store

• Users can store files in a secure cloud

• Synchronize data automatically to any device for offline viewing

• Includes 5GB of user storage in the HP cloud

• Changed files are automatically updated to the cloud and accessible by any PC or mobile device

• Files cached on the device are also available for offline viewing


• Provides private and public folders in the cloud with password protection

• Users can set access control rights for uploads and downloads

• Reduces email storage requirements by sending secure links to files instead of actual attachments.

• Request a file upload from co-workers or third parties by providing a secure link

Online workspaces

Share files securely in the cloud


• In-transit encryption for file transfers using 256 bit SSL for security

• Cloud storage encryption for files uploaded to Cloud File Management servers • File metadata has a unique encryption key,

stored separately from the file itself, and encrypted when uploaded

• Downloaded files are decrypted before their contents are sent to your browser

• On-device encryption is available through the Management Essentials package

Data encryption

Secure data both in transit and at rest in the cloud


• Provides flexible options for security administration

• Configure a standard mode with flexible access to files

• Configure a more secure mode with options like pin locks, file self-destruct, and disabling offline access

• Custom configurations are also available

Device security

Multiple security configurations for file accessConfiguration Management


Users get access to SharePoint files from within Cloud File Management with read and write access to documents stored there.


Additional integration opportunity


• Storage controllers allows administrators to manage on-premise storage centers alongside cloud-based storage

• Store select files within the HP converged cloud and/or on-premise to meet data governance rules

• Integrate storage controllers with any CIFS-based network share from any storage vendor

• Enable mobile access to file shares and document management systems through a single interface

Storage controllers

Create and manage local storage centers


ControlPoint 4.1Slide Demonstration


67


68


69


70


71


72


73


74


75


76



Slide Demonstration


78


79


80


81


82


83


84


85


86


87


88


89


90


91


Date post:	15-Jan-2015
Category:	Business
Upload:	oyster-ims
View:	522 times
Download:	1 times