RWDG Webinar: Big Data & BI Analytics Require Data Governance

transcript

Non-Invasive Data Governance™ is a trademark of Robert S. Seiner & KIK Consulting #RWDG@RSeiner

Real-World Data GovernanceBig Data & BI Analytics Require Data Governance

Monthly Webinar Series Hosted by DataversityRobert S. Seiner – KIK Consulting / TDAN.com

July 16, 2015 – 2:00 p.m. EST

Robert S. SeinerKIK Consulting & Educational Services, LLC - KIKconsulting.com

The Data Administration Newsletter, LLC - TDAN.com

• Real-World Data Governance – Monthly Webinar Series– August 20 – Data Governance & Data Steward Certification– September 17 – Data Modeling is Data Governance *– October 15 – Govern Metadata: Vocabulary, Dictionaries and Data– November 19 – Agile Data Governance: How to Apply Governance to Agile– December 17 – Data Governance and the Internet of Thing– Third Thursday at 2 pm EST– Register On-Line at DATAVERSITY.net– or TDAN.com or KIKConsulting.com

* with David C. Hay

Big Data & BI Analytics Require Data GovernanceUpcoming Webinars

• The Data Administration Newsletter (TDAN.com):Published Since 1997Partnership with DATAVERSITYRe-Launched July 1, 2015Embracing Social Media – @TDAN_comRSS Data News FeedAuthors & Practitioners for …

Articles, Columns, Features, BlogsWelcome Back & Please Register

Big Data & BI Analytics Require Data GovernanceCurrent Events

• Non-Invasive Data Governance:The Path of Least Resistance and Greatest SuccessISBN 9781935504856 / Technics Publishing – September 2014

• KIKConsulting.comHome of Non-Invasive Data Governance™Refurbished in 2014

• DATAVERSITY EventsData Governance Financial Services 2015 ConferenceWith DEBtech International – September 28-29 – Jersey City, NJConsiderations for Starting or EnhancingFinancial Data Governance Program

Enterprise DATAVERSITY 2015 ConferenceNovember 2-5 – Chicago, ILA Strategic Data Framework Based onGovernance Best Practices

• This webinar will cover:– Existing Governance Applications for BI– Future of Big Data & BI Data– Relationship between Big Data, BI and Governance– Articulating Value– True Intelligence Derived from Governed Data

Big Data & BI Analytics Require Data GovernanceAbstract

• Data Governance is the execution and enforcement of authorityover the management of data and data-related assets.

• Data Stewardship is the formalization of accountability for themanagement of data and data-related assets.

Big Data & BI Analytics Require Data GovernanceDefinitions

• Non-Invasive Data Governance™– The practice of:

• applying formal accountability & behavior • through a non-invasive framework of roles & responsibilities• to existing and / or new processes• to assure that the definition, production & usage of data• improves regulatory compliance, security, privacy & protection.

– Non-Invasive describes how governance is applied to assurenon-threatening management of valuable data assets.

– The goal is to be» Transparent» Supportive» Collaborative.

Big Data & BI Analytics Require Data GovernanceDefinitions

• Big Data– Broad term for data sets so large or complex that traditional data

processing applications are inadequate.

– Challenges include, but are not limited to:• Analysis• Capture• Curation• Search• Sharing

– Data Governance must be applied to all of these challenges.

• Storage• Transfer• Visualization• Information Privacy

Big Data & BI Analytics Require Data GovernanceApplications of Data Governance

Wikipedia

• Business Intelligence– Set of techniques and tools for the transformation of raw data into

meaningful and useful information for business analysis purposes.

– Common functions include, but are not limited to:• Reporting• Online Analytical Processing• Process Mining• Complex Event Processing

– Data Governance must be applied to all of these functions.

• Business Performance Management• Benchmarking• Predictive Analytics• Prescriptive Analytics

Wikipedia

• As companies generate increasingly more content (read big data),they are coming to recognize that [big data] analytics is anonnegotiable.

• Gleaning business insight from [big data] analytics is still in theearly-adopter and well-hyped phase, but companies are beginningto see results emerge in terms of smoother workflows, improvedsearch and better compliance [read governance] practices.

http://searchcontentmanagement.techtarget.com/tip/Content-analytics-becomes-an-enterprise-must-have

• [Big Data] analytics is a business practice that analyzes and derivesinsight from data about the amount of content being created, thenature of that content and how it is used.

• [Big Data] analytics can help companies manage their informationlifecycle management chains by creating relevant cost andconsumption metrics.

• With business metrics in place, companies can identify which digitalcontent [big data] is most valuable and adjust investments instorage and future analytics as needed.

http://searchcontentmanagement.techtarget.com/tip/Content-analytics-becomes-an-enterprise-must-have

• How Much Data is Big?– Bytes (8 bits)– Kilobyte (1000 bytes)– Megabyte (1 000 000 bytes)– Gigabyte (1 000 000 000 bytes)– Terabyte (1 000 000 000 000 bytes)– Petabyte (1 000 000 000 000 000 bytes)– Exabyte (1 000 000 000 000 000 000 bytes)– Zetabyte (1 000 000 000 000 000 000 000 bytes)– 2.7 Zetabytes of data existed in the digital universe as of 2012 – 35 Zetabytes to be created annually by 2020

– Yottabyte (1 000 000 000 000 000 000 000 000 bytes)– Xenottabyte (1 000 000 000 000 000 000 000 000 000 bytes)– Shilentnobyte (1 000 000 000 000 000 000 000 000 000 000 bytes)– Domegemegrottebyte (1 000 000 000 000 000 000 000 000 000 000 000 bytes)

Big Data & BI Analytics Require Data GovernanceMore Data Then We Can Count

• Why does Big Data require Data Governance?– Big Data as an Asset– Formalized Accountability– Following the Rules– Consistency in Big Data Application– Metadata to Improve Value & Understanding– Management of Big Data Definition– Management of Big Data Production– Management of Big Data Usage

Big Data & BI Analytics Require Data GovernanceWhy?

• Why do BI Analytics require Data Governance?– Data Drives Analytics– Formalized Accountability– Following the Rules– Consistency in Analytical Application– Metadata to Improve Value & Understanding– Management of BI Data Definition– Management of BI Data Production– Management of BI Data Usage

Big Data & BI Analytics Require Data GovernanceWhy?

• Metadata Past and Future– Metadata in the past was limited to :

• Names• Definitions• Labels

– Metadata now encompasses• Relationships between the

– Data– People– Processes– Use of Big Data

Big Data & BI Analytics Require Data GovernanceMetadata Lies at the Core

• Metadata Required– Metadata Continues to Evolve

• From a Data Management Asset• To a Business Intelligence / Data Warehousing Asset• To a Data Governance Asset• To a Privacy and Confidentiality Asset• To a Big Data & Analytical Asset

Big Data & BI Analytics Require Data GovernanceMetadata Lies at the Core

• Data Governance required for:– Structured data– Unstructured data– Content– Record– Logs– Social Media

Big Data & BI Analytics Require Data GovernanceGovernance Lies at the Core

• Data Governance required for:– Big Data– Small Data– Data Warehouse Data– Master Data– Metadata– Analytical Data

• Governing Perspectives for Big Data or Analytical Data– Accountability Perspective– Process Perspective– Rules Perspective– Inventory Perspective– Decision Making Perspective

• Governing Perspectives for Big Data or Analytical Data– Accountability Perspective

• Definition• Production• Use• Protection• Understanding• Retention / Elimination

• Governing Perspectives for Big Data or Analytical Data– Process Perspective

• Bill of “Rights”• Ownership• Participation• Input• Output / Outcomes• Throughput

• Governing Perspectives for Big Data or Analytical Data– Rules Perspective

• Business Rules• Compliance Rules• Protection Rules• Standard Rules

• Governing Perspectives for Big Data or Analytical Data– Inventory Perspective

• Quantitative• Qualitative• Ownership• Usage

• Governing Perspectives for Big Data or Analytical Data– Decision Making Perspective

• Who makes the decision• How they make the decision• When they make the decision• Who is impacted by the decision• How they track decisions

• To look at the future … Let’s start by looking at the past …– Business Intelligence Strategies have evolved from …

• Hierarchical Design• Relational Design• Dimensional Design• Enterprise Data Warehouses• Data Marts / Cubes / Distributed Databases• Data Lakes – object-based storage repository that holds data

in its native format until it is needed.• Unstructured Data & Big Data

Big Data & BI Analytics Require Data GovernanceFuture of Big Data & BI

• OK … Now let’s look to the future …– Business Intelligence Strategies have evolved to …

• Data Mining – looking for a needle in the haystack• Big Data Analytics – looking at the whole haystack

• Three Vs of Big Data– Volume– Velocity– Variety– You saw the chart – Growing exponentially– Present state handling of the Three Vs

• What are we going to do with all this data?– We need to be able to make sense of the data.– We need to be able to understand and trust the data.– We need to be able to protect the data.– We need to know where the data comes from.– We need to be able to access the data.– We need to be able to analyze the data.

• To achieve these goals:– We need to be able to make sense of the data.– We need to be able to understand and trust the data.– We need to be able to protect the data.– We need to know where the data comes from.– We need to be able to access the data.– We need to be able to analyze the data.

• What do we need? Data & Process & TechnologyAnd we need all these things to be governed.Some call this Information Governance.

• Value of Data Governance to Big Data– We just spoke about this so let’s reverse it!

• Value of Data Governance to BI Analytical Data– We just spoke about this so let’s reverse it!

» What is the value of Big Data to Data Governance?

» What is the value of BI Analytics to Data Governance?

Big Data & BI Analytics Require Data GovernanceValue of Governance

• What is the value of Big Data to Data Governance?

• What is the value of BI Analytics to Data Governance?

• Organizations need to use these things as their crutch to– justify the investment in …– maximize use of resources when doing …– provide trustable data from … – make better decisions using …

Big Data & BI Analytics Require Data GovernanceValue to Governance

• Bold Statement 1:– Organizations that govern their data effectively get more value

from their Big Data and BI Analytics effort.

• Bold Statement 2:– Organizations that do not govern their data effectively either:

• Do not embark on these activities• Get less value from their data-driven activities

• Deriving Intelligence from Governed Data requires:– Quality– Understanding– Protection– Availability– Accountability

Big Data & BI Analytics Require Data GovernanceTrue Intelligence

• Deriving Intelligence from Governed Data requires:– Quality

• Data Standards• Followed Process• Escalation Paths• Decision Rights• Certification

• Deriving Intelligence from Governed Data requires:– Understanding

• Glossary• Dictionary• Mapping• Validation• Education

• Deriving Intelligence from Governed Data requires:– Protection

• Classification• Security• Auditability• Process / Education• Enforcement / Compliance

• Deriving Intelligence from Governed Data requires:– Availability

• Access• Storage• Timeliness• Completeness

• Deriving Intelligence from Governed Data requires:– Accountability

• Formalization• Stewardship• Communications• Execution – Enforcement of Authority

• Main Points Covered:– Existing Governance Applications for BI– Future of Big Data & BI Data– Relationship between Big Data, BI and Governance– Articulating Value– True Intelligence Derived from Governed Data

Big Data & BI Analytics Require Data GovernanceSummary

• Real-World Data Governance – Monthly Webinar Series– August 20 – Data Governance & Data Steward Certification

Real-World Data GovernanceUpcoming Webinar

• Robert S. SeinerKIK Consulting & Educational Services – KIKconsulting.comThe Data Administration Newsletter – TDAN.comPost Office Box 112571, Upper St. Clair, Pennsylvania 15241412.220.9643, 412.220.9644 (Fax)rseiner@kikconsulting.comrseiner@tdan.com@RSeiner#RWDG

Real-World Data GovernanceContact Information