How to create a living SQL

Server data catalog

Data Mapping for a New Era of Oversight.

Richard Macaskill

Richard

MacaskillProduct Manager, Redgate Software

20 years Oracle and SQL Server

Jumped from Oracle 7.3 to SQL Server 7 in 2000.

Financial Systems, BI, Line-of-Business, Risk,

Performance.

London Financial Services

BI Dev for Lloyds of London

Hedge fund IT management

Product Manager at Redgate

Formerly SQL Clone, Data Masker, now Data Privacy and Protection

Currently boring everyone within earshot with Compliance and DevOps stories.

“Show me, don’t just tell me!”

Your colleagues can’t just trust you anymore

Statutory responsibility for compliance exists at board level.

Penalties have been designed to be “effective and dissuasive”.

“Show me, don’t just tell me!”

You will have to go beyond the basics

“The records you keep must be in writing. The information must

be documented in a granular and meaningful way.”

https://ico.org.uk

“Show me, don’t just tell me!”

Once peer scrutiny is out of the bag, the change

is permanent

Expect the tech-business conversations to continue long after 25 May.

Data Protection Impact Assessments.

Subject Access Requests.

Get ready for change

Data Breach <> hacking

24% of New York State

breaches in 2016 were the

result of inadvertent

disclosure

(source: https://ag.ny.gov)

Sources of data breaches

47%

28%

25%

Malicious or criminal attack

Human error

System glitch

Source: IBM 2017 Cost of Data

Breach Study

What does the law require?• Statements of the information you collect

and process, and the purpose for

processing (Article 13 of the GDPR).

• Records of consent from data subjects or

relevant holder of parental responsibility

(Articles 7 and 8 of the GDPR).

• Records of processing activities under your

responsibility (Article 30 of the GDPR).

Meet the regulators

Digging deeper

“Each controller … shall maintain a

record of processing activities

under its responsibility.”

• Processing INCLUDES STORAGE

So, just Production then?

Production vs Non-Production environments

Stag

ing

UA

TQ

AD

ev-I

nteg

rati

onD

evel

opm

ent

ProductionDev-Test

Dev

elop

ing,

Tes

ting

, Val

idat

ing,

Sec

uri

ng, H

arde

nin

g

De-

sen

siti

zing

dat

a co

pies

, mon

itor

ing

feed

back

, err

ors,

re

pla

ce P

rodu

ctio

n se

curi

ty f

eatu

res

Production Operations

Apply environment-specific permission sets

Add non-Prod encryption, DDM

Add availability features

Apply environment-specific permission sets

Apply environment-specific permission sets

What’s wrong with static documentation?

Fire and Forget Stale and Ignored

Do you expect your environment to remain

untouched?

Servers to keep the same configuration?

Zero refactoring of database schemas? (no

improvement?)

No changes in response to new exploits?

“Understanding the existing product

consumes roughly 30 percent of the total

maintenance time.”

Facts and Fallacies of Software

Engineering by Robert L. Glass.

Collaborative

improvement

Alerting to change & deviation from

agreed practice

Evergreen record-keeping involves oversight

Change over time

‘Where is it?’ & ‘What is it?’ are now required

It is impossible for organizations to assert that

they are protecting personal data adequately

(per GDPR) unless they can show

• Where the data is held (yes, all the places)

• What type of data it is

• How it is being protected

Data Catalog Challenges

Labels CatalogsDo you label the thing, or the container?

What if the container is empty?

What if the contents have changed?

Who is the audience?

What is on it beyond the basic data? Location?

Orphaned items (and lost labels)?

How can I know I’ve ‘got’

everything?

Have I looked everywhere?

What if there are new instances?

Can I change a vendor schema?

Will my XPs be blown away?

Support contract rules?

Is my schema static?

Is the data the same as it used

to be?

Orphaned records?

Data Catalog Challenges

Discovery 3rd Party Databases Change

DemoSSMS 17.x

Prepare database for secondary use

Replace personal data

Remap permissions

Update internal reference data

PROVISIONING

Sensitive

data

masked

Stay compliant and use production-like data in development

DemoSQL Data Privacy Suite

Microsoft

Confidential

Scan your network for SQL

Server instances.

Search multiple AD domains.

Manually add Azure SQL

Databases.

Generate suggestions across all

databases, all instances.

Persist to extended properties

using the same taxonomy as

Microsoft’s SSMS 17.5

Persist to a central metadata

store (for cases where schema

change is a problem).

Alert to changes, unclassified

columns.

Discover new instances,

databases, schemas.

Report on deviations from

best practice (customizable).

Redgate’s SQL Data Privacy Suite supports:

Discovery Classification Change over time

Building toward a SQL Estate managed for privacy

Ensure every database copy has

been masked appropriately.

Control sanitized database copies

from a central location, deliver

then revoke copies fit for

analytics, test use cases.

Backup and retention schedules

conform to balanced policy,

minimizing storage while

supporting operational stability.

Personal data is protected with

real-time monitoring, maximizing

availability and ability to meet

RTOs

Manage your remediation phase,

introduce then defend best

practice.

Security features are applied and

checked with context-aware oversight.

TDE and AG for Production, DDM in

UAT, static masking for Dev.

red-gate.com/gdpr

@datamacas /in/richard-macaskill

Thank you