Version Control meets Database Control

transcript

Webinar: Version Control Meets Database Control

DBmaestro Introductions

Presenters

Kyle Hailey @kylehhailey• Technical Evangelist at Delphix• Oracle ACE, member of the OakTable Network

Uri Margalit @UriMargalit • Director, Product Management• Presenter at world-wide conferences: ODTUG,

ilOUG, etc…

Before we start

• You will be on mute for the duration of the event

• We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first)

• There will be a Q+A session at the end but please feel free to type your questions in the Questions box in the Control Panel in advance

• A recording of the full webinar will be put up online

About Delphix

• Founded in 2008, launched in 2010• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))• Based in Silicon Valley, Global Operations• 10% of Fortune 500

About DBmaestro

• Founded in 2008• Headed by Yariv Tabac and Yaniv Yehuda• Headquartered in Israel

Version Control meets Data Control

Kyle Hailey & Uri Margalit

The Business Need

Copyright@2008, Juniper Networks, Inc.

80% of unplanned downtime is due to Change

50% More than

of this, half is due to human errors

40% of changes FAIL

The Technology Need

Dealing with Risk

Smaller and more focused changes are easier to manage (Agile…) Automation of repeating tasks lowers risk of (human) error Development and Operations should work in synergy (DevOps)

Source Control – Standard De Facto

Common version control tools:GitHubSVNPerforceTFSRTCVSS

The Database Challenge

• The Database is a crucial part of the Application— Schema Structure— PL/SQL Code— Lookup Content

• The Database is a central resource• Business Data Must be preserved

• The Database is not native to traditional version control

• Objects are not files on a file system• How can we manage Content?• How can we branch a Database?

Tradeoff: Speed, Quality, Cost

Good, Cheap, Fast : choose two

Good Cheap

What We’ve Seen

1. Inefficient QA: Higher costs of QA2. QA Delays : Greater re-work of code3. Sharing DB Environments : Bottlenecks4. Using DB Subsets: More bugs in Prod5. Slow Environment Builds: Delays

1. Inefficient QA: Long Build times

Build Time

QA Test

96% of QA time was building environment$.04/$1.00 actual testing vs. setup

2. QA Delays: bugs found late require more code re-work

Build QA Env QA Build QA Env QA

Sprint 1 Sprint 2 Sprint 3

Bug CodeX

1 2 3 4 5 6 70

10203040506070

Delay in Fixing the bug

Cost To

Correct

Software Engineering Economics – Barry Boehm (1981)

3. Full Copy Shared : Bottlenecks

Frustration Waiting

Old Unrepresentative Data

4. Subsets : cause bugs

Production

4. Subsets : cause bugs

Classic problem is that queries that run fast on subsets hit the wall in production.

Developers are unable to test against all data

The Production ‘Wall’

5. Slow Environment Builds: 3-6 Months to Deliver Data

Management

System Admin

Storage Admin

Developers Submit Request

Disk Capacity?

Approve Request $$ (2 Weeks)

Approve Request $$

(1 Week)

RequestAdditional

Storage?ProvisionCapacity

File SystemConfigured?

Configure LUNS & Build File System

Coordinate Replication w/ Infrastructure

Re-Parameterize & Configure DB

Mount Recovery DB to

Specific PIT

Begin Work

(3 Days)

(2 Days)

(3 Days)

.……1-2 Weeks of Approvals, Delays, and Provisioning……

5. Slow Environment Builds: 3-6 Months to Deliver Data

5. Slow Environment Builds: culture of no

What We’ve Seen

1. Inefficient QA: Higher costs2. QA Delays : Increased re-work3. Sharing DB : Bottlenecks4. Subset DB : Bugs5. Slow Environment Builds: Delays

Which of the following have you run into at your organization?1. Inefficient QA driving up costs2. QA Delays causing increased re-work of code3. Sharing DB causing development bottlenecks4. Subset DB database in development and QA

leading to bugs in production5. Slow Environment Builds causing project delays

60% Projects Over Schedule and Budget

CIO Magazine Survey:

Data is the problem

Solve the data problem.

TODAY.

UNLOCK YOUR DATA

Clone 1 Clone 3Clone 2

99% of blocks are identical

Clone 1 Clone 2 Clone 3

Thin Clone

Virtualization Layer

Virtualization

Three Physical Copies Three Virtual Copies

Install Delphix on x86 hardware

Intel hardware

Allocate Any Storage to Delphix

Allocate StorageAny type

One time backup of source database

Database

Production

Instance

File system

DxFS (Delphix) Compress Data

Database

Production

Instance

Data is compressed typically 1/3

File system

Incremental forever change collection

Database

Production

Instance

File system

Changes

• Collected incrementally forever• Old data purged

Time Window

File system

Typical Architecture

Production

Instance

Development

Instance

Database

File systemFile system

Database

With Delphix

Database

File system

Production

Instance

Database

Development

Instance

Database

Instance

Database

Instance

Three Core Parts

Production

Instance

Time Window

Instance

Virtual Database

1. Source Syncing

2. Storage (DxFS)

3. Self Service

Development

Fast, Fresh, Full

Database

Production

Instance

File system Time Window

Instance

Virtual Database

Database

Production

Instance

Virtual Database

Instance

Virtual Database

Branching to QA

Database

Production

Instance

Virtual Database

Instance

Virtual Database Dev

Self Service

What We’ve Seen With Delphix

1. Efficient QA: Low cost, high utilization2. Quick QA : Fast Bug Fix3. Every Dev gets DB: Parallelized Dev4. Full DB : Less Bugs5. Fast Builds: Fast Dev, Culture of Yes

1. Efficient QA: Lower cost

QA Test

1% of QA time was building environment$.99/$1.00 actual testing vs. setup

Build Time

QA TestBuild

2. QA Immediate: bugs found fast and fixed

Bug CodeX

Build QA Env QA Build QA Env QA

Bug CodeX

3. Private Copies: Parallelize

4. Full Size DB : Eliminate bugs

Production

5. Self Service: Fast, Efficient. Culture of Yes!

Management

System Admin

Storage Admin

Developers Submit Request

Disk Capacity?

Approve Request $$

(1 Week)

RequestAdditional

Storage?ProvisionCapacity

File SystemConfigured?

Configure LUNS & Build File System

Coordinate Replication w/ Infrastructure

Re-Parameterize & Configure DB

Mount Recovery DB to

Specific PIT

Begin Work

(3 Days)

(2 Days)

(3 Days)

.……1-2 Weeks of Approvals, Delays, and Provisioning……

What We’ve Seen With Delphix

1. Efficient QA: Low cost, high utilization2. Quick QA : Fast Bug Fix3. Every Dev gets DB: Parallelized Dev4. Full DB : Less Bugs5. Fast Builds: Fast Dev, Culture of Yes

Challenges of Development & Release to Operation

Organizing the development of changes• Code• Database • Configuration • Metadata• Work Items

Development Test/ Staging/ UAT

Enabling safe migration into production• Moving the right

components• Enabling

Rollback & Recovery

Production

Development Operations

Moving just the right components needed for the release

• Release Approved Items

Agility dictates frequent changes & new tools are needed to streamline the process

ChangeManagement

ReleaseManagement

What is DBmaestro TeamWork

• Database Enforced Change Management+ Database version control+ Plugs into the ALM (change request, tickets &

work items)

+ Database change impact analysis+ Database deployment automation

• DevOps Solution for databases + Deployment, rollback & recovery+ Plugs into release management

The Challenges that DBmaestro Addresses

• Development delays• Silos in development, DBA and operations• Delays in deployment (internally and to operations)• Errors in production

Development Delays

• Different methodologies for the application & database

• Code overrides• Lack of history of changes (who did what, where, when and

• Manual writing of delta scripts• Lack of automation

Silos in Development, DBA and Operations

• No sharing between the team• No visibility• Always looking for errors made by others

Delays in Deployment (Internally and to Operations)

• Deployment automation does not really include the database tier

• Database scripts generated out of the scope of automation

• Lack of confidence in automation

Errors in Production

• Missing changes• Deploying the wrong version of objects• What about the reference data?

Which challenges have you experienced?

1. Development delays

2. Silos in development, DBA and operations

3. Delays in deployment (internally and to operations)

4. Errors in production

• Database version control – Enforced Check Out/In – Labels– Rollback/Undo – Audit trail reports

• Database impact analysis – Utilizes version control repository information – 3 way analysis

• Database deployment automation– API – Baselines – Conflict resolution – Customized business logic

Without DCM - Two isolated Processes

Check-Out Script

Modify Script

Get updated Script from DB

Check-In Script

Compile Scriptin DB

Debug Scriptin DB

Version Control Process Development Process

Development & Version Control Process

With DCM - One Enforced Process

Check-Out Object

Modify Object in DB

Run Applications’

Check-In Object

Safety Net For Automation of Deployment

Source vs. Target

Action

= No Action

Source vs. Baseline

Target vs. Baseline

Action

= = No Action

≠ = Override

= ≠ Ignore

≠ ≠ Merge

You do not have all of the information

With Baselines and 3 way analysis the unknown is now known

Simple Compare & Sync Baseline aware Deployment

Benefits - Development

• Database change repository• Follow SCM best practices (Check-Out/Check-

In) • All changes are documented• Manage who can do what, where, when & why

Benefits - Operations

• Integrated deployment engine• Business level audit• Roles & responsibilities enforcement

Benefits - Management

• Complete visibility into changes in progress• Management reports• No silos

Live Demo

• Clone 2 virtual copies of the Trunk1. Dev1

2. Dev2

• Make changes & merge them into the Trunk: Developer1 modifies Dev1 Developer1 merges changes into the Trunk Developer2 modifies Dev2 Developer2 merges changes into the Trunk

• Rely on enforced changes & automation

Time Window

Instance

Virtual DatabaseDev1

Virtual Database

Developer 1 modify

Developer 2 modify

Merge Dev1 & TrunkMerge to dev2

Dev1Merge to dev1

Merge Dev2 & Trunk

Merge Dev1 & Trunk

Merge Dev2 & Trunk

Merge Dev1 to ForkMerge to dev2

Dev1Merge to dev1

Merge Dev2 to Fork

Merge Dev1 to Fork

Merge Dev2 to Fork

Q&AKyle Hailey @kylehhaileyDelphix: delphix.com

Uri Margalit @UriMargalit DBmaestro: dbmaestro.com

Thanks