Db trends final

transcript

Trends and Issues Impacting Database Management Systems

Circa 2004

Craig S. MullinsDirector, Technology Planning/BMC Software, Inc.

http://www.craigsmullins.com

General Industry Trends• Complexity

• Heterogeneity• Rapid change• Consolidation

• Lack of Resources• Skilled Technicians

– A lot of PFCSKs and IROCs

• Time– Already overworked

• Budget

• Speed-to-Market• Buy versus Build• Follow the Leader

DBMS Industry Trends

• Rapid DBMS Versioning• Enabling for the Internet• Online, real-time … • Java and .Net• XML• Multimedia• Procedural logic• ERP and CRM• Mo’ data! Mo’ data!• Integration

New DBMS Versions

• Analysis of New Features– Check all Requirements

• Hardware and Software

• Planning the Upgrade– Impact to system, applications– Scheduling

• Fallback Strategy

• Migration Verification

DBMS Subsumes Functionality

• XML

• ETL and Propagation

• OLAP

• Multimedia

• Objects

• Logic/Code – triggers, UDFs, stored procedures

The Internet

• From DBA to eDBAInternet-Age DBA Skills

Source: Gartner Group

It’s About Change...

Increasingly Complex Enterprise Infrastructure

A Virtual, Extended Infrastructure

Internet Infrastructure Weaknesses

Problem Symptom Effect

UnreliableSporadic crashesfor no apparent reason

Unplannedoutages

ComplexOperators do notunderstand how to resolve problems

Simple problemsresult in longoutages

Fragile IT mgrs. Must debug innocuous changes

Long debuggingcycles for newreleases

Vulnerable Viruses and bugsattack all systemsat once

Systems must berolled back toclean backups

Source: Forrester Research

The Cost of an Outage

Unplanned Unplanned outages - outages - driven by driven by problemsproblems

Application failureApplication failure Element FailureElement Failure PerformancePerformance Capacity limitsCapacity limits Transaction Transaction

BackoutsBackouts

Planned outages -Planned outages -driven by changedriven by change Management Management MaintenanceMaintenance MigrationsMigrations Version Version

ManagementManagement PropagationPropagation

Planned and Unplanned OutagesPlanned and Unplanned Outages Staff Shortages - 18 hr. workdaysStaff Shortages - 18 hr. workdays Short Implementation TimesShort Implementation Times

Executive Demands for Web PresenceExecutive Demands for Web Presence Management Tool ScalabilityManagement Tool Scalability

Unable to Keep Up With Own ScriptsUnable to Keep Up With Own Scripts Unpredictable Web Load/GrowthUnpredictable Web Load/Growth

Unknown Success of SiteUnknown Success of Site Customized EnvironmentsCustomized Environments

Need Flexibility of ManagementNeed Flexibility of Management

E-Business Applications Fail to Deliver Service Because...

70% of Outages

30% of Outages

Where is Application Downtime?

• Planned vs. Unplanned– Planned outages represents

70% of application downtime.

– Just 30% is due to unplanned outages and 50% of the unplanned downtime is due to problems during planned downtime.

Impact on eDBAs

• Downtime is not tolerated– Downtime was never “good” but it was tolerated – No longer, though as we move from availability to

e-vailability with intelligent techniques & solutions

• Avoiding downtime with automated tools– “On the fly” operational tuning

• ALTER SYSTEM - Oracle9i• SET SYSPARM - DB2 V7

– Monitor performance across multiple platforms– Redundant systems – HACMP, RAID, etc.

Action

Example: Keeping Your Systems Up!

Monitor

ARCHIVE FAILED?ADD LOGNOTIFY AML

LOGSLOGS

Change ZPARM

ActionAction

Monitor

SYSPLEX

EDM POOL FAILED?

Undo/Redo SQL

Example: Recover with no Down Time!

LOGSLOGS

Log Analysis

DB2CNo need totake the dataoff-line to runSQL against it!

Online and Real-time

• The need for more and more availability drives online and real-time maintenance– The DBMS begins to allow for more changes to be

made during normal operations

– The DBMS begins to gather statistics and performance metrics during normal operations

– ISVs deliver more online, real-time features and functionality that the DBMS does not yet deliver

– Less manual-intervention required

Database Design and Web Time

• When the Web is involved everything becomes “rush-rush” - do it now!

• Don’t let database design suffer - take your time and do it right.

• Apps are temporary but data is forever!– If you do not believe this, then consider: “How

often has your organization re-entered or re-keyed data into a new database when the data already exists elsewhere?”

Prepare for Global Scope

Who is accessing the database? Internal and External users Local, National, and

International Now versus then

24 x 7 x 365¼

Impact on the DBA

database schema

SQLapplication code

Java operating system

network software

bridge/router/hub

HTTPnetwork cabling

hardware

CGIconnectionZPARMs

Where is the performance problem? Most experts agree that 75% to 80% of

performance problems in relational applications is caused by poor SQL or application code, but on the web . . .

HTMLgateway

Java applet

DB2Connect

init.ora

SQL*Net

Java’s Popularity is Skyrocketing

1997 1998 1999 2000 2001 2002

Java Software Market ($U.S. Millions)

Source: IDC

Why Java?

• Portability

• Make web pages active and dynamic

• Like C/C++ with a smaller footprint

• Lower cost of change (DLLs)

• Download changes from the web

• But– slower - interpreted, not compiled

Java and Databases:Two Methods

• JDBC– Enables Dynamic SQL from Java– Uses API (CLI)

• SQLJ– Enables Static SQL for Java– Uses embedded SQL

Java Alphabet Soup

• J2EE - Java 2 Enterprise Edition– Standard services and specifications for making

Java highly available, secure, reliable, and scalable for enterprise adoption

• EJB - Enterprise Java Beans– Components that contain the business logic for

a J2EE application

Impact of Java on DBA

• Application tuning– Must understand Java

• To provide guidance during design reviews– Is the problem in the SQL or the application

• How can you tune the application if you do not understand the language (Java)?

– Optimizing SQL is not enough since it may be embedded in poor application code

– Must understand the SQL techniques used• JDBC and SQLJ

Microsoft .NET

• ... is a set of Microsoft technologies for connecting people, systems, and devices

• ... allows Internet Servers to expose functions to any client named as .NET web services

• … enables software to be delivered as a service over the web

• … is designed to let many different services and systems interact

Java versus .Net

• ...designed to enable applications to be deployed on any platform as long as they are written in Java

• …designed to enable development in multiple languages as long as the application is deployed on Windows

The Rush to XML

What is XML?

• XML stands for eXtensible Markup Language. – Like HTML, XML is based on SGML

(Standard Generalized Markup Language)

– HTML uses tags to describe the appearance of data on a page, whereas XML uses tags to describe the data itself, instead of its appearance.

– Allows documents to be self-describing, through the specification of tag sets and the structural relationships between the tags.

XML is a Meta Language

• XML is actually a meta language - a language used to define other languages.

– These languages are collected in dictionaries called Document Type Definitions (DTDs).

– The DTD stores definitions of tags for specific industries or fields of knowledge. So, the meaning of a tag must be defined in a DTD before it can be used.

– The DTD for an XML document can be either part of the document or stored in an external file.

Sample XML DTD<!DOCTYPE CUSTOMER [<!ELEMENT CUST (first_name, middle_initial, last_name, company_name, street_address, city, state, zip_code, country*)><!ELEMENT first_name (#PCDATA)><!ELEMENT middle_initial (#PCDATA)><!ELEMENT last_name (#PCDATA)><!ELEMENT company_name (#PCDATA)><!ELEMENT street_address (#PCDATA)><!ELEMENT city (#PCDATA)><!ELEMENT state (#PCDATA)><!ELEMENT zip_code (#PCDATA)><!ELEMENT country (#PCDATA)>]

Sample XML

<CUSTOMER><first_name>Craig</first_name><middle_initial>S.</middle_initial><last_name>Mullins</last_name><company_name>BMC Software, Inc.</company_name><street_address>2101 CityWest Blvd.</street_address><city>Houston</city><state>TX</state><zip_code>77042</zip_code><country>U.S.A.</country></CUSTOMER>

Why is XML Important?

• XML is used for exchanging and sharing data – Inter- and intra-organization

• XML may be used to define database structures – and vice versa

DOCTYPEXML definition lorem ipsum . . . Generate DDL

PhysicalDatabase

Generate XML

XML to DBMS Add XML functionality to the DBMS

• Extender capabilities similar to IBM DB2 video, image, audio, and other multimedia data types

• Combine UDT, UDFs, and triggers for functionality

XML document stored in a column -or- XML components stored as parts of multiple columns

in multiple tables Formulate XML documents from existing tables Search XML documents text and sections XQuery capabilities

XML and Data Management

• Middleware: Software called from your application to transfer data between XML documents and databases. For data-centric applications.

• XML-Enabled Databases: Databases with extensions for transferring data between XML documents and themselves. Primarily for data-centric applications.

• Native XML Databases: Databases that store XML in "native" form. For data- and document-centric applications.

• XML Servers: XML-aware J2EE servers, Web application servers, integration engines, and custom servers. For data- and document-centric applications.

• Wrappers: Software that treats XML documents as a source of relational data. These products typically query XML documents using SQL. For data-centric applications.

• Content Management Systems: Applications built on top of native XML databases and/or the file system for content/document management. For document-centric applications.

• XML Query Engines: Standalone engines that can query XML documents. For data- and document-centric applications.

• XML Data Binding: Products that can bind XML documents to objects. Some of these can also store/retrieve objects from the database. For data-centric applications.

Source: XML Database Products by Ronald Bourret

http://www.rpbourret.com/xml/XMLDatabaseProds.htm

Impact of XML on DBA• Database definition

– Perhaps a new way to create databases

• DTDs to build a database schema

• Create DTD from a database schema

• Interface to data modeling tools

• Data access– Database extenders

– XML defines the data it contains

• New XML DBMS products?– Tamino (Software AG)

– Xperanto (IBM)

• Over-enthusiasm!

Logic and the DBMS

Triggers vs. Functions vs. Procs

CodeCode

UPDATE

CodeCode

INSERT

DELETE

UDFs function( ) if this then that else do this stuff return x end

TriggersTriggers

StoredStoredProceduresProcedures

Procedural DBA Duties

DEVELOP

StoredProcedures

Triggers

Functions

ExternalLibraries

PerformanceMonitors

Admin.Process

DesignReview

Role of the Procedural DBA

DBCO Administration

(trigger firing order, proc set)DesignReviewsEXPLAIN

Analysis

DebuggingSQL

Coding

Complex

Queries

Tuning SQL

On Callfor DBCOAbends

DBCO Implementation(COMMIT in proc, write or guide)

EnsuringReuse

Schema

Resolution

DBCO = Database Code Object

Non-Traditional Data

“Universal” Data

ExistingDatabases

Unstructured Structured

Complex

Simple

Seismic Data

Images

VideoGraphics

Compound Docs

Design Data

Spatial Data

Temporal Data

Multimedia = Big Databases

Object Typical Size

HD TV 200 MB/second

Feature-length,high-resolution movie 5-6 GB

High-resolution video 3 GB/hour

Feature-length movie 2 GB

Video 1 GB/hour

Radiologic image 40-60 MB

Color image 20-40 MB

Large image 200 KB-3 MB

Text 30-40 KB/page

Check image 45 KB

Integration and Federation

• DBMSs are adapting to “handle” more types of non-traditional data– Spreadsheets– Word documents– Presentations

• How?– Integrate the data into the DBMS– Federate and manage the data “where it lies”

DBMS to Manage All Kinds of Data

Federated DBMS

• A federated approach allows the DBMS to manage data where it exists

Spreadsheet

DBMS to Manage All Kinds of Data

Or integration…

• An alternate approach “sucks” the non-relational data into the DBMS to be managed

Spreadsheet

Autonomic ComputingAutonomic Computing

SelfManaging

VirtualReorg

DatabaseWizards

Real TimeStatistics

Correct andNotify

IntelligentAutomation

Data Keeps Growing

Phenomenal Data Growth• “Global 2000 companies double the amount of data they own

every year, while the average dot-com’s data doubles every 90 days.”

– Mike Ruettgers, CEO of EMC Corp., Oracle Open World 2001

• “Inside IBM we talk about 10 times more connected people, 100 time more network speed, a 1000 times more devices and a million times more data.”

– Lou Gerstner, CEO of IBM Corp., eBusiness Expo 2000

• A recent research note from Giga Information Group estimates that there are about 201,000TB, or about 197 petabytes, on the planet. Of course, this is just an estimate that Giga deems to be accurate within an order of magnitude (that is, within a factor of 10).

• McKinsey & Company reports a CAGR of 76% for data storage.

Relational Database Size

Database Size Issues• Technology enables larger databases

• Web, multimedia, data warehousing, and data mining drive up database size

• Disk drives increase in capacity butspeed of access does not keep up withcapacity increases

• Cost of storage decreasing; sowhy not store more data? But...– What data do users need to store?

– How long must it be maintained?

– What are they willingto pay?

0010 001010100 101011101011 101011010010101 0100100110101011100001111001010100100101000100101

The Database Environment

• A lot of choices!– Vendor, platform, and architecture of DBMS

MVS, OS/390, z/OSWindows NT / 2000 / XPUnix

AIXSun SolarisHP-UXLinux

others?Others (VSE, VMS, MPE, OS/400, etc.)

Desktop OSWindows 98 / ME / XPLinuxMac?

Enterprise- Parallel EditionDepartmentalPersonalMobile (PDA)

Adabas, Teradata, PostgreSQL, Supra, Compaq Non-Stop SQL, Ingres, IMS, IDMS, Datacom, Teradata, others...

Heterogeneity Plus!Heterogeneity Plus!

• The DBA’s knowledge must span the entire enterprise– Operating Systems

– Networking Protocols

– Programming Languages

– Business Objectives

– And so on…

• The DBA’s knowledge must span the entire enterprise– Operating Systems

– Networking Protocols

– Programming Languages

– Business Objectives

– And so on…

Impact on the DBA• Unrealistic Expectations

– Impossible to master everything

• Education is the first thing cut!

– Impossible to specialize in a heterogeneous shop

• Reactive mode is encouraged

– Even though proactive mode is optimal

– Who looks for more problems when they don’t have enough time in the day to solve the problems of those complaining the loudest.

– DBA uses YBWJ method

Worldwide Spending for ERP Packaged Software

1995 1996 1997 1998 1999 2000 2001 2002

Manufacturing

Distribution

Human Resources

Financial

Source: IDC

The Computing “Platform” Evolution

The DBMS

Network (LAN/WAN)

Communications

The Operating System

Hardware

“Base” Applications

MVS, UNIX, WIN (NT,95)

Proprietary (IBM+) to Open (Wintel, HP/Intel)

SNA, TCP/IP, NFS

TCP/IP, Novell, WinNT

DB2, Oracle, SQL Server

SAP, Oracle, Baan, PeopleSoft

“Customer Service Apps”

60’s

This is the current battleground,NOT the DBMS

Simple› Numbers› Characters› Date› Time Complex › BLOBs› CLOBsVaried› Content

Management

Business› Data in

Context

Integration› Systems› PeopleSystems› Technology to

store and reportPeople› Identify› Capture› Maintain

Integration› Multiple

pointsof knowledge

Understanding› Knowledge

applied

The Data Continuum

Information

Knowledge

Wisdom

ContextExperience Perception

The Data Continuum

Information

Knowledge

Wisdom

DBMSFilesSpreadsheetsGraphicsFoldersEtc.

Data ModelsData DictionaryRepository

Technology +People

Knowledge Management

Knowledge +Application Systems

The Data Continuum

Information

Knowledge

Wisdom

ContextExperience Perception

An Example:

Other DBMS Market Trends

• ODBMS – no longer a threat to become “mainstream” but…

• “Post-Relational” DBMS?– XML DBMS– The Associative Model – Multivalue DBMS

• In Memory DBMS

Enter the DBA

• The job of database The job of database administration is getting administration is getting increasingly more difficult increasingly more difficult as database technology as database technology rapidly advances adding rapidly advances adding new functionality, more new functionality, more options, and more options, and more complex and comp-complex and comp-licated capabilities... licated capabilities...

The DBA is a “Jack of all Trades”

database schema

application code

Javaoperating system

network software

bridge/router/hub

network cabling

hardware

connection ZPARMs

HTMLgateway

Java appletDB2

Connect

OracleInformix

SQL Server

WindowsOS/390

SQL*Net

TCP/IP

V$ Tables

JCLCICS

• Intelligent automation of DBA tasks– because no one has all the skilled resources

they need– frees up more DBA time

So What is Needed?

AutomatedIntelligent“Scrubbing Bubbles”

They work hard so you don’t have tooooo…

A DBA Control Panel

DBA-focused on DBA tasks, not focused on systems managementtasks like framework products… but integrated with systemsmanagement functionality where it makes sense.

Contact Information

Craig S. MullinsTechnology Planning

Craig_Mullins@BMC.com

http://www.craigsmullins.com

http://www.bmc.com

http://www.craigsmullins.com/cm-book.htm

http://www.craigsmullins.com/dba_book.htm