Itec back on the ra cv2(smaller)

Information Technology Exchange Center • Dan Brint 1

Monday, December 12, 2011

http://about.me/DanBrintMonday, December 12, 2011

http://about.me/DanBrint

http://about.me/DanBrint

SUNYNet

SUNY OIT(Office of Information Technology)

3


Back on the RAC - Agenda

• 3 specific case studies regarding Oracle database management and virtualization at ITEC

• RAC001 - our first 11gR2 RDBMS cluster (with speeds and feeds)

• DBA 2.0

4


Virtualization / ConsolidationMonday, December 12, 2011

• 52 databases

• 3 largish servers

• Virtualized via IBM lPar technology

• Schema based consolidation with ≤ 3 campuses per DB

• 2 Oracle code trees per server

• 1+X application code trees per server (Aleph 500 by ExLibris)

• Advantage - Campus demand lifecycle

Seneca

8 CPU40 GB RAM

1 TB Disk

18 DB (p&d)

19 Campuses

Saranac

10 CPU50 GB RAM

2 TB Disk

18 DB (p&d)

19 Campuses

Delaware

8 CPU37 GB RAM

2 TB Disk

16 DB (p&d)

18 Campuses

Union

1 CPU10 GB RAM

3 TB Disk

1 DB56

Campuses

Mohawk

4 CPU5 GB RAM<1 TB Disk

6 DB 6

Campuses

56 8.4 M 303 K 1.3 M

2010

Library

6


search.sunyconnect.suny.edu

• Replaces a dedicated database, server and somewhat specialized vendor software implementation (i.e. the Union Catalog)

• Provides u-wide search capabilities

+

+

=

10,000 / month

7


SUNYConnect (Library)• Utilizes schema based

consolidation

• Multitenant friendly application

• ~ 18 databases per server

• 2 Oracle code trees per server

• A single database failure affects < 4 campuses

• But a single server failure affects 19 campuses

• Union catalog was a read only uber consolidation

• Replaced Union with an AWS (amazon web services) based search service

• Draws upon existing catalog services

• Little bit of custom code

• ... and yes it was impacted by the AWS outage in April 2011. However, each library specific search remained operational and a workaround was implemented prior to full AWS service restoration

8


Analysis

TrendsProjections

Information

Campus ERP

Insight

Campus ERP

Decisions

SUNYGoodness

Institution

majors

demographics

curriculum

courses

grades enrollment

programs

budget

SUNY BI Cloud by D. Brint (2010)

Knowledge

ITEC(Hosted ODS,

DataCenter BI, Online Learning)

OAS(Financial, Library ?)

RF(???)

Campus(ODS)

Business Intelligence

9


Sicas BI Cloud

MultitenantODS via VPD

Database

OS: OEL Linux 5.x 64bitCPU: 6RAM: 32 GBStorage: 7300 GB

Admin Server

OS: Windows Server 2003 32bitCPU: 1RAM: 4 GBStorage: 80gb

3 Node WebLogicCluster

Windows AdminToolset

MultitenantODS via VPD

Database

OS: OEL Linux 5.x 64bitCPU: 6RAM: 32 GBStorage: 7300 GB

Admin Server

OS: Windows Server 2003 32bitCPU: 1RAM: 4GBStorage: 80gb

2 Node WebLogicCluster

Windows AdminToolset

Primary Site DR Site

Middleware

OS: OEL Linux 5.x 64bitCPU: 2RAM: 8 GBStorage: 80GB

IdM(Identity Service)

IdM(Identity Service)

Middleware

OS: OEL Linux 5.x 64bitCPU: 2RAM: 8 GBStorage: 80GB

SISSIS

OBIEE OBIEE

10


SICAS BI Cloud• Shared infrastructure (consolidation AND virtualization)

• ‘schema like’ data consolidation via VPD (Virtual Private Database)

• Consolidation of application via a single shared OBIEE implementation

• Facilitates campus unique data and campus specific BI development as well

• Sized for 24 Campuses • 12,750 active users • 10% concurrency

• 3.6 TB Operational Data Store

• Dramatic reduction in infrastructure management costs (one database, one application)

11


Student Information System (SIS)

• Based upon SunGardHE Banner

• Managed in conjunction with SICAS and each campus

• Evolved over time

• Increasing value to campus

• Increasing complexity

• Increasing number of moving parts

SIS

12


Our Banner customers

13


DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS

Banner Server (PROD)

Software

Oracle Client

Oracle Precompilers Pro C, Pro Cobol

Oracle SQR

Databases Accessed

Apps

Proc CProc CobSQR

SQL / OS Shell Scripts

PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CProc CobSQR


DEVLPPRDWeb (App) Server

Software

Oracle Internet Application Server (10gR2,R3)

Databases Accessed

PROD

Apps

ReportsForms

mod pl/sql OC4J

PPRD DEVL

SIS

14


SIS

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR



SoftwareOracle Internet Application Server

(10gR2,R3)

Databases AccessedPROD

AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

14


App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

App DApp C

App B

DB Server 1

Databases

PROD

Software

Oracle RDBMS

DB Server 2

Databases

PPRD

DEVL

Software

Oracle RDBMS


Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR


PROD

Banner Server (Dev)

Software

Oracle Client


Oracle SQR

Databases Accessed

Apps

Proc CPro CobSQR




(10gR2,R3)


AppsReportsForms

mod pl/sql OC4J

PPRD DEVL

DB Server 3

Databases

A

Software

Oracle RDBMS

DB Server 4

Databases

AA

Software

Oracle RDBMS

App A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

PROD

App DApp C

App BApp A

Software

Oracle Client

Java

Databases Accessed

Apps

Comp 2Comp 1

DEVLPPRD

B C D AA

AA

AA

11 Banner Campuses

Numerous

Other

Apps

too ...

193+ Oracle

Databases

Thats a lot of ‘stuff ’ to manage !

15

Our hosted Banner customers


Database Cluster (RAC)

RAC Node1

db1

RAC Node2

db4

RAC Node3

db4

Banner on Linux n-tier

db2 db2

db5

db3 db3 db3

WWW 01

Binghamton

Buffalo StateDutchess

JCCNassau

SuffolkDB10DB 9 SICASDB 8 OldWestbur

DB 1 Binghamton

DB 7 NiagaraDB 6 NewPaltz

DutchessBuffalo State

JCCNassau

DB 2DB 3DB 4DB 5

Web tierINB and SSB

Banner tiergurjobs, command line ‘stuff’

Database tierdatabases

RAC NodeY

db10

db9

RAC NodeZ

db2

db1 db1

db10

Old WestburyNewPaltz

NewPaltzNiagara

Old Westbury

WWW 02

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW 03

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW x

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW y

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW z

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

Niagara

Binghamton Buffalo State Dutchess JCC

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICAS Suffolk

Nassau

• 10 application servers• 10 banner servers• 6 database servers

• 100,000+ student population• 10 Campuses• 19,000 students registered in 90 minutes

52 passenger bus every 15 seconds

But ... remember our Banner on RAC proof of

concept?

16


Double Payback• Performance yes but .... savings via server

consolidation as well as the management effort required to manage X ‘cookie cutter’ technology installations

• Application servers are exactly identical, database servers are very, very, very, nearly identical (special case for tape backups)

• 26 servers vs 40 (10 as + 10 banner + 6 dbvs 20 as + 10 banner + 10 db)

• Not demonstrated - each prod database has at least 3 non-prod databases behind it

• We can build a similar (smaller but more ram) non-prod cluster for that

• 20 servers vs 30 (6 as + 10 banner + 4 dbvs 10 as + 10 banner + 10 db)

17

Database Cluster (RAC)

RAC Node1

db1

RAC Node2

db4

RAC Node3

db4

Banner on Linux n-tier

db2 db2

db5

db3 db3 db3

WWW 01

Binghamton


JCCNassau

SuffolkDB10DB 9 SICASDB 8 OldWestbur

DB 1 Binghamton

DB 7 NiagaraDB 6 NewPaltz

DutchessBuffalo State

JCCNassau

DB 2DB 3DB 4DB 5

Web tierINB and SSB

Banner tiergurjobs, command line ‘stuff’

Database tierdatabases

RAC NodeY

db10

db9

RAC NodeZ

db2

db1 db1

db10

Old WestburyNewPaltz

NewPaltzNiagara

Old Westbury

WWW 02

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW 03

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW x

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW y

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

WWW z

Binghamton


JCCNassau

NewPaltzNiagara

Old Westbury

Niagara

Binghamton Buffalo State Dutchess JCC

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICASSuffolk

SICAS Suffolk

Nassau

30% less

30% less


Utopia

• THIS is where the campus uniqueness goes

• Balancing act WRT application location between consolidation and freedom to upgrade (technology wise)

• Fewer individual Oracle code trees

• Resource allocation on demand (perhaps even automated)

• Higher Availability

• Eliminate infrastructure uniqueness

• but ... Increased Complexity

• Fewer individual Oracle code trees

• Resource allocation on demand (perhaps even automated)

• Higher Availability

• Eliminate infrastructure uniqueness

• but ... Increased Complexity

WebLogic Clusters

RAC (database) Clusters

Banner servers (on OVM)

18


DB Virtualization and Consolidation Summary• OLIS (Library) schema based consolidation works

(and scales from a mgmt perspective)

• BI Cloud being a multi-tenant approach also works and scales from a mgmt perspective

• (Hosted) banner can be improved via consolidation and virtualization thanks to WL clustering and RAC clustering

19


Back onto the RACMonday, December 12, 2011

RAC - dev cluster 1

• With the assistance of the Oracle RAC pack and Oracle Advanced Customer Service we have constructed our first 11gR2 cluster

• 3 node cluster based on Dell m610 blades

21


RAC001 - speeds and feeds

• 3 m610 blades (currently all in the same devel chasis)

• 2 CPU, 8 Core running at 2.4 GHZ

• 64 GB ram

• iSCSI storage (Equalogic)

• 6 IP interfaces gigE

• 2 Storage (bonded)

• 2 Public Interface

• 2 Private Interconnect

22


RAC001 - technology included (and not)

23

ASM Finally - yes !

OCFS (for non Datafiles) Local FS for Software, NFS for the FRA.

Native (Oracle) NIC Bonding

We are trying it but we’re not entirely clear on the benefits

GNS (Oracle managed DNS) Our clusters will not be so dynamic as to need this.

SCAN(single client access name)

Yes (interesting challenge with direct external SQLNet access)

Connection Manager Will use this to control campus <-> instance access

Service (not SIDS) Obviously, but much more extensively (think INB, SSB, Batch, 3rd party app all as separate services)

Intelligent’ Connection Load Balancing

GridLink between RAC and WL looks very promising


Back on the RAC

24


Guiding Principles

25

• Document and Learn

• Walk before you run

• Creation of technical operating rules

• Separation of Duties

• Embrace the opportunity to change how we do things - seeking efficiencies and simplifications

• Build upon success


RAC adoption roadmap

• Building an adoption plan that looks at two aspects

• Application and business ‘readiness’

• RAC technology maturity (within ITEC)

26


27••• - many more apps listedMonday, December 12, 2011

Application Readiness1. Certified to run on RAC (minimum RDBMS version is going to be 11.2.0.2) -

possible values (y,n)

2. Does the app need HA (via datagard) - possible values (y,n)

3. External sqlnet required (because we have the scan firewall question) - possible values (y,n)

4. Size of the customer base - number of end users potentially impacted by an outage - possible values are small, medium large with small < 50, medium < 500, large is over 500

5. Internal or external users. Are the users of the application internal to ITEC - possible values (i,e, m)

6. Visibility within SUNY. The users of the app are limited to either a given department, a campus or multiple campuses - possible values are (one department, one campus, multiple campuses)

28


Application Readiness7. Business Criticality or Maximum outage permitted. Thinking here is that flimsy

tolerant apps can go first while those with much tighter requirements go later when we have our act much more together - possible values are (< 10 minutes, 1 hour, 1 day)

8. Is it a 'large capacity app' memory, cpu, network, activity (transactions, redo generation) ... hence making it a good candidate for multi node implementations - possible values are (y,n)

9. Does the app have good workload mixes and viability for workload segregation (a very good testing aspect as we've identified workload management as something we need to engage). Is node afinity something we can take advantage of via seperate services. - possible values are (y,n)

10. Workload profile (time based) 9-5 vs 6 - midnight vs midnight to 6am (so we can mix apps of varying profiles). within a given service varying the resource allotment (via resource planning) over time. - possible values are (y,n)

29


Application Readiness

Antipatterns - the following are potential indicators of apps that should NOT be moved into RAC

11. Excessive truncates - possible values (y,n)

12. Excessive DDL - possible values (y,n)

13. Excessive sequence usage - possible values (y,n)

14. Pipes - need special config to run on particular nodes - possible values (y,n)

30


RAC Summary

• By virtualizing databases and standardizing their configurations ...efficiencies will be realized.

• We’re doing this more so as a consolidation and database standardization mechanism. Increased availability and performance are by-products of this solution.

• RAC brings with it complexities - which industry tells us are outweighed by the efficiencies.

• We are focusing on the transition process. We must have our act together before we engage.

32


DBA 2.0Monday, December 12, 2011

34

“Progress is impossible without change; and those who cannot change their minds

cannot change anything.”George Bernard Shaw


DBA 2.0

" A forward looking plan for transformation of the ITEC DBA team; the ITEC DBA v2.0 organizational model seeks to maximize effectiveness of the DBA team in specific technologies for the benefit of our customers by providing a robust solution set supported by technicians with an unprecedented depth of experience. "

35


DBA 1.0

36

• Today, the ITEC Oracle DBA team is equally responsible for a myriad of technologies

• With respect to Banner - we have 8 DBAs with roughly the same skillset - Oracle RDBMS + Oracle Application Server

• We sprinkle these across campuses with primary and secondary designations (with some forethought by the way)


FYI - there is method to these assignments :)

37

Dan Deepa Ed Erik Gay Jiu Joe Kelly Paul Sayi Todd MaximumScadm Email Wed Fri Mon Thu TueIn/Out 1 1 1 1 1 1 1 1 1 1 1

ACCC 2 0.5 1 3.5Alfred 1 2 3Binghamton 2 1 3Brockport 1 2 3Broome 2 1 3Buff State 0.5 2 1 3.5Canton 2 1 3Corning 1 2Cayuga 1 2 3Cobleskill 1 2 3Delhi 1 0.5 2 3.5Downstate 1 2 3Dutchess 2 1 3Fredonia 1 0.5 2 3.5HCCC 1 2 3HVCC 2 1 3JCC 1 2 3Maritime 2 1 3Mohawk Valley 2 1 3Monroe 1 2 3Nassau 0.5 2 1 3.5New Paltz 1 2 0.5 3.5NCCC 1 2 3OLIS (Del, Nia, Sen)OLIS (Del, Nia, Sen) 2 0.5 1 3.5OLIS (Hud,Moh,Uni, Sar)OLIS (Hud,Moh,Uni, Sar) 1 0.5 2 3.5OldWestbury 2 1 0.5 3.5Orange 1 2 0.5 3.5SICAS 2 1 3SLN (Angel) 2 0.5 1 3.5SLN ( Oracle) 1 2 3Suffolk 1 0.5 2 3.5Oneonta 1 2 3UCAWD 1 2 3SUNYIT 2 1 3ITEC Internal (db) 1 2 3ITEC Internal (identity)ITEC Internal (identity) 1 2 3

Totals 0 3 3 17.5 13 13 15 12 11.5 11.5 14.5# of primaries 0 1 1 5 4 4 7 3 4 1 6

0

5

10

15

20

Totals

14.5

121212

15

1313

18

33

0

Relative Workload

Dan Deepa Ed Erik Gay Jiu Joe Kelly Paul Sayi Todd

Campuses in Common

Erik Gay Jiu Joe Kelly Paul Sayi Todd Ed Deepa

Erik

Gay

Jiu

Joe

Kelly

Paul

Sayi

Todd

Ed

Deepa

2 2 1.25 2.25 2 2.25 1 1 0.5

0 2 1 1 3 0.5 0.25 0

2.25 1 1.5 2 2 0 0

1.5 0 1 0.5 0 0

1 0 2 0 1

1.5 2 0.25 0

2.5 0 0

0 0

1


DBA 2.0

RAC Admins

WL AdminsTraditional

DBAs

“Jack & Jill of all trades” DBAs

38


39

WebLogic Clusterware

RAC Clusterware and ASM

WebLogic Containers & Databases

Traditional DBAsWebLogic Admins

RAC AdminsEM Admins

" Traditional DBAs focus on applications and databases that MAY

reside within weblogic and/or RAC clusters "

" RAC Administrators focus on database

clustering technology and Oracle based

storage (Clusterware and ASM)"

" EM Administrators focus on enterprise monitoring and management technologies utilized by ALL of the other

DBA teams"

" WebLogic Administrators focus on application server clustering technology (WebLogic Suite) "

“Incorporating a professional development

lifecycle; ITEC DBA 2.0 ensures that technical

staff are ‘skilled up’ as they move through the cycle.”


Traditional DBAs

40 WL AdminsRAC AdminsMonday, December 12, 2011

Traditional DBA• Manages Oracle databases within the

RAC Cluster

• Manages Oracle client access software on Banner and other 3rd party servers

• Does not install, patch or upgrade RDBMS, RAC, ASM or WebLogic software (this is done by the WL or RAC Admins)

• Works with the WL and RAC Admins to meet performance and availability requirements

• May manage WebLogic Containers within the WebLogic Cluster (typically this will be the SICAS Banner Application Admin)

41


WebLogic Admin• Manages the WebLogic Cluster

Infrastructure

• Provides a ‘node identical’ environment

• Manages resource allocations across the cluster

• Campus applications reside within individual WL containers IN the cluster and these are managed by either application admins from either SICAS or the Campus (or in some cases the ITEC ‘traditional’ DBAs)

42


RAC Admin

• Manages the RAC Infrastructure (Clusterware, ASM, Oracle RDBMS software)

• Provides a ‘node identical’ environment

• Manages resource allocations across the cluster

• Campus data resides within individual Oracle databases IN the cluster and these are managed by ITEC ‘traditional’ DBAs

43


Transition• Shift away from the campus specific ITEC DBA assignments model

• We will utilize the SMO (Service Management Office) to facilitate the customer interface

• SMO will draw upon the three teams to satisfy requests for service and will have technical expertise with them for guidance along the way

• Each of the three teams will be able to focus more on their specific areas of responsibility - becoming “insanely skilled” in those areas.

• This will lead to simplification and standardization.

• And ultimately - higher quality and more cost effective service.

44


Getting There

• We are putting considerable effort into the DBA 2.0 transition planning.

• You are already seeing the introduction of the SMO into hosted campus meetings

• SMO involved with WL and RAC transition planning and execution

45


DBA 2.0 Transition

46

January

• WebLogic Pilot campus chosen (thank you BSC)

• WebLogic training completed for all ITEC DBA staff (WL Admins, RAC Admins and Traditional DBAs)

• WebLogic Admins hand over campus specific Oracle responsibilities to their primaries/secondaries in order to focus on WL upgrade plan

• WebLogic upgrade plan developed and proven with pilot campus and the SMO


DBA 2.0 Transition

47

February - April

• WebLogic deployed to all campuses

• WebLogic administration requests communicated via the SMO or the campus primary DBAs or the helpdesk (not directly from the campus)

• SICAS (or campus) takes on application admin duties within WebLogic containers

• Forms, Reports, Ear, War, Jar file deployments


DBA 2.0 Transition

48

June

• RAC Pilot campus chosen

• RAC Admins slowly hand over campus specific Oracle responsibilities to the traditional DBA pool in order to focus on RAC upgrade plan

• RAC upgrade plan developed and proven with pilot campus


DBA 2.0 Transition

49

August - December

• Hosted campus databases moved to RAC infrastructure

• (BTW we will have numerous internal databases on RAC already)

• Database administration requests communicated via the SMO or the helpdesk to the traditional DBA pool


DBA 2.0 Transition

50

January 2013

• ALL WL administration performed by the WL team. Communicated via the SMO or the helpdesk

• Application administration performed by SICAS or the campus IT staff (forms, report, ear, war, jar file deployments)

• ALL DB administration performed by the traditional DBA team. Communicated via the SMO or the helpdesk

• ALL RAC administration performed by the RAC team. Communicated via the SMO


51

WebLogic Clusterware

RAC Clusterware and ASM

WebLogic Containers & Databases

Traditional DBAsWebLogic Admins

RAC AdminsEM Admins

" Traditional DBAs focus on applications and databases that MAY

reside within weblogic and/or RAC clusters "

" RAC Administrators focus on database

clustering technology and Oracle based

storage (Clusterware and ASM)"

" EM Administrators focus on enterprise monitoring and management technologies utilized by ALL of the other

DBA teams"

" WebLogic Administrators focus on application server clustering technology (WebLogic Suite) "


Uber Summary• Database virtualization and

consolidation in the past (OLIS, BI Cloud, Banner, RAC poc)

• ITEC climbs back on the RAC

• with an emphasis on planning and repetitive process

• DBA 2.0 - a transition towards specialization - intertwined with the SMO (service management office)

52

“Progress is impossible without change; and those who cannot change their minds

cannot change anything.”George Bernard Shaw


Thank you for your time. [email protected]

Please check out the ITEC Service Management Office presentation this

afternoon at 4:45 !

(I’ve heard that free alcohol follows)


mailto:[email protected]

mailto:[email protected]

Date post:	18-Nov-2014
Category:	Technology
Upload:	dan-brint
View:	298 times
Download:	0 times