Post on 28-Jan-2016
transcript
Databases in Databases in ContextContext
Wendy MoncurWendy Moncur
Department of Computing Science, Department of Computing Science,
University of AberdeenUniversity of Aberdeen
Databases in ContextDatabases in Context
Database design in a major bankDatabase design in a major bank
Database managementDatabase management
6000-table Personnel database6000-table Personnel database
Who am I ?Who am I ?
Wendy MoncurWendy Moncur
DataBase AdministratorDataBase Administrator (DBA) at one of UK’s (DBA) at one of UK’s largest banks.largest banks.
Designed databases for high performance & Designed databases for high performance & availability. availability.
19 years industry experience.19 years industry experience.
Platform: DB2 & SQLPlatform: DB2 & SQL
Largest database: 6000 tablesLargest database: 6000 tables
Why listen?Why listen?
DBA Average Minimum Salary DBA Average Minimum Salary £41,896£41,896
DBA Average Maximum Salary DBA Average Maximum Salary £47,147£47,147
Source: http://www.itjobswatch.co.uk Source: http://www.itjobswatch.co.uk
What does a What does a DBADBA do? do?
Database design & optimisationDatabase design & optimisation
Quality assurance of SQLQuality assurance of SQL
Performance managementPerformance management
Database administrationDatabase administration
Case study: the monster database
Case study: the monster database
• 6000+ tables• 18000+ indexes
Part1: ChallengesPart1: Challenges
““One size fits all”One size fits all” External supplierExternal supplier 6000+ tables6000+ tables 18000+ indexes18000+ indexes 1 tablespace1 tablespace Short timescaleShort timescale
Challenges: “one size fits all”?Challenges: “one size fits all”?
One size does One size does notnot fit all. fit all.
Performance of SQL statements Performance of SQL statements dependent on:dependent on:
Database designDatabase design Index designIndex design
The The DATADATA
Challenges: “one size fits all”?Challenges: “one size fits all”?
Every company has different requirements.Every company has different requirements.
Customers Customers demand demand high performance... and high performance... and control the budget. control the budget.
Service Level Agreements (Service Level Agreements (SLAsSLAs) dictate … ) dictate … Minimum transaction speedMinimum transaction speed Number of concurrent usersNumber of concurrent users Number of remote locationsNumber of remote locations Daily system availabilityDaily system availability
Database must be Database must be tailored tailored to achieve site-specific to achieve site-specific SLAs.SLAs.
Challenges: external Challenges: external suppliersupplier
Software package & database from Software package & database from external supplier.external supplier.
CannotCannot change this. change this.
Challenges: 6,000+ tablesChallenges: 6,000+ tables
Cannot Cannot change tables: no denormalisation change tables: no denormalisation allowed.allowed.
Supplied program code demands these tables Supplied program code demands these tables exist. exist.
Cannot change supplied program code unless Cannot change supplied program code unless essentialessential..
Challenges: 18,000+ indexesChallenges: 18,000+ indexes
Can Can change indexes:change indexes:
Unique indexesUnique indexes
Clustering indexesClustering indexes
Secondary indexesSecondary indexes
Unique indexUnique index
Defines what makes a row unique. Defines what makes a row unique.
Components of the index Components of the index cannotcannot be be changed. changed.
Order of componentsOrder of components cancan be be changed.changed.
Unique indexUnique index
E.g. – for Table “E.g. – for Table “EMPLOYEEEMPLOYEE” ”
Unique index =Unique index = DateOfBirth, Firstname, Surname.DateOfBirth, Firstname, Surname.
Most queries ask for data where only Most queries ask for data where only Surname, Surname, FirstnameFirstname are known.are known.
SELECT Surname, Firstname, DateOfBirthSELECT Surname, Firstname, DateOfBirthFrom EmployeeFrom EmployeeWhere Where SurnameSurname = “Jenkins” And= “Jenkins” And FirstnameFirstname = “Malcolm” ;= “Malcolm” ;
Recommendation: Change order of unique index to Recommendation: Change order of unique index to Surname, Firstname, DateOfBirth. Surname, Firstname, DateOfBirth.
Clustering indexesClustering indexes
Defines the physical order in which rows Defines the physical order in which rows of data should be stored.of data should be stored.
Components of the index Components of the index cancan be be changed. changed.
Order of components Order of components cancan be changed. be changed.
Clustering indexesClustering indexes
E.g. – Table “EMPLOYEE” E.g. – Table “EMPLOYEE”
Clustering index = Clustering index = DateOfBirthDateOfBirth
Yet most queries order by Yet most queries order by EmploymentStartDateEmploymentStartDate
SELECT EmploymentStartDate, Surname, FirstnameSELECT EmploymentStartDate, Surname, Firstname
From EmployeeFrom Employee
Where Surname = “Jenkins” And Firstname = “Malcolm” ;Where Surname = “Jenkins” And Firstname = “Malcolm” ;
Order by Order by EmploymentStartDate;EmploymentStartDate;
Recommendation: Change clustering index to use Recommendation: Change clustering index to use
EmploymentStartDate.EmploymentStartDate.
Secondary indexesSecondary indexes
Not unique.Not unique.
Do not dictate how the data is to be held.Do not dictate how the data is to be held.
Created to improve performance of queries and updates.Created to improve performance of queries and updates.
Increases cost of insert and update, as must be created and Increases cost of insert and update, as must be created and maintained along with the table.maintained along with the table.
Recommendation: Recommendation: Drop superfluous secondary Drop superfluous secondary indexes. indexes.
Challenges: One Challenges: One tablespacetablespace
Many tablespacesMany tablespaces
Create many new tablespaces. Create many new tablespaces. Split the tables between them, according to table Split the tables between them, according to table
function. function.
At least 4 test environments:At least 4 test environments:
96,000 objects!96,000 objects! ((6,000 tables + 18,000 indexes) * 4 environments)((6,000 tables + 18,000 indexes) * 4 environments)
3 months3 months
Challenge: Short Challenge: Short timescaletimescale
Vanilla Unit test System test Pre-live
ToolsTools
Use tools to…Use tools to…
Check performance of each SQL Check performance of each SQL statementstatement
Manage change processManage change process
Check performanceCheck performance
““EXPLAIN”EXPLAIN”
Evaluates route to data for every SQL statement.Evaluates route to data for every SQL statement.
Identifies what indexes are used Identifies what indexes are used
Doesn’tDoesn’t identify redundant indexes identify redundant indexes
Doesn’tDoesn’t identify indexes that need to be changed. identify indexes that need to be changed.
Manage change process Manage change process
Rigorous control neededRigorous control needed
Achieved through…Achieved through… Consistent naming standards Consistent naming standards Detailed record of every changeDetailed record of every change Consistent route through environments, no short cutsConsistent route through environments, no short cuts DBA tools DBA tools
Part1: Recap of Part1: Recap of challenges challenges
Can’t change:Can’t change:
““One size fits all”One size fits all”
External supplierExternal supplier
6000+ tables6000+ tables
CanCan change: change:
18000+ indexes18000+ indexes
1 tablespace1 tablespace
Short timescaleShort timescale
Part2: The Production Part2: The Production DatabaseDatabase
Does it perform?Does it perform?
Can the Can the rightright people use it? people use it?
If disaster strikes, can the data be recovered?If disaster strikes, can the data be recovered?
Does the database perform?Does the database perform?
Database performance monitored against Service Database performance monitored against Service Level Agreements (SLAs).Level Agreements (SLAs).
Regular health checks carried out:Regular health checks carried out: Data stored in sequence?Data stored in sequence? Enough space? Enough space?
If sub-standard performance, further database If sub-standard performance, further database design work done. design work done.
Can the right people access the data?Can the right people access the data?
PERSONNEL database
Can the right people access the data?Can the right people access the data?
Personnel team
Query & update data at individual or regional level
PERSONNEL database
Can the right people access the Can the right people access the data?data?
Personnel team
Query & update data at individual or regional level
PERSONNEL database
DBA
Backup/ restore data
Reorganise data
Change database definitions
Update statistics on data
Can the right people access the Can the right people access the data?data?
Personnel team
Query & update data at individual or regional level
PERSONNEL database
DBA
Backup/ restore data
Reorganise data
Change database definitions
Update statistics on data
Chief executive
Employee statistics
Can the right people access the Can the right people access the data?data?
Personnel team
Query & update data at individual or regional level
PERSONNEL database
DBA
Backup/ restore data
Reorganise data
Change database definitions
Update statistics on data
Chief executive
Employee statistics
Staff member
Their own data
Can the right people use the Can the right people use the database?database?
Different people, different information needs.Different people, different information needs.
Sensitive data – salary, health, discipline…Sensitive data – salary, health, discipline…
Solution Solution VIEWSVIEWS Transaction ManagementTransaction Management
If disaster strikes, If disaster strikes, can the data be recovered?can the data be recovered?
Robust backup & recovery strategies for:Robust backup & recovery strategies for: Hardware failureHardware failure Software failureSoftware failure
Part2: Recap of Part2: Recap of Production Database Production Database
issuesissues Database must perform to acceptable level.Database must perform to acceptable level.
Only the Only the rightright people should have access to any people should have access to any data item.data item.
No matter what, the data must be recoverable.No matter what, the data must be recoverable.