Grab Bag & Short Shouts

PASS Victoria BC ChapterFall 2014

Martin S. Stoller

http://ca.linkedin.com/in/martinsstoller/http://martinsstoller.ca/

GRAB BAG – SESSION 1

1. Schemas - use and misuse and my pet peeves

2. SQL Server Startup Procs - some "from the trenches" usage and script example

3. Collation 101 - Primer

4. SQL Server DDL Triggers in the development environment for auto change tracking

5. SQL Server Table Triggers - Best practices

SHORT SHOUTS – SESSION 2

6. SQL Server Database Diagrams 101 for Documentation

7. The Case for Server Case Sensitivity

8. A strategy to Index Low-Selectivity Columns - ie: Y/N columns

9. Intro to the HIERARCHID data type in SQL2008+ for hierarchical data

10. "ETL from the front lines" - when all you have is a screwdriver

11. The Trouble with Date and Time when dealing with international data

SCHEMAS

SCHEMA-> WHAT ARE THEY AND THEIR CONS

• Only interesting if you can configure path of each table, or are a data-developer with full control and access to the DB and any related application code base - or are lucky enough to use a flexible app that allows schema changes.

• In general, Schema in MSSQL are just naming convention lipstick.

• Some ORMs will not appreciate that you are using a Schema naming convention - looking at you .NET Entity Framework!

SCHEMA -> VALID USES• File groups - if your DB makes extreme use of file

groups, you could combine tables in each FG into their own schema - that way your table structure documents your FG structure. (Only really usable in a static system!)

FileGroupsSchemas

SCHEMA -> VALID USES• Security - in a large complex ERP type app,

schemas can help keep domains separate - simplifies security.

• If you are designing a DB that has a metadata component, you could place the metadata tables into their own schema, and then let the consumer use DBO for their own tables - this removes any possibility of name collision.

User Table

Metadata

SCHEMA -> VALID USES

• Similar to the metadata idea, if you have several APIs feeding off the same DB structure, you might consider grouping your stored procedures and functions into a segregated Schema - this is especially useful when you hijack an existing system and inject new functionality into it.

• Sometimes you end up building several related apps using the same DB, where the apps end up consuming some common table set.

• ETL staging - each Schema could represent different sources, which might have colliding table names.

MORE!

SCHEMA -> VALID USES

• Scalability - if you keep your schema discrete and create a connection string per schema, you can at any given time pull out a schema to a different instance - example would be a logging or auditing schema, which would not be polluted by other data.

• Finally, when you only have one DB available to work with, for example in a WEB-hosting or cloud situation, schemas can be used to keep all your apps separate.

EVEN

MORE!

SCHEMA -> CONCLUSION

SCHEMAS IN MS-SQL CAN ACTUALLY BE USEFUL.

BUT WE WOULD NOT BE TOOHINDERED OR SAD

IF THEY DID NOT EXIST...

The alternative for many of the above uses would be keeping metadata, etc, in a separate DB or DBs.

Of course that has its own downsides, such as backups need to be created simultaneously of all related DBs to maintain consistency – not too big an issue if you have

a good automated backup strategy.

STARTUP PROCS-> FROM THE TRENCHES

STARTUP PROCS-> FROM THE TRENCHES

Some uses for SP

1. Email on SQL service restart (steps follow).

2. Auto CHECKDB after restart.

3. Set a SQL-Agent job to start x min after restart. (NOT the same as an SQL-Agent job starting when SQL-Agent starts!)

4. Auto replication re-sync after restart.

5. Run cleanup after a restart.

STARTUP PROCS-> STEPS 4 RESTART EMAIL

1. Make sure you have your Database Mail configured.

2. Add your startup SP to [master].

3. Configure “scan for startup procs” = 1

4. Register SP as a startup proc.

5. Test

Bonus SQL! See what SPs are set to run on

startup!

USE MASTERSELECT * FROM SYS.PROCEDURESWHERE IS_AUTO_EXECUTED = 1;

Pro-Tip!

Use one single startup SP that runs other SPs to reduce wasting threads

and performance at startup!

/* SETUP MAIL */

USE [MASTER]GO

--- Incase you get "Ad hoc update to system catalogs is not supported."/*sp_configure 'Allow Updates', 0;reconfigure with override;go-- and just in case:sp_configure 'Show Advanced Options', 1;reconfigure with override;go*/

/* MAKE SURE WE CAN EDIT ADVANCED OPTIONS */exec sp_configure 'show advanced options',1;RECONFIGURE;GO

/* ENABLE DATABASE MAIL */exec sp_configure 'Database Mail XPs', 1; RECONFIGURE;GO

/* SETUP PROFILE AND ACCOUNT – edit SMTP server and EMAIL ADDRESES */EXECUTE msdb.dbo.sysmail_add_account_sp @account_name = 'DatabaseAdministrators',@description = 'Mail account for DBA e-mail.',@email_address = 'dba@mycompany.ca',@replyto_address = 'service.desk@mycompany.ca',@display_name = ‘MyCompany Database Adminisrators',@mailserver_name = 'smtp.mycompany.domain';GOEXECUTE msdb.dbo.sysmail_add_profile_sp @profile_name = 'THE_DBA_MAIL_PROFILE',@description = 'Profile used for DB administrative mail.' ;GOEXECUTE msdb.dbo.sysmail_add_profileaccount_sp @profile_name = 'THE_DBA_MAIL_PROFILE',@account_name = 'DatabaseAdministrators',@sequence_number =1 ;GOEXECUTE msdb.dbo.sysmail_add_principalprofile_sp @profile_name = 'THE_DBA_MAIL_PROFILE',@principal_name = 'public',@is_default = 1 ;GO

/* SET SP TO RUN ON REBOOT */USE masterGO

/* MAKE SURE WE CAN EDIT ADVANCED OPTIONS */exec sp_configure 'show advanced options',1;RECONFIGURE;

/* TELL THE SERVER TO SCAN FOR SPS WHEN IT STARTS */exec sp_configure 'scan for startup procs',1;RECONFIGURE;GO

/* CREATE THE EMAIL SP IN MASTER – edit subject and body as required */CREATE PROCEDURE [dbo].[LLFC_REPORT_REBOOT]ASBEGINSET NOCOUNT ON;declare @thebody as nvarchar(255)declare @thesubject as nvarchar(255)set @thebody = 'Hi there, '+@@SERVERNAME+' SQL has restarted. Thanks!'set @thesubject = '[ATTENTION] '+@@SERVERNAME+' SQL RESTARTED!';EXEC msdb.dbo.sp_send_dbmail@profile_name = 'THE_DBA_MAIL_PROFILE', @recipients = 'dba@mycompany.ca', @body = @thebody, @subject = @thesubject ;END/*******************/GO

/* REGISTER ABOVE SP AS A STARTUP SP */EXEC sp_procoption N'[dbo].[LLFC_REPORT_REBOOT]', 'startup', '1';RECONFIGURE;GO

/* CHECK IF SP IS REGISTERED */select name from sysobjects where type ='P' andOBJECTPROPERTY(id,'ExecIsStartup')=1;SELECT * FROM sys.objectsWHERE object_id = OBJECT_ID(N'LLFC_REPORT_REBOOT')AND type IN ( N'P', N'PC' ) GO

/* CHECK IF “scan for startup procs” = 1 */exec sp_configure GO

STARTUP PROCS-> CODE DUMP

Notes:1. Only sysadmin can setup. 2. SP must be in DBO schema.3. SP must be in [MASTER] DB.4. No SP parameters…Sorry.

COLLATION 101

?? ??

COLLATION 101-> WHAT DO THEY DO?

• Collation does not directly affect date, time or number formats - that is another story!

• Defines the default sort order for character types.

• Defines the code page used to store non-Unicode character types.

• Defines if compares are case, accent, kana and/or width sensitive - or not.

• Example: a case sensitive collation will always return ZERO rows with this query: SELECT GETDATE() WHERE 'Albert' = 'alBert'; CS=Case-Sensitive!

CI=Case-Insensi-tive!

COLLATION 101-> WHERE DEF INED?

• On the instance level. This affects what the default collation of all system DBs, any new DBs, tables, fields and temporary tables will be!

• On the DB level.

• On the table level.

• Finally, on the field level!

????

COLLATION 101-> CAVEATS

• Once an instance has a defined collation, it cannot be easily changed. You need to backup all user DBs, then rebuild the master DB with the new collation, and finally reimport the user DBs.

• Collation of DBs can be relatively easily changed, but underlying tables and columns will still need to be changed by script or in some cases manually.

• If a field has a constraint associated with it, you cannot change collation by script - you can however use the UI, because that drops and rebuilds the table. Note, large tables are a pain to change field collation on, and you end to have UI table rebuild option turned on (off by default)...

COLLATION 101-> SELECT LEVEL & SOME TOOLS

• Sometimes you have to deal with collation in a select, either in a join or where - simplest way to handle it if you do not need to care too much is:

• Select * from T1 left join T2 on T1.a = T2.b collate database_default;

• Bonus -> Source code reference for changing collation on each level.

COLLATION 101-> CODE – SET DB COLLATION

Update underlined items with your own DB name(s) as needed.

USE mastergoSELECT name, collation_name FROM sys.databases where name in ('METADATA','WAREHOUSE');GOALTER DATABASE [WAREHOUSE]SET OFFLINE WITH ROLLBACK IMMEDIATEGOALTER DATABASE [WAREHOUSE]SET onlineGOALTER DATABASE [WAREHOUSE] COLLATE SQL_Latin1_General_CP1_CS_AS GOSELECT name, collation_name FROM sys.databases where name in ('METADATA','WAREHOUSE');GO

COLLATION 101-> CODE – TABLE HELPER

DECLARE @collate nvarchar(100);SET @collate = 'SQL_Latin1_General_CP1_CS_AS';

/* List Tables not in desired Collation */SELECT t.name,c.*FROM sys.columns c INNER JOIN sys.tables tON t.object_id = c.object_idWHERE c.collation_name <> @collateand c.object_idIN (SELECT object_id FROM sys.objects WHERE type = 'U')AND c.collation_name != 'NULL'ORDER BY t.name,c.name

/* Create ALTER STMTs to set tables to desired Collation - Set Output to TEXT first */SELECT 'ALTER TABLE [<Table>] ALTER COLUMN [<Column>] <Column_Type> COLLATE ' + @collate as [AlterThis]FROM sys.columns c INNER JOIN sys.tables tON t.object_id = c.object_idWHERE c.collation_name <> @collateand c.object_idIN (SELECT object_id FROM sys.objects WHERE type = 'U')AND c.collation_name != 'NULL'ORDER BY t.name,c.name

Use this to check what tables do not fit the desired collation.

Set query output to TEXT and use this to create ALTER statements to set all tables

to the desired collation, that are not yet correct.

Set underlined to desired collation.

COLLATION 101-> CODE – F IELD HELPER

DECLARE @collate nvarchar(100);DECLARE @table nvarchar(255);DECLARE @column_name nvarchar(255);DECLARE @column_id int;DECLARE @data_type nvarchar(255);DECLARE @max_length int;DECLARE @row_id int;DECLARE @sql nvarchar(max);DECLARE @sql_column nvarchar(max);DECLARE @is_Nullable bit;DECLARE @null nvarchar(25);

SET @collate = 'SQL_Latin1_General_CP1_CS_AS';

DECLARE local_table_cursor CURSOR FOR

SELECT [name]FROM sysobjectsWHERE OBJECTPROPERTY(id, N'IsUserTable') = 1

OPEN local_table_cursorFETCH NEXT FROM local_table_cursorINTO @table

WHILE @@FETCH_STATUS = 0BEGIN

DECLARE local_change_cursor CURSOR FOR

SELECT ROW_NUMBER() OVER (ORDER BY c.column_id) AS row_id , c.name column_name , t.Name data_type , c.max_length , c.column_id , c.is_nullable FROM sys.columns c JOIN sys.types t ON c.system_type_id = t.system_type_id LEFT OUTER JOIN sys.index_columns ic ON ic.object_id = c.object_id AND ic.column_id = c.column_id LEFT OUTER JOIN sys.indexes i ON ic.object_id = i.object_id AND ic.index_id = i.index_id WHERE c.object_id = OBJECT_ID(@table) and c.collation_name <> @collate ORDER BY c.column_id

OPEN local_change_cursor FETCH NEXT FROM local_change_cursor INTO @row_id, @column_name, @data_type, @max_length, @column_id, @is_nullable

WHILE @@FETCH_STATUS = 0 BEGIN

IF (@max_length = -1) SET @max_length = 4000; set @null=' NOT NULL' if (@is_nullable = 1) Set @null=' NULL' if (@Data_type='nvarchar') set @max_length=cast(@max_length/2 as bigint) IF (@data_type LIKE '%char%') BEGIN TRY

/* Cont from prev side */

SET @sql = 'ALTER TABLE ' + @table + ' ALTER COLUMN [' + rtrim(@column_name) + '] ' + @data_type + '(' + CAST(@max_length AS nvarchar(100)) + ') COLLATE DATABASE_DEFAULT' + @null PRINT @sql --EXEC sp_executesql @sql /* uncomment to have the ALTER run immediate - works sometimes! */ END TRY BEGIN CATCH PRINT ‘-- -> ERROR: Some index or contraint rely on the column ' + @column_name + '. No conversion possible.' PRINT @sql END CATCH

FETCH NEXT FROM local_change_cursor INTO @row_id, @column_name, @data_type, @max_length, @column_id, @is_Nullable

END

CLOSE local_change_cursor DEALLOCATE local_change_cursor

FETCH NEXT FROM local_table_cursor INTO @table

END

CLOSE local_table_cursorDEALLOCATE local_table_cursor

Change this to the new collation you need all fields to be in!

This script creates a set of ALTER statementsthat can be run to set each field’s collation which did not match the desired collation, and sets the fields to the default collation of the database..

DDL TRIGGERS

DDL TRIGGERS-> HELP ING YOU DEVELOP• DDL (Data Definition Language) triggers are

new since 2005.

• Can be used to audit, or even block DDL changes.

• Can be set with Server level or Database level scope.

• Reference -> Full list of SQL DDL trigger events including their scope:

• http://msdn.microsoft.com/en-us/library/ms189871.aspx

DDL TRIGGERS-> DATABASE DDL EVENTS

Common triggers on DDL events on Databases:

• Create, Alter or Drop Table

• Create, Alter or Drop Index

• Create, Alter or Drop View

• Create, Alter or Drop Function

• Create, Alter or Drop Stored Procedure

• Create, Alter or Drop Trigger

DDL TRIGGERS-> STRATEGIES

1. Create an AUDIT DB.

2. Create one or more audit tables, depending on what you wish to audit, and how granular , who has access to what, etc.

3. In the target DB, add DDL triggers to fire when those actions are run that you need to catch.

4. To create a baseline, modify all objects without actually changing them. This will save a current snapshot into the audit table.

DDL TRIGGERS-> CODE - AUDIT

CREATE TRIGGER AuditTablesON DATABASEFOR CREATE_TABLE, ALTER_TABLE, DROP_TABLEASBEGINDECLARE @ed XMLSET @ed = EVENTDATA()

INSERT INTO myAuditDB.dbo.DDL_Table ([PostTime], [DatabaseName], [Event], [ObjectName], [TSQL], [Login]) VALUES(

GetDate(),@ed.value('(/EVENT_INSTANCE/DatabaseName)[1]', 'varchar(256)'),@ed.value('(/EVENT_INSTANCE/EventType)[1]', 'nvarchar(100)'),@ed.value('(/EVENT_INSTANCE/ObjectName)[1]', 'varchar(256)'),@ed.value('(/EVENT_INSTANCE/TSQLCommand)[1]', 'nvarchar(2000)'),@ed.value('(/EVENT_INSTANCE/LoginName)[1]', 'varchar(256)')

) ENDGO

This is an example trigger that audits any new, changed, or dropped tables, and saves the information into a previously set up Audit DB (and I am leaving that as an exercise for you ).

DDL TRIGGERS-> CODE – WATCHDOG!

CREATE TRIGGER WATCHDOG_DROP_TABLEON DATABASEFOR DROP_TABLEASBEGINPRINT 'SORRY, TABLES CANNOT BE DROPPED! CONTACT YOUR DBA!'ROLLBACKENDGO

CREATE TRIGGER WATCHDOG_DROP_TABLE_W_EXCEPTIONSON DATABASEFOR DROP_TABLEASBEGINDECLARE @ED XMLDECLARE @TABLENAME VARCHAR(256)SET @ED = EVENTDATA()SET @TABLENAME = @ED.VALUE('(/EVENT_INSTANCE/OBJECTNAME)[1]', 'VARCHAR(256)')IF @TABLENAME NOT IN ('MYEXCEPTION1','MYEXCEPTION2')BEGIN

PRINT 'SORRY, TABLES CANNOT BE DROPPED! CONTACT YOUR DBA!'ROLLBACK

ENDENDGO

This is an example trigger that actively prevents any tables from being dropped, even if the user has access!

The second is an example that filters whichhardcoded tables to allow, anyway…

TABLE TRIGGERS

TABLE TRIGGERS-> BEST PRACTICES

1. If at all possible DO NOT USE TRIGGERS!

2. Triggers WILL impact performance, even simple ones.

3. Keep. It. Simple. Seriously!

4. Avoid locking and blocking!

5. See #1.

Assum

ptio

n:

You

have

a b

asic

unde

rsta

ndin

g of

tabl

e tri

gger

s.

TABLE TRIGGERS-> WELL, I NEED THAT TRIGGER , SO. . . NOW WHAT?

1. If your trigger can fire for different events, for example Update and/or Delete and/or Insert, ensure the trigger only executes code that is meant only for the desired events. Do not try and read DELETED records when an INSERT event has taken place.

2. When handling an UPDATE event, check that the columns you are watching really updated with “IF UPDATE(COLUMNNAME)…”

3. At all costs, avoid CURSORS in triggers!

4. If possible, avoid using WHILE statements.

TABLE TRIGGERS-> WELL, I NEED THAT TRIGGER , SO. . . NOW WHAT?

5. Double, triple and quadruple check that no infinite loops are possible in your trigger!

6. Instead of sending an email from inside the trigger itself, consider saving to an audit or “to be emailed” table, and have a scheduled job send the actual email. This reduces the chance that issues with the mail subsystem will affect performance.

7. Avoid using User Defined Functions in triggers. Scalar UDFs are known performance hogs!(Table-Valued UDFs are generally fine…)

MORE!

END OF SESSION 1

THANK YOU ALL!

(Tiki Stoller, 20140902)