23
Injecting TDD into your ETL
Contact: Andrew Holowaty
614.448.8196
About Andrew
• Consultant for over 18 years
• Experience – Fortune 500, State Government, Finance, Insurance, Healthcare, and small businesses
• Education – MBA, PMP, MCDBA, MCSD, MCAD, BS/BA
• Family –
2
Agenda
• TDD Overview
• Demo
• Nuts & Bolts of Functionality
• Benefits with Pros and Cons
• Q & A
3
TDD Overview
4
WIKIPEDIA.com - Test-driven development (TDD) is a software development
technique that relies on the repetition of a very short development cycle: First
the developer writes a failing automated test case that defines a desired
improvement or new function, then produces code to pass that test and finally
refactors the new code to acceptable standards. Kent Beck, who is credited with
having developed or 'rediscovered' the technique, stated in 2003 that TDD
encourages simple designs and inspires confidence.
en.csharp-online.net - TDD is a computer programming technique involving
writing test cases first ; then, implementing just the code needed to pass the
test. Test-driven development gives rapid feedback as part of an Extreme
Programming methodology. TDD is not merely a method of testing: Adherents
emphasize test-driven development as a method of designing software.
www.methodsandtools.com - TDD is radical process that promotes the notion
of writing test cases that then dictate or drive the further development of a class
or piece of code. This is often referred to as "writing tests first". Indeed, I will
refer to the phrase "writing tests first" as one of TDD’s primary principles.
TDD Overview
5
• Steve McConnell’s Code Complete V1 released in 1993
– Outlines Best Practices
– Packed with checklists
• Requirements
• Architecture
• Test Cases
• Effective Inspections
• Many others
– One of the many takeaways - Focus on testing after coding was complete
• What does code complete mean to you?
TDD Overview
6
• Waterfall and Structure Systems Analysis
– Dominant in the 70s and 80s
• Object-Oriented Programming – OOP/OOD/OOA
– Became mainstream in the early 1990s
• Computer-Aided Software Engineering – CASE tools
– At their peak in the early 1990s
• Rapid Application Development
– Flavors – Agile, Extreme Programming (XP), JAD, LD, Scrum
TDD Overview
7
TDD Overview
8
Source: Wikipedia
TDD Overview
9
Write a
failing test
Write some
codeRun tests Refactor Code
No
Yes
Development
Finished?
No
Yes
Requirements
List
Code
Complete
Tests
Pass?
Another View…
Source: ASP.NET MVC 1.0 Test Driven Development, WROX
TDD Overview
10
An example with Visual Studio 2008 Pro…No tools like this for BI Studio
2008.
It’s good to
see green!
11
DEMO of creating your own TDD
environment for ETL
Demo
12
Nuts & Bolts of Functionality
1. SQL Server Control Tables
2. SSIS Steps
3. Main Stored Procedures
4. Control Table Population
5. SSRS Reports
13
Nuts & Bolts of Functionality
1.SQL Server Control Tables
14
Nuts & Bolts of Functionality
2. SSIS Steps – Set the Stage and Final Check
15
Nuts & Bolts of Functionality
3. Main Stored Procedures
1) upEtlControlBatchInsert
2) upEtlControlBatchUpdateEndDateTime
3) upEtlControlSystemLogInsert
4) upEtlControlAuditLogInsert
5) upEtlAuditCheck
6) upEtlAuditReportGet
Called From Main Package
Called From Main Package
Called From ETL Audit Status
Report
Called From Main Package
16
1) Exec upEtlControlBatchInsert ?
2) exec upEtlControlSystemLogInsert @pi_ETLBatchID = ?,
@pi_vcLoggedBy=?,
@pi_vcSystemLogDesc='Start/End of Package',
@pi_vcSource = ?
3) exec upEtlControlSystemLogInsert
@pi_ETLBatchID = @pi_iETLBatchID,
@pi_vcLoggedBy = 'upEtlDimRegionLoad',
@pi_vcSystemLogDesc = ‘Start/End of Procedure',
@pi_vcSource = 'upEtlDimRegionLoad‘
4) Insert Into dbo.ControlAuditLog ( ETLBatchID, AuditEntryDescription, AuditEntryMeasurement )
Values ( @pi_iETLBatchID,
'FactClaimOccurrence Records Inserted',
(Select COUNT(1) From @tblTemp Where Action = 'INSERT') );
5) Insert Into dbo.ControlAuditLog
( ETLBatchID, AuditEntryDescription, AuditEntryMeasurement, TestCaseNumber )
Values ( @pi_iETLBatchID, 'Source Average LOS', @dLOSFromDestination, 22 );
Insert Into dbo.ControlAuditLog
( ETLBatchID, AuditEntryDescription, AuditEntryMeasurement, TestCaseNumber )
Values ( @pi_iETLBatchID, 'Destination Average LOS', @dLOSFromSource, 22 );
Nuts & Bolts of Functionality
4. Control Table Population Call From Main Package
Call From Any Package
Call From Any SP
Call From Any SP
17
Nuts & Bolts of Functionality
5. SSRS Reports
18
Nuts & Bolts of Functionality
5. SSRS Reports (continued)
19
Nuts & Bolts of Functionality
5. SSRS Reports (continued)
20
Benefits with Pros and Cons
Benefit: Testing upfront can reduce total development time.
Source: http://www.methodsandtools.com/archive/archive.php?id=20
21
Benefits with Pros and Cons
Pros:
• More predictive – Writing test cases is more predictive than fixing bugs – http://www.renaissancesoftware.net/blog/archives/16
• Productive – So, why use TDD? - arguably because you end up with half the defects but similar productivity.
– http://www.eventyr.co.uk/summary_of_points/
• Flexibility – Iterative process that can change with client’s ever changing requirements
– http://www.methodsandtools.com/archive/archive.php?id=20
• Malleability – TDD can lead to more modularized, flexible, and extensible ETL code
– http://en.wikipedia.org/wiki/Test-driven_development
• TDD makes development fun again. It is fun to see green. – http://www.adaptionsoft.com/tdd_how.html
• The collection of tests give a full regression test suite
Probably the
most
important.
22
Benefits with Pros and Cons
Cons:
• Overhead – Tests are coded and need to be maintained
• Limited – Unit versus Full functional tests
• False sense of quality – Still need to engage other quality control processes such as integration and compliance testing
– Source: http://en.wikipedia.org/wiki/Test-driven_development
23
Q & A
Questions
And
Thank You!
