Post on 16-Apr-2018
transcript
Injecting TDD into your ETL
Contact: Andrew Holowaty
614.448.8196
Andrew.Holowaty@Perficient.com
About Andrew
• Consultant for over 18 years
• Experience – Fortune 500, State Government, Finance, Insurance, Healthcare, and small businesses
• Education – MBA, PMP, MCDBA, MCSD, MCAD, BS/BA
• Family –
2
Agenda
• TDD Overview
• Demo
• Nuts & Bolts of Functionality
• Benefits with Pros and Cons
• Q & A
3
TDD Overview
4
WIKIPEDIA.com - Test-driven development (TDD) is a software development
technique that relies on the repetition of a very short development cycle: First
the developer writes a failing automated test case that defines a desired
improvement or new function, then produces code to pass that test and finally
refactors the new code to acceptable standards. Kent Beck, who is credited with
having developed or 'rediscovered' the technique, stated in 2003 that TDD
encourages simple designs and inspires confidence.
en.csharp-online.net - TDD is a computer programming technique involving
writing test cases first ; then, implementing just the code needed to pass the
test. Test-driven development gives rapid feedback as part of an Extreme
Programming methodology. TDD is not merely a method of testing: Adherents
emphasize test-driven development as a method of designing software.
www.methodsandtools.com - TDD is radical process that promotes the notion
of writing test cases that then dictate or drive the further development of a class
or piece of code. This is often referred to as "writing tests first". Indeed, I will
refer to the phrase "writing tests first" as one of TDD’s primary principles.
TDD Overview
5
• Steve McConnell’s Code Complete V1 released in 1993
– Outlines Best Practices
– Packed with checklists
• Requirements
• Architecture
• Test Cases
• Effective Inspections
• Many others
– One of the many takeaways - Focus on testing after coding was complete
• What does code complete mean to you?
TDD Overview
6
• Waterfall and Structure Systems Analysis
– Dominant in the 70s and 80s
• Object-Oriented Programming – OOP/OOD/OOA
– Became mainstream in the early 1990s
• Computer-Aided Software Engineering – CASE tools
– At their peak in the early 1990s
• Rapid Application Development
– Flavors – Agile, Extreme Programming (XP), JAD, LD, Scrum
TDD Overview
9
Write a
failing test
Write some
codeRun tests Refactor Code
No
Yes
Development
Finished?
No
Yes
Requirements
List
Code
Complete
Tests
Pass?
Another View…
Source: ASP.NET MVC 1.0 Test Driven Development, WROX
TDD Overview
10
An example with Visual Studio 2008 Pro…No tools like this for BI Studio
2008.
It’s good to
see green!
12
Nuts & Bolts of Functionality
1. SQL Server Control Tables
2. SSIS Steps
3. Main Stored Procedures
4. Control Table Population
5. SSRS Reports
15
Nuts & Bolts of Functionality
3. Main Stored Procedures
1) upEtlControlBatchInsert
2) upEtlControlBatchUpdateEndDateTime
3) upEtlControlSystemLogInsert
4) upEtlControlAuditLogInsert
5) upEtlAuditCheck
6) upEtlAuditReportGet
Called From Main Package
Called From Main Package
Called From ETL Audit Status
Report
Called From Main Package
16
1) Exec upEtlControlBatchInsert ?
2) exec upEtlControlSystemLogInsert @pi_ETLBatchID = ?,
@pi_vcLoggedBy=?,
@pi_vcSystemLogDesc='Start/End of Package',
@pi_vcSource = ?
3) exec upEtlControlSystemLogInsert
@pi_ETLBatchID = @pi_iETLBatchID,
@pi_vcLoggedBy = 'upEtlDimRegionLoad',
@pi_vcSystemLogDesc = ‘Start/End of Procedure',
@pi_vcSource = 'upEtlDimRegionLoad‘
4) Insert Into dbo.ControlAuditLog ( ETLBatchID, AuditEntryDescription, AuditEntryMeasurement )
Values ( @pi_iETLBatchID,
'FactClaimOccurrence Records Inserted',
(Select COUNT(1) From @tblTemp Where Action = 'INSERT') );
5) Insert Into dbo.ControlAuditLog
( ETLBatchID, AuditEntryDescription, AuditEntryMeasurement, TestCaseNumber )
Values ( @pi_iETLBatchID, 'Source Average LOS', @dLOSFromDestination, 22 );
Insert Into dbo.ControlAuditLog
( ETLBatchID, AuditEntryDescription, AuditEntryMeasurement, TestCaseNumber )
Values ( @pi_iETLBatchID, 'Destination Average LOS', @dLOSFromSource, 22 );
Nuts & Bolts of Functionality
4. Control Table Population Call From Main Package
Call From Any Package
Call From Any SP
Call From Any SP
20
Benefits with Pros and Cons
Benefit: Testing upfront can reduce total development time.
Source: http://www.methodsandtools.com/archive/archive.php?id=20
21
Benefits with Pros and Cons
Pros:
• More predictive – Writing test cases is more predictive than fixing bugs – http://www.renaissancesoftware.net/blog/archives/16
• Productive – So, why use TDD? - arguably because you end up with half the defects but similar productivity.
– http://www.eventyr.co.uk/summary_of_points/
• Flexibility – Iterative process that can change with client’s ever changing requirements
– http://www.methodsandtools.com/archive/archive.php?id=20
• Malleability – TDD can lead to more modularized, flexible, and extensible ETL code
– http://en.wikipedia.org/wiki/Test-driven_development
• TDD makes development fun again. It is fun to see green. – http://www.adaptionsoft.com/tdd_how.html
• The collection of tests give a full regression test suite
Probably the
most
important.
22
Benefits with Pros and Cons
Cons:
• Overhead – Tests are coded and need to be maintained
• Limited – Unit versus Full functional tests
• False sense of quality – Still need to engage other quality control processes such as integration and compliance testing
– Source: http://en.wikipedia.org/wiki/Test-driven_development