Proprietary and Confidential
Continuous Integration using Hudson and Fitnesse
Norman Boccone | Lead Engineer | twitter: @dropslowlyJennifer Wong | Lead QE Engineer | twitter: @jenlwong
Vasu Durgavarjhula | Director of Development | twitter: @d_vasu
2Proprietary and Confidential
Overview
► Who we are ?► The Challenge► Tools and Processes► Test Strategy► Conclusion
Proprietary and Confidential 3
• We are a leading provider of information and analytics solutions for life science researchers
• Our goal is to help life science researchers get biological insights from their data quickly and reliably
• Our products are used by thousands of researchers worldwide
Ingenuity Systems
Partial Customer List
Tagline
Experimental Data Search
?
Publishing and Collaboration
Cancer
Enormous volume and complexity of biological
and chemical data…
Enormous volume and complexity of biological
and chemical data…
Hard to synthesize the picture of what that data
means in a broader biological context, and how the pieces work together…
Hard to synthesize the picture of what that data
means in a broader biological context, and how the pieces work together…
Traditional publishing models are good for
learning but not applying knowledge …
Traditional publishing models are good for
learning but not applying knowledge …
Toxicology Biomarkers PharmacogenomicsDiscovery
Use understanding of disease
mechanisms to identify and
validate targets
Elucidate biological mechanisms of drug action and
toxicity
Identify and prioritize novel biomarkers by
understanding role in disease pathways
Understand mechanisms behind
differential response to R(x)
Proprietary and Confidential 4
Scientists Need a Tool to Make Good Decisions From Complex Biological Data
Discover existing relationships and function, as well as inference of relationships that may not have been studied in a particular context yet
The Ingenuity® Knowledge Base Is the core technology behind all of Ingenuity’s products and services
THE INGENUITY KNOWLEDGE BASEANALYSIS
HYPOTHESIS GENERATION
VISUALIZATION
Generate testable hypotheses, build pathways, ability to inference
Get answers by asking detailed and in-depth scientific questions
ENABLES SCIENTIFIC UNDERSTANDING
ECOMMERCE ENABLEMENT
PATHWAY, REAGENT &
GENE SEARCH
► Structured - to capture relevant details
Scientific statements are modeled into Findings using the Ingenuity Ontology
► Expert Review Process - for accuracyFindings go through QC process
► Comprehensive - leverage knowledge in one placeLargest scientific knowledge base of its kind with modeled relationships between proteins, genes, complexes, cells, tissues, drugs, pathways and diseases
► Timely - for access to up-to-date knowledge Findings are added weekly
The Ingenuity® Knowledge Base How is it different?
THE INGENUITY KNOWLEDGE BASE
Proprietary and Confidential 7
The Challenge
Proprietary and Confidential 8
Single product
Proprietary and Confidential 9
multiple products
Ingenuity Product Search
Reagent Portals for BD, Sigma, Thermo Fischer
Ingenuity Variant Analysis
Next Generation Sequencing Analysis Reports
10
Proprietary and Confidential
Large monolithic code base
Proprietary and Confidential 11
Modular Services
UserManagement
ContentService
Master Graph
User Data
Views & Reports
iReport
Single Sign On
Proprietary and Confidential 12
Long release cycles
Proprietary and Confidential 13
Faster release cycles
Proprietary and Confidential 14
Continuous Integration
Speeding up the feedback loop
Stakeholders Developer
Proprietary and Confidential 15
Do all of this while supporting the existing product
Proprietary and Confidential 16
Approach• Move to a Service Oriented Architecture
– Don’t miss Jeff’s talk on Sunday 10:45 am “Learning to Love Services: Experiences from the Transition to a Service-oriented Architecture”
• Develop a test strategy focused on automated testing
• Build a continuous integration system• Adopt agile product management• Improve infrastructure to create and deploy
services• Foster a Dev/Ops culture
Proprietary and Confidential 17
Continuous Integration Workflow
Nightly Build
(Clover)
ApplicationBundle
Run JUnitTests
Hudson Dashboard(JUnit, Fitnesse summary, Code Coverage)
FitnesseBundle
SVN
Deploy Application
Deploy Fitnesse
Run Fitnesse Tests
(Nightly suite)
Fitnesse Wiki(Test history, Details,
Test Case Management)
Link
pu
blish
pu
blish
publish summary
Commit(Test Cases)
Proprietary and Confidential 18
Tools and Processes
Proprietary and Confidential 19
Service Oriented Architecture
Proprietary and Confidential 20
SOA: Before• lots of modules, separate but dependent
– easy to develop on but also easy to add circular dependencies
– hard to keep track of purpose/value of all the modules
• product consisted of bunches of modules put together– reference was typically to latest dev code– to ensure "latest," individual modules of a product
were rebuilt for every release– junit tests were run every time as well
• new product version meant build everything, test everything– builds were slow
Proprietary and Confidential 21
SOA: “Current”• components: module grouped together. Access
through api module only– less chance of circular dependencies– easier to comprehend modules– more flexibility to change code: just do not change
api• components built/tested separately
– faster builds– less repetitive testing
• some shared modules still exist (they are not part of any component)
Proprietary and Confidential 22
SOA: Conversion Notes• find a logical group of modules• Convert groupput group into hierarchical
structure, hide things behind an api layer• convert outside modules to use apis• see Jeff's talk
Proprietary and Confidential 23
Build System
Proprietary and Confidential 24
Build System: Before• in-house ant scripts
– lots of control but it required lots of resources– no real industry standard
• global scripts/property files, overrides allowed– offered consistency but (busy) people would
override instead of follow pattern– scripts got complicated
• anyone could add a new feature– if anyone wanted a new feature, just tell them to
add it– inefficient additions, different code styles made
scripts hard to read– very little dependency management
• multiple repositories for binary files
Proprietary and Confidential 25
Build System: “Current”• maven
– not much control, and very rigid (a good thing, in a way)
– steep learning curve– different from previous way of doing things– standard structure; difficult to not follow structure– hierarchical structure easily followed– versions for all dependencies well defined– easy to add new features (after a short burst of
intense agony)– only one repository
Proprietary and Confidential 26
Build System: Conversion Notes• get a maven repository manager (nexus? Turn on
search!)• add antrun plugin to publish artifacts to old
system• choose a simple component to start, make lots of
little submodules (1 module per artifact, plus assembly)
• manually add needed dependencies to repository• build up company parent pom, use hierarchy• Put all plugin management into parent• Key maven plugins: buildnumber, buildhelper,
release, assembly,
Proprietary and Confidential 27
Build System: TODO• cleaner release plugin• continue learning maven• contribute to plugins• Improve internal maven FAQ• Look into git
Proprietary and Confidential 28
Continuous Builds
Proprietary and Confidential 29
Continuous Builds: Before• Hudson (no plugins)
– auto build start for each module, with artifacts published
– manual build start for products– shell scripts to copy artifacts to release dir
Proprietary and Confidential 30
Continuous Builds: “Current”• Hudson
– dev master for auto build of modules/submodules, auto published to repository manager (dev area)
– release master for components/products, auto published to repository manager (release area)
– modules/submodules built on checkin– components/products built on schedule or on
demand– process includes various tests and deployments
(Jen)
Proprietary and Confidential 31
Continuous Builds: Conversion Notes• use Jenkins, not Hudson (more activity for
Jenkins)• startup new, clean servers
– copy/create job that we want• Try to make Jenkins be the portal to everything • fingerprinting• Plugins we use: clover, fitnesse, findbugs,
checkstyle, disk-usage, downstream build view, (custom) dashboard view, promoted builds, ssh slaves
Proprietary and Confidential 32
Continuous Builds: TODO• better job rollup• job templates• more maven/hudson plugins• submit hudson plugin(s) to community
Proprietary and Confidential 33
Deployment: Before• proprietary shell/perl scripts• targeted to install app only (not machine set up)• multiple proprietary property config scripts
Proprietary and Confidential 34
Deployment: “Current”• Puppet
– Configures machines and apps– Work on our machines and EC2
• Use of .erb files everywhere for property configuration
Proprietary and Confidential 35
Deployment: Conversion Notes• modify tar format• moved to one tomcat• used erb files• split properties into app
Proprietary and Confidential 36
Deployment: TODO• standard server configuration types (e.g. “build
machine”, “app machine”, …)• more puppet scripts
Proprietary and Confidential 37
Visibility
Proprietary and Confidential 38
Visibility: Before• cfmap (IT server diagnostic tool)• Hudson status page• FitNesse server
Proprietary and Confidential 39
Visibility: “Current”• cfmap• Hudson status page• Hudson dashboard• extra tests built in to Hudson• FitNesse server• Service status page
Proprietary and Confidential 40
Visibility: Conversion Notes• use Hudson plugins, with drill down• get everything to report in a standard way (junit
xml)
Proprietary and Confidential 41
Visibility: TODO• more widgets for the dashboard• Integrate outside reports into dashboard
– cfmap – status page– sdm? (service discovery manager)
Proprietary and Confidential 42
Test Strategy
Proprietary and Confidential 43
Testing For CI/CD
• Test Automation is key• How and what to automate
Cohn
Mezaros
Crispin
Proprietary and Confidential 44
Test Challenges
• Testability: some products are difficult to test at lower levels
• Legacy Apps: one of our main products was a ‘Legacy App’. Tests used to look like this:
Manual Tests
Silk Tests
Unit
Proprietary and Confidential 45
Test Challenges
• varying levels of coverage– some of our newer components have great
coverage– other components have lower coverage: legacy,
proof of concept
Proprietary and Confidential 46
Solutions
• Test and code refactoring• Legacy Apps: Strangulation approach
– automate new and refactored features– incremental work on tests: reserve time in each
iteration for adding to tests• Change in culture: team ownership of tests and
status• Cycle time for ipa was 2.5 weeks• Now for most of our services, it is 20-40 minutes
Proprietary and Confidential 47
Test Infrastructure and Environments
Proprietary and Confidential 48
Process: Build Promotion
• Build Promotion through Quality GatesDev
• Build succeeds
• Junit tests pass
auto
Continuous Integration (CI)
• Deployed to CI
• Automated
• Functional Tests Pass
auto
Stable
• Deployed to multiple nodes in Stable
auto
Beta
•Deployed to Beta
Proprietary and Confidential 49
Test Environments
CI Environment- Single nodes- functional tests
Stable Environment- Multiple nodes- Deployment Tests- System and performance tests- Manual/exploratory
Service1 Service2
Service3 Service4
Service5 Service6
Service1
Service2
Service3
Service4
Service5
Proprietary and Confidential 50
Test Types
Proprietary and Confidential 51
Test Types: Deployment and Health
• Each of our components has a built-in status page
Proprietary and Confidential 52
Test Types: Deployment and Health
• Status page– Reflects app status– Resource availability: DB Connections, Services
Proprietary and Confidential 53
Test Types: Deployment and Health
• Information is used for:– Deployment– Health Monitoring– Service Compatibility and Dependency
checking
Proprietary and Confidential 54
Test Types: FitNesse
• FitNesse is a wiki-based web server test tool– Helps abstract test definition from technical
implementation– Provides visual reporting and result history tracking
Proprietary and Confidential 55
Test Types: FitNesse
• The FitNesse Server – http://localhost:8080/FitNesse.UserGuide.TwoMinuteExample
http://localhost:8080/FitNesse.UserGuide.TwoMinuteExample?pageHistory
Wiki view Editing the wiki Test Results
Proprietary and Confidential 56
Test Types: FitNesse
• Integration with Hudson- Views, drilldown, configuration
- Test results converted to junit format
View of fitnesse test job in Hudson
Proprietary and Confidential 57
Drill downs on fitnesse test results in Hudson
Proprietary and Confidential 58
FitNesse Plugin Configuration
Convert results to Junit format
Proprietary and Confidential 59
Publish FitNesse Results and Coverage
Proprietary and Confidential 60
Test Types: FitNesse• What we use it for:
– Functional tests– Integration tests– UI Tests (com.jbergin.HtmlFixture)– DB Tests (dbfit)
• What we’ve done with it that is different– Use as execution framework for more complex tests – Extension of fitnesse server for data-driven tests– json fixture – pass in javascript– Execution of Selenium tests
Proprietary and Confidential 61
Test Types: FitNesse
• Lessons learned– FitNesse is good for straightforward verification of
data– To do more, you have to get creative– Fixture and test ownership needs to be a shared
responsibility
Proprietary and Confidential 62
Test Types: Selenium• We fire off Selenium tests in two ways: via
Hudson job or through FitNesse.• Test results converted to junit format for display in
hudson
Hudson job in dashboard:
Hudson job view:
Proprietary and Confidential 63
Test Types: Selenium
• Hudson job configuration: trigger execution on Selenium grid using Ant
Proprietary and Confidential 64
Test Types: Selenium
• Second way we fire off Selenium tests is using a FitNesse fixture: WebTest
Proprietary and Confidential 65
Test Types: Backward Compatibility
• No staging environment• How do we know that when we release a new
service version to prod that it won’t break?• Service version compatibility
– Leverage Service Discover Manager (SDM) and Status pages to check for availability of services
• Jar compatibility– static code analysis could detect changes in
method signature but not underlying object or serialization changes
Proprietary and Confidential 66
What’s Next: Getting to the End of the Road
Proprietary and Confidential 67
What’s Next
• What do we need to do to get to the end of the path to Continuous Integration and Deployment?
• Continue to invest in standardized tools and processes– Maven– Puppet– Jenkins– Metrics and Immune Systems
• Continue to improve our test coverage and practices
• Culture:– Continue to support, reinforce, educate
Proprietary and Confidential 68
The Finish Line
• Continuous deployment• Reduced Cycle Time• Improved quality• Predictable metrics• Reduced risk
Proprietary and Confidential 69
• Slides are posted here: http://www.slideshare.net/jenlwong/ingenuity-svcc-ci-presentation-20111007
• Plugs– Learning to Love Services: Experiences from the
Transition to a Service-oriented Architecture• Speaker: Jeff Green 10:45 AM Sunday
– Ingenuity is hiring: http://www.ingenuity.com/company/careers.html
• Q&A• References
– Todd. Test Automation Pyramid – review. Retrieved September 29, 2011 from: http://blog.goneopen.com/2010/08/test-automation-pyramid-review/
– Humble, Jez, and Farley, David. Continuous Delivery. Boston: Pearson Education Inc, 2011. Print.– Crispin, Lisa, and Gregory, Janet. Agile Testing: A Practical Guide for Testers and Agile Teams. Boston:
Pearson Education Inc, 2011. Print.