TEST AUTOMATIONA PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
2
TEST AUTOMATION A PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONSThe time has finally come for healthcare test automation. In fact, a strong
case can be made that test automation is a must-have asset in today’s
healthcare environment. Simply put, best practices in IT system testing have
outpaced almost any organization’s ability to keep up with largely manual
methods. The proof of this reality can be found in the very difficult (and risky)
choices organizations often make when laying out a testing plan for an EHR
implementation:
• What will be tested (and what must be ignored)?
• How often can we test (what is enough or too little)?
• What should we test in the ecosystem outside of our EHR?
Fortunately, test automation has come a long way over the past few years,
making major strides in the areas of deployment and proven effectiveness.
There are many areas of consideration healthcare organizations need
to evaluate when establishing a test automation strategy. The following
addresses many of these major factors.
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
3
PRIMARY ADVANTAGES OF TEST AUTOMATION
Test automation is the gift that keeps on giving. One of the most time and resource-intensive
activities in an EHR implementation – or any major IT system such as an ERP or revenue cycle
solution – is the extensive manual testing required before, during and after go-live.
EHR implementations would be far more successful if not for the manual testing needed before, during and after go-live.
The routine is all too familiar. A large group of end users spend multiple days testing keyboard
functions in a classroom setting. The problems identified are fixed, and the cycle starts over
again.
Test automation changes the process, team composition, capacity to test and the monotony
of the effort. A proper test automation program, supported by the necessary tools, allows
healthcare organizations to streamline and strengthen test processes, while providing
tangible advantages and improved employee morale. Simply put: an improved EHR project.
��� The primary benefits of test automation include:
• Greater Test Accuracy/Quality: Even the best manual testers make mistakes. Automation reduces errors while increasing confidence in testing metrics and results.
• Expanded Scope and Test Capacity: The volume of test workflows and iterations/scripts conducted increases significantly with automation and dynamic scripting. This allows for greater capacity and far more departments and functions to be included within the testing scope.
• Reduced Time and Effort: Not only does automation support a greater volume of tests, but tests can be completed in a far shorter period of time as critical tests can be repeated via full regression testing on a daily basis. In the end, you are not performing the actual test but simply validating the error results and repairing the problems identified.
• Cost Savings: Automation means your staff spends far less time away from their primary duties executing test scripts, which decreases backfill and other employee-related costs. Not to mention the significant increase in test capacity that automation provides.
• Decreased Risk: Improper testing, or testing shortcuts, can allow small yet critical bugs and problems to slip through into a live patient care setting. Many of these can create significant patient and financial impact if left undetected or remedied.
• Higher Satisfaction: Test automation results in fewer bugs and challenges discovered ‘the hard way’ by end users in the production environment, which, in turn, minimizes end user (including physicians and clinicians) frustration while maximizing satisfaction and adoption.
• Sustainable Value: While the benefits are considerable during initial EHR implementation, they exponentially increase over time as vendors move toward quarterly (or more frequent) upgrades and the library of tailored test automation scripts constantly grows.
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
4
KEY STAFFING ASSUMPTIONS
Cost per FTE (Annual and Fully Burdened) for IT/Business/Clinical Test $115,000
Work Hours per Year 2,080
Hourly Fully Burdened FTE Cost $58.20
ANNUAL TEST CYCLE DESCRIPTION
Number of Test Cycles per Year 3
Typical Test Cycle Duration (Weeks) 10
AVERAGE TEST CYCLE DESCRIPTION
Total FTEs Available to Execute Testing 16
Percent FTEs Assigned to Testing + Error Validation (To be Automated)* 60%
Equivalent FTE’s Assigned to the Test Cycle 9.6
Percent Work Assigned to Resources Dedicate to Testing 70%
Total Hours Devoted to Testing per Test Cycle 2,688
Total Cost per Test Cycle $148,620
AVERAGE TEST PRODUCTIVITY
Number of Scripts Run (Current) per Test Cycle 100
Average Cost per Script During Test Cycle $1,486
ESTIMATED COSTS
Total Annual Cost under Standard Manual $445,859
3-year cost $1,337,576
CURRENT APPROACH COST YEAR 1 YEAR 2 YEAR 3 TOTAL
Total Number of Scripts Run 300 300 300 900
Estimated Annual Cost $445,859 $445,859 $445,859 $1,337,576
EST. TEST AUTOMATION COSTS YEAR 1 YEAR 2 YEAR 3 TOTAL
Total Scripts Run 15,600 15,600 15,600 46,800
Initial Custom Development + Annual Fee $150,000 $100,000 $100,000 $350,000
One-Time Service Fee $50,000 $50,000
Total $200,000 $100,000 $100,000 $400,000
ANTICIPATED SAVINGS YEAR 1 YEAR 2 YEAR 3 TOTAL
Annual and 3-Year $245,859 $345,859 $345,859 $937,576
Average Annual Savings $312,525
Estimate Current (Standard Manual) Costs
Test Automation - Financial Comparison
THE FINANCIAL IMPACT OF TEST AUTOMATION
*Simplified Example for approximately 300-bed hospital
A Comparison of a Standard Manual Test Effort vs. Full Test Automation
*Testing and validation activities to be automated
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
5
THE RETURN ON INVESTMENT (ROI) OF TEST AUTOMATION
As with any investment, it is important to have a clear understanding of where and how test automation can deliver
the highest ROI. Healthcare organizations can realize significant returns from relatively modest investments in
test automation within two primary areas.
Regression TestingAs implied above, test automation generates the greatest ROI during repeated regression testing that can be leveraged for each and every EHR update or upgrade, as well as, other applications that intersect with the EHR in your ecosystem. Many vendors are rapidly moving toward releasing new versions on a quarterly (or even on-demand) basis. As such, regression testing becomes even more critical for ensuring EHR usability and safety. In addition, and as the technology ecosystem expands in healthcare, testing your EHR quickly, after additional systems have been introduced to the ecosystem, has exponential value.
Automation allows regression tests to be completed reliably in a few hours, rather than over several weeks or even months if done manually. This efficiency is essential to comprehensive testing of regular software updates and new releases. It doesn’t take long for the cost savings of regression testing to add up to a significant ROI for test automation. In fact, test automation may deliver a one-month ROI if a health system tested everything they would like to with their existing resources or budgets.
Operational EfficienciesHealthcare organizations also reap sizable and tangible ROI results from test automation by allowing staff testers to remain focused on their primary duties and the many needs that surround an EHR implementation. Automation allows them to spend more time addressing/fixing test errors versus merely finding them. However this additional time is used, giving your staff time back in their day results in more efficient workflows and lower operational (and project) costs overall.
These two areas are the most common areas to yield a financial ROI from an investment in test automation. However, for many organizations, the non-financial key advantages cited above are often as important (or more important) than an ROI alone.
Test automation by no means eliminates all manual testing. Instead, a sound testing plan employs test automation where it
has the greatest value and reserves manual testing to a select subset of activities.
FINANCIAL ADVANTAGES CURRENT TEST AUTO IMPACT SUMMARY
3-Year Estimated Cost $1,337,576 $400,000 -$937,576 30% of current
Estimated Cost per Script Run $1, 468 $9 -$1,555 0.6% of current
Approximate B/E 9.4 months
TEST SCOPE & CAPACITY CURRENT TEST AUTO IMPACT SUMMARY
Testing Cycles per Year 3 52 +49 17.3 times more
Scripts Run per Cycle 100 300 +200 3 times more
Scripts per Year 300 15,600 14,300 52 times more
Weeks per Cycle 10 1 -9 10% of current
Beneficial Impacts
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
6
User & DataValidation
FOUNDATION
Validate WorkflowCreation
(Function/Module)
CORE
“End-to-End”Integrated Scripts
IMPROBABLE
User Security & System Access
ImmediateValidation Points(Typically Multiple Steps)
•
•
Automating the Registration Workflow
Positive or NegativeTests Possible
•
•
Data Passage between SystemsApplication or Ecosystem
•
•
01
02
03
PHASES OF IMPLEMENTATIONA proper approach to test automation implementation first
validates foundational “low hanging fruit” before converting manual scripts to E2E automation.
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
7
TEST PLANNING: THE 80�20 RULE
When deploying test automation, one of the earliest decisions to make is which test activities
are best suited for automation. It is rare for a healthcare organization to automate all
testing as certain activities require personal judgment to determine the best combination
of possible test scenarios. The following guidelines have proven helpful for organizations
preparing initial testing plans.
What to Automate
Generally, automation should cover 80% of all testing activities in an EHR implementation or upgrade, and virtually all activities requiring keyboard or mouse functions.
For example, patient registration is a prime candidate for automated testing. A manual test for registering a patient, with minimal dynamic data, takes approximately eight to twelve minutes to complete. By comparison, an automated test for the same task can be completed nightly in a regressive model in two to three minutes. Couple that with the use of dynamic data and the differences in effort is massive.
Additionally, all orders (medication, labs, radiology, etc.) should be completed via test automation. Doing so can mean starting the test in the EHR, moving into a downstream system to record the result from the order, and going back into the EHR to validate that the result returned as expected. This eliminates the need for experts to be “on-call” and prevents delays in manual script completion.
Beyond registration and orders, patient scheduling and claims processing should also be considered for automation, as the entire end-to-end patient experience is covered.
What Not to AutomateThe remaining 20% of EHR testing activities still should be completed manually when an organization is creating an initial test automation plan. This is considered “single source testing” and covers tests against a specific build that would not be used with each subsequent release. These are random, unexpected tasks, and there is not much benefit to automating their process. It is better to focus automation efforts on the 80% of tasks that are repeatable and will deliver sustainable value through repeatability during implementation and future upgrades.
Again, these guidelines have proven useful to healthcare organizations deploying test automation on a broad scale for the first time. Organizations with test automation experience, bolstered by the steadily growing library of tailored scripts, may tackle increasingly challenging testing through automation and move the 80/20 rule toward 90/10, and beyond. They do so, however, armed with first-hand lessons learned.
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
8
LESSONS LEARNED IN TEST AUTOMATION
While no two situations are exactly alike, there are some time-proven lessons learned
beyond understanding what to automate (and what not to) in getting maximum value and
benefit from test automation. These include:
1. Tool vs. Talent: A good test automation tool is important but not as important as the experience and skills of the resources wielding the tool. Knowing what to automate, how best to do so, when to run automation and what to do when things go wrong is critical. No tool can make those pivotal decisions.
2. Timeline Expectations: Test automation saves time, but it takes time for the first deployment. The exact timeline depends on the scope of automation, and third-party involvement, but, in general, testing begins within a month or so after the test automation system is installed, and test patients and end user workflows have been set up. An organization can then expect to see results very quickly.
The entire test automation process—including running automation scripts for 80% of EHR tasks—can be completed over a three- to six-month time period.
3. Testing Design: Automation without good test design may lead to a lot of activity but little value. The primary role of test automation specialists is to not only to establish a solid design but to understand when and how that design can be enhanced over time to fine-tune results and expand scope. Deriving increasing value from test automation, over time, is a central and critical part of its ongoing upkeep.
4. Staying Current: Speaking of upkeep, it is imperative that test automation design and scripts keep up with the continual updates and releases of the EHR (or whatever system is being tested). Changes to the EHR require at a minimum a careful review to determine the extent to which related scripts must be changed. This rather obvious lesson is worth pointing out because over time testing results (automated or manual) become unreliable and even misleading.
These proven and basic lessons learned in test automation can be valuable as a starting point to develop your program. There obviously are many, many more that may be highly relevant to an individual organization’s environment and situation. In the end, it pays to tap into as much available test experience as possible.
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
9
THE CASE FOR EXPERIENCED HELP
Many healthcare organizations can and will deploy test automation on their own. They
may have the knowledge and resources to do so. Other organizations, however, have found
partnering with a highly experienced partner to pay great dividends. A capable partner will
help prepare an appropriate test automation plan, integrate test automation software with
the EHR(s), automate an entire library of tailored testing scripts, and run the initial wave of
automated tests, if needed. When deciding on the need for experienced, outside help, most
healthcare organizations consider several pivotal factors:
• System Expertise: By working with an experienced partner, healthcare organizations tap into their consultants’ deep system expertise. Consultants have certifications with the EHRs being tested and are knowledgeable of the exact workflow changes required during updates and upgrades. They can provide recommendations around regression testing and help a testing team prioritize scripts for automation.
• Cost Savings: A partner completes automated tests in a far shorter period of time than a hospital’s own testing team, which in turn shortens the overall EHR implementation or upgrade timeline. Less time and fewer internal resources needed for an implementation project results in cost savings for the entire organization, as described above.
• Scripts Library: Most organizations with common EHRs benefit tremendously from the extensive library of ‘starting point’ test scripts that outside help can provide. Many of these scripts can be applied with minimal, if any, modifications. Reinventing the wheel is a waste of time and money, and the library dramatically shortens time to generate ROI.
• Test Coverage: Organizations without test automation often find it very difficult, if not impossible, to determine with certainty whether they’ve covered 100% or 50% of the most common test cases. They don’t know what they don’t know. A proven and professional test partner has the tools and knowledge to accurately determine test coverage.
• Staff Availability: Most organizations simply do not have the staff required in the near-term to successfully deploy test automation, especially against tight timelines. A partner augments IT and clinical teams, so a hospital can meet implementation deadlines and realize results sooner. Employee morale will also increase when team members can stay focused on their daily jobs without testing interruptions.
TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.
10
About Santa Rosa ConsultingSanta Rosa Consulting provides management advisory services and technical consulting across the spectrum of vendor and IT products and systems. We empower better healthcare by designing, creating, staffing and sustaining IT solutions that deliver quantified value. Founded in 2008, Santa Rosa Consulting is managed by industry veterans with an average of 20 years of healthcare information technology experience. Santa Rosa is recognized for delivering world-class services and solutions and has received Modern Healthcare’s Best Places to Work in Healthcare award multiple years in a row.
Visit: santarosaconsulting.com
SUMMARY
Test automation is a true game changer for implementations and
upgrades of major IT systems such as EHRs. The days of making difficult
and risky decisions on testing are over as test automation delivers
tangible advantages while positioning a healthcare organization to have
far greater confidence in, and satisfaction with, new systems.
Santa Rosa Consulting has more experience leading test automation
than any other firm and proven success in helping clients save time and
improve project efficiency. Our team understands the ins and outs of test
automation software, and we know how to seamlessly integrate it with
any EHR on the market. We bring significant experience and expertise,
along with a growing library of test automation scripts, to your table.
Through test automation services, we’re able to provide a proactive, ROI-
driven solution that meets a need all healthcare organizations will face
during EHR implementations.
THE STRENGTH YOU NEED IN EVERY STEP OF THE HEALTHCARE IT JOURNEY