35
1 How to Juggle Multiple Beta Tests at Once
| Date post: | 07-Jan-2017 |
| Category: |
Technology |
| Upload: | centercode |
| View: | 26 times |
| Download: | 1 times |
1
How to Juggle Multiple Beta Tests at Once
Meet the Speaker
2
Austin Meyer
Managed Beta Director
Agenda
1. Introductions
2. The Pitfalls of Juggling Multiple Betas
3. How Do We Juggle Multiple Tests at Once?
4. Clearly Defined Roles & Test Stages
5. Time Management & Workload Balancing
6. Recruiting the Right Testers
7. Righting the Ship
8. Q&A
3
4
Who is Centercode?
Services Fully managed beta tests and programs
Software Complete beta program management platform
Testers Access to 150k+ deeply profiled global candidates
Content Free tools and best practices
We've been running beta tests since 2001
Beta test management solutions are our core competency
5
Beta Testing =Customer Acceptance
Targeted customers using your product in real environments
Increases quality, improves user experience, validates performance
Customer Validation: CAT, UAT, FUT, Customer Evaluation, Field Trials, Pre-release
6
The Pitfalls of Juggling Multiple Betas
Lack of Project Plan
Underestimating Workload
Delays
Recruiting Bad Testers
Poor Resource Management
Frustrated Testers
Low Tester Participation
Bad Data
Delayed Product Release
Negative Product/Company Perception
7
The Value of Getting It Right
High participation rates
Unique, clear, actionable data
Successful product launch
Reputation for quality products
8
How Does Centercode Do It?
Clearly defined roles with unique responsibilities
Clearly defined test schedule with four stages
Effective time management & workload balancing
Recruiting the right testers
Righting the ship
9
Clearly Defined Roles
10
Test Manager - Client lead, responsible for success of the project
Tester Lead - Testers’ main POC, responsible for support and managing Ongoing Feedback
Community Manager - Responsible for finding and keeping the best testers
Program Manager - Program lead, responsible for workload management
Roles - Defined
11
Centercode Managed Test Team Structure
Program Manager
Community Manager
Test Manager Test Manager
Tester Lead Tester Lead
Pod 1 Pod 2
12
Project Preparation - Coordinating all pre-test activities
Platform Expert - Setting up and managing the Centercode Platform
Client Contact - Expert resource for the client
Conduct Research - Preparing, executing, and analyzing research
Project Management - Utilizing PM skills to ensure success of the project
Roles - Test Manager
13
Tester Support - Acting as Tier 1 support for testers
Feedback Filtering - Ensuring feedback is prioritized, unique, clear, and actionable
Tester Compliance - Managing participation expectations with testers
Project Logistics - Shipping and receiving beta product
Incentive Management - Distributing “Thank-You” gifts to testers
Roles - Tester Lead
14
Community Manager
Managing a community of potential testers
Developing Recruitment Plan & strategy
Program Manager
Assigning projects to the appropriate resource
Issue escalation
Program vision & direction
Scaling Roles - Community & Program Manager
15
Clearly Defined Test Stages
16
Planning - Developing a strategy for your beta, ensuring you have the right resources
Preparation (Prep) - Getting ready for your beta to start, logistics, set-up, recruitment
Test - Managing testers and client, collecting data, compliance
Closure - Aggregating and analyzing data, incentive distribution
Stages - Defined
17
More testers or longer tests ≠ better results
1 week = 1 activity
Consider your team’s schedule
Always have a plan
Assign a codename to each project
Plan for delays!
Stages - Planning
18
Don’t underestimate the work that needs to be done during prep!
Set up as much as you can beforehand
Develop a curated, targeted community of testers
Shipping logistics are usually a time sink
Plan for delays!
Stages - Prep
19
Prioritize work appropriately
Be ready to pivot
Block time to work on feedback filtering
Establish report cadence with client
Plan for delays!
Stages - Test
20
Report continuously during the Test period
Plan time for cumulative analysis
Ship devices with return shipping labels / software time bomb
Allocate budget ahead of time for thank-you gifts
Set a meeting for a “Post-mortem” to improve processes for future
Stages - Closure
21
Effective Time Management & Balancing Workload
22
Visual representation of workload
Serves as a meeting place
Identify areas of overlap & heavy workload
Lego allow for flexibility & delays
Project Board
23
Centercode Project Board
24
“Lay out the timelines for each of your beta tests. You will be able to check and see when you need to send out surveys to your testers for each of your projects, when you need to follow up with them to ensure participation, etc.” - Doug Sorensen, Sr. Beta Manager
“If the schedule allows, plan to start your beta tests on Monday. Avoid Fridays as testers may run into issues during the weekend where support may not be available, and the feedback may be overwhelming on Monday.” - Rebecca Lam, Beta Manager
“Set aside some time during the day where you are in do not disturb mode. Limiting interruptions can help you to knock out a project much more quickly.” - Mike Fine, Sr. Beta Manager
“Leverage work you have done with other projects. If you have a great survey designed for one project, you can often repurpose it for other projects.” - Michelle Williams, Beta Manager
Test Manager - Tips
25
“Make a to-do list for the next day right before leaving work for the day prioritizing from Urgent to Important. Review to-do list first thing in the morning and adjust when necessary. And commit to it through the day.”
- Diego Gomez, Tester Lead
“The emails you send to testers at the beginning of a project are extremely important. Be sure to be as specific as possible and try to explain instructions in a way that anyone could understand them. Have someone who isn’t familiar with the project read your instructions and see if they have any questions or if anything isn’t clear. The fewer emails you receive from confused testers the more time you save.”
- Omid Karger, Tester Lead
“Adjust titles in feature request and bug reports to make it as easy as possible for other testers to find. This will help you avoid having to close and respond to duplicates.” - T.J. McCabe, Tester Lead
“Dedicate chunks of time to one thing and try not to jump back and forth between filtering feedback, responding to emails, or any other task. You’ll be more efficient by focusing on one thing at a time.”
- Matt Kapitan, Tester Lead
Tester Lead - Tips
26
Recruiting the Right Testers
27
Having the right testers reduces the work during the test period
Look for people who are:
● Strangers to your product (if possible avoid using family/friends/co-workers)● Volunteers (people who are “volun-told” typically don’t perform well)● Part of your target market● Enthusiastic & passionate about your product and about testing in general
Treat testers with respect - remember they’re volunteering to help you!
Recruiting the Right Testers
28
Righting the Shipor How to Salvage a Test Headed South
29
Extreme delays
Limited test time
High touch client
Frustrated testers
Little or no participation
Risks & Red Flags
30
Extreme Delays
● Keep testers informed! Don’t go 2 weeks without letting them know where things stand.● Give them the opportunity to bow out● Reset expectations with client
Limited Test Time
● Communicate the short test duration to your testers● When possible, recruit proven testers● Do your best to stay on schedule
Altering Course
31
High Touch Client
● Outline and set expectations ahead of time● Schedule frequent check-ins● Be proactive vs. reactive● Communicate reasonable timelines
Frustrated Testers
● Communicate more frequently● Identify areas of confusion● Fall on your sword● Thank & encourage your testers regularly
Altering Course (cont.)
32
Little or No Participation
● Reach out individually via phone or email● Experiment with additional incentive strategies (raffle for weekly compliance, compounding
incentive, etc.)● Are your surveys too long? Are you asking too much of your testers?● Remind testers about the incentive
Altering Course (cont.)
33
Review
Clearly defined roles with unique responsibilities
Clearly defined test schedule with four stages
Effective time management & workload balancing
Recruiting the right testers
Righting the ship
34
Q&A
Austin Meyer
Managed Beta Director @ Centercode
35
Thank You!
