INTRODUCTIONMake Idea Evaluation More Objective with Idea Screeners and Scorecards
A recent Booz & Company study2 showed that being highly effective at idea conversion is more important to financial success than being highly skilled at generating ideas.
Only 36 percent of participants said their efforts to convert ideas to product development projects were highly effective.
The accompanying Excel file (worksheet_screener_scorecard.xls) contains the customizable idea screener and scorecard that you can use in your organization. Here, we give all the instructions you need to use them effectively.
The reality in every organization is that innovation comes down to money—or more accurately—the lack of it. Businesses do not have a limitless pot of money and must therefore be extremely smart about spending the right way.
Unfortunately, organizations don’t seem to do this well. As many as 88.6 percent of companies spread their resources across too many development projects1. In this hyper-competitive marketplace, businesses cannot afford to spend money poorly.
Focus on the best and weed out the rest. To make sure you are investing effectively, you must evaluate ideas and projects in a fair and objective manner. The idea screener and scorecard is a suite of evaluation tools that will help you weed out the weakest ideas and identify the strongest ones.
Eliminate the maybes. Companies without evaluation tools often say “yes” to more projects than they can afford, and fail to exercise appropriate caution. They become reluctant to drop projects and rely on instinct and emotion to evaluate them. With an idea screener and scorecard, you have a basis to stop investing in ideas that don’t hold up to a certain standard.
Define what makes a good idea. Creating your idea screener and scorecard will force your organization to define the criteria that make an idea worthy of further investigation. When these tools are shared with everyone, it will make your innovation efforts more transparent and purposeful. People who are generating ideas or working on innovation projects will have a clear understanding to shape their thinking. They’ll also know why their ideas get rejected or approved.
“The process of choosing which ideas to convert to full-scale product development is perhaps even more critical to a company’s innovation success than is the ideation stage.”— Booz & Co3.
Within the Larger Process ContextIncorporating These Two Essential Tools into Your Idea Evaluation Process
The screener and scorecard are critical support tools as part of your idea evaluation process. They help gauge potential, mitigate risk, gain buy-in, and provide objectivity to an otherwise subjective process.
The screener and scorecard are comprised of a few, fundamental criteria that are important to your organization (for example, strategic alignment, brand fit, and feasibility). A team assigned to evaluate ideas will use the tools to determine if innovation ideas/projects meet these criteria to make objective go/no-go decisions based on the results. Use both tools for more comprehensive evaluations. The screener and scorecard complement each other and we recommend that you use both in your evaluation process.
> Evaluation #1: Idea Screener At a high level, the Screener should be used early in your innovation process. It’s used to see if a raw idea (or batch of ideas) meets enough strategic criteria to warrant further investigation.
> Evaluation #2: Idea Scorecard The scorecard is built on the same criteria as the Screener, but contains more rigorous questions. It’s used to evaluate projects later in your process—these are the ideas that passed the Screener and have had a certain amount of analysis and investments put behind them. It will help you make more “serious” decisions about the projects (for example, making more investments, assigning greater resources, going to market).
Within the Larger Process Context (continued)Incorporating These Two Essential Tools Into Your Idea Evaluation Process
Quickly weed out ideas that are weak or unnecessary.
Weigh the benefits of good vs. mediocre projects.
Evaluate projects that you’ve put some thinking/resources/investments behind to see if there’s a real business case or “go-to-market” strategy. For example, these projects might have an assigned team that has worked on a market study, business plan, or prototype.
Used by a group of “scorers” (those responsible for go/no-go decisions on projects). The scorers are typically cross-functional, higher-level resources.
…you’re evaluating more detailed projects and need to juggle investments between them. A scorecard will help you weigh the relative advantages between the projects and determine where more significant resources can be best spent.
Evaluate raw ideas (for example, ideas generated after a brainstorm) in early stages of consideration.
Typically used by an individual screener (the resource that handles your innovation pipeline). The screener is usually a lower-level resource.
…you’re expecting a large number of raw ideas being generated in your company. A Screener can filter many ideas and focus your initial resources on a smaller, more qualified list of ideas.
Building Your Evaluation ToolsHow to Select Your Evaluation Criteria
Screener Question: Scorecard Question:
Strategic Fit Does this idea fit within our current innovation strategy?
How well does this project fit within our current innovation strategy?
Feasibility Does this idea have a reasonable likelihood of success in the marketplace?
How likely is it that the project will be successful in the marketplace?
The criteria you choose become the cornerstones of both tools, so selecting your criteria is an extremely important exercise. The following pages offer tips for crafting your screener and scorecard.
In order to make evaluation consistent, you should use the same criteria in both the idea screener and scorecard. This ensures that you’re judging ideas and projects using a common set of standards throughout your process.
The difference is the level of detail within each tool. In the Screener, the criteria are framed as high-level
“Yes/No” questions. You simply add up the number of YES and NO answers to quickly ascertain whether an idea has potential. In the scorecard, the criteria go a level deeper. Questions are framed such that projects are rated on a 1–5 scale. Ultimately, you’ll arrive at a total numerical score. With these scores, you’ll be able to weigh the relative benefits of different projects.
See how the same criteria are framed differently across the two tools.
Develop the screener and scorecard in parallel. It’s important to make evaluation consistent. Make sure you’re thinking of both your screener and scorecard as you finalize your criteria. You should use similar criteria for both tools.
Building Your Evaluation Tools (continued)How to Select Your Evaluation Criteria
1
2
Criteria What it means
1. Strategic Alignment Does this idea fit your current strategy? Will it help meet your goals?
2. Feasibility Is it technically feasible?
3. Customer Need Does it meet a customer need?
4. Brand Fit Does it support/expand your brand strategy/positioning?
5. Markets/Revenue Does it have an attractive revenue/market potential?
6. Competitive Advantage Will it give you the ability to stay ahead of competitors?
7. Capabilities Do you have the capabilities (time/money/people/R&D) to implement this?
8. Innovativeness Is it breakthrough/unique/one-of-a-kind?
Consider these eight fundamentals. We provide eight basic criteria in the Excel worksheets. Use them as a starting point as you develop your own. Your goal is to choose the most important ones based on your innovation strategy, and refine them for your organization. The eight criteria are:
Stick to a few. Every organization has unique needs and demands. Those with
higher appetites for risk tend to have fewer criteria, while more risk-averse businesses (or those with high-budget decisions to make) use a longer list. A good rule-of-thumb is to have about four to six criteria. This will ensure a more holistic evaluation without “handcuffing” innovation.
Inspire; don’t debilitate teams. Remember, your criteria have a
“psychology” associated with them. Present them in a way that enables teams, rather than restricts them. For example, you might brief a team by telling them that the criteria will make their projects better. Present them as challenges, not obstacles.
Get a jump-start on your criteria. You might not need to start from scratch
to develop your criteria. Ask your team some of these questions:
> Do you have a definition of innovation that is being used currently? Can evaluation criteria be culled from it?
> How have you made choices about go/no-go decisions in the past? Even if it was informal, what have been the biggest issues that weigh on your minds?
> What are the biggest “deal- breakers” when you’re thinking about implementing projects?
Building Your Evaluation Tools (continued)How to Select Your Evaluation Criteria
Building Your Evaluation Tools (continued)How to Select Your Evaluation Criteria
1. Strategic fit. Does the idea meet the company’s objectives and strengths?
2. Market attractiveness. What’s the size of the pie?
3. Technical feasibility. Can this idea be implemented?
4. Supply and entry point. How do we supply this new innovation?
5. Competitive advantage. Will this idea keep us ahead of competition?
6. Macro-environmental aspects. Does the idea answer legal/public policy/safety/health/environmental issues?
7. Financial attractiveness. What’s the potential gain for the company?
8. Killer variables. Are there specific events, changes in market conditions, or new technologies that could dramatically alter the situation?9. Plan to proceed. Is there a plan for how the project will move
forward?
Exxon-Mobil
1. Size of the pie. What is the total value created, and how much of that value is captured by Shell?
2. Why Shell? Do we have the competencies to implement it? Does it fit within our strategic objectives?
3. Sustainability. Is it robust enough to adapt to a changing world? What is the environmental impact? Does it fit society’s expectations?
4. Strategic control. Can we protect the business opportunity?
5. How much does it change the game? Does it have a unique value proposition? Is it revolutionary?
6. Go-forward plan. Can we assign it resources/leadership team to implement?
Shell
Leading innovators use consistent criteria to evaluate their projects. See how organizations have tweaked the eight criteria to suit their needs.
The idea screener helps you weed out the weakest ideas at the beginning of your evaluation process. It helps you quickly answer the question “Is this idea even worth investigating?”
Use the accompanying Excel file (worksheet_screener_scorecard.xls) to customize the Screener for your organization.
InstructionsHow to Customize and Use the Idea Screener
1. Add specific instructions to ensure consistency. For example, “Ideas have to meet X number of criteria to move to next phase.” You could also refer to specific materials/ guidelines posted on your Intranet.
2. Make ideas trackable. You need a unique identifier as an idea moves through your evaluation process. It will help you track progress of each idea and compile metrics, like the number of ideas submitted (perhaps, by region/department). Log all ideas (name, number, status) in a central pipeline.
3. Not sure if it’s a YES or NO? Sometimes, you may not have an easy YES/NO answer. It can be a judgment call and we recommend that you give every idea a fair chance. If it’s not possible to commit to a single answer, give it a “YES” and indicate your reservations in the notes column. These notes can help build more robust concepts in the next stage of the process.
4. Make a decision. Ensure that screeners put their decisions on the Screener. We recommend three types of
decisions: > GO: Move idea to next phase.> HOLD: Keep idea on back-burner
until next review (because it might not be the right time).
> STOP: File the idea away—not to be reviewed again.
5. Define clear next steps. If an idea is approved, it should be assigned an owner who will investigate it further.
Assign a specific resource. It’s most effective to have a single individual in charge of your screening process. This individual should be responsible for maintaining your innovation pipeline and managing the screening process.
Decide how many “YES” answers you require. There are a few approaches that organizations use to decide when ideas should get a “GO” decision. Depending on your needs, you have a few options:
Ideas get a “GO” decision when...
> Every criterion gets a “YES.” This is the ideal—yet rigid—approach. If your organization has too many ideas and too few resources, this approach ensures that fewer ideas will bubble up your pipeline.
> A significant percentage of criteria get a “YES” (for example, four out of six criteria). This is a more flexible approach in which you’ll be investigating a higher number of potential ideas. This is a better option if you’re just beginning your innovation efforts or you can afford to be more exploratory.
InstructionsHow to Use the Idea Screener More Effectively
Develop a screening process. It’s important to have your resource (see
Step 1) manage a consistent screening process:
> Review the idea pipeline on a periodic basis. For example, you might choose to screen every quarter/month (depending on the number of ideas you’re receiving) to keep the pipeline flowing smoothly.
> Filter out the most obvious ideas. The individual resource will use the idea screener to weed out unnecessary ideas. The screener should be trained to identify
“bad eggs.” These are ideas that you do not want to waste time on.
> Organize a group screening (if needed) for ideas with potential. Ideas that cannot easily be shelved might need a larger set of minds to weigh in. This group could include a more senior and cross-functional team. At this meeting, let the group review each idea. They should understand why the fate of this idea wasn’t easily decided. Notes from your screening
resource can be valuable. To make a decision, you could:
Let a majority of votes win. Conduct individual YES/NO
“votes,” where the majority votes decide the fate of an idea.
Conduct a joint vote. Have a group discussion, and then conduct a joint vote on whether the idea should move ahead.
> File away the killed ideas. Ideas that are weeded out must be filed away. Their status should be noted on your pipeline. Also communicate to the people who submitted ideas.
Make background information accessible. With criteria like
“Strategic Alignment,” “Brand Fit,” or “Markets/Revenues,” make sure screeners have the information necessary to mark a YES/NO. An online bank of background information (for example, your innovation strategy, a background on the markets your organization serves, brand strategy/positioning, etc.) will be extremely useful.
InstructionsHow to Use the Idea Screener More Effectively
3 4 5Create consistency with how your ideas are submitted. Consider using a
standard idea submission form in your organization. It can help you collect ideas in a consistent and systematic format. This way, you can compare apples to apples when making screening decisions.
In this example, the organization selected five criteria for evaluation. Here, the “Flexible (rubber-based) credit card” idea was given a “GO” decision.
While the screener is used to evaluate raw ideas, a scorecard is used to evaluate projects—those ideas that have passed the Screener, have assigned team leads, and have some thinking and resources behind them.
The scorecard will help you weigh the relative advantages between projects and make more involved decisions (for example, making more investments, assigning greater resources, going to market). Projects that receive higher scores at this phase get “GO” decisions and move forward to more serious development.
Use the accompanying Excel file (worksheet_screener_scorecard.xls) to customize the scorecard for your organization.
InstructionsHow to Customize and Use the Idea Scorecard
1. Move from YES/NO to a sliding scale. In an idea scorecard, you use the same criteria as the Screener to ensure consistency. While the screener uses “YES/NO” questions, the scorecard questions are rated on a 1–5 scale. Add up the criteria scores to get the total.
2. Frame your questions carefully. Questions should be framed so that evaluators can score ideas on a sliding scale. For example, for the FEASIBILITY criteria, you cannot simply ask, “Will this project be successful in the marketplace?” because you cannot assign a score to it. Instead, the question should be, “How likely is it that the idea will be successful in the marketplace?”
3. Use weight factors. Weight factors help to fine-tune your scorecard and indicate some criteria as “more important” than others:
> Distribute 100 index points across the criteria. The more important a criterion, the more weight it should receive. For example, if “Brand Fit” is more important than “Strategic Alignment,” you should assign it a higher weight factor.
> Calculate a weighted score. The weighted score for each question = Weight Factor x Score. This formula is built into the Excel worksheet.
> Round off your weight factors. Weight factors need not be exact. For example, a weight factor of 23 vs. 28 doesn’t make a significant difference in the final score. Assign them in units of five or 10. Just remember, the weight factors reflect the relative importance of each criterion.
> Weights or no weights? If you’re just beginning your innovation efforts, consider omitting weight factors in your scorecard. You can add it at a later time as your evaluation program evolves, and your team gets a better understanding of innovation.
Instructions (continued)How to Customize and Use the Idea Scorecard
4. Continue the tracking. Each idea in your pipeline should get a unique tracking number (generated by an idea pipeline). The number should be carried over from the Screener.
5. Indicate who’s in charge. By this time in your evaluation process, an idea will be “owned” by a specific individual. List that resource on the scorecard.
6. Use the scorecard guideline. To ensure that all evaluators have a common understanding in terms of scoring, use the “scorecard guideline” worksheet in the Excel file. It explains what a score of 3 vs. 5 means. This should be distributed to all evaluators. Look at the following pages for more information on the scorecard guideline.
7. Encourage notes. Scorers should document questions and comments when they’re evaluating ideas. We offer tips on running a scorecard meeting on the next page.
8. Make a decision. As in the Screener, there are three possible decisions:
> GO: Move project to next phase.> HOLD: Keep project on back-
burner until next review (because it might not be the right time).
> STOP: Stop work on project—not to be reviewed again.
9. Identify clear next steps. If a project is given a “GO,” a team needs specific direction on how to take it forward. Document your next steps here.
How to Run a Scorecard MeetingEnsure You’re Using the Scorecard Effectively
Ensuring that the scorecard is used effectively is just as important as having it in place. Here are some useful tips on how to run a scorecard meeting.
BEFORE THE MEETING
Assign an evaluation team. Scoring is holistic when it reflects a wide set of opinions. A typical evaluation team should be:> Cross-functional: From different
> Empowered: Senior managers who have the ability to make final go/no-go decisions
> Diverse: A mix of personalities (visionaries vs. bean-counters)
Get them comfortable. After the team has been selected, ensure they have comfort, confidence, and agreement on the criteria, weight factors, and scorecard guidelines. We recommend some dry runs on actual projects. Expect some changes to the scorecard after these “pilot tests.”
Formalize scorecard meetings. Ensure that you’ve got scoring meetings on everyone’s calendars (either monthly or quarterly).
DURING THE MEETING
Present each project in a standard manner. To make sure evaluators compare apples to apples, ensure that each project is presented in a consistent format. For example, each project is:
> Given a set time (30 minutes). You can elect to have the project/team leader make the presentation.
> Presented using a standard template. This could be a presentation (with consistent sections) that each team uses. The sections should address the different evaluation criteria.
Define “rules of engagement” for evaluators. Scorecard meetings tend to go off-track, so create a set of rules that evaluators will abide by:
> No “last-minute reading” at the meeting. If evaluators are given pre-reads before the meeting, they should come fully prepared.
> Review each criterion and reach a conclusion. Don’t be tempted to say, “Let’s come back to this later.”
> Treat all projects fairly and consistently. There should be no hidden agendas, favorite ideas, etc.
> Avoid the “beauty pageant.” Projects should not be evaluated on the basis of their presentations. Don’t allow evaluators to get carried away by well-designed presentations.
> Make a decision within the working day to increase response times and speed up decision making.
> Stop off-topic issues immediately. Put them in the “parking lot” (an easel pad dedicated to potentially distracting issues raised in the session).
How to Run a Scorecard Meeting (continued)Ensure You’re Using the Scorecard Effectively
DURING THE MEETING (continued)
Encourage active discussion about the idea. Discussions at scoring meetings are especially productive since they raise many important issues to help improve the ideas. These conversations can be of invaluable help to a project team if their ideas are approved. Ensure that detailed notes are taken.
Score/vote individually. After the discussion is over, each evaluator should individually score and vote on the idea. Collect the scorecards. Quickly average the scores and count the final votes. Present the results to the evaluators and make a decision.
Define next steps. There should be an air of finality after an idea has been scored. Here are some decisions that your team might make:
> If an idea is approved, what should the team leader focus on in the next step?
> If the team has asked for additional resources, when can they receive them?
> Does someone need to be responsible for taking certain ideas forward?
AFTER THE MEETING
Summarize. Communicate decisions made (or outstanding items) to all evaluators/project teams.
Save Scorecards for later. File away all scorecards in a central place and update your pipeline. Also inform participants (team leaders) about the status of their projects.
The Scorecard GuidelineGet Evaluators on the Same Page
A scorecard can make evaluation more objective. However, the reality is that there’s still a certain level of subjectivity to the scores. Every person on your evaluation team will have a different opinion on what a score of 3 or 5 means.
We highly recommend that you use a “scorecard guideline”—it will help get everyone on the same page and create more consistency in your scoring. We’ve included a scorecard guideline in the accompanying Excel file. Once you’ve designed your scorecard, customize it for your organization and distribute it to all evaluators. Here, we highlight how to develop your guidelines.
InstructionsHow to Customize and Use the Scorecard Guideline
1. Define guidelines for all criteria. Be as descriptive you can with each criterion. We’ve included guidelines for each of the eight fundamental criteria.
2. Develop a range of scores. You don’t need to define a guideline for each score. Instead, use a low/medium/high score-range for each criterion, and define a guideline for the range. We recommend the following score-range:
LOW: 1 to 2MEDIUM: 2 to 4HIGH: 4 to 5
3. Describe a clear demarcation between ranges. Describe in clear terms the difference between the low, medium, and high ranges. For example, for the “Innovativeness” criterion, the descriptors we recommend are: LOW: Incremental innovation MEDIUM: Substantial innovation HIGH: Revolutionary innovation
4. Qualitative vs. quantitative criteria. Some criteria are qualitative where you should be vivid and descriptive with your guidelines. Some examples include: “Innovativeness,” “Competitive Advantage,” and “Strategic Fit.” For other criteria, you can define specific numbers for the guidelines. “Markets/Revenue” can have actual dollar figures based on your organization’s needs. For example, you can define the guidelines as: LOW (scores 1–2): Up to $50 million MEDIUM (scores 2–4): $50–$100 million HIGH (scores 4–5): $100 million + “Capabilities” is another criterion that is sometimes quantified. Define the guidelines in terms of the investment (dollars) you need to make in order to successfully implement the project.
Little Known TruthsLook Out for These Blind Spots While Using Both Tools
In many organizations, evaluation tools exist, but are rarely used effectively. Simply mandating a tool doesn’t ensure you’ll move the best ideas forward. Here are some often-overlooked issues that you should be aware of in order to use the screener and scorecard more effectively.
Be prepared to evolve the tools. Tools have a limited shelf life. Invariably, markets, competition, and people change. You’ll find yourself shifting your priorities on innovation or changing direction to adapt to the marketplace. It’s natural that tools go through periodic fine-tuning. With the information in this document, you’ll find it easier to fine-tune and tweak your evaluation tools.
Don’t mandate the tools and expect them to work. Finalizing your tools is just one side of the coin. They need to be
“massaged” into your organization to be used effectively. Often times, a senior champion is needed to shepherd these tools into how you do business everyday.
Communicate, communicate, communicate. No matter what decisions you make about the ideas in your pipeline, it’s critical to let those involved (idea-submitters and project-teams) know the status of their work. With the tools in your hand, you’ll be able to have more fact-based conversations with the people who are making innovation happen in your organization. Remember, the best innovation programs are based on open conversation and transparency.
