More than 1,000 organizations have taken the initiative to implement ROI, based on the number of organizations participating in a comprehensive certification process designed to help individuals and teams to implement the ROI methodology. With the acceptance of ROI, much of the focus has now turned to best practices for ROI implementation. The 11 best practices outlined here represent the state of the art with those organizations that have successfully implemented ROI.
1. The ROI methodology is implemented as a process-improvement tool and not a performance-evaluation tool for the L&D staff.
L&D staff acceptance is critical for the implementation of this process. No individual or group is willing to create a tool that will ultimately be used to evaluate that individual’s or group’s performance. Consequently, many organizations recognize that ROI is a process-improvement tool and communicate this posture early. The ROI methodology not only shows the success of a particular project, program or solution, but also provides the details of how the project can be revised to add additional value. Barriers and enablers to success are always identified.
2. The ROI methodology generates a micro-level scorecard with six types of data.
As Figure 1 shows, the data represents a scorecard of performance, representing both qualitative and quantitative data, often taken at different time frames and from various sources. This approach builds on the four Kirkpatrick levels. The methodology generates a balanced, micro-level scorecard for that particular program.
Figure 1: ROI Data
|
Level |
Measurement Focus |
|
1. Reaction and Planned Action |
Measures participant satisfaction with the program and captures planned actions. |
|
2. Learning |
Measures changes in knowledge, skills and attitudes. |
|
3. Application |
Measures changes in on-the-job behavior. |
|
4. Business Impact |
Measures changes in business impact variables. |
|
5. ROI |
Compares program benefits to the costs. |
|
Intangibles |
Measures that are purposely not converted to monetary value. |
3. ROI methodology data are being integrated to create a macro scorecard for the L&D function.
As more and more studies are conducted, data are rolled up to create a macro-level scorecard, showing the value of the function. As shown in Figure 2, the individual micro scorecard evaluation data are integrated into the overall macro-level scorecard. This approach requires a few similar questions to be asked each time that are then integrated, using technology to create the L&D macro-level scorecard. In essence, as each program is evaluated (every program is evaluated at level 1), data are generated from a few questions at each level that can ultimately be developed in a scorecard format.
4. ROI impact studies are conducted selectively, usually involving 5 percent to 10 percent of all programs and solutions.
Usually, the programs that are targeted for levels 4 and 5 are those that are strategically focused, expensive, high-profile, controversial and have generated management�s interest. This does not mean that other programs are not evaluated. It is recommended that all programs be evaluated at level 1 and the vast majority at level 2, but only a few select programs are taken to levels 3, 4 and 5. More important, those involving the actual ROI calculation�taken to the fifth level�are evaluated at all five levels.
5. ROI evaluation targets are developed, showing the percentage of programs at each level.
Organizations are targeting the desired number of programs to evaluate at each level, expressed as a percentage. Figure 3 shows a typical profile of a best-practice organization, targeting a percentage of programs at each level. Target levels are developed, reflecting the resources available and the feasibility of evaluation at each level. Targets usually begin at 100 percent of programs at level 1 and conclude with 5 to 10 percent of programs at level 5.
� Figure 3: ROI Evaluation Targets
�
| RECOMMENDED TARGETS | |
| Evaluation Level | Target* |
|
Level 1 � Reaction |
��������������� 100% |
|
Level 2 � Learning |
����������������� 60% |
|
Level 3 � Application |
����������������� 30% |
|
Level 4 � Business Impact |
����������������� 10% |
|
Level 5 � ROI |
������������������� 5% |
�
*% of programs evaluated at that level
�
6. A variety of data collection methods are used in ROI analysis.
ROI evaluation is not restricted to a particular type of data collection method, such as monitoring of business data. Instead, questionnaires, built-in action plans, focus groups and observations are used in developing the complete profile of six types of data in the ROI methodology.
7. For a specific ROI evaluation, the effects of L&D are isolated from other factors.
Although a difficult issue, best-practice organizations realize that to make a business linkage to a specific L&D effort, there must be some method in place to show the direct contribution of the L&D program. Using a variety of techniques ranging from control-group analysis to expert estimation, best-practice organizations are tackling this difficult issue with each impact study. Some argue that this is too difficult or impossible. In reality, it must be done for executives to understand the relative contribution of L&D. Otherwise, there�s a temptation to slash the budgets of major programs because there�s no clear connection between the program and the business value.
8. Business-impact data are converted to monetary values.
These days, it may not be enough to show the actual outcome from an L&D program expressed in numbers such as quality improvement, cycle-time reduction, turnover reduction or enhancement in customer loyalty or job satisfaction. The actual value in monetary terms is needed. Best-practice organizations are tackling this with a full array of approaches to develop the monetary value. This is absolutely essential in developing ROI because an ROI calculation compares the monetary value with the cost of the program.
9. The ROI methodology is being implemented for about 3 percent to 5 percent of the L&D budget.
One of the common fears of ROI implementation is the excessive cost in both time and direct funds. Best-practice firms report that they can implement the ROI methodology for roughly 3 percent to 5 percent of the total budget, using targets set in Figure 3. Using a variety of cost-savings approaches keeps the cost and time commitment at a minimum. Some of the most common cost-savings approaches are:
�
- Plan for evaluation early in the process.
- Build evaluation into the training process.
- Share the responsibilities for evaluation.
- Require participants to conduct major steps.
- Use shortcut methods for major steps.
- Use sampling to select the most appropriate programs for ROI analysis.
- Use estimates in the collection and analysis of data.
- Develop internal capability to implement the ROI process.
- Utilize Web-based software to reduce time.
- Streamline the reporting process.
�
When implementing ROI, many organizations have migrated from a very low level of investment (around 1 percent or less) to the 3-to-5-percent level by a process of gradual budget enhancements. These enhancements sometimes come directly from the cost savings generated from the use of the ROI methodology.
10. ROI forecasting is being implemented routinely.
Senior executives are sometimes asking for a forecast of ROI before a project is developed and implemented. Consequently, ROI forecasting is being used routinely in best-practice organizations to enhance the decision-making process. Recognizing the shortcomings of forecasting, conservative adjustments are made and steps taken to ensure that the best expert inputs are secured to develop the forecast.
11. The ROI methodology is used as a tool to strengthen and improve the L&D process.
One of the important payoffs of the use of ROI over a period of time is that it transforms the role of L&D in the organization. The process focuses more attention on alignment with business needs, improves the efficiency of design, development and delivery, and enhances the value of learning and development in the organization. More important, it builds respect, support and commitment from a variety of groups, including senior executives and major program sponsors.
Collectively, these best practices are evolving as hundreds of organizations are using ROI each year. The best practices underscore the progress in the evolution of ROI application and use.
Jack J. Phillips, Ph.D., a renowned expert in accountability, is with the Jack Phillips Center for Research, a division of the FranklinCovey Company. He provides consulting services for Fortune 500 companies in the United States and to organizations in 36 countries. He is also the author or editor of more than 30 books and 100 articles. E-mail Jack at jphillips@clomedia.com.
