Research sponsored by ATD shows that the number one measure from talent development programs desired by CEOs of Fortune 500 companies is business impact, with 96 percent indicating it’s their number one measure, with ROI at number two.
Together, these priorities show that executives want learning and development to connect to the business. The same is true of executives in nonprofit organizations, as every organization has “business” measures such as output, quality, cost and time.
If you’re not sure that’s true for your organization, just ask your top executive, “Would you like talent development programs to connect to the business of the organization?” We all know the answer. That answer points to the program’s central purpose: Impact is the “why” of the program.
If you start with the end in mind, you’re more likely to reach it. The starting point for a talent development program is a business measure—start with why. From there, the program should influence one or more business measures. While reaction, learning and application are important, impact is more vital.
So, how does this work? How can you have a system that starts with why and ends with business success? This article explains the methodology that has become the most widely used evaluation system in the world.
The value chain
Before describing the ROI Methodology, it’s helpful to understand the datasets that are captured and how they connect. Figure 1 shows the value chain for any type of project or program: a logic model with a series of data types that build on each other. For example, in the model, application isn’t possible without learning, and impact won’t occur without application. Although this classic logic model dates back to the 1800s, Raymond Katzell published this version in 1952. Don Kirkpatrick popularized these steps and claimed them as his own.
Level 0, Input, involves the number of participants, the time they are involved, and the cost. This is followed by five levels of outcomes that build on each other.
Level 1, Reaction, focuses on how the participants in a project or program see value in the project or program. Do they see it as relevant, important, something they would recommend or something they’re committed to making successful? Reactions are powerful, and an adverse reaction can kill any new project or program.
Level 2, Learning, involves acquiring or enhancing skills and knowledge.
Level 3, Application, involves action by the participants. The extent of use, frequency of use, and success with use are potential measures. The barriers and enablers to success are captured.
Level 4, Impact, is the consequence of application. Impact measures are in every organization as output, quality, cost and time measures. To be credible, the effects of the program have to be isolated from other influences.
Level 5, ROI, is where the impacts are converted to money (for the monetary benefits) and compared to the total cost of the program, with direct and indirect costs. This is calculated as the benefit-cost ratio.
| BCR = | Benefits | |
| Costs |
The ROI calculation is
| ROI (%) = | Benefits – Costs | X 100 |
| Costs |
ROI addresses the question: ”Was it worth it?” This is increasingly important to those who fund and support programs.
The ROI Methodology
Figure 2 shows the ROI Methodology, a twelve-step process for delivering results. It’s not just an evaluation system; it’s a system to influence the design and delivery of a program. This system has enjoyed wide-scale adoption, with more than 10,000 organizations using it, and it’s now the most widely used evaluation system in the world. Why? Because it delivers value. It’s CFO and CEO friendly. It delivers the connection to the business that executives want to see. Here’s a brief review of the steps.
Designing for the delivery of results
Planning the evaluation
Planning the evaluation involves three critical steps that set the program up for success.
Step 1 is to start with why by aligning the program to the business measure. Is this a problem worth solving? If so, what measures reflect the problem? Or is this an opportunity worth pursuing? If so, what measures reflect success if you address that opportunity? If these aren’t readily available, it will require some discussion, analysis, or comparisons to identify the impact measures. This usually involves the requestors or supporters of the program, who can guide you in that direction if it’s not already obvious to you.
Step 2 is to make sure you have the right solution. If it’s a problem that has been given to you, you may need to analyze what’s causing it, and your solution will address the problem. But if it’s a requested program, make sure it’s the right solution to deliver the desired results. The good news is that most requested programs are there for a good reason.
Step 3 is expecting success. To put that expectation into practice, three design thinking actions are taken in this step.
- It’s made clear to all stakeholders that success isn’t achieved until the impact has occurred.
- Specific objectives are set for reaction, learning, application, impact and ROI, if the evaluation is going to the ROI level.
- The objectives are provided to all stakeholders, including the designers, developers, program owners, facilitators, participants, and the managers of the participants so everyone does their part to ensure that this program delivers the intended impact.
These three steps prepare the program for success. Two planning documents are prepared: A data collection plan and an ROI analysis plan (if the ROI is pursued).
Collecting data
Once planning is complete, data collection is fundamental to evaluation, and the methods used are traditional, except that action plans are used more often than with other evaluation models.
Step 4 involves collecting data for reaction and learning, which organizations do routinely.
Step 5 involves collecting application and impact data. You need both to make a difference. Application without impact is just being busy. This step obtains the impact data, the same data identified in Step 1.
Analyzing data
Now for the credibility challenge. This involves five steps that are designed to make the evaluation results credible to those who need to see them, usually senior leaders, sponsors, investors and particularly the CFO.
Step 6 isolates the program’s effects. This can always be done and is required by definition, process and standards in the ROI Methodology. All 11,000 certified ROI professionals have completed this step in a way that can be defended to the CFOs. The techniques range from experimental versus control groups to estimates. When estimates are used, the data are collected in a non-threatening, unbiased way from the most credible source and adjusted for error.
Step 7 converts impact data to money. Many of the data sets have already been converted into money using standard values, approved by management and provided to individuals who need to know them. If that’s not available, there are experts who can help you. Also, you may research the value externally, or maybe you can use estimates. The output of this step is the monetary benefits for the ROI calculation. The philosophy is “when in doubt, leave it out,” to make it more credible. Most impact data can be converted to money. If not, it becomes an intangible measure.
Step 8 is to identify the intangibles. These are measures that can’t be credibly converted to money within a reasonable amount of time. Intangibles are still important.
Step 9 involves capturing all costs, direct and indirect, including the time of everyone involved in the program. The philosophy is “when in doubt, leave it in,” which adds credibility.
Step 10 is to calculate the ROI, using the benefit-cost ratio and ROI, as presented earlier. These are the two most common ROI measures. The benefit-cost analysis originated in governments, and the ROI calculation was derived from the BCR about 400 years ago. ROI has become the dominant method for the private sector. But they’re both showing the benefits versus costs, just in two different formats.
Optimizing results
In Step 11, stories engage stakeholders, derived from collecting data at reaction, learning, application and impact. Six data sets are available with five levels of outcomes: reaction, learning, application, impact, ROI and intangibles. These make a convincing story with credibility tied to the measures.
Finally, Step 12 involves optimizing results. The results are used to influence future funding, obtain more support for talent development and make programs better. Other important levers include recognizing the people who delivered the results and gaining respect from executives who need to support the talent development function.
As discussed in this article, conservative principles make the process acceptable to the top executives who fund the programs. Any estimates must be conservative, not overstating the results, and the processes must be credible. The standards appear in Figure 3 and represent 12 important rules for planning data collection, analysis and reporting.
Let’s recap
So, there you have it. Twelve steps that can deliver success. For readers who want to explore the methodology further, more detail on this methodology is found in “The Business Case for Learning: Using Design Thinking to Deliver Business Results and Increase the Investment in Talent Development.”
