CLOs have a measurement dilemma. They must demonstrate training’s business impact but can’t garner required resources without showing the connection between training and performance. But even in a difficult economic environment, some progress has been made in this endeavor. In January, about half of CLOs reported measurement activities are fully aligned with their learning strategies. Compared with 2008 data, far fewer CLOs are dissatisfied with their organizational approach to measurement in 2012, but progress is uneven.
Every other month, IDC surveys Chief Learning Officer magazine’s Business Intelligence Board (BIB) on an array of topics to gauge the issues, opportunities and attitudes that make up the senior learning executive role. For this month’s analysis more than 350 CLOs shared their thoughts on learning measurement.
Generally, training impact metrics collection and reporting can be described as basic. Two-thirds of enterprises routinely report the progress toward training objectives — courses, students and hours of training — for specific organizational initiatives, up from about half in 2009. And measurement of stakeholder and learner satisfaction with training programs is frequently completed. But more advanced measurement application is still constrained by time, resources and support.
Taking Measures
Learning professionals generally agree on measurement’s value. When done properly, it can demonstrate training’s impact on the company’s top and bottom lines. Effective measurement of the training impact also ensures that organizational initiatives will more frequently include appropriate learning participation. This helps the learning organization increase its relevance: As one CLO reports, “We have a very robust and comprehensive measurement system in place which helps us to articulate our impact on the business across our entire portfolio of solutions. Those measures are included in our organization scorecard.”
Key metrics may include employee performance, speed-to-proficiency, customer satisfaction, improved sales numbers and more. As the survey shows, the challenge lies in gaining access to these metrics and finding the time and resources to conduct measurement.
In the 2008 version of this survey, a majority of enterprises indicated a high dissatisfaction with the training measurement within their organizations. Two years later, sentiment was significantly more moderate. In 2012, dissatisfaction increased again — probably due to higher expectations, greater tool availability and greater awareness on the part of the learning organization combined with a continuing challenge to fully leverage tools, staff projects and effectively influence leaders (Figure 1).
As some CLOs report:
• “There is a large misconception at the executive level about the need to conduct reliable and qualitative training measurements.”
• “I think we could gather more data and make better use of technology.”
• “We need to do more but don’t have the funding or the will.”
• “We do measure, but it is inconsistent and doesn’t leverage all the data to drive better business decisions.”
• “We are not able to run good reports, nor do we get executive management support in training.”
When organizations were very satisfied with the approach to measurement, they thought learning and development played a very important role in the achievement of organizational priorities. On the other hand, when the measurement programs were weak, 60 percent of CLOs reported their influence and role in helping achieve organizational priorities was also weak. Learning organizations that can demonstrate their impact on learning initiatives can expect to have greater influence when needed.
Compared with prior years, the common forces working against satisfaction with measurement programs remained a combination of capability and support. Specifically, CLOs believe their ability to deploy effective measurement process is limited by lack of resources, level of funding, level of leadership support and availability of technology.
Leadership support is essential to develop an effective measurement program. But getting that support is challenging. One survey respondent said: “Our management has not yet bought into the value of training and employee development.”
Others report the measurement program is not as far along as they would like because they have “a very small group trying to balance a very large workload.” For several years, a common refrain has been: “It is imperative to get management involved in facilitating knowledge transfer in the workplace.” A combination of the inability to generate impactful measures and the unwillingness of executives to participate and leverage data creates a difficult barrier to cross.
With the various forces aligned to prevent advancements in measurement, there has been little change in its use (Figure 2). Consistent with past results, about 80 percent of companies do some form of measurement — with half of those using a manual process and half using some form of automated system. There has been a slight decrease in the percentage of respondents who indicate they use a manual process and an increase in the percentage of enterprises that use a combination of LMS and ERP systems to develop their learning metrics.
Organizations report that technology-based learning platforms give them greater ability to make correlations between training and performance, and the absence of technology can be a significant burden. “Because we use manual collection and analysis, measurement and analysis are done when we can. But, the data is often dated,” one respondent said.
Correlating Training to Outcomes
There has been an increase in the percentage of enterprises working to correlate training to changes in the organization. Organizations are frequently evaluating the impact training has on employee satisfaction and even engagement scores. Training’s impact on customer satisfaction is also frequently measured. However, there are still areas where more work could be done.
Less than half of all organizations evaluate training’s impact on product or service quality. And fewer measure the impact on either employee retention or productivity. Part of the reason for this is because training programs aren’t expected to impact quality or retention, but the opportunity for increased analytics use to establish the impact training has on a wider set of business outcomes is apparent (Figure 3).
When training is effective, the benefits of measurement extend to the bottom line for the training organization. Organizations that can consistently tie training to specific changes are more likely to train less. They focus training efforts on the most appropriate topics, abandon useless courses and spend less overall.
Things Are Looking Up
Despite obvious challenges, organizations are making progress. As Figure 1 showed, while 35 percent of CLOs are satisfied with the extent of measurement going on at their companies, nearly 80 percent of CLOs are planning to increase the level of training impact measures they undertake — that is more than the 50 percent who reported intentions to increase measurement activities in 2010.
About 30 percent of CLOs report they will be developing measurement programs to track training’s impact on employee capability — equal to about half of the organizations that don’t currently measure it. And about half of enterprises that don’t already will begin to track training’s impact on employee engagement — about 25 percent of the total survey population. Measuring the impact of training on customer satisfaction and overall business performance is also on the agenda for many CLOs. But training’s impact on sales is still somewhat ambiguous, with just under 60 percent of enterprises reporting they have no plans to add that to their measurement repertoire.
Measurement will remain a challenge, but it is still on the agenda. CLOs can take several steps to demonstrate training impact. Three of the most significant practices are to:
Define success early. By defining with stakeholders what success will look like upfront, learning professionals can more easily identify and benchmark key metrics for measurement before training is delivered and make post-training results easier to quantify.
Establish metrics at the project or business unit level. While it may be tempting to demonstrate training’s value at the enterprise level, successful measurement programs typically start off as smaller initiatives that focus on projects or the business unit level. When working with smaller groups there are typically fewer obstacles to interfere in the measurement process as well.
Set expectations upfront with stakeholders. Help those interested in measurement understand the commitment required to see assessment projects through to the end.
Cushing Anderson is program director for learning services at IDC. He can be reached at editor@CLOmedia.com.
