Measurement means numbers, yet there are many learning professionals who would rather change careers than deal with these pesky digits. But you don't need to be a mathematician to be successful in a world of numbers — it's not about how to "d
by Patti P. Phillips
November 22, 2006
Some say today is the Quantitative Information Age, which is surely the case for the learning function. At the beginning of this year, measurement was the No. 1 issue on which to focus. Measurement means numbers, yet there are many learning professionals who would rather change careers than deal with numbers. But one need not be a mathematician or statistician to be successful in a world of numbers — it’s not about how to “do” the numbers but about knowing what the numbers say.
The Picture Numbers Paint
Numbers paint a picture — good, bad or indifferent. They show us where we are in relation to where we want to be or where we were. Learning analytics systems offer a variety of pictures that show us how successful or unsuccessful we are, depending on how one defines success. With just a stroke of the brush, we paint by numbers, showing all that we want stakeholders to see.
The Real Story
While the picture displays a message, the real story requires more thought and consideration. The details put numbers into context and can be extremely important when critical decisions are being made based on those numbers. So, while a picture might look good, there is more to the story.
Meaning Behind the Number
Understanding and explaining what is meant by numbers is important. For example, airline industry regulations determined by the Department of Transportation consider an arrival “on time” if the flight arrives fewer than 15 minutes after the published arrival time. The airline industry understands what “on time” means, but for passengers, the definition is different — if the flight arrives 14 minutes later than the published arrival time, would you consider it on time? So when the industry compares online arrival metrics among carriers, and you see your favorite carrier at the top, knowing how many times you’ve been 14 minutes late, just remember that your definition and the industry’s definition of “on time” might not be the same.
Evolution of the Number
Two factors are considered in the development of numbers: quality of numbers and quantity of inputs into the development of numbers. Counting an observable object is the most accurate way to develop a single metric, which guarantees the quality of the number, in essence. When FedEx counts the number of packages shipped to Omaha, Neb., simple addition allows the company to accurately report the number shipped. Mathematics is the only way we can be exact — anything else is a guess.
Reality dictates that the luxury of simply counting is nonexistent for most numbers reported in organizations. How do you count the number of satisfied customers? One possible answer: Counting the customers who rated us 95 percent satisfied, according to J.D. Power and Associates. But how does J.D. Power and Associates define customer satisfaction, and how did the company ask customers their level of satisfaction?
Organizations that rely on subjective measures such as customer satisfaction know that going customer to customer and observing how satisfied they are is virtually impossible. But through the use of surveys, an organization can collect enough information to estimate how satisfied a customer is given a certain point of time and based on a specific set of questions.
Organizations rely on surveys to provide quantitative measures on a routine basis. Yet, surveys are the weakest technique through which these data can be developed. Why are these tools so weak yet so beloved? Surveys depend on the subjectivity of the respondent, which relies on the quality of questions asked, among other factors. Surveys, however, allow large quantities of information to be gathered for minimal cost. Using statistics, data collected via surveys can provide an estimate of the true value of the variable being measured. The key is collecting data from enough people that the estimated values have some level of reliability — or that they are statistically significant. Statistics 101 tells us that if statistical significance exists at the 0.90, 0.95 or 0.99 level of confidence, the findings are true. Significance depends, however, on the acceptable risk of making a decision based on a number that might not be true. Considerations in defining significance include:
So, while a number might not meet the 0.90 level of significance, as long as the risk of accepting an untrue value is acceptable to the decision-maker, the number represents a truth. But meeting a defined level of significance is only part of the story — the “significant” number also must be important. Significance from a statistical perspective does not mean important — an unimportant number can have a greater level of significance than an important number.
Source of the Number
According to The Dallas Morning News, Tulsa World, Times-Herald of Vallejo, Calif., The Humane Society of the United States and The American Society for the Prevention of Cruelty to Animals, a single female cat and her offspring produce 420,000 cats over just seven years.
That many births in just seven years seems unbelievable. But look at the sources of information — not only does the Humane Society report it, but multiple sources report the same number.
The source of data is an important consideration when reporting numbers. Who says it and why? While the number might or might not be accurate, perceived bias can discount any credibility inherent in the number. When reporting or making decisions with numbers, keep the source in mind. But don’t forget: Even if the source is the most credible, understanding how the source developed the number is important.
Carl Bialik, aka The Numbers Guy for The Wall Street Journal online edition, investigated the cat story in an Oct. 12 story. He found that none of the sources claimed responsibility for the number of births. Even the Humane Society, which lists 420,000 births on its Web site, reported to Bialik that it did not come up with the number. The American Society for the Prevention of Cruelty to Animals didn’t claim the number either — it credited the Humane Society.
Relative Value of the Number
Finally, when numbers are presented, they often are presented as stand-alone values. But how does one know what the number really means unless it is compared with some other meaningful number? For example, the Labor Department reported job growth for nonfarm payrolls for October as 92,000 jobs. Alone, the number looks acceptable, unless compared with the 110,000 benchmark the Federal Reserve thinks is needed each month to keep up with the population growth. This puts the number in perspective. When averaged with August and September, the monthly job growth looks more promising — 157,000 new jobs (a three-month average) compared with a target of 110,000. Relating a number to another important number gives us the final context we need to know what the number really says.
Questions to Ask About Numbers
What does this rant about numbers mean to the CLO? Every day chief learning officers make decisions based on numbers regarding:
By knowing what numbers say, CLOs can make better decisions and can present their case so that it influences decision-makers in their favor. Below are a set of questions the CLO can ask about numbers to put them into clear context:
By putting a number into context, the CLO can clearly know what the number says.
Patti Phillips, Ph.D., is president and CEO of the ROI Institute and co-author of “Show Me the Money,” published by Berrett-Koehler Publishers. She can be reached at pphillips@clomedia.com.