Every learning investment eventually faces the same question: Did it actually change anything in the work? Too often, learning answers that question too late and with the wrong evidence, such as completion rates and satisfaction scores.
Artificial intelligence did not create the impact problem, but it has certainly exposed it anew.
The evidence that AI makes individuals faster is solid: In a study of 5,179 customer support agents, researchers found that access to a generative AI assistant raised issues resolved per hour by 14 percent on average, and by 34 percent for novice and less-skilled agents. Sponsors now expect value just as fast as the tools deliver it.
But faster people don’t automatically make for a better organization. That also depends on who reviews the output, where the saved time goes and whether quality holds. When it doesn’t, a draft can be created in minutes, but the review that follows takes longer than the draft ever did. The effort hasn’t disappeared; it’s moved to someone else’s desk.
The fix isn’t a longer evaluation; it’s three questions asked before launch: What should people be doing differently? What’s actually stopping them? And what result would make us stop? They apply to any learning investment, from an AI rollout to mandatory data-protection training. I learned that by failing them myself.
A plan that looked finished
Earlier this year, I wrote an AI enablement plan for client-facing consultants. I was happy with it. It identified three specific use cases per role: proposal drafts, research synthesis and meeting write-ups. The same pattern applies to any team whose work involves writing things like sales proposals, policy drafts or client reports. The plan avoided demos in favor of 90-minute practice labs on live client work. Each practice got a champion with protected time, and client-approved AI sat inside the role pathway instead of in a separate module.
The goal was to convert saved time from faster proposal creation into more capacity for billable work. We would measure that through usage, confidence and time saved on drafts. The plan named a specific barrier (people not knowing what good looks like) rather than a blanket skills gap. It also set a week-four pulse check, a 60-day impact report and a recommendation for phase two.
Next, I pressure tested the plan against those three questions, and it failed all three. It measured usage, confidence and draft speed, not whether proposals got better. The plan assumed that time saved on drafts would become billable capacity, and it assumed the barrier. Its review points were dates on a calendar, not decisions anyone had agreed to make.
This is the gap learning has faced for decades: We measure the learning instead of the work, and we have no concrete way to decide whether to keep going.
3 questions that give speed a direction
Learning has too often chosen perfection over speed. But the answer isn’t speed for its own sake. It’s to agree on the evidence you need before you build. Making the traditional build a little faster with AI still leaves the question of value until the end, because no one decides in advance what would make them stop. Moving fast toward a measured outcome is the only speed that compounds, and these three questions supply that direction.
What should people be doing differently? Activity is not improvement. A respected learning leader at a media company recently shared that their organization’s mandatory AI course had reached 95 percent completion. Nearly everyone finished because they were required to, so that number says nothing about what people now do differently or what new value they create.
Completion is worth tracking, but it is not worth celebrating as an outcome. In my plan, “confident using AI” and “time saved on drafts” became this: consultants draft proposals with AI, and the drafts reach review needing less rework. Total effort per proposal falls, and quality holds.
Behavior drives work output, and work output drives business results, so the discipline is simple: pair every speed or volume measure with a quality condition. Identify these targets by defining what good looks like. You can do this by asking who already does this well and what they produce that others don’t.
Then track:
- Use: Did people use the tool or the intervention at all?
- Total effort: How many review cycles did a proposal take, not just how fast the first draft appeared?
- Quality: Did the standard hold, according to the people who review and buy the work?
- Time saved: Where did it go? Manager reports, self-reports and timesheets each give a partial answer.
Be sure to avoid vague indicators like states of mind (“understands,” “confident,” “embracing”), usage reported as the outcome and counts with no quality condition attached. A useful test: How would an outsider spot the change on a normal Tuesday?
What’s actually stopping them? Most learning requests arrive with the solution (not the desired outcome) already decided. My favorite example is the sales program. Sales are down, so the business demands a sales skills program. But if you ask the sales team why deals aren’t closing, they may tell you they have no leads. Teaching people to have better conversations helps only if there are conversations to be had.
Before assuming a skills gap, check the work before the worker, in this order:
- Clarity: Do people know what good looks like?
- Ability to do the work: Do they have the tools, access, time, process and handoffs?
- Consequences: What happens if they do it? What happens if they don’t?
- Skills: Only then look at skills, capacity (what comes off the day?) and motivation.
For each, separate what you know from what you are assuming. Then ask one more question: What will people do to make this look successful, even if it isn’t? That question matters because measures shape behavior. If the organization celebrates usage, people will find ways to use the tool whether or not it helps the work, and some may go around review steps to look productive.
Each workaround can register as a win on a dashboard while creating risk the dashboard never sees: client material in the wrong tool, or unchecked output reaching a customer.
My plan assumed the barrier was an unclear standard. That was plausible. So was a tool policy that kept client material out of the AI tool, a poor fit with how proposals get written, or no spare time at bid deadlines.
Any of those would have sunk the plan, and training would have fixed none of them. I could have checked every one of them in the first two weeks by asking reviewers to mark up a few recent drafts and say why each needed rework, and by asking someone what the tool policy actually allows. My diagnosis was a hypothesis, and I should have tested it before building anything.
None of this is new. Thomas Gilbert mapped these barriers in the 1970s, and performance consultants have used them since. What is new is that AI has removed the excuse for not taking them into consideration. When anyone can build a course in an afternoon, the build is no longer the hard part. The diagnosis is.
What result would make you stop? A review date is not a decision. My AI enablement plan had three review checkpoints, and each one would have produced a report, but none was tied to a choice anyone had agreed to make.
Before launch, agree on four things with the sponsor: the evidence you will examine, the date you will examine it, the decision each result triggers and who makes the call.
Set specific targets. For example, review cycles per proposal fall from [baseline] to [target], and reviewers rate quality [at least as high] compared to today. If you don’t have a baseline yet, your first commitment should be to set one by a named date.
Then agree up front what each result would mean. For a plan like mine, the cause and effect would go something like this:
- If total effort falls and quality holds: expand.
- If results are mixed, or it’s too early to tell: continue once, with a narrower test and a new date. A second mixed result means redesign.
- If drafts are faster but need more rework: redesign.
- If there’s little use, even after barriers are resolved: stop this approach.
Stopping is the option most teams avoid. But stopping an approach isn’t abandoning the need; you’re choosing a different way to get there. And “too early to tell” is a valid decision, as long as it comes with a new date. Think back to that mandatory AI course. Will everyone retake it next year? Unless someone agrees on when and how its value will be judged, and who will do the judging, it will run indefinitely and no one will know whether it’s actually adding value.
The same questions, without the AI
None of this depends on AI. Take mandatory data protection training, a program most organizations judge on completions.
What should people be doing differently? First, they should be sending client files by the secure route, and reporting a suspected breach the very same day. These are things you can actually see.
What’s stopping them? Is it really that they don’t know the rules? Or is the secure route slow, and does reporting a breach feel like asking to be blamed? If it’s either of those, another round of training won’t help.
What result would make you stop? The result, and your decision, might look something like this: If, after two cycles, files still travel by insecure routes and same-day reporting hasn’t increased, keep the obligation to protect data, but change the approach. An increase in reported incidents may be the first sign it’s working, not failing.
The outcome stays mandatory. What changes is what you deliver, and how you judge it.
Give it a decision date
Learning functions are good at starting things. We are less practiced at finishing them, or at saying out loud that a decision is due. Programs pile up because nothing forces a decision—a set date is that forcing point.
Put your answers on one page and take it to your sponsor before launch. It comes down to four sentences:
- Here’s what people should be doing differently, and the standard their work has to meet.
- Here’s what may be getting in the way, and how I’ll check by a set date.
- Here’s the evidence that would change our direction, when we’ll assess it and what we’ll do depending on what we see.
- I’d like you to agree on the decision and who makes it.
If you can’t get the sponsor in a room, send the one-pager with a single question and a date: “Can you agree to these conditions by Friday?” If a sponsor won’t commit to a decision, that’s a sign the program may not have a clear purpose, which is worth knowing before launch, not after.
These conversations are getting harder to avoid because AI has made the impact problem impossible to ignore. When building is fast and cheap, nobody will be impressed that we built something. Sponsors will ask what changed in the work, and they’ll ask before the budget renews.
Learning has spent years asking to be judged on performance rather than attendance. I think this is our moment to show we meant it.
So, pick the initiative you’re least sure about, not your showcase, and run it through the three questions this week. Every program has a start date. Give yours a decision date.
