There are numerous circumstances where a statistically rigorous, high-level impact study is not only appropriate but strongly recommended. In each case, the underlying reason for conducting the study should include one or more of the following:
- Ensure planned results are being delivered
- Optimize the deployment
- Learn in order to improve the design and deployment of future initiatives
Before listing the circumstances that warrant high-level measurement, however, let’s make sure the basics are in place. By this I mean that learning leaders have spent time with the senior stakeholder to carefully plan the learning in support of the organization’s goals. This includes ensuring that the learning is aligned with one of the organization’s high-priority goals, identifying the right target audience, type and duration of learning, discussing the likely cost of the learning program, and agreeing on the expected benefits or impact of learning on achieving the goal. This proactive, strategic approach will help ensure not only that the learning you design really does meet the organization’s needs but that you and the stakeholder agree on how success will be measured. (These early discussions should be with the most senior stakeholder, not their staff.)
So, at this point (before the program is even developed) we have a forecast of the expected impact or value of learning jointly determined by the stakeholder and the learning professional. The actual impact, of course is likely to differ from the expected impact. This is the nature of a forecast. The actual impact will be more or less than expected. Hopefully, the actual impact will be close to the expected or forecasted impact, but it may not be. This raises the question, “When does it make sense to do a high-level impact study to find out what the actual impact was?”
I think it makes sense to conduct a high-level impact study in the following circumstances:
- The program is in support of a strategic or high-priority organization goal
- The program will be deployed to many participants over a long period
- The program is very expensive and represents a significant investment
- You have little experience with this type of a program
- The stakeholder is very interested in the actual, isolated impact of learning
- You are very interested in the actual, isolated impact of learning
- The stakeholder would like to know the relative impact of all the factors that contribute to achieving the organization goal.
Often several will apply which makes an even stronger case. Many high-priority, expensive initiatives touch numerous employees and are spread out over multiple quarters or even years. Wouldn’t you want to know if the initiative is delivering the planned benefit before deploying to everyone, especially if you have no experience with a program like this? Wouldn’t you want to know how the program is impacting subsets of the target audience and what opportunities there might be to fine tune the target audience? And, if deployment is already completed, wouldn’t you want to know whether the initiative had the expected impact and what could be done in the future to make such programs even more successful?
Note that justifying your expenditures on the program or justifying the existence of the L&D function is NOT a good reason to do the study. If you find yourself in that defensive position of having to prove your worth, you very likely skipped the proactive discussion with the stakeholder discussed above, and honestly, at this point the stakeholder is not likely to believe the results of any study you do. That is why it so important to start at the beginning with the senior (not staff) stakeholder to carefully plan the learning, and to agree on expected impact, measures of success, and a measurement strategy which may include a high-level impact study.
David Vance is the president of Manage Learning, LLC. He consults with organizations on learning and performance issues with a particular focus on launching corporate universities and designing effective strategies for managing the learning function, including alignment, governance, organization and measurements David is the former president of Caterpillar University. He was responsible for ensuring that the right education, training and leadership were provided to achieve corporate goals and efficiently meet the learning needs of Caterpillar and dealer employees. Prior to this position, Dave was Chief Economist and Head of the Business Intelligence Group at Caterpillar Inc. with responsibility for economic outlooks, sales forecasts, market research, competitive analysis, and business information. David’s blog for Chief Learning Officer can be found here.