How to measure whether a training session worked

Write down the result you want before you plan the session, then plan backwards from it. The Kirkpatrick model for evaluating training has four levels: reaction, learning, behavior and results. Kirkpatrick Partners insist on starting with the last one, the result the training is for, and working back to the behavior that produces it.

Make the goal something you could watch someone do. "Understand pivot tables" is not measurable. "Builds a pivot table from the monthly sales export without help" is, and it tells you what the exercises and the final task should be.

Most training is judged by a form handed out at the end. In 2003, Winfred Arthur and colleagues quoted an industry benchmarking survey in which 78% of the organizations measured reactions, against 32% that measured learning and 9% that measured behavior at work. Their review concluded there is very little reason to believe that whether trainees liked a program tells you much about what they learned or whether their work changed.

If you ask for reactions, ask the right one. A meta-analysis by George Alliger and colleagues found that ratings of how useful the training was were more strongly related to learning and to performance at work than ratings of how much people enjoyed it. Ask "How useful will this be in your work?", not "Did you enjoy it?". How to get useful feedback at the end of a workshop covers the rest of the form.

The session itself produces the most direct evidence, if you collect it:

• Completion of each exercise. Who finished step 3, and how long it took them.
• Where people got stuck. The step with the most help requests is the step that needs rewriting, or the skill that needs more practice.
• Checks for understanding. A one-question quiz after each new idea shows what the room understood, while you can still re-explain it (how to check for understanding without stopping the class).

Kirkpatrick Partners make the same point about reactions: rather than waiting for a survey at the end, use quick checks during the program so you can correct course while it's running.

At the end, have everyone do the thing the session taught, on their own, with a result you can check: build the pivot table from a new file, write the query that answers a new question, fix a broken configuration. Arthur's review notes that learning is typically measured this way, with written and performance tests rather than self-ratings.

If you can, give a similar task at the start. The difference between the two is what the session added. Keep the final task short and new: an exercise copied from the session measures memory of that exercise, not the skill.

The real test happens back at work, and it usually looks worse than the end-of-session result. Arthur's meta-analysis found the effects of training on behavior noticeably smaller than its effects on learning, and gave one reason: people can't show a skill they never get the chance to use. So check that there is a chance. If nobody builds pivot tables in their job, the session can't change how they work.

Two practical rules for the follow-up:

• Ask a few weeks later, about specific use. "In the last two weeks, have you built a pivot table at work? What stopped you?"
• Don't rely only on the learner's own report. A meta-analysis of transfer studies by Brian Blume and colleagues found that results collected from the same source in the same setting consistently looked stronger than they were. Ask the manager, or look at the work itself.

When the class ends, Workshopy builds its report. A sentence at the top says how many finished every step and which step the class got stuck on, and the Step by step table shows, for every step, how many students finished it, how many got stuck, how many asked for help and how long they waited. The Polls & quizzes card shows how many answered each quiz question and how many got it right, and the report exports as CSV or Markdown.

Run the same template with the next group and the Reports list shows each class with the share of students who finished every step, so you can see whether a rewritten step made a difference. What happens back at work is outside Workshopy: that part of the measurement is the follow-up above.