are we measuring?
EVALUATION, as in life, there are many ways of measuring success. When you set your life
goals - for health, wealth, fame or happiness (and all those unprintable ones that don't
quite fit into this list) - you also determine your criteria for success. And so with
evaluation. What you measure is a factor of your original goals.
The form of evaluation that we undertake is determined by the
criteria that we choose, or are told to use, to measure success. The following four levels
of evaluation were defined by Kirkpatrick back in 1975:
Level 1: Reactions
Reactions are what you measure with the happy sheet. Reactions
are important because, if learners react negatively to your courses, they are less likely
to transfer what they learned to their work and more likely to give bad reports to their
peers, leading in turn to lower numbers.
Level 2: Learning
Learning, in terms of new or improved skills, knowledge and attitudes, is
the primary aim of a training event. Learning can be measured objectively using a test or
exam or some form of assessed exercise. If a learner has to achieve a certain level of
learning to obtain a pass mark, then the number of passes may be used as an
evaluation measure. Another important aspect of learning is the degree of retention
how much of the learning has stuck after the course is over.
Level 3: Application
If a learner has learned something from a course, you hope that this will be
reflected in their behaviour on the job. If a learner employs what they have learned
appropriately, then their work behaviour will meet desired criteria. Behaviour can be
measured through observation or, in some cases, through some automated means. To assess
behaviour change requires that the measurements are taken before and after the training.
Level 4: Business results
If, as a result of training, learners are using appropriate behaviours on
the job, then you would expect that to have a positive impact on performance. A wide
variety of indicators can be employed to measure the impact of training on performance
numbers of complaints, sales made, output per hour and so on. It is hard to be sure
that it is training that has made the difference without making comparisons to a control
group a group of employees who have not been through the training.
Recently, attention has focused on the financial
implications of training and, consequently, a fifth level has been defined (Phillips,
Level 5: Return on investment
Return on investment (ROI) is a measure of the monetary benefits obtained by
an organisation over a specified time period in return for a given investment in a
training programme. Looking at it another way, ROI is the extent to which the benefits
(outputs) of training exceed the costs (inputs).
But leaving aside the five levels of Kirkpatrick and
Phillips, in the real world we are measured in other ways as well:
One way of measuring the success of training is the good old bums on
seats. Although by no means a true measure of the effectiveness of training, learner
numbers do reflect the fact that the training is addressing a need and that the design and
methodology is meeting expectations.
Direct costs are those costs that are incurred directly as a result of a
training programme external design and development, consultancy fees, travel
expenses and so on. If the programme did not take place, these costs would not be
incurred. Many organisations only ever take direct costs into consideration when measuring
Indirect costs are costs that may or may not be directly associated with a
training event, but which would have been incurred anyway, whether or not the training
took place. Examples are salaries of in-house tutors and learners and the costs of rooms
and equipment. Any analysis of the true costs of training will include both direct and
Efficiency is a measure of the amount of learning achieved relative to the
amount of effort put in. In practical terms this means the amount of time it takes to
complete a piece of training. Efficiency has a direct relation to cost the more
efficient a training method is, the less it will cost.
Performance to schedule
Sometimes with a training programme, time is of the essence
the training needs to be completed by a given date if a particular business
objective is to be achieved. In these situations, the extent to which a training programme
performs to schedule is a critical measure of success.
If you are a training provider operating externally to a client
organisation, then income received is a vital measure of your success. Its the
financial equivalent of bums on seats the more courses you run or
places you fill, the greater the benefit. Some internal training providers may also
cross-charge their clients, although, because this correspondingly increases the cost to
the organisation, this would not be regarded as a benefit when assessing return on
The extent to which learners mix
A justification often made for training, particularly group events, is that
it provides an opportunity for learners who work in different departments or regions to
interact with each other, share experiences and make contacts. Because this is a valued
outcome of training, it needs to be considered when comparing training methods. Similarly,
some training may be regarded as a perk, a benefit of some value, even if this is not
directly related to learning.