Effective Training Evaluation

Okay, so you have spent a great deal of time and effort to develop lesson plans, curriculums, and course materials and you are executing training as planned… but is it good training?  Objectively evaluating training to determine where improvements can be made to increase effectiveness is always a challenge.  There are two methods that are most commonly used, each has its strengths and weaknesses, and each may or may not produce a useful data.

The Nominal Group Technique

Simply put, the end of course survey given to students to evaluate the effectiveness of the training received.  A properly structured end of course critique using the nominal group technique should focus student comments on specific areas to be evaluated.  This way we do not end up with random comments about the quality of the coffee or the temperature of the classroom and we can focus the comments on areas of the training that we are looking to improve.  Ideally, the comments should allow the student to quantify the experience to quantify the results for analysis.  In this way, we can perform an apples to apples comparison between classes on the same subject areas.  While student comments can be invaluable, the comments need to be examined and analyzed on the basis of the context in which the survey was given.  In other words, it is difficult to establish how much learning actually occurred and how well the student will retain that training on the basis of end of course critique.  The critique’s strength will illuminate areas of the training where the delivery was weak and could be improved; however, even poorly delivered training that is properly planned can have the desired impact on the student with regard to retention (we tend to remember the extremes… good and bad).  The weakness of the survey has to do with asking the right questions and removing ambiguity.  An improperly constructed survey will result in less predictable or analyzable data because each student will interpret the question differently.  A question not asked is an answer that we will simply not know or hear.  Either defect can result in an incomplete picture in the final analysis.

The Delphi Method

Another method of evaluating training is using the Delphi Method… or finding subject matter experts (SMEs) who have the expertise and/or experience to evaluate the training to be performed or to evaluate the training that has been performed to determine weaknesses in the program or curriculum.  To keep the evaluation honest, the expert comments should be anonymous; thus, free to offer open criticism without fear of confrontation or reprisals.  The benefit of the Delphi Method is that SMEs often have different perspectives that may not have considered in the development or delivery phases.  Additionally, the evaluation is free to be completely objective as the SMEs have not connection to or responsibility for the success or failure of the training to be evaluated.  Where the Delphi Method can fall flat is that SMEs tend to forget what it is like to be a student and that perspective can taint the evaluation process by focusing on the structure or the delivery rather than the technical content where their observations will tend to be most valuable.  Additionally, SME experience and unfamiliarity with the non-traditional training methods may lead the SME to believe that the training is less effective than traditional methods and to evaluate the potential effectiveness harshly.

Putting it all together

For a complete evaluation of training, a mixture of techniques combined with building metrics into the course structure (e.g. course written and/or hands on performance examinations, outcome-based practical exercises, etc.).  Additionally, follow up field surveys issued to managers and supervisors on the quality of trained personnel conducted months after the training is complete can yield a wealth of information on training gaps.  No single metric is going to provide a complete picture of the total effectiveness of the training performed.  Combining the various crowd sourcing techniques using the students, subject matter experts, managers, and supervisors will most likely provide enough perspectives to create a more complete picture of the overall effectiveness of the training performed.