Strengths and weaknesses

Paul, I want to thank for your fascinating description of the complex training and assessment system you've implemented at ROMATSA. I suspect that anyone in training at any weather service would be interested to see what your organization has put in place. Whether or not we intend to implement something so rigorous, we can learn from what you've done and how it is working out.

First, a question. With such a rigorous system in place, what do you do to keep training and learning FUN. I've always felt training should be fun and tackled willingly if lasting learning is to occur, and if there are so many requirements, does this diminish the intrinsic motivation and reward for students? Or do you ensure this motivation remains via the design of your courses and the way your trainers work with students?

Second, your concept of blended assessment is a big and important one. I understand your need for outside certification of competency, but that seems only a small piece of what you are after at ROMATSA. I really like that you have a goal to keep on-the-job training and assessment an integral part of your system. To me it seems that this is the only way to fully assess the impact of training and whether it is actually transferred to the job. You seem to apologize for the fact that this assessment is somewhat subjective, but I think that makes it all the more richer. Standardized and objective assessments can only measure the quantities and qualities they are designed to measure. But the on-the-job subjective assessments, when done by someone skilled and open-minded, can likely undercover gaps (and skills!) that no one would have anticipated beforehand.

Third, you mention using COMET modules as tools in the assessment process. At COMET,we've traditionally hoped that our modules are seen as valuable components within an overall, more comprehensive training system. We've not traditionally seen them as assessment tools, and don't invest the same effort in creating assessment instruments for them as we do in creating the education and training content. Have you considered using COMET modules as "elective" training tools that forecasters can use for their ongoing refresher (and extension) training? Have you considered developing your own assessment techniques that "localize" the content in the modules? For examples, case studies or assessment questions that ask learners to apply the conceptual or procedural knowledge to local events and using local tools?

Fourth, what other assessment techniques have you considered using in the potential "toolbox" of blended assessment approaches? Have you considered customizing assessment for individual students needs, or is the desire for standardizing too strong?

Fifth, it seems to me that the goal of any assessment system is not just proof to outside or inside authorities that competencies are met and that training is working, but also to strengthen the learning process. As you pointed out in your lecture, assessment can also be used to identify gaps that point to what additional training might be useful. But also, assessment lets the learner self-assess and test their knowledge and skill to focus their learning efforts, which is integral in the learning process--not outside it or at the end. Can these smaller, mini-assessments that are are part of the teaching process be made to count toward the total required assessment process, or is it too ingrained that it is the "final" assessment that matters? (The portfolio assessment approach, which you don't mention, is a step back from the bias toward "final" assessment.)

Ok, so ends my long-winded posting to the forum. I hope others join. Let's keep this one open for some time if you agree. With summer holidays, some people may not see your presentation or read the forums until later this summer. :-)