Week 2: Enhancing mental models

Week 2: Enhancing mental models

by Bruce Muller -
Number of replies: 7

Welcome to Week 2 of the session, Teaching with Conceptual Models.

This week, we have a pretty traditional activity for you: reading and reflection. Here's a research article, Competence in Weather Forecasting, that was published in the book, Psychological Investigations of Competence in Decision Making. The article describes characteristics of competent forecasters. 

For those of you interested in evaluation techniques, read the whole article. To jump to the results and content for discussion, start reading on page 11 of the .pdf file (page 60 as marked on the article pages), 'Characteristics of Competent Weather Forecasters'. 

A common theme here is the use of complex, dynamic mental models (i.e. conceptual models). That being the case, how do we go about fostering the development and enhancement of these mental models? Please share your thoughts on any of the following questions or related topics. 

And here's a 1-5 scale to refer to regarding the sophistication of a meteorologist's mental model of the atmosphere. This is a really goofy scale and could certainly be improved upon - but it should serve for now :-)

1 = Lives in planview, one moment at a time (just like a squashed fly on a print out of a 12-hour model forecast of 500hPa heights)

3 = Has a good handle on the 3-dimensionality of complex atmospheric processes but relies heavily on NWP to create and verify their mental model of the current situation.

5 = Moves forward and backward through time and space (like an imaginary air parcel traveling forward and backward through the evolution of a severe convective storm cell) and readily finds discrepancies in NWP 

1. How do the characteristics outlined in the article compare to your experience working with and training competent forecasters?

2. If you have forecasting experience, share your insights on what it takes to work at a level 5.

3. How can we enhance our training to help move forecasters towards developing level 5 thinking? Some examples were cited in Week 1 postings; let's expand on those.

In reply to Bruce Muller

Re: Week 2: Enhancing mental models

by Vesa Nietosvaara -

Bruce,

thanks for sharing this article, it was interesting to compare the characteristics of a competent vs non-competent forecaster. I can easily get back in time and remember some of my forecaster fellows from old times. They had a unique skill to be able with a few charts and checks be able to settle into the day's shift, and have already an understanding of how the day's general outlook will be. Whereas I was often that person waiting for the latest NWP, or the latest observation, and then changing my mind on the last minute. Decision-making was not my top skill.

I think those forecasters who have somehow artistic mind, a creative soul and curiosity for exploring the unknown, often are very good in conceptual models. They do not stick to the NWP guidance as a rule, but dare to question the computer models and look for alternative solutions.

In reply to Bruce Muller

Re: Week 2: Enhancing mental models

by Tsvetomir Ross-Lazarov -

Hi all,

I can only comment on the questions 1 and 3 since I have never been a forecaster nor have played one on TV.

In our work helping forecasters create three-dimensional (3D) mental models, we found that some forecasters think they are working in 3D but aren't. Others struggle with imagining things in 3D.

Most forecasters think of the atmosphere in 3 slices - surface, mid and upper levels.  Many of them equate that to being able to create 3D mental models.  Usually they do not connect the levels, or their thinking about the connections is not accurate nor as detailed as it can be.  

Other forecasters find it difficult to imagine the atmosphere in 3D and then imagine how those features will move with time.  The educational literature in the US refers to this skills as "spatial ability."  Research findings suggest that this skill is not very common but can be developed with exercises.

One of the techniques we use to help forecasters develop their spatial ability is to build a 3D model using water vapor imagery.  The group uses blue, red and orange streamers to represent the cold, warm, and dry conveyor belts respectively.  Then they arrange themselves to represent those features while holding the streamers in their hands.  Using sitting, standing, or crouching, the students then position themselves to represent the relative height of these features.  The group discusses what they see, and then builds their translation of the 2D data into a 3D mental model.

Tsvet

In reply to Tsvetomir Ross-Lazarov

Re: Week 2: Enhancing mental models

by Bryan Guarente -

One of the biggest problems in the meteorological world when it comes to 3D mental models is actually Numerical Weather Prediction (NWP) output.  Although it can be very useful to have NWP output for times beyond 24 hours, forecasters have the ability to out-forecast NWP in areas where the NWP and observations have a mismatch (in the first 24 hours).  However, because we have so much data to slice and dice with NWP, we often just want to believe the NWP because it offers us a more "elegant" solution than our own (when it is under-developed).  

In the past two weeks, we convinced forecasters to believe in their own predictions of the atmospheric evolution through their 3D mental model.  The best part (as facilitator) was when the forecasters had to try to actually build a 3D mental model (playdoh model in our case) out of NWP data.  They came to realize that they didn't get extra useful information from looking at the NWP.  In fact, sometimes they lacked information like they had from the observations.  It was a struggle for them to build a 3D mental model from the NWP in the end.  Multiple forecasters mentioned how hard it was to use NWP to make a 3D mental model even though they thought it was going to be easy with all the extra data.

An easy exercise aside from making forecasters build a 3D mental model out of NWP data is to layer a bunch of standard NWP output into a 3D grid, then ask forecasters to point to locations where they don't have any information.  Holes in the analysis process become evident whether it is from observations or from NWP.  Both have their holes when forecasters don't build it all out mentally.  

We use NWP as such a crutch (in a lot of instances) that we tend to forget its physical/dynamical meaning.  We can get away with far less NWP output than we think we need in a lot of instances.  We love the forecast funnel (for good reason), but some use it inappropriately by just thinking in isobaric surfaces from the top of the troposphere down.  It is truly a scale issue (much like mental models).  Use more appropriate output and your 3D mental model grows by leaps and bounds with far fewer maps.  Robust 3D mental models often trump NWP in the first 24 hours of a forecast. 

On a personal note, we often mention the oncoming "fire hose" of data that the new GOES-R satellite will be bringing to the western hemisphere.  I find this issue to be different than some believe it to be.  We should be working on using the basic channels better and only using the advanced products when they are necessary.  We tend to jump to the advanced products right away (similar to NWP) rather than look at the more "basic" products.  I don't find the advanced products to be useless, I just find we leap over the basics that help us make mental models before getting to the advanced products.  We could use the basic products to find areas where we need more information from different sensors or different NWP fields or other observations.  All of these products can work together to make the best possible answers.

Another thing forecasters (and I) have a tendency to forget (or lack time to do) is to make a forecast based on the observations before moving into the NWP realm.  If you have a guess (3D mental model) when you go into the NWP output, you can test your hypothesis.  If you go into the NWP without a hypothesis, then you can only pull up products and search for meaning.  It all "looks right" though because you have nothing to rate it against.  Without a 3D mental model, you have nothing but the NWP to work with.  For this reason, in labs during our courses, we always make the forecasters build their 3D mental model and make a forecast with it before letting them look at the NWP.  This past week, I had a forecaster say to me, "I should have believed my mental model and not gone with the NWP.  It lead me astray."  This was validation for the methods we are employing.

If in the end we can enable forecasters to trust their own perception, we are doing better than ever before.  Confidence without overconfidence is one of the keys to making forecasts outside of the NWP envelope.  I feel we have taught the students in our courses some form of confidence to help them make those predictions in the face of NWP.  Without 3D mental models and truly defining them with some visible shapes (playdoh), we wouldn't have ever noticed the gaps.  

Bryan

In reply to Bryan Guarente

Re: Week 2: Enhancing mental models

by Larisa Nikitina -

Hi, Bryan!

Really enjoyed reading your message! It is so close to my opinion as former forecaster with more than 25 years of aviation forecasting.

In spite of we in Russia are not so familiar and not so good with NWPs as you in USA or Europe (as we are doing only first steps in some offices), we might be good with our mental models, as our forecasts are quite good enough (I can tell that confidently as a chief of forecasts verification group). ;)

I managed to work with NWPs for some year before I become Olympic forecaster (that has changed so much in forecasting in Russia and in my live too). And my first experience was the same as you describe in you texts – I believed too much to NWPs, even I had found my favorite NWP to make forecasts! However, my old forecasting school and my intuition (that is the pure forecasting experience) made me gradually doubting in NWP models a little bit. J

Lucky me that I was chosen to be an Olympic forecaster, on trainings I met a lot of researchers and developers of NWPs from different countries, got familiar with a lot of high resolution models. And I agree with you – for 24 hours lead time forecast nothing is better than your own mental 3d model but.. but.. if you was born a forecaster!

Not all forecasters can make such 3d models in their mind, but together with a good NWPs they can make a good enough forecast! Could we tell them then as incompetent forecasters? NO or we will have not enough forecasters soon (at least in Russia so!). Could we teach then them to make 3d mental models? Not every time…

In addition, I have another experience of using NWPs and mental models during Olympics.

In the case of lack of the time and high stress situation, we invented our type of doing forecast using NWPs and synoptic models. During Olympics we (in Alpine Ski Venue, when I was working) had 7 weather stations in 5 levels, 2 radars, 2 microrain radars, 4-daytimes sounding, about 10 high resolution NWPs and 5 nowcastings. And we have only 1- 1.5 hour in the morning before the Sport Weather Briefing to make forecasts for next 12 and then 24 hours.

In the lack of time we were unable together in our team to check all observation, NWPs and then make our 3d mental models. We were 2 for the shift so we divided our duties in 2 parts  - my partner was checking all observations, satellites, maps and was making 3d synoptic mental model, whilst I was checking all NWPs, nowcasting and so on to make 3d NWP model. Then in the last half an hour we were changing our searches and opinions and then were doing forecasts together.

Maybe it is a little bit naïve and unprofessional, but it worked all the time during Olympics!

Lara

 

Attachment my team.jpg
In reply to Bruce Muller

Re: Week 2: Enhancing mental models

by Larisa Nikitina -

Good article! Have read it only now! Thanks a lot, Bruce, for sharing it!

I have never thought before whether I used metal models in my mind or I worked another way, but it is it. Yes – firstly large synoptic scale, then more detailed and 3d 0r better to say 4d weather image and only then NWPs.  And yes – my first belief or forecast was usually the correct one :)

At the end of article they tell about recommendations how to improve forecasting performance. Very interesting! Could we read this recommendations?

Regarding NWP, another thing I want to mention is that soon (maybe in 5 years or some more) the most meteorological service for aviation is going to be changed from provision meteo products (such as TAF, SIGMETS and so on) to transmitting meteo info (such as visualized NWP products) directly to aircraft board and pilot on the air. So what then about NWPs confidences and forecasters mental models?

 Lara

In reply to Larisa Nikitina

Re: Week 2: Enhancing mental models

by Tsvetomir Ross-Lazarov -

Great question Lara!

I don't have a good answer, just a few thoughts.  As more and more forecasters rely on NWP to create their short-term forecasts, their analysis, diagnosis and prognosis skills atrophy.  We are approaching a situation in which many forecasters have become so dependent on NWP that when NWP is wrong, their forecast is wrong (or vice versa).

If this becomes prevalent, decision-makers will have little choice but to dismiss the forecasters.  If they are only as good as NWP, why are forecasters needed?  No matter how much one argues that it is important to know the models, their biases, etc., if the forecasters are correct only when NWP is correct, forecasters could be working themselves out of a job.

So strengthening forecasters' abilities to create and adapt 3D mental models based on real-life observations may be a good way to help them question NWP forecasts and improve on them.  In that situation, NWP will be one tool among many, not the primary tool (crutch as Bryan called it).

In reply to Bruce Muller

Re: Week 2: Enhancing mental models

by Bruce Muller -

It's Friday! At least here in Boulder - for most of you, it will already be the weekend when you read this. In any case, I'm happy to be wrapping up the end of the work week here at COMET. 

Thank you, Vesa, Tsvet, Bryan, and Lara for posting to the forum this week. I can only hope that more folks took the time to read the article of interest for this week's session. My guess is that anyone with forecasting and training experience will be able to relate to the findings and it should spark ideas on how to enhance your approach towards training competent forecasters. Tsvet's last posting may come across as quite provocative but I certainly can see how his logic could become a reality. I sure would like to hear some of your thoughts on that...hint, hint...

We'll kick off the last week of this session on Monday: collecting examples of conceptual model lessons, images, animations, and ideas for creating conceptual models that are currently not available on MetEd or other venues. Have a great weekend, everyone!

Bruce