Forum: Question Challenges

Hi Jose,

This is the place!  I will study your objectives and questions and get back to you with any suggestions.

Thank you for choosing to participate!

Tsvet

Jose,

Thank you for posting your objectives and questions!  It is delightful that you have decided to give the ideas from the live session a try.  As you mentioned, sometimes it is difficult to distinguish among various outcomes.  Here are a couple of examples that you might find useful for distinguishing between comprehension and analysis:

• “Comprehension is interpreting information in one’s own words.” An example of that is to “describe the relationship between pressure and volume” (Morrison, Ross, Kemp, 2001, p. 87). If we were conducting a test in person, we’ll listen to the learner describe the relationship in their own words and determine whether the description was correct.
   
• “Analysis is breaking an idea into component parts and describing the relationships between the parts.”  An example of that is “explain why an automobile’s tire will not appear underinflated after being driven several miles at a high speed” (Morrison, Ross, Kemp, 2001, p. 87). The learner will have to consider the processes that are involved in the tire scenario, perhaps break them down (tire is spinning at high speed that will experience friction; friction causes rise in temperature and so on).  Then they may need to break down their own explanation of the relationship between pressure and volume, and identify the role played by temperature and figure out how that relates to the tire.

The first objective and question you created are a great starting point for an objective measuring comprehension.  You also did a good job of limiting untended clues in the question and options.  I’d like to offer you a few suggestions about both the objective and the question.

You would like the learners to: “realize that some ground scenes produce enough energy to be detected by satellites through solar channels.”  Realization or understanding happens inside our minds yet they have outward expressions as well.  It is the outward expression that enables us to determine if the realization has occurred.

One expression of this understanding is being able to identify the ground scenes that produce enough energy to be detected through the satellite’s 1.6 µm solar channel.  If we were to observe a meteorologist working with the 1.6 µm solar channel images and he/she was able to determine the ground scene that produced the image, we would accept that as evidence that they have the understanding.  The understanding is necessary in order to successfully complete the task.  I’d like to encourage you to state the objective in more performance-oriented terms and avoid using realize, understand, see, believe.

With this in mind, if I were to write this objective it will say something like:
Identify ground scenes that produce enough energy to be detected through the satellite’s solar channels.

You have correctly identified the task that directly measures the objective and the question type that directly measures the task:
1. The task is to identify the ground scenes.
2. A question that measures this directly is to list a number of ground scenes and ask the learner to select the ones that will show up on the images.  As you will see, I have modified the original question.

Original Question
Can any heat sources on the Earth surface appear on the solar channel centred at 1.6 µm?
A)     No, the image is only made of sun radiation reflected on Earth during the day, and just instrument noise at night.
B)     Yes, night light generated in towns shows weakly in that channel.
C)     Yes, strong fires can be detected at night.
D)     Yes, pixels with areas of hot lava from eruptions give a signal at night, and even under day light.


Modified Question
Which of the following heat sources on the Earth’s surface will appear on the 1.6 µm solar channel?  Select all that apply.
a. night light from towns
b. strong fires at night
c. lava from volcanic eruptions
d. cooking fires at night

The modified question no longer contains option A from the original question.  The question is a multiple select question and option A precludes the other three from being correct.  If the learners select A, it will be illogical to select any of the other options; thus since this question is supposed to have multiple answers, the learners will eliminate option A right away.

It is quite fine to create two or more questions to test a single instructional objective.  In my experience, an objective can require three or more questions.  So, in this case, I’d suggest that you consider creating a simple comprehension question for the information in option A.  For example:

Select True or False.
True/False     The solar channel centered at 1.6 µm is made of solar radiation reflected on Earth during the day and instrument noise at night.

When writing question options, it is also a good idea to avoid words like only, always, never, etc. as they give students clues about the correctness of the answer.

I hope that you will find this discussion helpful.

I need a bit more time to compose my feedback about your second objective and question.  I will post it as soon as it is ready.  Thank you for your patience!

Jose,
Thank you for your patience!  Here is my feedback about the second question.

The analysis objective and question are more difficult to create.  You have done a good job at avoiding unintended clues in the answer options.   The stem presents a definite problem but will need to be revised as the current question operates on the level of comprehension rather than analysis.  The learning objective also needs to be edited in order to reflect a task on the level of analysis.  As I was preparing my feedback, I realized that the task that you selected for this question has a challenging aspect, so it is very understandable that the objective and questions need some work.

It seems that you would like your learners to choose the right satellite channel that will enable them to identify the presence of a cloud and determine its thickness.  (I hope that this is correct, otherwise you will find the discussion below only marginally useful.)  This objective has elements of comprehension and application.  So a performance-oriented objective would read something like this:

Choose the satellite channel best suited for identifying the presence of a cloud and determining its thickness.

This objective has three tasks: “select the channel,” “identify the presence of a cloud,” and “determine the thickness.”  The first task operates on the level of comprehension, the last two on the level of application.  Identifying the cloud and determining its thickness require the learner to use their knowledge about satellite images to accomplish these tasks.

When thinking about an analysis level objective, we come up against the challenging aspect of the task you selected.  Identifying the presence of the cloud and determining its thickness is one information point that by itself cannot be used to make a forecast decision.  It needs to be combined with other information like the cloud top temperature, in order to determine the potential for precipitation or aviation icing, for example.  (Not being a meteorologist, I consulted with my COMET colleagues about this, so please let me know if my understanding is incorrect.)

Because of this aspect, a learning objective for analysis based on a real-world task will also reach to the levels of synthesis and perhaps evaluation.  Here is one possible version of a learning objective:

Analyze satellite products from the thermal, solar, and near infrared channels and estimate the probability of ice forming on aircraft.

The task is to analyze three satellite products and combine the information gleaned from them into an estimate of the probability of icing if an aircraft flew through the cloud.

Here is one possible question:

Review the three satellite products and answer the question below. (Here we’ll show thermal, solar, and near infrared images or animation loops.)

An aircraft is running out of fuel and needs to land at an airport with heavy cloud cover.  The airport tower has called your office and would like to know, what is the probability that ice may form on the aircraft as it descends through the cloud?

a. high probability, because the cloud top height is …. the approximate thickness is …. and the clouds contain particles in … phase.

b. high probability, because the cloud top height is …. the approximate thickness is …. but the clouds contain particles in … phase.

c. low probability, because the cloud top height is …. the approximate thickness is …. but the cloud contain particles in … phase.

d. low probability, because the cloud top height is …. the approximate thickness is …. and the clouds contain particles in … phase.

Naturally, the wording of the options may need to be adjusted to match the information available in the satellite products.  We’ll need to avoid unintended clues as well.

Please let me know if I have understood your objectives correctly.  Do the suggestions make sense?  If others are monitoring this discussion thread, do you have any thoughts or suggestions?

Tsvet