Scenarios: What are they good for?

“Why do you want to use scenarios?” your client asks. “Why can’t we use the quizzes that we’ve always used?”

Sometimes the best way to convince a client is to show them through examples. Present one of their quiz questions three ways, so the client can see for themselves the deeper thought required by a scenario-style question.

Here’s an example. What kind of thinking is required by each type of question?

1. Quiz question

Which of the following is the most secure way to carry sensitive data?

    A. On a laptop

    B. On a USB drive chained to your wrist

    C. On a CD titled “The Chipmunks Sing Disco Duck”

Feedback for incorrect answer: Incorrect. Try again.

2. Mini-scenario with correct/incorrect feedback

Bob wants to work on the salary data at home. He has a long commute on a train. How should he carry the data with him?

    A. On his laptop

    B. On a USB drive chained to his wrist

    C. On a CD titled “The Chipmunks Sing Disco Duck”

Feedback for incorrect answer: Incorrect. Try again.

3. Mini-scenario with “showing” feedback

Bob wants to work on the salary data at home. He has a long commute on a train. How should he carry the data with him?

    A. On his laptop

    B. On a USB drive chained to his wrist

    C. On a CD titled “The Chipmunks Sing Disco Duck”

Feedback for A: Bob falls asleep during the commute, and a thief steals his laptop and sells the data. Try again.

Feedback for B: Bob falls asleep during the commute. A thief sits next to him, plugs his USB drive into his laptop while Bob is unconscious, and later sells the data. Try again.

Feedback for C: Bob falls asleep during the commute, and a thief steals all his belongings. The thief breaks the CD into pieces in disgust and no one ever sees the data. This is the best choice.

Version 1, the quiz question, asks learners to regurgitate a fact with no context.

Version 2 puts the facts into a realistic context but directly tells the learner when they’ve made an incorrect choice.

Version 3 includes context and lets learners conclude on their own from the results that they’ve made an incorrect choice. The results also show why that choice wasn’t the best one.

We could include “telling” explanation in the feedback for version 2, such as, “Incorrect. A laptop is appealing to thieves and is likely to be stolen.” But that type of feedback abandons Bob on the train and drags the learner into the bland world of abstraction, plus it directly tells the learner that they screwed up.

Instead, the “showing” feedback of version 3 both keeps the feedback in the context of the question and requires slightly more thinking from the learner. It also emulates the way we learn in the real world — from experience, not from a disembodied voice that immediately tells us “incorrect.”

But quizzy questions take less time to write!

Yes, it’s quicker to crank out a bunch of abstract fact checks. Writing any type of scenario takes a little longer and requires you to work closely with your subject matter experts to make sure everything is accurate and realistic.

However, as I hope you can see from our silly example, a scenario-style question that puts the challenge into context and shows the result of each decision requires learners to think a little more deeply and independently. This at least makes your elearning more engaging and could lead to better transfer on the job.

Image © iStockPhoto: Menno van Dijk

Comments

  1. Luke Stollings says:

    Awesome post. I loved it and shared it widely this morning as soon as I read the email in my inbox. This helps me explain to people in a very clear, concrete way the difference between regular multiple choice and something approaching a scenario, and something that IS a scenario. Though I already believed “telling stories” is better, I hadn’t seen so clearly that it’s the feedback on correct and incorrect answers within the context of the story that can make the difference.

    By the way, this inability to give different feedback for each wrong answer that is a real frustrating limitation of the current version of SoftChalk. I’ve asked them to change that as a feature request.

    • Cathy Moore says:

      Luke, thanks for your comment, and please keep pushing your tool developer to allow for option-specific feedback. That small change can make a big difference!

  2. Tracy Thomson says:

    Thank you for the excellent post and the reminder to clarify our communication with others by providing examples. I’m currently working with a group of subject matter experts to develop an in-house certification exam. Each is responsible to write questions (draft versions) and the tendency is definitely to write ones that are “quizzy” and follow a more traditional format.

    If your tool allows you to include an image, scenario questions can be made even stronger by having learners pull information from things such as a diagram, building blueprint, or a map.

    • Cathy Moore says:

      Tracy, thanks for mentioning visuals. I agree that they (as well as audio and any other type of media) definitely have a place in scenario questions that simulate real-world decisions.

  3. Pat Tymchatyn says:

    It would be great to see the scenario that goes with that silly quiz since easy to say write one but what does it look like? I like the info.

    • Cathy Moore says:

      Pat, thanks for your question. The silly quiz *is* the scenario. It’s a one-scene mini-scenario. It contains the main ingredients of a scenario: a character that faces a realistic challenge, and feedback that shows the results of the learner’s decision. In my view, a scenario can be a one-scene decision point like the one in this post as well as a multi-scene branching adventure. This question would be included in material that includes many other similar questions covering many other security scenarios.

  4. Mary Vivit says:

    Thank you for another great article. I’ve used the more verbose feedback along with “quizzy” ones in the same course. For example (still using data security), I wouldn’t give much information up front (except maybe a link for learners to open the policy), then present the question with the verbose answer. After the question, I might give a quick summary screen. At the end of the course, I’d provide only the quick responses in the quiz.

    Great stuff, though!

    • Cathy Moore says:

      Mary, thanks for your comment. I agree (and should have described in the post) that the scenario-style question works well if you just plunge learners into it, with access to job aids as applicable (like in “How to create a memorable mini-scenario”). The feedback then supplies the information that the learner appears to lack, based on their choice. And the less-verbose mini-scenario like version 2 can do nicely as a quiz.

  5. Marcella Simon says:

    Great post! Roger Schank I believe said that a great motivator for learning is the embarrassment of trying something and failing. Scenarios are a safe way to do this. I have had groups write their own scenarios for their own workplaces with realistic choices and feedback. Talk about engagement!

    • Cathy Moore says:

      Marcella, you make a great point that it’s good to have learners write their own scenarios to use with their colleagues. A scenario requires deep knowledge of the context and of all the possible ramifications, and writing one in a small group is an excellent way to debate the grey areas and learn deeply.

  6. Awesome example, thanks! Visualization would not only empower the effect like Tracy wrote, but would also make the question easier and faster to analyze.

  7. Kathy Sierra says:

    Top-selling tech book authors Eric Freeman and Beth Robeson have said it takes them 80% of their time “writing” to develop the scenarios, and the remaining 20 to actually create the whole 500+ page book. I completely agree on the value of scenarios and also on how little it takes to set one up.

    I am not a big fan of giving the results in quiz feedback, though. While it is still a huge improvement over your first two versions, if possible I would prefer scenarios where the participant imagines for himself what could happen. So rather than being TOLD the guy falls asleep on the train, for example, a slightly smarter interaction would simply ask (after they choose a, b, or c) “are you sure? What could happen?”. One more level of not telling them what to think (e.g. The person could fall asleep) but what to think ABOUT (e.g. What might happen while the person is on the train?). Like Dan Meyer’s goal of being “strategically less helpful”.

    I realize this is much tougher to do in a lot of brain-dead software, and that your improvement is certainly much better than than the first two. And being shown a poor result (guy falls asleep) is still a good prompt for helping them start thinking through other potential outcomes.

  8. Anders says:

    Excellent examples, Cathy! Most useful! The main problem I see with multiple-choice based questions is that they quite often model real decisions poorly, whatever writing style is used. Many decisions (most?) are too complex to be represented by selecting the best way to go from a given set of options. As an alteranative to MC-based questions or exercises, I like the idea of rating (recorded) scenarios from a set of criteria.

  9. Anna says:

    The beauty of Cathy’s scenarios is that they can be created in a very low-tech and cost effective way. We use this methodology for our face to face as well as computer based training. All the people in our department were excited to get this blog link in their email this morning! Everyone can clearly see how effective it is to have users make decisions in a real life based scenario. Making the story and the wrong answers as appealing and realistic as possible takes careful consideration on the part of the instructional designer and the SME. This indeed is the majority of the project time in our experience.
    Thanks for another simple-yet-brilliant ISD methodology idea, Cathy!

  10. Cassaundra Wells says:

    Cathy,
    I think scenarios are an effective thought provoking communication tool to get individuals to think about their responses independently and with the subject matter experts. I agree that creating scenario’s can be a bit time consuming the overall objective is to obtain tangible feedback. The traditional “quizzy” questions remind me of learning about close-ended interview questions. If you want a short and sweet answer, then scenarios are not the best form to use unless it’s in addition to other methods.

  11. Jacinda Ferguson says:

    After reviewing the information provided Cathy, I appreciate the way that you presented the material in the various forms to visualize why the correct answer was what it was. The examples show potential clients how to look at the cognitive thought process of their employees in different eyes. Simply stated the scenario evokes thought and it was a pleasure to learn something new as an individual who is a novice instructional design.

  12. Felicity says:

    I have been looking through your blogs, and it all makes so much sense. My question is how we get from developing the Story, that is our quiz to make it live. We use Moodle as our LMS, so I am not really sure what to do with the next stage…..so I use something like captivate or articulate to make this happen? I really like this scenario below, and have used it to facilitate learning, it gets people very excited. http://icu-10.med.usyd.edu.au/ce/case1/index0.html
    I know it is pretty old in terms of the type etc, but the content and the process is excellent. This is the sort of thing we want to do for our health organisation….but is there some simple software I can do it on that I can then upload into Moodle?

  13. Michelle says:

    Cathy
    Utilization of scenarios in instruction enables IDT professionals to create learning that is more likely to be retained and retrieved at a later time. Scenarios activate the mind to make associations between the current content and prior knowledge and experiences. By connecting to prior knowledge, the learner is more likely to not only remember the new content, they will also be able to utilize the information in new ways. In my opinion, that higher level of thinking is where true learning takes place. Scenario questioning techniques bring the learner’s mind to a level of application and creation of similar scenarios to ultimately bring understanding of new material.
    Thanks for sharing
    Michelle

  14. TEasley says:

    Thank you so much for the post and for including examples. I am a secondary education teacher who incorporates eLearning into my brick-and-mortar classroom. Your post and examples really got me to thinking about what a difference I can make by using scenario type questions in this way with my adolescent students. I struggle with ways to involve higher-order thinking; this technique definitely fits the bill. In studying this concept more, I found another great posting that offers suggestions on how to develop these types of questions. The post, from a blog called Rapid eLearning, is entitled “7 Tips for Better eLearning Scenarios.” I am not associated with this blog at all; I simply found it helpful. If anyone is interested, the link is http://www.articulate.com/rapid-elearning/7-tips-for-better-e-learning-scenarios/.

  15. Heather Borelli says:

    Hi Cathy
    I am new to your blog and just starting out in instructional design, I really appreciated your focus on providing examples to demonstrate the value of a scenario style- question versus the standard multiple choice or true/false quizzes. The ability to give learners contextual feedback that encourages them to review and reevaluate the course’s content in order to better understand and encode the new information. Scenario-style questions within an assessment test enable learners to apply the new information learned in the course to their existing schemata to make associations and facilitate transfer of the information from working memory into long term memory (Ormrod, Schunk & Gredier , 2009) . I believe that scenario-style assessment tests are more representative form of assessment as they require learners to apply what they have learned without relying on memorization of facts. Thank you for posting this information and the accompanying examples.

  16. Jen says:

    Cathy,
    Great post! I not only agree with your mini scenario type of questioning as a great learning tool, but I find it hard to believe that this is not more utilized. For me, this type of question is so much more beneficial than just a simple true/false or multiple choice question with no feedback on why an answer is correct or not. I came across a website that identified different ways in which we learn best (http://globalcrisis.info/teachproblemsolving.html). Some of the reasons listed in this website that support your reasoning refer to when we come up with the wrong answer we will “retrace” our thoughts as to where we went wrong and this will help reinforce the correct answer. Isn’t this so true? Don’t we all learn best from our own mistakes and are therefore more likely to remember the correct answer? I know this is true for me. This website also states that we are more likely to learn when we have thought through the process of how we came up with the answer. Giving a scenario allows the learner to go through the proces and come up with a solution.
    I was recently reading the text for my Instructional Design course (Ormrod, Schunk, & Gredler, 2009). The chapter was about Cognitive Information Processing Theory and I came across a section regarding retrieval of information. What stood out to me the most was the more meaningful information is the more likely it will be stored in long term memory. Using the scenario based questions allows the learner to activate schema and retrieve information that has already been learned.

    Thank you for reinforcing ideas and methods that are important to for effective teaching.

    References:

    Global Crisis Solution Center. (2003, April 9). Teaching and Learning Through Problem Solving [Blog Posting]. http://globalcrisis.info/teachproblemsolving.html
    Ormrod, J., Schunk, D., & Gredler, M. (2009). Learning theories and instruction (Laureate custom edition). New York: Pearson.

  17. Jeff Davis says:

    This is such a great post. It reminded me of the Choose Your Own Adventure storybooks from my much, much younger days.

  18. Stacie says:

    Cathy,

    I couldn’t agree with you more on the quality and importance of scenario-based questions. I’m part of a team of Air Force instructional designers and one of our responsibilities is to write scenario-based test questions for each lesson we develop. Also, we include simple and complex scenarios in each lesson in order to engage students critical thinking. By doing so, we are asking our students to not only step outside of the box, but to get rid of the box by stretching their thinking beyond what is possible. When we are designing test questions and lesson scenarios, we do so in a way that draws from someone’s personal experiences. We do this because we want to be able to reach students on both the cognitive and affective levels, by saying “hey, this could you in this scenario faced with this problem, good, bad or indifference. What should you do based on the lesson principle(s) you just learned?” Again, we are addressing the students’ cognitive and affective levels. We have found by doing this, our students value the lesson(s) and the classroom discussions, which allows the students to bring in their personal experiences in order to enhance the lesson(s).

  19. Johanna says:

    Cathy, I was excited to encounter this article as it confirmed for me the efficacy of the method I’ve been using to develop testing of learners’ knowledge & application. Quiz questions absolutely take less time to write however I completely agree with the methodology of the mini-scenarios because it doesn’t just force the learner to recall facts but to put their problem-solving to work which is aligned with what they would encounter in real life. I’m currently developing an online course which has extensive short multiple choice exercises however the questions force them to reason and for every single wrong answer selected by the student, there is an explanation for why the question they chose is wrong but it doesn’t give them the correct answer making it possible to go through the questions multiple times for more comprehensive understanding. I believe this method also encourages metacognition in students as they are being empowered to understand the why and how of their incorrect thought patterns. I’ve experienced the increase in time/cost with heavier involvement of SMEs & developers, but the end result of a more meaningful and engaging learner experience is worth it. Thanks for posting!

Trackbacks

  1. [...] Text: Scenarios in elearning: Why they’re better than quiz questions. X .nrelate_popular .nr_sponsored{ left:0px !important; } /* .nrelate_related .nr_sponsored{ [...]

  2. [...] You may find a possible answer in The Guiding Principle for all Rapid E-Learners….. or maybe Scenarios: What are they good for? relates more to your training context.  Moving from a classroom to online does require some [...]

  3. [...] Here‘s an excellent article about using scenarios to test a learner’s understanding of the material being taught.  The author, Cathy Moore, provides three variations of answers to a multiple choice question. [...]

  4. [...] her blog, Cathy Moore explains that using scenarios will help significantly in, “make[ing] your elearning more engaging and [...]

  5. [...] her blog, Cathy Moore explains that using scenarios will help significantly in, “make[ing] your eLearning more engaging and [...]

  6. [...] her blog, Cathy Moore explains that using scenarios will help significantly in, “make[ing] your elearning more engaging and [...]

  7. […] Designing questions so that it doesn’t matter if students look for answers online. Multiple choice questions that test recall make it easy to cheat. Short answer questions that require higher order understanding are not only harder to cheat, they do a better job of assessing what your students have learned. The University of Leeds has some helpful resources for developing exam questions, and blogger Cathy Moore offers a short but good post on scenario based questions. […]

x
Scenario design workshops

This September and October! Click here to learn more.