"Providing Feedback in Computer-based Instruction: What the Research Tells Us"

Prepared by Beryl Jean Maxon and Roger Bruning

Previous Thought in Research
Feedback was believed to either weaken or strengthen the learner's responses.
It enabled educators to think more positively about feedback but was a narrow view of what feedback could ultimately contribute.

Types of Feedback
There are two basic types of feedback.
Verification labels an answer as correct or incorrect.
Elaboration gives more information to lead the learner to a correct answer.

• Informational elaboration gives information that the learner can use to reach the correct answer.
• Topic-specific elaboration gives more detailed information about the topic and leads the learner through the correct answer. It does not address incorrect responses.
• Response-specific elaboration gives information on both correct and incorrect responses. The learner finds out why the incorrect answer was wrong.

The current belief is that feedback needs both verification and elaboration.

Types of Feedback (used individually and in combination)

• No-feedback: no verification and no elaboration 
• Knowledge-of-response feedback: verification and no elaboration 
• Answer-until-correct feedback: verification and no elaboration (The learner answers it again and again until he or she chooses the correct answer.) 
• Knowledge-of-correct-response feedback: verification and no elaboration except telling what the right answer is 
• Topic-contingent feedback: verification and elaboration containing information with the correct answer 
• Response-contingent feedback: verification and elaboration based on the learner's response 
• Bug-related feedback: verification and elaboration on the error the student may have made (It allows the learner to discover where they went wrong and determine the correct answer.) 
• Attribute-isolation feedback: verification and elaboration on the general concept

Factors That May Influence the Feedback's Effectiveness
Elaboration. some studies show that elaboration has no effect, but more studies show that elaboration helps increase learning. There is a "trend for increased learning in response to more elaborate feedback," but there are apparently other variables at work.
Student achievement levels. immediate feedback better for low-achieving students while higher ability students may receive more benefit from feedback that allows them to process information on their own
Depth of knowledge. There weren't exact answers in this area (in fact, results conflicted) but learners with more understanding could get more out of answer-until-correct feedback than learners with less understanding. Feedback effects seem to be weaker for higher-order learning, but learners at a lower level could benefit more from delayed and knowledge-of-correct-response feedback than from answer-until-correct or no-feedback.
Attitude toward feedback. Students seem to want "more elaborative, immediate feedback" (11). So giving learners feedback that they believe helps is probably a good idea even though the attitude may not actually impact learning outcomes.
Learner control. It may be beneficial to allow high ability learners control over the type of feedback they receive. However, it may not be as good of a policy to allow low ability learners to choose their feedback type, because low ability learners may just want the correct answer instead of the feedback that will provide the highest amount of learning.
Response certitude. A correct answer may be from a guess or from total understanding. An incorrect answer may be from a simple mistake or from a total lack of understanding. The impact of the feedback would be different depending on the reason.

1. High certitude and correct answers. the learner does not spend much time on feedback.
2. High certitude and incorrect answers. the learner wants to find the reason for the error. (This situation is possibly when feedback is the most helpful.)
3. Low certitude and correct/incorrect answers. the learner may not receive as much benefit from the feedback if the understanding (and thus certitude) is low. However, a study by Mory (1994) found that learners with low certitude spent more time studying the feedback.

There was not a significant difference in learning when the feedback was adjusted for certitude and correctness.Timing. There is a debate between whether immediate feedback is better due to immediate correction before the error is ingrained or whether delayed feedback is better so that correct information can be studied without interference. It seems immediate feedback is better for applied studies, list learning, decision-making, novel information tasks, and lower level knowledge based tasks. It helps with learning the content. Delayed feedback is better for learning test content, abstract concepts, and application/comprehension skills. It helps with remembering content for the long-term. Both types could be used. For example, the student can know immediately whether they were correct (verification) but learn later about the information behind the correct answer after having some time to think for themselves (elaboration).

Conclusions and Recommendations
"The challenge therefore is to identify the type of feedback that is most effective in specific educational settings" (14).

Student Achievement	Lower Level		Higher Level
Task Level	Lower level task	Higher level task	Lower level task		Higher level task
Timing of Feedback	Immediate feedback		Immediate feedback		Delayed feedback
Prior Knowledge	Low prior knowledge	High prior knowledge	Low prior knowledge	High prior knowledge	Low prior knowledge	High prior knowledge
Type of Feedback/Level of Elaboration	Knowledge-of-correct-response feedback with response-contingent feedback	Knowledge-of-correct-response feedback with topic-contingent feedback	Knowledge-of-correct-response feedback with RC feedback	Knowledge-of-response feedback with TC feedback	Knowledge-of-response feedback with delayed knowledge-of-correct-response feedback + RC feedback	Answer-until-correct feedback and delayed TC feedback

Questions

1. Is it fair to split up lower order learners and higher order learners? Is it a reality or a label? How can we create higher order learners from low level learners? Can we instill it in them from the very beginning?
2. Is there a way to overcome the problem in student's scores reflecting guesses and simple mistakes? After all, a learner could memorize a chapter and spit it out but still not understand it. Is there a new type of "question" we haven't thought of?
3. Is our view of feedback still narrow? In what ways could it be broadened?

Backwards Design and Outcomes

Backward Design for Forward Action"
Summary. The article focused on a "backwards" process of raising student performance. It's only backwards compared to the normal order people go in. Instead of immediately rushing in with an action to fix student performance like usual, the article stressed that other steps should be taken before trying to fix the performance. The first step is identifying desired results. After all, how do you know what to do to "fix" the performance, if you don't even know what you want to achieve? It emphasizes teaching key concepts that are behind the standardized test questions rather than teaching the narrow aspects of the specific test questions. The second step is to analyze multiple sources of data. Get the students to "apply their learning . . . and explain their responses" instead of giving them strictly multiple choice examinations (53). Improvement teams should then write up the data in a summary, which is easier for educators to understand. The third step is developing the action plan. Identify the root causes of the low achievement, and then finally implement the action plan.
Experience. I think it would have helped me in school to be taught the key concepts behind the narrower aspects of learning. I thought history was a rather useless subject; in fact, I can remember having a conversation with a fellow student about which subjects we could understand the reasons for and which were unnecessary. I believe history was placed in the "unnecessary" category. We were still in elementary school. I wouldn't come to the same conclusion today, but that is because I understand how history applies to life and can teach us about what mistakes to avoid in the future. The article gave an example about identifying key concepts in history and gave the question, "What can we learn from studying other places and times" (53). I believe if we had been taught such key concepts to apply our learning to, I would have been able to be more excited about learning history because I would have understood the purpose it served.


"What Is Backward Design?"
Summary. This chapter starts off identifying the teacher as a designer. The audience for this design is the students and the constraints of the design include both the standards for the teacher to meet and also the students needs. The reason it is called "backward design" is because it's backward from the usual method not backward in effectiveness. The first step is identifying the desired results based on what is worth the students being familiar with, what is important for students to know and do, and what students should keep as "enduring" understanding. The second step is determining acceptable evidence. The ways of gathering this evidence include casual checks for understanding, observation, and conversations with the students. The ways also include quizzes, tests, academic prompts, and application tasks and projects. The third step is to plan the learning experiences and instruction that go along with the decisions made in the first two steps.
Experience. I think a simple negative experience can be seen in my reaction to math problems in high school. The teacher would work through problems on the board that I understood and could handle. The difficulty came because the homework had more complicated problems. I either didn't understand how to apply what I had learned in the simple problems, so the information was not presented in the broader frame, or the teacher legitimately did not identify the more difficult problems as the learning assessment goals and then did not teach me so that I could meet those assessment goals well.


"Understanding by Design"
Summary. The introduction gives several key definitions and explains that the rest of the book will be looking for the definition of understanding. The first chapter was "What Is Backward Design" and is summarized above. The second chapter, entitled "What Is a Matter of Understanding," was new material that I will summarize here. It is important to determine what the audience needs to have uncovered. To do so, the teacher needs to know the subject. There are different levels of specificity in the standards that teachers are given: topical statements, general understandings, and specific understandings. The last part of the chapter focused on the importance of questions. Questions help get the students from being passive knowledge accruers to active learners. Asking questions also helps to encourage rethinking. Teacher's shouldn't be satisfied with just getting answers from students but should look for the students to start asking more questions from the answers they come up with. Entry-point questions help lead students to the more general and overarching questions. The entry-level questions help students relate the subject to their real life. 
Experience. I don't think I was encouraged to ask questions in my education. Sure, teachers would try to pound into us that "good students" don't just accept textbook answers. But those lectures just made me feel like a bad student, because I didn't. The teachers didn't teach me how to get into the deeper questions. The classes just weren't set up that way. Every once in a while, a teacher would say something that would get me thinking. For example, one professor (not an editing professor) said, "The best editing is invisible." I've continued to reflect on this statement and question it and come up with my own answer in my mind. I think I would have enjoyed a discussion in an editing class on that whether the best editing is invisible or not. Questions really would have improved my thinking.


"Developing Learning Outcomes"
Summary. Learning outcomes use course lessons as a means to the ends of students' learning. They help the teacher focus the teaching and provide students with a clear understanding of where their learning is going. The outcomes should be specific with action verbs.
Experience. A learning outcome for my major is the following: Identify the linguistic structures of present-day and historical varieties of the English language in terms of sounds, sound patterns, word-formation processes, grammar, meaning and discourse. This outcome uses the action verb "identify" and the terms in which we are expected to identify the structures are specific. It might have helped to be introduced to these more in my classes. I have some teachers who went through the class's learning outcomes, but I don't think I've ever had a class that related them to the overall outcomes of the major. It would have helped me relate to how the class would help me achieve my learning goals.


"Techniques & Methods for Writing Objectives & Performance Outcomes"
Summary. All three formats for writing learning objectives were similar. The Mager Format is the most basic. It includes the performance (what a student should be able to do), conditions (what situation the student is working in), and criterion (how well the student should be able to do and what expectations they should be able to meet in the performance). The second format, Gagné and Briggs, includes situation (conditions), learned capability (performance), object (performance), action (performance), and tools and other constraints (criterion). Object and tools are additions to the Mager Format. The ABCD format includes audience, behavior (performance), condition (condition), and degree (criterion). The audience is added to the Mager Format's basic items for inclusion
Experience. I see these in the syllabi of my courses, such as the following one in a course this semester: "Students who successfully complete the requirements for this course [audience] will be able to demonstrate [performance] that they have acquired an understanding of the Isaiah text in its historical and cultural context." This is less specific. It includes nothing of what the performance is that the student will do with this acquired understanding. It does not mention the criterion at all. Nor does it mention the conditions the student will be performing in. It does mention the audience and the presence of a performance, but that is about all. There are no mentions of action or tools.


"Bloom's Taxonomy"
Summary. This article began with a short history of Bloom's Taxonomy. It was going to be one leg of a three-part study. The researchers came up with six learning levels, and then a new team, which included one of Bloom's students and also included a co-editor of Bloom's taxonomy, performed a new study and created the Revised Bloom's Taxonomy after six years of work. It changed all of the nouns to verbs and changed "synthesis" to "creating" and "knowledge" to "remembering." It also switched the order of the top two levels so that "evaluating" was then second from the top and "creating" became the top level. The revised taxonomy also added a second dimension with the four knowledge dimensions: factual, conceptual, procedural, and meta-cognitive. The revised taxonomy was also designed for a broader audience. The article then stressed the usefulness of the revised version in classrooms, giving an example of how it was used to design a history and literature team course.
Experience. I was in a class on language and literature--not the most common of course combinations--that moved us through the revised Bloom's Taxonomy. On the midterm, our instructor gave us rhetorical terms and short quotes from Emily Dickinson poems, and the class was tested on our understanding of the terms by whether we could apply the terms to the correct quote from the poem. In the next big project, a paper, I analyzed the use of quantifiers to determine how they were related to each other and the characters that used them. For the final, we moved on to the "creating" stage and wrote a hymn using the devices we'd been learning about during the semester.

My Definition of Design

To me, design is organizing materials to achieve some effect under the given constraints.

The first key component of this definition is "organizing materials." It is an important aspect because it implies that choices need to be made when organizing the materials. The word "materials" is general enough to apply to diagrams in a CAD program, couches in interior design, fabric in clothing, visual elements on a poster, etc.


The second key component is "to achieve some effect." There needs to be a purpose guiding the choices made when organizing the materials. When I make a flyer, I don't throw the elements on willy nilly. I have a purpose (generally to advertise something, such as that a journal is calling for submissions) and a target audience (students at Brigham Young University). I then organize the materials (dates for submissions, graphic[s], journal name, what types of submissions are desired, etc.) according to what the target audience will need/want to see in order to answer the purpose of the flyer (submitting papers and artwork to a journal).

The third key component, "constraints," came from a professor and is the difference between my current definition and the definition I had before I began my class in instructional design. I had originally left this component out of my definition, probably because the constraints are always present so I took them for granted. Some obvious constraints would be time, money, space, and materials. All of these factor in to limit the design in some way. Whether these constraints cause or prevent the "perfect" design is a separate debate and will be saved for some future date if I address it at all.

I will settle for now on addressing what makes for "good" design. I think this depends on whether we're discussing "good" in terms of "good enough" or as the lowest level in the progression of "good, better, best" or just as a positive adjective acting in opposition to "bad." They all kind of tie in to each other, but I'll start with the last one.

So what makes good design as opposed to bad design? I'd say a "good" design is a design that accomplishes its purpose. Little details (such as whether a shirt has a wide belt or a skinny belt, whether the bathroom door has a hinge on the right or the left, or whether the date is above the time on a flyer) all fade into the background if neither option prevents the purpose of the design from being accomplished. If the design of the earlier flyer convinces students to submit artwork or papers to a journal, then it is a good (enough) design, whether or not it could have been better or was in fact the best design possible.