Formulate open exam questions
This article contains tips and tricks for drawing up open exam questions.
Open-questions are closer to professional situations than closed-ended questions (like for example multiple choice questions and true-false questions). In addition, students typically describe their thought process in their answers, which allows lecturers to reward partially correct answers and give feedback.
In case of remote exams, the risk of fraud is higher than for on-campus exams. Answers to closed-ended question are more difficult to (illegally) share compared to answers to closed-ended questions. Therefore, open-ended questions are preferable.
The guiding principles for exam questions are the quality requirements for assessment.
How to write open-ended exam questions?
So, you are about to write open-ended questions. Watch this video or read the following ideas/points to get started:
- Your exam questions should be challenging, interesting, fair and, if possible, fun!
- Exam questions should reflect the learning objectives of the course (validity).
Identify the action verbs in your learning objectives, and then make sure that your questions cover the same cognitive level (level of difficulty, e.g., remembering vs analysing).
- Make sure you are testing what you promised students would learn. Let them know which course learning objective(s) is/are covered in the exam.
- The learning objectives will also determine the type of exam questions: for example, would you ask your students to write an essay if you wanted to quickly assess how well they have understood a broad range of topics? Likewise, asking students to summarize something in 1-2 sentnces does not make sense if what you want is students to explain their understanding and demonstrate creativity.
- Write clear instructions; if there’s any ambiguity in what they need to do, students will struggle.
- Ask a colleague to proofread your questions. They could use the checklist below.
- Simplicity is best! Don’t write complex sentences or use idiomatic language that students may find difficult to understand (there are a lot of international students in TU Delft!).
- To test a deeper level of understanding and insight, ask the students to justify their answers.
- You can also write a case study (real-world examples) and ask a couple of short questions on the case to test.
- Ask your students what kind of questions they would expect to see in the exam.
When constructing exam questions, use the following 3-part structure:
- Context (optional)
- Question (assignment)
- Directions for answering, for example, "Justify your answer, showing which formulas you used. Write no more than 3 sentences."
Context (1) and question (2) should be linked
- Make sure the context is relevant for the question. If not, delete it.
- If the question can be answered without using the context, then change/remove the context OR change the question.
Unless a learning objective is to filter out irrelevant information, of course.
- Avoid “anything goes” questions such as "What do you think…"
Directions for answering (3)
- Use imperative sentences (e.g. “List three characteristics of X”).
- Specify what you expect in the answer (e.g., “List the three characteristics of X”).
Developing your answer model will help you tweak your exam questions. Therefore, develop them in parallel. The reason for this is that you will check back and forth:
- Do my students have all the information to produce a correct answer?
If not, do you want to add background information?
- Could the question lead to other answers that correctly answer the question?
Are these answers that you wanted them to produce or do you want to clarify the question?
- Do my students have all the information to produce a correct answer?
If you are doing short-answer essays or extended essays, these are subjective and allow students to organize their arguments and mostly present an original answer. You will be giving them a lot of freedom to respond, but in return, essay questions are always more difficult and time consuming to grade. You will be asking your students to analyse, classify, compare and contrast, define, give an example, discuss, summarise, and so forth.
- Do not focus on snippets of content. Try to put the questions in context so that the students can see how the knowledge is used in the real world.
- Always include the point allocation for every answer, and/or include an estimate of how much time they should be spending on the task.
- Indicate on what criteria and aspects you will be marking their answer and if some aspects will have a higher weight than others.
- Give instructions on the expected length of the answer, or number of arguments you expect them to bring up.
Use tools to check your exam and questions quality
Information for students on the exam:
- During the test/assignment, are the points to be earned by each question or subquestion announced? This way students can budget their time to be most impactful for them. They should not spend a lot of time on a question that will not earn them a lot of points.
- Before taking the test/assignment, do students know ahead of time what will be on the test both in structure and in content?
- Before taking the test/assignment, did your students get experience with the types of questions with which you will be testing?
- After getting the grade and feedback, does the student get information on how her grade has been calculated, and on how she can improve her performance, for example per learning objective, criterion or subquestion?
Lay-out / language of the exam:
- Is the layout clear?
- Are the figures clear?
- Are there any spelling errors or typos?
- Have double negatives been avoided? Is the question concisely formulated?
- Any copy-paste errors?
- Is the answer model in line with what the test questions ask?
- Is the question unambiguous and is it clear what is being asked?
- Is the tested knowledge or skill necessary to answer the question?
- Does each subquestion cover no more than one learning objective?
You can use an assessment matrix (for an example, see Table 1) to check whether the division of questions over the learning objectives (or topics) is in line with the time spent on this learning objective / topic during the course. Furthermore, it can help you to check whether the level of Bloom of the questions is in line with the level of Bloom of the learning objectives. This is a tool to ensure validity and constructive alignment (see this video on constructive alignment) of your exam.
Table 1: Assessment matrix for an existing exam based on the learning objectives listed previously. Q = (sub)question number, P = points per (sub)question. Intended level of Bloom are bold in the matrix.
Intended Bloom level Bloom’s cognitive levels Total points (% of total score) Remember Understand Apply Analyse Q P Q P Q P Q P LO1 Remember 1a
10 LO2 Remember 1c
10 LO3 Understand 1e
8 LO4 Analyse 1e
2c 5 3 5 15 LO5a Understand, apply 1d 7 1e
17 LO5b Understand, apply 3 5 3 10 20 LO6 Analyse 3
20 Total 20 30 25 25 100