Formulate open exam questions

Open-questions are closer to professional situations than closed-ended questions (like for example multiple choice questions and true-false questions). In addition, students typically describe their thought process in their answers, which allows lecturers to reward partially correct answers and give feedback.

In case of remote exams, the risk of fraud is higher than for on-campus exams. Answers to open-ended question are more difficult to (illegally) share compared to answers to closed-ended questions. Therefore, open-ended questions are preferable.

The guiding principles for exam questions are the quality requirements for assessment.  

So, you are about to write open-ended questions. Watch this video or read the following ideas/points to get started:

1. Your exam questions should be challenging, interesting, fair and, if possible, fun!
2. Exam questions should reflect the learning objectives of the course (validity).
   Identify the action verbs in your learning objectives, and then make sure that your questions cover the same cognitive level (level of difficulty, e.g., remembering vs analysing).
3. Make sure you are testing what you promised students would learn. Let them know which course learning objective(s) is/are covered in the exam.
4. The learning objectives will also determine the type of exam questions: for example, would you ask your students to write an essay if you wanted to quickly assess how well they have understood a broad range of topics? Likewise, asking students to summarize something in 1-2 sentnces does not make sense if what you want is students to explain their understanding and demonstrate creativity.
5. Write clear instructions; if there’s any ambiguity in what they need to do, students will struggle.
6. Ask a colleague to proofread your questions. They could use the checklist below.
7. Simplicity is best! Don’t write complex sentences or use idiomatic language that students may find difficult to understand (there are a lot of international students in TU Delft!).
8. To test a deeper level of understanding and insight, ask the students to justify their answers.
9. You can also write a case study (real-world examples) and ask a couple of short questions on the case to test.
10. Ask your students what kind of questions they would expect to see in the exam.

When constructing exam questions, use the following 3-part structure:

1. Context (optional)
2. Question (assignment)
3. Directions for answering, for example, "Justify your answer, showing which formulas you used. Write no more than 3 sentences."

Context (1) and question (2) should be linked

• Make sure the context is relevant for the question. If not, delete it.
• If the question can be answered without using the context, then change/remove the context OR change the question.
  Unless a learning objective is to filter out irrelevant information, of course. 
• Avoid “anything goes” questions such as "What do you think…"

Directions for answering (3)

• Use imperative sentences (e.g. “List three characteristics of X”).
• Specify what you expect in the answer (e.g., “List the three characteristics of X”).

Developing your answer model will help you tweak your exam questions. Therefore, develop them in parallel. The reason for this is that you will check back and forth:

1. Do my students have all the information to produce a correct answer?
   If not, do you want to add background information?
2. Could the question lead to other answers that correctly answer the question?
   Are these answers that you wanted them to produce or do you want to clarify the question?

If you are doing short-answer essays or extended essays, these are subjective and allow students to organize their arguments and mostly present an original answer. You will be giving them a lot of freedom to respond, but in return, essay questions are always more difficult and time consuming to grade. You will be asking your students to analyse, classify, compare and contrast, define, give an example, discuss, summarise, and so forth.

1. Do not focus on snippets of content. Try to put the questions in context so that the students can see how the knowledge is used in the real world.
2. Always include the point allocation for every answer, and/or include an estimate of how much time they should be spending on the task.
3. Indicate on what criteria and aspects you will be marking their answer and if some aspects will have a higher weight than others.
4. Give instructions on the expected length of the answer, or number of arguments you expect them to bring up.
    

Information for students on the exam: 

• During the test/assignment, are the points to be earned by each question or subquestion announced? This way students can budget their time to be most impactful for them. They should not spend a lot of time on a question that will not earn them a lot of points. 
• Before taking the test/assignment, do students know ahead of time what will be on the test both in structure and in content? 
• Before taking the test/assignment, did your students get experience with the types of questions with which you will be testing? 
• After getting the grade and feedback, does the student get information on how her grade has been calculated, and on how she can improve her performance, for example per learning objective, criterion or subquestion? 

Lay-out / language of the exam: 

• Is the layout clear? 
• Are the figures clear? 
• Are there any spelling errors or typos? 
• Have double negatives been avoided? Is the question concisely formulated? 
• Any copy-paste errors? 

Question quality: 

• Is the answer model in line with what the test questions ask? 
• Is the question unambiguous and is it clear what is being asked? 
• Is the tested knowledge or skill necessary to answer the question? 
• Does each subquestion cover no more than one learning objective? 

You can use an assessment matrix (for an example, see Table 1) to check whether the division of questions over the learning objectives (or topics) is in line with the time spent on this learning objective / topic during the course. Furthermore, it can help you to check whether the level of Bloom of the questions is in line with the level of Bloom of the learning objectives. This is a tool to ensure validity and constructive alignment (see this video on constructive alignment) of your exam.

Table 1: Assessment matrix for an existing exam based on the learning objectives listed previously. Q = (sub)question number, P = points per (sub)question. Intended level of Bloom are bold in the matrix.