Ans at the TU Delft
Ans is a web-based system that supports the creation of interactive assignments for Formative and Summative assessment. It has won the digital assessment tender and will be TU Delft’s assessment system for all basic assessments in the coming years.
How does Ans relate to other assessment tools?
Figure: The assessment tool landscape of the TU Delft
1. Ans: Dedicated exam tool with practice options:
Ans is the standard exam tool, and a dedicated exam tool that portrarys some options for students to practice. Lectures can use Ans for:
-
Digital exams are administered in digital exam rooms, in the TU Delft secure exam environment. Students enter answers by clicking and/or typing.
-
Students take the exams on their laptops. Students can use the laptop’s camera to insert their pen-and-paper answers into the answer field in case of open questions. Ans can be taken in combination with RPNow (online proctoring software). This requires permission from the Board of Examiners and can only be used in specific situations.
-
In paper-based exams (this is called ‘written assignments’ in Ans), students answer questions in answer boxes on paper. Each exam booklet has a unique QR code at the bottom, which is linked to the student number that the student enters on the cover page. Completed exams are scanned, and graded online, per question, by the examiner’s grading team. Students can review the graded exam (including notes) online.
-
This is a combination of both digital and paper-based exams.
-
These are assignments are linked to a Brightspace course. The reason to link individual Ans assignments to Brightspace and not ask students to enter Ans directly or approach the individual assignments on Ans is because lecturers cannot set the order of the assignments in Ans.
The order of assignments in Ans depends on which one opened was currently viewed by the student. The order cannot be set to be fixed. Another good thing is that you can sync the scores from the assignment to the Brightspace gradebook. You can also sync the scores from the assignment to the Brightspace gradebook.
Here you can find how to link to Ans assignments from Brightspace.
2. Dedicated practice tools with a summative layer
In addition, there are dedicated practice tools that have a summative layer. This means that these tools were created to help students learn during the course (so for formative use). They do have options for use in exams (‘summative layer’), but they were not designed to be used for exams and therefore have some limitations. Examples are:
3. Learning management system
Our learning management system Brightspace is used for formative and summative assessments:
- Students can hand in assignments that are graded using e.g. rubrics
- Students can do formative quizzes
- Lecturers can keep track of students' partial grades in the Gradebook
4. Other tools
- Fraud prevention and detection tools, like Ouriginal (plagiarism tool) and RPNow (online proctoring tool)
Why are we happy with Ans?
- Useful analyses: It provides insightful analyses that help you directly improve your answer model.
- Insight into consequences of changing a subquestion to a bonus question: In addition, it directly shows the consequences for the passing rate and average score if you change a subquestion to e.g. become a bonus question.
- Positive student experience: The student experience is good, and it has standard options that increase the accessibility for students with e.g. dyslexia.
- Communication with students during an exam: It is possible to send announcements to all students and answer questions from individual students via chat during exams.
- Voluntary question bank use: Work with and without a question bank.
- Students can review (formative) assignments: Students can review each other’s or their answers, using an answer model you provide.
- Online student review: (students going over their reviewed work) The review can take place online. Students can add questions/comments online. This review can take place online, or in the secure digital exam environment of a digital exam room.
- Group assignments are also possible.
- Exam and grading preview: It is possible to test the student view and grading view with the answer model.
General information on using Ans
-
Ans uses slightly different terms compared to the TU Delft. Here is an overview of the differences:
TU Delft
Ans
Assessment/exam
Assignment
Question (1)
Exercise
Subquestion (1a, 1b, etc.)
Question
-
The workflow in Ans is shown in the picture above and listed below:
- You will edit the assignment (exam/test) which holds the exercises (questions).
- Students will take the assignment. This tab will lead you to an overview of the students taking the exam.
- You will review (grade) the students’ answers with your grading team. You can do that under the review tab.
- You can check an overview of the student results (scores) under the ‘Results’ tab.
- You can find insights (analyses) under the ‘Insights’ tab.
- You can change the settings of the assignment (exam/test) at any time. This also applies to what students can see when while reviewing the test.
-
Yes, you will. The one for exams-summative (including print-scan exams) will have the extension ‘Exam’, while the formative one that will be coupled to your course’s Brightspace page will have the extension ‘Brightspace’.
As of academic year 23/24, grades from Ans can be pushed to Brightspace Gradebook.
The flow is as follows:- Your 'Brightpace Ans course’ (which is called ‘course code Brightspace’) will be accessible via Brightspace and links you create from Brightspace to assignments in Ans. As of academic year 23/24, you will be able to push grades from Ans to Brightspace Gradebook. Grades from the gradebook need to be exported to a .csv file and imported into Osiris.
- Your ‘Exam Ans course’ (which is called ‘course code Exam’) is accessible for students who registered for a specific exam. The Digital Exam Deks (Digital Exams) will give access to all students who enrolled for this exam via Orisis. You will need to export the grade list from Ans to either the Gradebook in Osiris, or Brightspace (in the case that a partial grade is not registered in Osiris).
Multimedia resources on Ans
-
You can find additional instructional videos on Ans on the following topics:
-
Click here to watch the introductory Ans webinar. The webinar will in the end be changed into a PowerPoint with a professional voice-over.
Creating assignments, exercises, and questions
An assignment refers to an assessment or exam (in TU terminology), a question to an exercise, and a question to a subquestion.
-
See this short video on how to create a written assignment in Ans.
- Go to ans.app (login via SSO/single sign-on)
- Go to your course (email Digital Exams with subject ‘'Could you please create my course in Ans?’, they can create your course if it is not there, yet)
- Click on the pink button ‘new assignment’, bottom right:
- Choose a name
- Choose a type. Here is an overview of the available types in the overview above:
Ans assignment type
Explanation TU Delft
Digital test
Digital exam/test/quiz
Written assignment
Print/scan exam
Bubble sheet
Multiple choice forms: replacement of Contest
Hand-in assignment
Do not use, no mandatory plagiarism tool available. Alternative: Brightspace assignment
Appraisal form
Assessment form (e.g. presentation)
Peer-reviewed digital test
Digital test, reviewed by students’ peers
Self-reviewed digital test
Digital test, self-reviewed by students
Digital test for groups*
Group-based digital test
Hand-in assignment for groups*
Do not use, no mandatory plagiarism tool available. Alternative: Brightspace assignment
*Assignments to be taken in groups simultaneously
-
The assignment page shows the following actions:
- Create a new exercise (question)
- Import an exercise (question) from your question bank
- Edit exercises (questions)
- Edit the grading scheme (answer model) of each question
- Add objectives (learning objectives) to each question (subquestion)
- Create flows: per student, pick a random question from several questions
- Preview your test, and after completing the exam, preview the grading of your test.
-
Per assignment (test, exam), you can indicate their status. You can choose from the statuses below:
Label/ status
Explanation
0. Trash
Consider this assignment trashed
1. Drafting (start status)
Lecturers are in the process of creating a test.
2. Reviewing content
Colleagues/assessment advisor is giving feedback on the questions. The examiner is reviewing their feedback.
3.Content ready
The concept is ready for the test, no action is currently needed.
4. Ready Test taking
Ready for delivery of the test, after adding test settings (time slots, including for extra-time students, and remote students) & adding students to the test.
5. Grading Student Work
The test is in process of scoring. In the case of peer feedback or self-evaluation, this is done by students.
The examiner checks the analysis.
6. Result Analysis & Adjustments
The examiner checks the analysis.
7. Scoring & Grade Communicated
For summative tests: adapt answer model/conversion to enable fair grades.
8. Student Review & Comments
Students review their work and can add comments on questions/requests for higher scores/questions. Examiners react to that
9. Appeal Period
During this period, students can file an appeal, for which they need a copy of the exam, answers, scores, and grade calculation. Until 6 weeks after publishing course grades.
10. Archived
6 weeks after publishing course grades
No changes are possible unless the admin changes the status (e.g. in case of individual CBE- or CBHO-appeal cases)
11. Student Data is Removed
After 2 years, student data is deleted; detailed performance data of questions and tests are kept (this is anonymous).
12. Exam is Removed
After 7 years, the exam is destructed. Lecturers can copy their questions to the question bank before that date.
-
Add an exercise (main question)
After clicking “new exercise”, add a question name that will help you retrieve the question at a later time. You can choose to show or hide the question name from your students in the assignment settings.
Do not add a time limit, and do not tick that the exercise can only be viewed once, as it will increase the stress level of your students and decrease their performance.Add a question (subquestion, e.g. 1a) and description (question-wide information)
- Use ‘New description’ to add for example a case description that will be used in all subquestions.
- Use ‘New question’ to add subquestions (e.g. 1a, 1b, etc.) to the main question (e.g. question 1).
-
Here’s an overview of question types. Use the links to go to the Ans help page.
We categorized them as follows:- Grading type: automatically or manually graded
- Paper/digital type: whether they can be used in paper-based tests, in digital tests, or both
- How students enter information: clicking, typing, or by pen
Grading type
Answer entry
Digital/paper
Ans assignment type
Example
Automatically graded
Clicking on the answer option
Both
Digital
Put the process step in the correct order
Match the cities in the left column to the countries in the right column
Mark areas in the picture that match a condition
Drag tool names to tools that are depicted in the picture
Typing short answers
Fill in gaps in a sentence. This can be numeric answers as well (e.g. 20.54 kN). Numerical answers cannot be combined with randomization if automatic grading is needed.
Automatically graded numeric answers, that can include parameterization.
Numbers that include exponentials are currently not allowed as answers, and therefore checking rounding neither. Use fill-in instead or ask for the answer to be given in specific units.An equation is the final answer.
Manually graded
Multiple types
Digital
Students can insert text, pictures, audio, webcam captures, and files into an answer field.
Students draw with a very minimal, Paint-like tool. You can provide a background image so that students can draw on top of it.
File upload ➔ warning: no plagiarism check available!
Students upload a file, which you grade. Unsuitable for graded assignments, since no mandatory plagiarism check is available.
Paper
Students complete the answer within an answer field. Answer fields can be empty, or can contain writing lines or squares. If students need more space because of errors, invigilators provide white stickers to cover the errors and create fresh space.
Students answer by drawing on top of an image.
You can edit the place where students enter or draw answers freely.
Does Ans have their own overview?
- Yes, you can find their overview of question types here.
Why would you want automatically graded questions?
- Grading will be less work.
- You can provide automated feedback to the students, so that’s great for formative use!
Good to know:
- If you make a mistake in the answer to an automatically graded question, you can fix it at a later time.
-
If you click the three dots, you open the options per question within an exercise (i.e. on subquestion level):
The following two tables give an overview of the available options(setting) depending on the type of assignment chosen (digital vs. paper) and depending on the question type.
Digital assignments (test) overview 1/4 (question types in alphabetical order)
Options
Question type
Bonus
O
O
O
O
O
Skippable
O
O
O
O
O
Show answer options
O
Partial scoring
X
X
X
X
X
Predefined answer
X
Word limit
Use variable
Model answer
T
T
T
T
T
Use automatic scoring
O
Start with zero points
X
X
X
X
Limit minimum to zero points
X
X
X
X
X
Limit maximum to total points
X
X
X
X
X
O= option is off
X= option in on
T= text can be added without selecting an option
Digital assignments (test) overview 2/4
Options
Question type
Bonus
O
O
O
O
O
Skippable
O
O
O
O
O
Show answer options
Partial scoring
X
X
X
X
X
Predefined answer
Word limit
Use variable
Model answer
T
T
T
T
Use automatic scoring
O
O
O
Start with zero points
X
X
X
X
Limit minimum to zero points
X
X
X
X
X
Limit maximum to total points
X
X
X
X
X
O= option is off
X= option in on
T= text can be added without selecting an option
Digital assignments (test) overview 3/4
Options
Question type
Bonus
O
O
O
O
O
Skippable
O
O
O
O
O
Show answer options
Partial scoring
X
X
X
X
Predefined answer
X
X
Word limit
O
Use variable
O
Model answer
T
T
T
T
T
Use automatic scoring
O
Start with zero points
X
X
X
X
Limit minimum to zero points
X
X
X
X
X
Limit maximum to total points
X
X
X
X
X
O= option is off
X= option in on
T= text can be added without selecting an option
Print-scan assignments (test)
Options
Question type
Bonus
O
O
O
O
O
Skippable
O
O
O
O
O
Show answer options
Partial scoring
X
X
X
X
Predefined answer
X
X
Word limit
O
Use variable
O
Model answer
T
T
T
T
T
Use automatic scoring
O
Start with zero points
X
X
X
X
Limit minimum to zero points
X
X
X
X
X
Limit maximum to total points
X
X
X
X
X
Print-scan assignments (test)
Options
Question type
Multiple choice (paper)
Bonus
O
O
O
O
O
Skippable
O
O
O
O
O
Start on new page
X
O
O
O
O
Image full page width
O
Stack alternatives vertically
O
Separate answer boxes
X
Partial scoring
X
X
X
X
X
Model answer
T
T
T
T
T
Use automatic scoring
O
Start with zero points
X
X
X
X
X
Bonus
O
O
O
O
O
Skippable
O
O
O
O
O
-
You do that by accessing the code editor in an exercise (question level) or question (subquestion level). The parameters you use will be available throughout the entire exercise (question level).
Ans has a detailed Ans help page on this topic, including an overview per question types on the parameterisation options (table at the bottom of the page). Below, you will find a more concise example.Where can I access the code editor?
Here is where you can access the code editor:
Can you give a simple example?
This is an example question (see above): Petra wants to calculate the number of tiles on a floor. The answer (ans) is the number of tiles on the short side (p) times the number of tiles on the long side (q). We want to give all students different numbers of tiles on the long and short side.
In the code editor, we enter the following code:
- p is a random number out of all numbers between 3 and 10 that are separated by step size 1 (3, 4, 5, 6, 7, 8, 9, 10)
- q should be longer than p, so we make this a random number between one-more-than-p, and 20 (p+1, p+2, p+3, …. 18, 19, 20)
- the answer ans = p*q
- print (p,q,ans) prints the values of p, q, and ans below the word ‘Code editor’ and just above the black box in which you edit the code (grey numbers 9, 17, and 171)
- By pressing ‘run’ you can create another random sampling of numbers.
-
Warning: this will deteriorate the quality of the test result analysis. Rir and Rit will not be calculated for these questions.
If you to create unique exams by having Ans pick a random question out of 5 questions, you use the ‘Flow’ tab in a digital test.
-
Learning objectives are added to subquestions (‘questions’ in Ans). As a result, you can see the statistics per learning objectives, and also students can see their performance per learning objective. It is quite a bit of work, though:
- It takes four clicks per subquestion to assign an objective:
- You need to define objectives on the course level first
- Each objective needs to be linked to a ‘domain’, which is not useful for most degree programs at the TU Delft (except for at least IDE). For IDE, each learning objective is explicitly linked to a final attainment level. Lecturers from IDE can therefore use the final attainment level as a ‘domain’.
You must choose what to use as a ‘domain’ (you cannot work without this):
- Course code (which ignores the domain)
- Level of Bloom
- Final attainment/ILO. Use this if you are in the bachelor of IDE, or if you know what ILOs/final attainments your learning objectives belong to. Do not use them if you expect them to change soon.
Background
We assume that the partition into domains and objectives originates in the fact that some educational institutions divide learning objectives into three domains with three different taxonomies (cognitive domain with Bloom’s taxonomy, psychomotor domain, and affective domain). At TU Delft, we focus on objective learning objectives, which implies that all ‘affective’ learning objectives (critical thinking, open-mindedness, etc.) are operationalised using the Bloom taxonomy. Psychomotor skills like lab work can also be expressed using Bloom.
Ans suggest using the levels of Bloom as if they were domains. You would then first select the level of Bloom, and then the learning objective. You would have to retype the objective if it has questions on different levels of Bloom. The main objection is that in the analysis, the results are grouped per level of Bloom, instead of the other way around, making it harder to interpret the results.
Question banks
-
Like this!
It is a long list of questions that you can select to create an assignment within the question bank, which you can then ‘schedule’ (move) to a course, in which students can take the assignment.
-
You will get an Ans course per year. After two years, the data will be anonymised and after 7 years, the course will be removed (see information on archiving). The idea of a question bank is that you store all questions into one question bank so that you can re-use them for formative use, or adapt and re-use them in exams.
It depends on your faculty and board of examiners whether and under what conditions you are allowed to re-use (adapted) exam questions.
Do I need a question bank?
Usually not: you can start creating your exam/assignment right away!
You will use a question bank if:
- You have a question bank already
- If you want to invest time in setting up a question bank for your course in ANS (for support contact digitalexams@tudelft.nl or teaching-support@tudelft.nl
It is possible to copy assignments (tests) from course to course.
-
- Use logical exercise names, you can search for names.
- Filter on the following:
- Labels (status of the exercise, e.g. ‘draft’)
- Author
- Psychometric measures:
- P-value (average relative score of students on this question)
- Rit-value
- Rir-value
- Question type
- Tags (see below)
- Objectives (learning objectives). Warning: learning objectives are defined per question (subquestion), so if you filter on an objective, it may only occur in a small subquestion.
- Etc.
You can define tags yourself, and they are a very good way to organise your question bank.
-
Warning: you can only add tags in a question bank, not in a course.
Info: tags are assigned to exercises (questions), as opposed to objectives that are assigned to questions (subquestions, e.g. 1a)
Tags consist of a key (tag name) and a value and they can be assigned to exercises (questions that can consist of subquestions). You can define both the key and values on the go.Tag suggestion
Here’s a list of examples of tags that might be useful for you:Key (tag name)
Values
Explanation
Duration
1, 2, 3, 4, …
Estimation of time students will spend on the exercise, in minutes. Helps to control the duration of a test.
Nr.Subquestions
1, 2, 3, ….
Number of questions (subquestions) in the exercise (question)
Bloom
1, 2, 3, 4, 5, 6
Bloom level in numbers
Bloom
remember, understand, apply, analyse, evaluate, create
Bloom level in words
Difficulty
1, 2, 3
The difficulty, expressed on a scale from 1 to 3 (3 being the most difficult)
LO
1, 2, 3, 4, …
Learning objective, expressed in numbers
LO
‘create questions in Ans’, ‘take the exam’, …
Learning objectives, described in words.
topic
‘bridge’, ‘question’, …
Short description of a topic
This is how tags will look in Ans:
It is important to keep the tag values short, for a clear overview.
How can I use tags?
You can filter tags by selecting them in the question bank.
You cannot sort on tags in the question bank, since they are not columns.
Tags are not visible in the question bank, and therefore also not visible in assignments in courses.
-
Pros question bank
Cons question bank (compared to creating assignments directly in a course)
You can test questions on the correctness and scores for different student answers in automatically graded questions. In courses, you can only test entire assignments.
You cannot test whether automated feedback works; you will only receive “ correct/incorrect” as feedback. This should be tested in a course
Questions are saved ‘forever’
(In courses, assignments are saved for 7 years.You cannot test parametrization in a question bank. This should be tested in a course.
Share exercises with other Ans users, inside and outside the TU Delft. Use the dedicated role that allows them to copy questions to their question bank, but not edit them in the question bank.
Question bank assignments do not have an assignment test-taking preview & grading preview. This should be tested in a course.
The process of drafting an assignment from exercises and then moving it to a course keeps the course clean.
It is currently unclear which exercise version is newer (if the question bank exercise or the course exercise is edited).
Tags are a nice, flexible way to structure questions.
In a (course) assignment, you can select questions from the question bank, but you cannot use the filter option when finding questions (i.e. tags are unavailable)
Quality control: Sort on p-values & Rir-values
Tags can only be added in a question bank
You can improve your formative questions for next year
-
Yes, that’s very easy. It should be done per exercise:
-
Yes, that’s very easy. Here’s how.
-
If you applied for this service, your question banks were imported. The folder structure in Möbius was converted to tags in Ans.
Reviewing and grading
-
You can find more information on reviewing in these clips.
You can review per student or question. Reviewing per question is recommended, as it removes some of the temporal bias. When a large number of students’ work needs to be reviewed, the grading team typically divides the questions among the members of the grading team. This increases consistency in grading.
Below is an example of an exam that needs to be graded. The left side holds the question plus student answers, and the right side the criteria on which you will grade, You can access this by either clicking it or by using shortcut keys (see below), e.g. typing the number before each criterion.
On the top right, the following options are shown:
- Review: do the actual reviewing of student answers by using the criteria/slider/rubric
- Annotations: add inline comments in student’s work
- Flags: check the flags of yourself and colleagues to discuss reviewing issues (e.g, ‘Should I give this student 4 or 5 points for this unique answer?’)
- Discussion: Review the discussion points that students created after reviewing their graded work.
-
Yes, you can use the following shortcut keys:
Key
Action
ß
Previous question
à
Next question
ENTER
Next question
B
previous student
N
next student
1/2/3/4/…
toggle criteria
SPACE
toggle positive criteria
0
0 points
F
view flags
-
You can add ‘quick comments’ per criteria by using the ‘Quick comment’ button within the ‘+Comment’ option:
-
The following analyses are available and they are grouped under the tabs ‘Results’ and ‘Insights’:
- Results:
- Marks: Student results
-
- Exercises: visual representation per subquestion
- Results:
-
Source animation: Ans @ https://support.ans.app/hc/en-us/articles/360027234814-Analyse-questions
Students’ perspective of test-taking
-
You can have a look at these videos that show the student's experience during the test and the review of their graded work.
As shown above, you can add a cover page (left screenshot) and a formula sheet (middle screenshot) to a digital exam. Students can also hide these (right screenshot) so that they focus on the questions.
Students have the following features available:
- Students can bookmark a question, for example, if they are not sure about the answer. This will add a dot to the question overview at the bottom of the page.
- Students can see which questions they answered at the bottom of the page. These questions are orange.
- In multiple-choice questions, students can strike through options that they deem incorrect. The teacher cannot see that.
- Students can change their accessibility settings (e.g. dyslexia font, increased contrast, etc.) by themselves, so they do not have to bother the teaching staff. The system remembers their settings for the next test.
-
Students and lecturers can use LateX to enter formulas. In addition, there are several formula editor keyboards available. Here are some screenshots to give you an impression:
Unless students are fluent in LateX, typing formulas will still take more time than expanding formulas on paper. Consider using print-scan for formula-heavy exams.
Need support?
Get in touch with us! We are happy to help.
+31 (0)15 27 84 333