Mauricio Aniche: “There are things you can’t assess automatically that would matter in real life.”
An increase in student numbers, followed by Covid, inspired Mauricio Aniche to innovate his software quality and testing course. As an Education Fellow, he developed a tool that can both assess students’ exams and help them during the preparatory exercises.
It was a moment he will not forget: as a young software developer, Mauricio Aniche was team lead for an American company. Here, he developed petrol station management software for Central America. “Back in 2006 that was still new”, he says. After many months of development, he travelled to the Dominican Republic to see it in action. “We were celebrating the successful launch on the beach, when in the middle of the night the software crashed. It had not been properly tested and that meant no petrol coming out of the nozzles at petrol stations.” It was a revelation for Aniche, who turned his focus on software testing. “It is also what drew me to becoming a researcher, so we could learn how to do things better.”
Aniche joined TU Delft in 2016 as a postdoc, working in Professor Arie van Deursen’s software engineering department; in 2018, he became an assistant professor. He worked alongside his MSc and PhD studies, and he’s always liked having a foot in both camps, academia and industry. At TU Delft, he also acquired a passion for teaching. “I taught master courses and did thesis supervision, but my most popular course is the bachelor course in software testing. It is part of the first year of the computer science programme, and now typically draws some 500 students.”
State of the practice
Aniche had a clear vision for this course, Software Quality and Testing: “99 percent of our graduates who become software engineers will have to solve problems right now. That is why I first teach them the state of the practice and then the state of the art”, he says. “So a lot of my course is showing students how to make the best use of what we know. Later on you can start thinking of how to advance the field.” A bit less theory and more practical tips: this is what he sees as his contribution to a course which was already “pretty good” before he came. However, a sudden spike in student numbers for computer sciences in 2018 called for more action. “We went from 150 students to 900, so we had to scale up. Before, we would rely a lot on teaching assistants, but with so many students that was impossible.”
That is when Aniche put forward his Fellowship proposal for an automated feedback tool: “Students have to write their own test cases during exercises and exam. I wanted to create a tool that could assess students’ solutions and let them know if they are on the right track or not, and give them hints, so they can keep trying to solve the exercise by themselves. Only when that doesn’t work, they can contact a TA or me. Now that we have the tool, it only takes an additional ten minutes to add the rubrics the tool can use when I create a new exercise, so this really saves a lot of time”, Aniche explains “The beauty of it is that we also use the tool to assess exams. When the exam is over, I just press a button, and by the time I get home, the exam results of 500 students await me.”
[... ] the number of complaints about grades is really low, and we got a lot of positive feedback
Mauricio Aniche
The tool can assess the exam questions on the components that are taught during the course. “The tool checks if their solution touches on every part of the programme, as there could well be a bug in one of those ten lines of programme you are not testing. This is also the kind of hint the tool can give: ‘you are not testing some lines…’ Another example is the minimum amount of testing. “You can test programmes from many different angles, but there are certain tests I want students to come up with. Only a solution that tests enough and that has ‘the right tests’ gets a 10.”
During the course, students get to try their hands at more than 60 exercises, in practice mode with hints, or in exam mode where they just get a grade. “Students already know the tool once they enter the exam, and the exam is close to what they have practiced. The tool also removes subjectivity in grading, so the number of complaints about grades is really low, and we got a lot of positive feedback”, says Aniche. “There is a trade-off. There are things you can’t assess automatically that would matter in real life. However, during lectures we discuss a lot of real life examples of bad software and why it is bad, and that will make them better engineers.”
There is a trade-off. There are things you can’t assess automatically that would matter in real life. However, during lectures we discuss a lot of real life examples of bad software and why it is bad, and that will make them better engineers.
Mauricio Aniche
Open source
The Fellowship allowed Aniche to hire TA’s to create the tool that was first used during the 2022 course. “We learned a lot from actually running it, so last summer we did some improvements. Minor ones, as it was already good the way it was.” The tool, which now goes under the name of Andy, is fully integrated in WebLab, EEMCS’s tool for online exams. It is also available open source, so other universities can use it as a TU Delft certified plug-in. Nijmegen University will be the first to do so. A great success, but one he did not achieve alone, he stresses. “I am very grateful to Frank Mulder, my co-lecturer, and to Martin Mladenov and all the students that helped me develop Andy.”
While Aniche is very happy with Andy as a tool for formative and summative assessments, he would like to see another shift in the way students are graded. “There is too much emphasis on grades alone, and this puts lots of pressure on students, so they will focus too much on their exams. We do need to assess them, but we should do so in a way that they will concentrate on learning, not just on passing. That is something we should discuss in the Teaching Lab.”
Now employed as Tech Lead at payment platform Adyen, Aniche plans to return every year to teach this particular course. While he misses the inspiration he got from his students, and being free to explore his own research ideas, he still has his hand in teaching in a way. “I am involved in upskilling engineers in the sense of lifelong learning. We also try to improve the on-boarding experience, because of the complexity of the software systems”, he says.
He would like to bridge the gap between graduation and being able to work in industry. “We should organise summer programmes, like three-month boot camps to ready our students for their postgraduate career. There is so much students need to know if they want to work in industry. Front-end technologies, like React or VueJS, cloud-native technologies, like Kubernetes, and day-to-day development tools, like Spring Boot or PostgreSQL – the tools that most companies use. Graduates, from TU Delft and elsewhere, often have amazing theoretical knowledge, and that will support them throughout their careers. Twenty years from now, that will enable them to learn technologies that don’t exist today with ease. But that doesn’t count on day one of your career in industry.”