Report on E-ATP Madrid 2019

Nieuws - 12 november 2019

Summary of the European conference by the Association for Test Publishers

 

The E-ATP is the European edition from the originally North American Innovations in Testing conference by the Association for Test Publishers, ATP. The ATP originates from the test publishing industry in the U.S.A., who design and maintain large question banks and deploy assessments for K12 schools, colleges and professional (re)certification institutions. The event nowadays offers a wider view on the testing industry: From research on question development (question types, Artificial Intelligence) to services around test deployment (test security, proctoring services).

Last September the E-ATP took place in Madrid. The topic Artificial Intelligence is getting more prominent in the world of testing. 

Artificial Intelligence and Machine Learning were leading topics during this conference. It is being used in test development, language translation, grading or ranking writing assignments and performance assessment by recruitment.

Interesting developments (for me) I want to report here:

Test development:

  • Question creation. Based on a 3 to 4 page text, items can be created (multiple choice/selection, fill the gap, drag and drop, ordering or matching question types). 50% of the items can be used immediately, 33% is unusable, while the remainder needs editing. The test system Televic Education provides such functionality. The system is being developed in cooperation with KU Leuven
  • Translation of (English) test items to different languages. Research shows that the quality of the item translation highly influences the test outcome. Automated, high quality translation of items is typically interesting for test publishers that deploy their tests in different countries. Using automated translation, now, can save up to 50% of the translation time. In the future it is expected this can be reduced even further. www.shl.com/nl/

Grading:

  • Adaptive Comparative Judgement: (from Wikipedia: Adaptive comparative judgement is a technique borrowed from psychophysics which is able to generate reliable results for educational assessment – as such it is an alternative to traditional exam script marking. ) In the approach, judges are presented with pairs of student work and are then asked to choose which is better, one or the other. By means of an iterative and adaptive algorithm, a scaled distribution of student work can then be obtained without reference to criteria. RM Results is a company that has a system (RM Compare) that works with this principle to rank student’s papers. https://rmresults.com/digital-assessment-solutions/rmcompare 

Authentic testing:

  • Bringing the work environment into the exam ..
    • Microsoft exams Using labs to better assess whether the examinee can not only tell what the correct answer is, but also show that he or she can perform the task or solve the problem.
  • or bringing the exam into the working environment:
    • Using Big Data and the Internet of Things (IoT) in the assessment process (Vantage Labs). Using the data captured by wearables in the workplace (When tradionnal Assessments Won’t do). Examples given are in the landscaping business and sports (game analysis).

Augmented Intelligence:

This is defined as ‘Computers improving human capacity to solve complex problems by relying on human-computer interaction.’

BYOD

At the conference the topic Bring your own device (BYOD) was discussed in a peas in a pot session that was rather disappointing. The discussion did not get beyond the setting of students having to login into the test system and taking the test. The regular suspects were shared: preparation of students prior to the exam (practice test, system check, charged batteries), availability of back-up systems, invigilator training (or the presence of more skilled support workers), the necessity of ample wifi and sockets, and cheating issues.

It seems that the Dutch Higher Education institutions (gathered by Surf’s SIG Digital Testing) are already beyond that point. Enabling not just test system (Q&A) exams but also supporting more authentic examination by offering an environment that not only (or not even) a test system, but also the tools (excel, simulation tools, etc.) to perform tasks or solve problems in a more authentic way. Perhaps even moving away from BYOD and providing centrally managed Chromebooks, so more locations on campus can be used for digital assessment.

About the Author

Meta Keijzer-de Ruijter is an IT in Education Specialist at TU Delft. Find her at TU Delft or LinkedIn.

"Report on E-ATP Madrid 2019" by Meta Keijzer-de Ruijter is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.