Julian Urbano | Research of How We Do Research
12 OCTOBER 2023
Information Access (IA) is the broad field concerned with connecting users with the right information at the right time, including fields such as Information Retrieval or Recommender Systems. Research in these fields is very much task-oriented: there are user needs, datasets, annotations, evaluation metrics, etc. Development is carried out through "how well" questions, such as "how well does model X retrieve relevant documents?", ultimately executing evaluation experiments to provide the answers. But we also question whether our evaluation methods are appropriate in the first place. This is what Evaluation research is about: we ask "what if" questions, such as "do we have a better estimate of user satisfaction if we use metric X?", or "if we do X to reduce annotation cost, can we still accurately estimate system performance?". In this talk I will discuss Evaluation research with two concrete examples regarding what data we use in our experiments, and what is it that we measure.
Julián Urbano is an Assistant Professor in the Multimedia Computing Group, EEMCS at TU Delft. His interest is in Information Access and related disciplines, with an emphasis on Information Retrieval (IR). His research is primarily concerned with the evaluation of IR systems, covering topics such as construction of datasets, design of metrics, reliability of experiments, statistical analysis and simulation.
In the past, Julián was more active in the ISMIR community (music IR), where he was Program Chair of ISMIR 2019 in Delft and Associate Editor of the Transactions of ISMIR. In the past 5-10 years he moved away from the music domain and is now more active in the ACM SIGIR community, serving for example as Short-papers Chair of SIGIR 2023 and as Editor of the SIGIR Forum.