Open AI research at TU Delft: The beginning of a conversation

Authored by Emmy Tsang, reviewed by Eliška Greplová

From forecasting the progression of the COVID-19 pandemic, to modelling and designing green electricity systems, machine learning and artificial intelligence (AI) is an increasingly integral part of our research and scientific processes. This presents new challenges and opportunities for those who care about open science, the movement to make scientific information, data and outputs more widely accessible with active engagement of all stakeholders: what does it mean for AI systems to be “accessible”, and how can stakeholders – researchers and everyone using/affected by research- engage effectively with AI algorithms and models?

Thankfully, these are not entirely new questions: the responsible development of AI technologies is a much-discussed and researched topic within the AI research community, and open science practitioners and advocates have been experimenting with the design and implementation of policies, infrastructure, governance models, etc, to increase the accessibility and transparency of scientific output. A few of the areas both communities are concerned with are highlighted below.

Transparency and reproducibility: The lack of reproducibility in research is perhaps no longer shocking to many researchers. To increase transparency and reproducibility throughout the lifecycle of research, the research community has been actively promoting more openness to data, source code and other research artefacts. With that, new guidelines and training, technical infrastructure, international instruments, etc have been established. AI researchers have also played an active role, discussing and creating solutions for their community: the open-source platform Papers with Code allow researchers to share machine learning papers with code, datasets and evaluation tables publicly; container and virtualisation systems such as Binder and Code Ocean facilitate the capturing of model software dependencies and environmental parameters; Distill pioneered a template for dynamic and interactive visualisations of complex neural networks to encourage readers to interrogate them and their

Photo by ThisIsEngineering from Pexels

results. As the utility of AI methods broaden in research, the communities are aligned in their interest to ensure that code and data are well-managed, versioned and released following ethical guidelines, dependencies are cited, and validation and verification of published results are made easy technically and systematically encouraged.

Fairness and inclusivity: Central to the ethos of open science is the idea that everyone, no matter their background, geographical location, or abilities, should be able to access and build upon the knowledge generated by research. The open science community has been discussing inclusivity and diversity in research: scrutinising current research practices like experiments that only including male subjects, critically reflecting on barriers to participating in research, such as language, access to technology, etc. Likewise, those passionate about fair and trustworthy AI have been working on frameworks and tools to avoid the marginalisation of vulnerable groups and to foster AI systems that are accessible and fair to all. As AI methods become more widely employed in research, we need to design mechanisms and frameworks that mitigate inherent biases and issues of exclusion in the data collection and model training processes, and allow stakeholders with diverse expertise and backgrounds, including end-users and those affected, to take part as much as possible in all stages of research and design, such that the resulting research output work for diverse individuals and do not end up harming under-presented, vulnerable groups of society. 

Accountability: AI systems are taking over more parts of our decision-making processes: self-driving cars save us the cognitive energy to scan the roads and make decisions to brake, accelerate, or turn; nano-second trading algorithms make split-second buying and selling decisions through ingesting huge volumes of real-time data, something human brains will never be capable of. Yet, who is accountable for the consequences of the decisions made by these algorithms? How do we design auditability – the ability for users and other stakeholders to assess algorithms, data and design processes into AI systems? This is also an aspect that is perhaps under-explored in the discourse around research, as we see in the decisions made by governments in the current pandemic, supposedly made based on “science”. If there is one lesson we learnt from the pandemic, it is the fact that scientific results and predictive models have inherent uncertainties. The research and AI communities have practices around how to quantify, account for and evaluate uncertainties, but are decision-makers, users of AI systems and consumers of scientific information able to effectively understand and interpret them? This ability would be crucial for them to participate in discourse and contestation around research. 

So – many converging questions and exploration between open science and responsible AI. At TU Delft, we are fortunate to have vibrant communities of designers, engineers and researchers who care deeply about open science and/or responsible AI. While we might have come from different starting points, we share the vision of creating inclusive and equitable technologies and systems, through engaging people with diverse perspective- scholars of other disciplines, policy makers, citizens - throughout the lifecycle of our research, design and engineering processes, from ideation to the sharing and communication of research output. This series of stories highlight the work of some AI researchers at TU Delft who are passionate about AI systems that are FAccT â€“ Fair, Accountable and Transparent. We hope to inspire more at TU Delft and beyond to reflect, discuss with others and take steps forwards to make more parts of their work more transparent and inclusive. 

The untapped opportunities

Where do we go from here?  The TU Delft Open Science Strategic Programme 2020-24 aims to build policies and guidelines, infrastructure and capacities to support TU Delft researchers and teachers in practising open science – what kinds of institutional support are needed for our researchers – not only AI researchers, but also those applying AI methods in their work or collaborating with AI researchers - to engage in FAccT AI practices?

There is an extra layer of complexity: even where scholars are not aware that they have used AI in their research, AI technologies are often part of their research workflows. From our machine-assisted literature and data discovery engines to GUI-based image analysis software that has built-in segmentation algorithms, as these AI research tools evolve, their exact ways of working become harder and harder for the average researcher to fully comprehend. How does that affect our research output, and what measures can we take to ensure openness and accountability?

There is no doubt that there are more questions to be explored. We invite members of the open science and responsible AI communities to come together, converse and learn from each other, to build towards our shared vision of transparent, fair and accountable research.