A decade ago, Luís Cruz worked on developing software tools to bring artificial intelligence (AI) to smartphones. It quickly triggered the question of energy efficiency. Not with the noble goal of making them more sustainable but to prevent them from draining the battery too quickly. Cruz went searching for tools and knowledge, but encountered an empty field. “It made me realize there was a need for research,” he says. Soon thereafter he started his PhD research project on ‘green software’, starting simple by studying the energy efficiency of basic mobile apps. As an assistant professor at TU Delft, Cruz remains passionate about sustainable software, specifically AI.
Your work on the energy efficiency of mobile apps made quite an impression.
“Yes, our paper from 2017 was recently recognized as the most influential paper of International Conference on Mobile Software Engineering and Systems. Since then, the field has evolved. The AI we couldn’t yet study before, is now everywhere. ChatGPT, the famous AI model by the company OpenAI, has made headlines around the world. And my group is now fully engaged in studying the energy efficiency of AI.”
How much energy does AI consume?
“It is very difficult to answer that question. Not just because it is technically challenging to get reliable energy data, but also because companies aren’t keen on disclosing them. But we do know that training the ChatGPT 3.5 model—so a single training round, not even the experimental training and re-training that was done during development of the model—costs around 500 tonnes of CO2 emission.”
Is that a lot?
“I usually convert it to a distance driven by a car: 500 tonnes CO2 is roughly equivalent to 1000 cars each driving 1000 km. And after the training is done, the energy needed for the AI tool to do its work is generally about the same. Of course, for ChatGPT, it will be different, as it went viral. The energy cost will be bigger.”
Education is the solution
“Exactly. Education doesn’t change things in 2-3 years, but in 10-15 years it can change the industry. That’s why I started a course in sustainable software engineering for students at TU Delft. To make sure that the software engineers of the future will be able to consider the energy consumption of software, to identify and tackle sustainable software issues.”
How do you make AI more energy efficient?
“In general, so not considering the specifics of the neural networks involved, there are many ways to save energy. There are strategies that are not new and some of them are quite cheap. My favourite is simplification. For example, not all the data you use to train the model make a real difference. So, instead of dumping all the data you’ve got into the training process, a clever data selection strategy that only collects data that matters for the model and application at hand, will save a lot of energy. There is a trend of continuously retraining the model as new data become available. We should ask ourselves, is it really necessary? Can we come up with a more energy-efficient solution for users who really need the new data? Once the model has been trained, energy can also be saved by reducing the complexity of the model. This means training the most complex AI model you can, and then coming up with a simpler model that behaves the same but only learns the parts of the model that really matter. All obsolete neurons are removed. This could reduce the model complexity by a factor of ten while maintaining high performance. In this way, you reduce energy consumption for free.”
What can be done at the level of hardware?
“If we discuss hardware, we’re used to talking about CPUs, central processing units. Since AI relies on matrix operations, AI hardware concerns GPUs, graphics processing units. We know quite well that current AI systems use GPUs for only 30-50% of their capacity. More than half of the hardware capacity is left unused, which adds up to a huge waste of computer hardware and energy. You know, when you train an AI model, the choice of GPU is very important. But it shouldn’t be just about finding the overall best GPU but finding the GPU that best fits the model. Looking for the most powerful GPU gives you a false idea of doing the best you can. An older or smaller GPU might also get the job done. But here’s the thing: there is not a lot of knowledge about finding the appropriate GPU. Specialists in developing deep learning networks, like myself, don’t know too much about the hardware. The combination of these two disciplines doesn’t really exist.”
You’re bringing these topics closer?
“I can’t tell you which GPU to use. I can only ask you to look at different GPUs because there might be surprising differences in the energy usage. In our research we are trying to find ways to identify the best hardware for each AI model, as well as looking into other strategies to make AI greener. My goal is to make sure that, if there’s a company wanting to build a green AI pipeline, they have the knowledge to do it. That’s where I want to help. The field is evolving so fast, we are trying so many things, so many technologies, it’s difficult to say: we should all do this one thing to be more sustainable. There are many things we can do, we don’t even need to invent new things, we just need to start the conversation.”
My goal is to make sure that, if there’s a company wanting to build a green AI pipeline, they have the knowledge to do it.
“One way of being successful is being the first. Another way is to do it right. Sustainability is unfortunately not compatible with being the first. I understand that, it’s a race and that’s how it works. Now that OpenAI has created the buzz, now that people have experienced the power of AI for themselves, companies with similar capabilities will look for new ways to bring value to their customers. Sustainability is one such value. As soon as we get this hype behind us, let’s make it right. And when we reach that point, I want to have everything in place to help people and companies do so.”
Sustainability as a unique selling point
“No, if sustainability is only a selling point, it won’t work. There are cheaper ways of selling sustainability, like changing the colours in your logo to green. That’s more effective than changing your AI pipeline. Instead, we need to make AI more democratic. We know that people care about sustainability. So, reporting the energy cost is the first step, after that everyone can start making informed decisions, and change the industry.”
Instead of regulation
“I’m not in favour of regulating everything. Smaller players might be left out of the race. A large company like Microsoft can easily buy all the green energy that’s available, use it to train their AI models and leave smaller companies with coal-based energy.”
Are you confident this shift in awareness will happen?
“I am confident we will get there. I am optimistic. But this does not mean I cannot see a scenario where everything goes wrong. I think education is the most powerful thing out there. We, the people, we are the users of AI. If we have literacy in the concept of green AI, if we start asking questions about the impact of the AI we’re using, then companies will see that we care, and they will do something about it. We need to bring more sustainable software engineering into curriculum of Computer Science programmes, and to the general public. If we start talking about it, then we don’t need regulation.”
There are many things we can do, we don’t even need to invent new things, we just need to start the conversation.
How are the students responding to the course?
“I see that students are interested in the theme, but know little about it in advance. This shows how little we talk about it, how few resources there are. There are other courses at Dutch universities that pay some attention to this topic but no other fully dedicated courses. It was difficult to create, we had to come up with the teaching materials from scratch. That’s also why we made the materials public, so that everyone wanting to teach it can download my slides. I hope they will be useful to someone out there. Actually, I get a lot of messages about them, also from people working in the industry. That’s a nice feeling.”
What’s the plan for the next seven years?
“In 7 years’ time, when someone comes up to me and says they want to build energy-efficient AI, I want to be able to guide them with a set of resources, like a recipe book. I hope that the ideas about green AI will grow and become more democratic. Of course, I also hope that more researchers will join me. So that, eventually, all AI will be green AI. But of course, such an enormous change will probably take more than 7 years. Eventually, in maybe 10-15 years, AI will be energy-efficient by default. Ideally, sustainable digital technologies will have become the industry standard and nobody will have to think about this topic anymore — myself included!”
