How Cooling is keeping up with the AI revolution

Moore’s law is very overheated at the moment

For semiconductors, this decade is all about vertical integration—stacking chiplets on top of each other and stacking transistors, and that’s great for their performance, but very problematic for cooling.

The heat problem

We’ve managed to continuously improve performance by packing more and more transistors into the size of a chocolate square. But now, there are so many transistors that they cannot all be used at once without the chip overheating. How much heat a chip dissipates is measured in so-called TDP, which stands for thermal design power. It’s based on the maximum heat flux that can be removed from a chip.

If we take NVIDIA’s H100 GPU, it’s around 700W TDP. The latest NVIDIA Blackwell GPU dissipates about 1kW of heat. One of the problems with such powerful chips is that while it performs operations, about half of this chip is dark silicon.

Dark silicon is a phenomenon where a significant portion of the transistors on the chip cannot be computing at the same time due to power and thermal constraints.

What makes it even worse is the ongoing vertical integration. It all started by stacking chiplets, smaller pieces of chips that have their own function, on top of each other. That’s great for their performance but very problematic for cooling. The same trend is happening with the chip’s building blocks - transistors - as we’re now transitioning towards stacking nanosheets vertically. Well, what are we going to do with all the additional heat that will come as a result of this?

Keeping it cool

There are many ways to keep chips cool. Typically, what we used to do is conduct this heat out somewhere and then dissipate it. But by far, the most popular solutions involve cooling with air or liquid. Cooling with air works for some desktop chips and some server processors like AMD’s EPYC Milan CPU, which dissipates about 280W.

However, somewhere around 300W TDP, we reach the limit of how much we are able to cool with air alone, and above that, we have to switch to liquid cooling, which can conduct up to 3,000 times more heat than air can. This works, for example, with NVIDIA GPUs like A100 and H100, which can dissipate up to 1000 Watts of heat.

Cooling the Hottest Chips

Eventually, we end up guiding as much heat as we can to heat sinks or cold plates. In any case, even heat sinks have limits. So, if we want to cool down something like a DOJO training tile that dissipates up to 15,000W of heat, or when we cool down Cerebras’ Wafer Scale Engine that dissipates up to 25,000W, clearly we need to use something much more advanced. I explained how it’s done in this video.

Now, air or liquid cooling is what we typically think of when we think of cooling. But actually, work on cooling starts during the process of designing chips. When I was working as a chip designer, already during the physical design phase, we consider the most thermally-aware ways of placing the blocks and cells on the chip. We also have to keep in mind the switching activity of the blocks, as some of the chip’s parts will be switching more than others. So we place these blocks in a way that minimizes both the peak temperature as well as the temperature gradients across the chip.

Through Silicon Vias. Source: Okmetic

But that’s usually not enough, so for 3D chips, we have to create heat corridors by placing TSVs in a particular way to help spread the heat evenly. TSVs are through-silicon vias, which are copper connections that travel through the silicon die. These are used to connect chiplets in designs like AMD’s MI300, which has 13 chiplets stacked together and acts as one single chip, and Intel's Ponte Vecchio GPU. Also, with TSVs, we provide vertical and lateral pathways for heat dissipation. So they do help to guide some heat away, but it’s not a silver bullet, unfortunately.

Recent Cooling Innovations

Now, it's clear that we will see more powerful chips and AI ASICs coming in the next few years. Managing the heat produced by these chips is going to be one of the biggest challenges in electronics going forward. Already, we now have to go for immersion cooling and have sunk whole racks into tanks of liquid. The next step is obviously to bring this fluid even closer to the source of that heat—closer to the transistors.

Just think for a moment how efficient it would be if coolants flowed inside processors!

This idea is called embedded cooling—when we bring the cooling to the interior of the silicon, super close to the computing cores that are running the job. This helps to light up those dark silicon parts that we discussed earlier, giving a strong boost to chip performance. I explain how it works in every detail in this video. Watch it now:

According to the paper, it improves the cooling efficiency by a factor of at least 50. This cooling technology is so cool, that is has its own fan club 😂. If you enjoyed this post, please share it with your friends and on social media. Thanks a lot!

While we’re talking about Hot Chips today - the Hot Chips Conference will take place at Stanford University between August 25 to 27. There will be discussions about AI in chip design, all the recent advances in Hot Chips and the cooling technologies of the future. Check it out and register here. I’ll be attending it in person. If you are there, please come and say hi.