SSD overheating during scrub in laptop

[Martin Steigerwald <martin@lichtvoll.de>]

Hi!

Just to share an experience and ask whether others have experienced this.

Already during summer it happened, that the Samsung 990 Pro 4 TB SSD in 
this ThinkPad T14 AMD Gen 5 said goodbye during scrubbing a 2 TB BTRFS 
filesystem with almost 2 TB of data in big files running at about 3,5 to 
4,2 GiB/s. I concluded this being due to excessive heat.

However it still succeeded with the 500 GiB /home with about 300-350 GiB 
of data in it back then. I worked around it, by scrubbing two minutes, 
then canceling, waiting to cool down, resuming for two minutes until the 
7-9 minutes of scrubbing were complete. I tried to work around with 
lowered speed, but even then the temperature slowly rose till the SSD said 
goodbye. I think I went down to 1 GiB/s of speed maximum, maybe even below 
that. But then the scrubbing takes longer so more time for SSD to heat up. 
However… I would have thought it would not heat that much with a slower 
speed. Maybe it would have worked with 50 or 100 MiB/s. But this takes 
long.

Now I had these SSD goodbyes during regular use in times of heavy I/O and 
in the end it could not even scrub that /home partition anymore in one go. 
Linux hung and only way to recover was to forcefully power off the laptop.

I opened the laptop and removed dust with high pressure air can while 
holding the fan still so it would not generate current. And with disabled 
laptop battery.

This fixed the SSDs goodbye issue and I could even scrub that 2 TiB 
filesystem again. However, according to sensors command it still had about 
80,8 °C composite temperature and even 100,8 °C for sensor 2 for the NVME 
SSD at "nvme-pci-0300" shortly before the end of the scrub, with critical 
for composite at 84,8 °C, but no critical set for sensor 2. That is still 
quite high. Granted, it took about 7-8 minutes of scrubbing at 3,5 to 4,2 
GiB/s in one go for it to heat up like this. But on the other hand it is 
not summer anymore and room is not as hot as in summer.

Anyone had similar experiences?

My solution to this will be to remove the dust inside laptop about every 
half year. However… I was a bit surprised that the NVME SSD would not 
throttle itself before saying goodbye. Or maybe it tried and it was not 
enough or to late? The laptop is a bit less than 15 months old. So I 
conclude that it takes less than a year for the cooling system to become 
quite a bit less effective due to dust. Good old ThinkPad T520 took way 
longer for that. But it is way larger on the other hand with more space to 
distribute heat.

I bet it is out of scope for btrfs scrub command to monitor NVME SSD 
temperature and pause when to high. And I conclude the NVME layer of the 
kernel is not throttling to hot NVME flash either.

I checked my data filesystems, two BTRFS and one BCacheFS by scrubbing and 
fsck'ing without changing anything. They are okay.

Best,
-- 
Martin