SSD overheating during scrub in laptop

SSD overheating during scrub in laptop

[Martin Steigerwald]

Hi!

Just to share an experience and ask whether others have experienced this.

Already during summer it happened, that the Samsung 990 Pro 4 TB SSD in 
this ThinkPad T14 AMD Gen 5 said goodbye during scrubbing a 2 TB BTRFS 
filesystem with almost 2 TB of data in big files running at about 3,5 to 
4,2 GiB/s. I concluded this being due to excessive heat.

However it still succeeded with the 500 GiB /home with about 300-350 GiB 
of data in it back then. I worked around it, by scrubbing two minutes, 
then canceling, waiting to cool down, resuming for two minutes until the 
7-9 minutes of scrubbing were complete. I tried to work around with 
lowered speed, but even then the temperature slowly rose till the SSD said 
goodbye. I think I went down to 1 GiB/s of speed maximum, maybe even below 
that. But then the scrubbing takes longer so more time for SSD to heat up. 
However… I would have thought it would not heat that much with a slower 
speed. Maybe it would have worked with 50 or 100 MiB/s. But this takes 
long.

Now I had these SSD goodbyes during regular use in times of heavy I/O and 
in the end it could not even scrub that /home partition anymore in one go. 
Linux hung and only way to recover was to forcefully power off the laptop.

I opened the laptop and removed dust with high pressure air can while 
holding the fan still so it would not generate current. And with disabled 
laptop battery.

This fixed the SSDs goodbye issue and I could even scrub that 2 TiB 
filesystem again. However, according to sensors command it still had about 
80,8 °C composite temperature and even 100,8 °C for sensor 2 for the NVME 
SSD at "nvme-pci-0300" shortly before the end of the scrub, with critical 
for composite at 84,8 °C, but no critical set for sensor 2. That is still 
quite high. Granted, it took about 7-8 minutes of scrubbing at 3,5 to 4,2 
GiB/s in one go for it to heat up like this. But on the other hand it is 
not summer anymore and room is not as hot as in summer.

Anyone had similar experiences?

My solution to this will be to remove the dust inside laptop about every 
half year. However… I was a bit surprised that the NVME SSD would not 
throttle itself before saying goodbye. Or maybe it tried and it was not 
enough or to late? The laptop is a bit less than 15 months old. So I 
conclude that it takes less than a year for the cooling system to become 
quite a bit less effective due to dust. Good old ThinkPad T520 took way 
longer for that. But it is way larger on the other hand with more space to 
distribute heat.

I bet it is out of scope for btrfs scrub command to monitor NVME SSD 
temperature and pause when to high. And I conclude the NVME layer of the 
kernel is not throttling to hot NVME flash either.

I checked my data filesystems, two BTRFS and one BCacheFS by scrubbing and 
fsck'ing without changing anything. They are okay.

Best,
-- 
Martin

Re: SSD overheating during scrub in laptop

[Qu Wenruo]

在 2025/9/14 21:27, Martin Steigerwald 写道:
> Hi!
> 
> Just to share an experience and ask whether others have experienced this.
> 
> Already during summer it happened, that the Samsung 990 Pro 4 TB SSD in
> this ThinkPad T14 AMD Gen 5 said goodbye during scrubbing a 2 TB BTRFS
> filesystem with almost 2 TB of data in big files running at about 3,5 to
> 4,2 GiB/s. I concluded this being due to excessive heat.
> 
> However it still succeeded with the 500 GiB /home with about 300-350 GiB
> of data in it back then. I worked around it, by scrubbing two minutes,
> then canceling, waiting to cool down, resuming for two minutes until the
> 7-9 minutes of scrubbing were complete. I tried to work around with
> lowered speed, but even then the temperature slowly rose till the SSD said
> goodbye. I think I went down to 1 GiB/s of speed maximum, maybe even below
> that. But then the scrubbing takes longer so more time for SSD to heat up.
> However… I would have thought it would not heat that much with a slower
> speed. Maybe it would have worked with 50 or 100 MiB/s. But this takes
> long.
> 
> Now I had these SSD goodbyes during regular use in times of heavy I/O and
> in the end it could not even scrub that /home partition anymore in one go.
> Linux hung and only way to recover was to forcefully power off the laptop.

Can you setup netconsole and catch the dying message?

I doubt if it's really the drive dying.


> 
> I opened the laptop and removed dust with high pressure air can while
> holding the fan still so it would not generate current. And with disabled
> laptop battery.
> 
> This fixed the SSDs goodbye issue and I could even scrub that 2 TiB
> filesystem again. However, according to sensors command it still had about
> 80,8 °C composite temperature and even 100,8 °C for sensor 2 for the NVME
> SSD at "nvme-pci-0300" shortly before the end of the scrub, with critical
> for composite at 84,8 °C, but no critical set for sensor 2. That is still
> quite high. Granted, it took about 7-8 minutes of scrubbing at 3,5 to 4,2
> GiB/s in one go for it to heat up like this. But on the other hand it is
> not summer anymore and room is not as hot as in summer.

I have hit similar situation, but the symptom is very different, the 
death come silently at boot, the drive is no longer recognized by the 
BIOS thus no longer bootable, and Linux kernel from liveUSB will not 
recognize it either.

That's why I'm asking if it's really dying caused by the heat.


> 
> Anyone had similar experiences?
> 
> My solution to this will be to remove the dust inside laptop about every
> half year. However… I was a bit surprised that the NVME SSD would not
> throttle itself before saying goodbye. Or maybe it tried and it was not
> enough or to late? The laptop is a bit less than 15 months old. So I
> conclude that it takes less than a year for the cooling system to become
> quite a bit less effective due to dust. Good old ThinkPad T520 took way
> longer for that. But it is way larger on the other hand with more space to
> distribute heat.

You can refer to man page of btrfs-scrub, it provides the way to limit 
the bandwidth of scrub using cgroup or even the btrfs sysfs interface.

Thanks,
Qu

> 
> I bet it is out of scope for btrfs scrub command to monitor NVME SSD
> temperature and pause when to high. And I conclude the NVME layer of the
> kernel is not throttling to hot NVME flash either.
> 
> I checked my data filesystems, two BTRFS and one BCacheFS by scrubbing and
> fsck'ing without changing anything. They are okay.
> 
> Best,

Re: SSD overheating during scrub in laptop

[Qu Wenruo]

在 2025/9/15 06:58, Qu Wenruo 写道:
> 
> 
> 在 2025/9/14 21:27, Martin Steigerwald 写道:
>> Hi!
>>
>> Just to share an experience and ask whether others have experienced this.
>>
>> Already during summer it happened, that the Samsung 990 Pro 4 TB SSD in
>> this ThinkPad T14 AMD Gen 5 said goodbye during scrubbing a 2 TB BTRFS
>> filesystem with almost 2 TB of data in big files running at about 3,5 to
>> 4,2 GiB/s. I concluded this being due to excessive heat.
>>
>> However it still succeeded with the 500 GiB /home with about 300-350 GiB
>> of data in it back then. I worked around it, by scrubbing two minutes,
>> then canceling, waiting to cool down, resuming for two minutes until the
>> 7-9 minutes of scrubbing were complete. I tried to work around with
>> lowered speed, but even then the temperature slowly rose till the SSD 
>> said
>> goodbye. I think I went down to 1 GiB/s of speed maximum, maybe even 
>> below
>> that. But then the scrubbing takes longer so more time for SSD to heat 
>> up.
>> However… I would have thought it would not heat that much with a slower
>> speed. Maybe it would have worked with 50 or 100 MiB/s. But this takes
>> long.
>>
>> Now I had these SSD goodbyes during regular use in times of heavy I/O and
>> in the end it could not even scrub that /home partition anymore in one 
>> go.
>> Linux hung and only way to recover was to forcefully power off the 
>> laptop.
> 
> Can you setup netconsole and catch the dying message?
> 
> I doubt if it's really the drive dying.
> 
> 
>>
>> I opened the laptop and removed dust with high pressure air can while
>> holding the fan still so it would not generate current. And with disabled
>> laptop battery.
>>
>> This fixed the SSDs goodbye issue and I could even scrub that 2 TiB
>> filesystem again. However, according to sensors command it still had 
>> about
>> 80,8 °C composite temperature and even 100,8 °C for sensor 2 for the NVME
>> SSD at "nvme-pci-0300" shortly before the end of the scrub, with critical
>> for composite at 84,8 °C, but no critical set for sensor 2. That is still
>> quite high. Granted, it took about 7-8 minutes of scrubbing at 3,5 to 4,2
>> GiB/s in one go for it to heat up like this. But on the other hand it is
>> not summer anymore and room is not as hot as in summer.
> 
> I have hit similar situation, but the symptom is very different, the 
> death come silently at boot, the drive is no longer recognized by the 
> BIOS thus no longer bootable, and Linux kernel from liveUSB will not 
> recognize it either.
> 
> That's why I'm asking if it's really dying caused by the heat.
> 
> 
>>
>> Anyone had similar experiences?
>>
>> My solution to this will be to remove the dust inside laptop about every
>> half year. However… I was a bit surprised that the NVME SSD would not
>> throttle itself before saying goodbye. Or maybe it tried and it was not
>> enough or to late? The laptop is a bit less than 15 months old. So I
>> conclude that it takes less than a year for the cooling system to become
>> quite a bit less effective due to dust. Good old ThinkPad T520 took way
>> longer for that. But it is way larger on the other hand with more 
>> space to
>> distribute heat.
> 
> You can refer to man page of btrfs-scrub, it provides the way to limit 
> the bandwidth of scrub using cgroup or even the btrfs sysfs interface.

And I forgot my physical solution, with a thick thermal pad, you can 
connect the NVME to the back plate of the laptop to dissipate heat.

If there is enough space (I doubt for laptop though), you can add some 
thin heatsink for the drive.

It works fine for one of my laptop, and the drive keeps a healthy 
temperature since then.

Thanks,
Qu

> 
> Thanks,
> Qu
> 
>>
>> I bet it is out of scope for btrfs scrub command to monitor NVME SSD
>> temperature and pause when to high. And I conclude the NVME layer of the
>> kernel is not throttling to hot NVME flash either.
>>
>> I checked my data filesystems, two BTRFS and one BCacheFS by scrubbing 
>> and
>> fsck'ing without changing anything. They are okay.
>>
>> Best,
> 
> 

Re: SSD overheating during scrub in laptop

[Martin Steigerwald]

Thanks.

Qu Wenruo - 14.09.25, 23:33:01 CEST:
> 在 2025/9/15 06:58, Qu Wenruo 写道:
> > 在 2025/9/14 21:27, Martin Steigerwald 写道:
[…]
> >> Now I had these SSD goodbyes during regular use in times of heavy I/O
> >> and in the end it could not even scrub that /home partition anymore
> >> in one go.
> >> Linux hung and only way to recover was to forcefully power off the
> >> laptop.
> > 
> > Can you setup netconsole and catch the dying message?

I suppose I could do that as I have several laptops at hand. However… 
after cleansing out at least some of the dust it did not happen anymore. 
And I do not quite feel like putting some dust back or otherwise provoke 
the issue. I do not quite feel like trying to overheat the SSD again. :) I 
looked at it more closely, it looks fine enough. Nothing melted 
apparently, but what do I know? And I think it does not add to the 
lifetime of the SSD to overheat it on purpose.

I might have made a photo of some earlier time, but I am not sure. I will 
have a look and see whether I can find any. I remember the kernel wrote a 
lot about NVME opcodes. However I do not recall the details. Unfortunately 
at this recent occasion I did not make a photo.

In case it happens again by itself, I am at going for a netconsole log – 
in case I can reproduce it then. Otherwise I would need to run netconsole 
permanently to be sure to catch it on the first occurrence already. Not 
sure whether that is a good idea.

> > I doubt if it's really the drive dying.

Why? The issue went away after removing at least some of the dust. Of 
course that is a correlation and not necessarily a causation, but what 
makes you think that something different is happening?

I checked the drive smart status. SMART status is passed. So everything 
okay. There is 2% of spare area used, but that is still a very good value.

An indication that there is something to your suspicion is:

Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

So the drive itself recorded no critical composite temperature times. And 
if it has fields for that, I suppose it could still record it on overheat 
condition.

Oh by the way, it is a Samsung 990 Pro with a firmware version above that 
firmware version that was known to be broken regarding SMART status 
reporting. In the first I bought I made sure of that myself, but this one 
came with a newer firmware version already.

> >> I opened the laptop and removed dust with high pressure air can while
> >> holding the fan still so it would not generate current. And with
> >> disabled laptop battery.
> >> 
> >> This fixed the SSDs goodbye issue and I could even scrub that 2 TiB
> >> filesystem again. However, according to sensors command it still had
> >> about 80,8 °C composite temperature and even 100,8 °C for sensor 2 
> >> for the NVME SSD at "nvme-pci-0300" shortly before the end of the 
> >> scrub, with critical for composite at 84,8 °C, but no critical set 
> >> for sensor 2. That is still quite high. Granted, it took about 7-8
> >> minutes of scrubbing at 3,5 to 4,2 GiB/s in one go for it to heat up
> >> like this. But on the other hand it is not summer anymore and room
> >> is not as hot as in summer.
> > 
> > I have hit similar situation, but the symptom is very different, the
> > death come silently at boot, the drive is no longer recognized by the
> > BIOS thus no longer bootable, and Linux kernel from liveUSB will not
> > recognize it either.

Hmm, did you capture any logs?

> > That's why I'm asking if it's really dying caused by the heat.

Ah okay… You mentioned your physical solution being a thermal pad. So did 
this symptom go away with it? Or was your physical solution a general 
solution to reduce temperature, a solution unrelated to that symptom you 
described?

> >> My solution to this will be to remove the dust inside laptop about
> >> every half year. However… I was a bit surprised that the NVME SSD
> >> would not throttle itself before saying goodbye. Or maybe it tried
> >> and it was not enough or to late? The laptop is a bit less than 15
> >> months old. So I conclude that it takes less than a year for the
> >> cooling system to become quite a bit less effective due to dust.
> >> Good old ThinkPad T520 took way longer for that. But it is way
> >> larger on the other hand with more space to distribute heat.
> > 
> > You can refer to man page of btrfs-scrub, it provides the way to limit
> > the bandwidth of scrub using cgroup or even the btrfs sysfs interface.
> 
> And I forgot my physical solution, with a thick thermal pad, you can
> connect the NVME to the back plate of the laptop to dissipate heat.

Well it has some violet colored pad below the NVME SSD. So a pad that is 
between NVME SSD and motherboard. Not sure whether it is a cooling pad. It 
might be more about isolating the motherboard from the heat of the SSD? It 
has been there for the initial SSD Lenovo put into the laptop that I 
replaced with the larger Samsung 990 Pro one.

But yeah, I may order a cooling plate to connection to the back plate. 
Good idea.
 
> If there is enough space (I doubt for laptop though), you can add some
> thin heatsink for the drive.

Ha ha, no way. I would like to put a 8 TB SSD in there, but it would be 
double layered and I doubt it would be a good idea to put a double layered 
SSD into the laptop.

Best,
-- 
Martin

Re: SSD overheating during scrub in laptop

[Qu Wenruo]

在 2025/9/16 21:06, Martin Steigerwald 写道:
> Thanks.
> 
> Qu Wenruo - 14.09.25, 23:33:01 CEST:
>> 在 2025/9/15 06:58, Qu Wenruo 写道:
>>> 在 2025/9/14 21:27, Martin Steigerwald 写道:
> […]
>>>> Now I had these SSD goodbyes during regular use in times of heavy I/O
>>>> and in the end it could not even scrub that /home partition anymore
>>>> in one go.
>>>> Linux hung and only way to recover was to forcefully power off the
>>>> laptop.
>>>
>>> Can you setup netconsole and catch the dying message?
> 
> I suppose I could do that as I have several laptops at hand. However…
> after cleansing out at least some of the dust it did not happen anymore.
> And I do not quite feel like putting some dust back or otherwise provoke
> the issue. I do not quite feel like trying to overheat the SSD again. :) I
> looked at it more closely, it looks fine enough. Nothing melted
> apparently, but what do I know? And I think it does not add to the
> lifetime of the SSD to overheat it on purpose.
> 
> I might have made a photo of some earlier time, but I am not sure. I will
> have a look and see whether I can find any. I remember the kernel wrote a
> lot about NVME opcodes. However I do not recall the details. Unfortunately
> at this recent occasion I did not make a photo.
> 
> In case it happens again by itself, I am at going for a netconsole log –
> in case I can reproduce it then. Otherwise I would need to run netconsole
> permanently to be sure to catch it on the first occurrence already. Not
> sure whether that is a good idea.
> 
>>> I doubt if it's really the drive dying.
> 
> Why? The issue went away after removing at least some of the dust. Of
> course that is a correlation and not necessarily a causation, but what
> makes you think that something different is happening?
> 
> I checked the drive smart status. SMART status is passed. So everything
> okay. There is 2% of spare area used, but that is still a very good value.
> 
> An indication that there is something to your suspicion is:
> 
> Media and Data Integrity Errors:    0
> Error Information Log Entries:      0
> Warning  Comp. Temperature Time:    0
> Critical Comp. Temperature Time:    0
> 
> So the drive itself recorded no critical composite temperature times. And
> if it has fields for that, I suppose it could still record it on overheat
> condition.
> 
> Oh by the way, it is a Samsung 990 Pro with a firmware version above that
> firmware version that was known to be broken regarding SMART status
> reporting. In the first I bought I made sure of that myself, but this one
> came with a newer firmware version already.
> 
>>>> I opened the laptop and removed dust with high pressure air can while
>>>> holding the fan still so it would not generate current. And with
>>>> disabled laptop battery.
>>>>
>>>> This fixed the SSDs goodbye issue and I could even scrub that 2 TiB
>>>> filesystem again. However, according to sensors command it still had
>>>> about 80,8 °C composite temperature and even 100,8 °C for sensor 2
>>>> for the NVME SSD at "nvme-pci-0300" shortly before the end of the
>>>> scrub, with critical for composite at 84,8 °C, but no critical set
>>>> for sensor 2. That is still quite high. Granted, it took about 7-8
>>>> minutes of scrubbing at 3,5 to 4,2 GiB/s in one go for it to heat up
>>>> like this. But on the other hand it is not summer anymore and room
>>>> is not as hot as in summer.
>>>
>>> I have hit similar situation, but the symptom is very different, the
>>> death come silently at boot, the drive is no longer recognized by the
>>> BIOS thus no longer bootable, and Linux kernel from liveUSB will not
>>> recognize it either.
> 
> Hmm, did you capture any logs?

Nothing suspicious at all, except no PCIE device detected at all.

So in my case it's really toasted so that it can not even be detected 
anymore.
> 
>>> That's why I'm asking if it's really dying caused by the heat.
> 
> Ah okay… You mentioned your physical solution being a thermal pad. So did
> this symptom go away with it?

The nvme drive went away literally, and never came back.

> Or was your physical solution a general
> solution to reduce temperature, a solution unrelated to that symptom you
> described?

Unrelated to my long-gone drive, but a new drive never suffered critical 
high temperature anymore.

Thanks,
Qu

> 
>>>> My solution to this will be to remove the dust inside laptop about
>>>> every half year. However… I was a bit surprised that the NVME SSD
>>>> would not throttle itself before saying goodbye. Or maybe it tried
>>>> and it was not enough or to late? The laptop is a bit less than 15
>>>> months old. So I conclude that it takes less than a year for the
>>>> cooling system to become quite a bit less effective due to dust.
>>>> Good old ThinkPad T520 took way longer for that. But it is way
>>>> larger on the other hand with more space to distribute heat.
>>>
>>> You can refer to man page of btrfs-scrub, it provides the way to limit
>>> the bandwidth of scrub using cgroup or even the btrfs sysfs interface.
>>
>> And I forgot my physical solution, with a thick thermal pad, you can
>> connect the NVME to the back plate of the laptop to dissipate heat.
> 
> Well it has some violet colored pad below the NVME SSD. So a pad that is
> between NVME SSD and motherboard. Not sure whether it is a cooling pad. It
> might be more about isolating the motherboard from the heat of the SSD? It
> has been there for the initial SSD Lenovo put into the laptop that I
> replaced with the larger Samsung 990 Pro one.
> 
> But yeah, I may order a cooling plate to connection to the back plate.
> Good idea.
>   
>> If there is enough space (I doubt for laptop though), you can add some
>> thin heatsink for the drive.
> 
> Ha ha, no way. I would like to put a 8 TB SSD in there, but it would be
> double layered and I doubt it would be a good idea to put a double layered
> SSD into the laptop.
> 
> Best,