Re: [ata] : SATA device speed down after multiple power reset.

Re: [ata] : SATA device speed down after multiple power reset.

[Damien Le Moal]

On 2/23/23 19:51, Deepak Ukey wrote:
> Hi All,
> 
> I have SATA SSD that I am using for robust device functionality testing.
> I have a setup where SATA SSD is attached to AHCI controller and power 
> to the device is given by power management unit.
> 
> For the particular test I am doing mutliple power on/off to the device,
> it is similar to hot unplugged and plug operation. 
> 
> After the 4th iteration of power off and on to device, device link speed
> is going down to next lower speed, i.e. if device link speed is 6Gbps then
> it will reduce to 3Gbps.
> 
> I tried to debug the issue and found that ata_eh_speed_down function is
> recording the error as ATA_EH_SPDN_SPEED_DOWN and limiting the speed to
> next lower speed. Afterward SSD starts working on the next lower speed.
> 
> So can you please tell me why this specific functionality is added. Also 
> is there any way to recover the speed to max speed so that SSD
> performance should be intact. 

Please send a dmesg output of your test to see what is happening.

The issue is likely that cutting the drive power is may be not handled the same
way as unpluging the drive and replugging it (try to see if you get the same
result or not). If that is the case, then the drive is simply
disappearing/appearing and that is handled as an error. Too many of these and
libata decides to lower the link speed to try to reduce the number of errors. So
your test is probably not entirely identical to plug/unplug. Losing power is an
error. Unplugging is not.

> 
> Thanks & Regards,
> Deepak Ukey
> 
> 

-- 
Damien Le Moal
Western Digital Research

Re: [ata] : SATA device speed down after multiple power reset.

[Deepak Ukey]

> On 2/23/23 19:51, Deepak Ukey wrote:
> > Hi All,
> > 
> > I have SATA SSD that I am using for robust device functionality testing.
> > I have a setup where SATA SSD is attached to AHCI controller and power 
> > to the device is given by power management unit.
> > 
> > For the particular test I am doing mutliple power on/off to the device,
> > it is similar to hot unplugged and plug operation. 
> > 
> > After the 4th iteration of power off and on to device, device link speed
> > is going down to next lower speed, i.e. if device link speed is 6Gbps then
> > it will reduce to 3Gbps.
> > 
> > I tried to debug the issue and found that ata_eh_speed_down function is
> > recording the error as ATA_EH_SPDN_SPEED_DOWN and limiting the speed to
> > next lower speed. Afterward SSD starts working on the next lower speed.
> > 
> > So can you please tell me why this specific functionality is added. Also 
> > is there any way to recover the speed to max speed so that SSD
> > performance should be intact. 
> 
> Please send a dmesg output of your test to see what is happening.

Thanks for the response. As you mentioned, we are cutting the power and
device is disappearing/appearing. Please find the below logs for the reference.

---------------------
[  226.463486] ata_eh_link_autopsy: IN
[  226.463492] ata_eh_link_autopsy: taking speed down action
[  226.463496] speed_down_verdict_cb: Returning zero
[  226.463497] speed_down_verdict_cb: Returning zero
[  226.463497] speed_down_verdict_cb: Returning zero
[  226.463498] speed_down_verdict_cb: Returning zero
[  226.463500] speed_down_verdict_cb: Returning zero
[  226.463501] speed_down_verdict_cb: Returning zero
[  226.463501] speed_down_verdict_cb: Returning zero
[  226.463502] speed_down_verdict_cb: Returning zero
[  226.463504] ata_eh_speed_down_verdict: ATA_EH_SPDN_SPEED_DOWN_1
[  226.463504] ata_eh_speed_down: Verdict:2
[  226.463506] ata_eh_speed_down: Calling limit Speed 8........
[  226.463507] sata_down_spd_limit: speed limit:0
[  226.463509] sata_down_spd_limit: Out speed limit:3
[  226.463511] ata4: limiting SATA link speed to 3.0 Gbps
[  226.463512] ata_eh_autopsy:IN
[  226.463514] ata_eh_reset: IN
[  230.669238] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[  230.669505] ata4.00: supports DRM functions and may not be fully accessible
[  230.671982] ata4.00: supports DRM functions and may not be fully accessible
[  230.674234] ata4.00: configured for UDMA/133
---------------------------

But is there any way that we can recover the speed back to normal if device is
functioning properly after power cut for considerable amount of time.

> 
> The issue is likely that cutting the drive power is may be not handled the same
> way as unpluging the drive and replugging it (try to see if you get the same
> result or not). If that is the case, then the drive is simply
> disappearing/appearing and that is handled as an error. Too many of these and
> libata decides to lower the link speed to try to reduce the number of errors. So
> your test is probably not entirely identical to plug/unplug. Losing power is an
> error. Unplugging is not.
> 
> > 
> > Thanks & Regards,
> > Deepak Ukey
> > 
> > 
> 
> -- 
> Damien Le Moal
> Western Digital Research

Thank and regards,
Deeepak Ukey

Re: [ata] : SATA device speed down after multiple power reset.

[Damien Le Moal]

On 3/30/23 20:12, Deepak Ukey wrote:
>> On 2/23/23 19:51, Deepak Ukey wrote:
>>> Hi All,
>>>
>>> I have SATA SSD that I am using for robust device functionality testing.
>>> I have a setup where SATA SSD is attached to AHCI controller and power 
>>> to the device is given by power management unit.
>>>
>>> For the particular test I am doing mutliple power on/off to the device,
>>> it is similar to hot unplugged and plug operation. 
>>>
>>> After the 4th iteration of power off and on to device, device link speed
>>> is going down to next lower speed, i.e. if device link speed is 6Gbps then
>>> it will reduce to 3Gbps.
>>>
>>> I tried to debug the issue and found that ata_eh_speed_down function is
>>> recording the error as ATA_EH_SPDN_SPEED_DOWN and limiting the speed to
>>> next lower speed. Afterward SSD starts working on the next lower speed.
>>>
>>> So can you please tell me why this specific functionality is added. Also 
>>> is there any way to recover the speed to max speed so that SSD
>>> performance should be intact. 
>>
>> Please send a dmesg output of your test to see what is happening.
> 
> Thanks for the response. As you mentioned, we are cutting the power and
> device is disappearing/appearing. Please find the below logs for the reference.
> 
> ---------------------
> [  226.463486] ata_eh_link_autopsy: IN
> [  226.463492] ata_eh_link_autopsy: taking speed down action
> [  226.463496] speed_down_verdict_cb: Returning zero
> [  226.463497] speed_down_verdict_cb: Returning zero
> [  226.463497] speed_down_verdict_cb: Returning zero
> [  226.463498] speed_down_verdict_cb: Returning zero
> [  226.463500] speed_down_verdict_cb: Returning zero
> [  226.463501] speed_down_verdict_cb: Returning zero
> [  226.463501] speed_down_verdict_cb: Returning zero
> [  226.463502] speed_down_verdict_cb: Returning zero
> [  226.463504] ata_eh_speed_down_verdict: ATA_EH_SPDN_SPEED_DOWN_1
> [  226.463504] ata_eh_speed_down: Verdict:2
> [  226.463506] ata_eh_speed_down: Calling limit Speed 8........
> [  226.463507] sata_down_spd_limit: speed limit:0
> [  226.463509] sata_down_spd_limit: Out speed limit:3
> [  226.463511] ata4: limiting SATA link speed to 3.0 Gbps
> [  226.463512] ata_eh_autopsy:IN
> [  226.463514] ata_eh_reset: IN
> [  230.669238] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
> [  230.669505] ata4.00: supports DRM functions and may not be fully accessible
> [  230.671982] ata4.00: supports DRM functions and may not be fully accessible
> [  230.674234] ata4.00: configured for UDMA/133
> ---------------------------
> 
> But is there any way that we can recover the speed back to normal if device is
> functioning properly after power cut for considerable amount of time.

Please try to unplug and re plug the drive, to see if plug-and-play works
correctly. Really unplug the drive (not just the power cut).

From the above messages (not standard), it seems that the system did not see the
drives going away, and so when power is restored, the state is as before and eh
assumes that there was an error and so decides to lower the drive speed.

If you want to save power, use the proper ata power management features and put
the drive to deep sleep (PM3 mode with SLEEP command) or Standby_z power state
(STANDBY IMMEDIATE command). See man hdparm.


-- 
Damien Le Moal
Western Digital Research

Re: [ata] : SATA device speed down after multiple power reset.

[Ioannis Barkas]

Hello,

Since you have reported this problem, I suggest to provide some more
details in case you wish to find its root cause.

You have not provided any useful information at all! You have just
sent the error but everything else is kept in dark...

Having that in mind, please elaborate to see what goes wrong. Damien
will help you if it caused by any coding error. First of all, provide
a full dmesg so that we can see the interesting details. In detail, we
need to see the CPU, the SATA host capabilities and the IC used.
Moreover, we must see the SSD device you are using as well as the
kernel version. Are you using a custom kernel or one of the official
releases? Please provide a link to its source in the case of a custom
kernel.

To track this error, I suggest to increase the kernel buffer and
enable CONFIG_ATA_VERBOSE_ERROR. Repeat testing and provide the latest
dmesg.

Concerning the storage device, does your SSD have power loss
protection? Have you tested this configuration with an HDD? If the
controller is PCI based, please provide lspci -nnnvvvxxx for all PCI
devices run with elevated rights. Also please post hdparm and check
your SSD device health. Have you detected any filesystem related
corruption? Why do you want to power it down?

In regards to the Power Management Unit(PMU), is the system running
while the SSD is powered off or does it enter a lower power state as
well? For example S3 or S4 speaking ACPI wise.

There have been a lot of broken storage devices as well as SATA hosts
but no one will be able to help you on this unless you provide
additional information.

Kind regards.