2.6.18-rc2, problem to wake up spinned down drive?

2.6.18-rc2, problem to wake up spinned down drive?

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

Hi folks,

I tried to spin down my harddisk using hdparm, but when it is
supposed to spin up again, then it is blocked for quite some
time. dmesg says:

ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
ata1.00: (BMDMA stat 0x20)
ata1.00: tag 0 cmd 0xca Emask 0x4 stat 0x40 err 0x0 (timeout)
ata1: port is slow to respond, please be patient
ata1: port failed to respond (30 secs)
ata1: soft resetting port
ata1.00: configured for UDMA/133
ata1: EH complete
SCSI device sda: 312581808 512-byte hdwr sectors (160042 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back

The disk is a SAMSUNG SP1614C.


On another machine (with a SAMSUNG SP2504C inside) there is no
such problem: The disk is back after just a few seconds.


Is there some trick to wake up the disk a little bit faster?


Regards

Harri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Johannes Weiner]

Hi,

On Sun, Jul 30, 2006 at 02:01:02PM +0200, Harald Dunkel wrote:
> Hi folks,
> 
> I tried to spin down my harddisk using hdparm, but when it is
> supposed to spin up again, [...]

When is the point reached to spin up again?

> On another machine (with a SAMSUNG SP2504C inside) there is no
> such problem: The disk is back after just a few seconds.

Same kernel?

Hannes

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 982 bytes --]

Hi Hannes,

Johannes Weiner wrote:
> Hi,
> 
> On Sun, Jul 30, 2006 at 02:01:02PM +0200, Harald Dunkel wrote:
>> Hi folks,
>>
>> I tried to spin down my harddisk using hdparm, but when it is
>> supposed to spin up again, [...]
> 
> When is the point reached to spin up again?
> 

When some process tries to access a file that is not
cached in RAM, AFAICT. All partitions are mounted with
noatime.

>> On another machine (with a SAMSUNG SP2504C inside) there is no
>> such problem: The disk is back after just a few seconds.
> 
> Same kernel?
> 

Yes. But both machines are very different. The working
machine is an amd64 with nforce2. The machine that gets
stuck is a 2 GHz Pentium M with some Intel chipset:

00:1f.2 IDE interface: Intel Corporation 82801FB/FW (ICH6/ICH6W) SATA Controller (rev 04)
00:1f.2 0101: 8086:2651 (rev 04)

Sorry, surely I should have mentioned this before. The
problem isn't _that_ reproducible.


Regards

Harri



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Tejun Heo]

Harald Dunkel wrote:
> Hi folks,
> 
> I tried to spin down my harddisk using hdparm, but when it is
> supposed to spin up again, then it is blocked for quite some
> time. dmesg says:
> 
> ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
> ata1.00: (BMDMA stat 0x20)
> ata1.00: tag 0 cmd 0xca Emask 0x4 stat 0x40 err 0x0 (timeout)
> ata1: port is slow to respond, please be patient
> ata1: port failed to respond (30 secs)
> ata1: soft resetting port
> ata1.00: configured for UDMA/133
> ata1: EH complete
> SCSI device sda: 312581808 512-byte hdwr sectors (160042 MB)
> sda: Write Protect is off
> sda: Mode Sense: 00 3a 00 00
> SCSI device sda: drive cache: write back
> 
> The disk is a SAMSUNG SP1614C.
> 
> On another machine (with a SAMSUNG SP2504C inside) there is no
> such problem: The disk is back after just a few seconds.

In standby mode, the drive's interface and state machines stay online 
and are supposed to spin up and process the command when it receives 
one.  The above message is printed because an IO command hasn't finished 
in 30 secs meaning that it didn't wake up when it should have.  The 
drive seems to act incorrectly.

> Is there some trick to wake up the disk a little bit faster?

Can you try the following instead of hdparm?

echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state

It will make libata involved in putting the disk to sleep and waking it 
up, and, when waking, it will kick the drive in the ass by resetting the 
channel.  Please try with the latest -rc kernel.

-- 
tejun

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

Tejun Heo wrote:
> 
> Can you try the following instead of hdparm?
> 
> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state
> 
> It will make libata involved in putting the disk to sleep and waking it
> up, and, when waking, it will kick the drive in the ass by resetting the
> channel.  Please try with the latest -rc kernel.
> 

Sorry to say, but this did not work:

# echo 1 > /sys/bus/scsi/devices/0:0:0:0/power/state
bash: echo: write error: Invalid argument
# ll !$
ll /sys/bus/scsi/devices/0:0:0:0/power/state
-rw-r--r-- 1 root root 0 Aug  1 20:00 /sys/bus/scsi/devices/0:0:0:0/power/state
# cat !$
cat /sys/bus/scsi/devices/0:0:0:0/power/state
0
# uname -a
Linux bugs 2.6.18-rc3 #2 PREEMPT Sun Jul 30 16:26:22 CEST 2006 i686 GNU/Linux


Regards

Harri



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Tejun Heo]

Harald Dunkel wrote:
> Tejun Heo wrote:
>> Can you try the following instead of hdparm?
>>
>> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state
>>
>> It will make libata involved in putting the disk to sleep and waking it
>> up, and, when waking, it will kick the drive in the ass by resetting the
>> channel.  Please try with the latest -rc kernel.
>>
> 
> Sorry to say, but this did not work:
> 
> # echo 1 > /sys/bus/scsi/devices/0:0:0:0/power/state
> bash: echo: write error: Invalid argument
> # ll !$
> ll /sys/bus/scsi/devices/0:0:0:0/power/state
> -rw-r--r-- 1 root root 0 Aug  1 20:00 /sys/bus/scsi/devices/0:0:0:0/power/state
> # cat !$
> cat /sys/bus/scsi/devices/0:0:0:0/power/state
> 0
> # uname -a
> Linux bugs 2.6.18-rc3 #2 PREEMPT Sun Jul 30 16:26:22 CEST 2006 i686 GNU/Linux

You probably should do 'echo -n 1', the parsing function is pretty picky.

-- 
tejun

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Pavel Machek]

Hi!

> I tried to spin down my harddisk using hdparm, but when it is
> supposed to spin up again, then it is blocked for quite some
> time. dmesg says:
> 
> ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
> ata1.00: (BMDMA stat 0x20)
> ata1.00: tag 0 cmd 0xca Emask 0x4 stat 0x40 err 0x0 (timeout)
> ata1: port is slow to respond, please be patient
> ata1: port failed to respond (30 secs)
> ata1: soft resetting port
> ata1.00: configured for UDMA/133
> ata1: EH complete
> SCSI device sda: 312581808 512-byte hdwr sectors (160042 MB)
> sda: Write Protect is off
> sda: Mode Sense: 00 3a 00 00
> SCSI device sda: drive cache: write back
> 
> The disk is a SAMSUNG SP1614C.
> 
> 
> On another machine (with a SAMSUNG SP2504C inside) there is no
> such problem: The disk is back after just a few seconds.

How do you manage to spindown SATA disks? I tried hdparm -y, but that
did not work iirc.
							Pavel

-- 
Thanks for all the (sleeping) penguins.

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Bill Davidsen]

Tejun Heo wrote:
> Harald Dunkel wrote:
>> Tejun Heo wrote:
>>> Can you try the following instead of hdparm?
>>>
>>> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state
>>>
>>> It will make libata involved in putting the disk to sleep and waking it
>>> up, and, when waking, it will kick the drive in the ass by resetting the
>>> channel.  Please try with the latest -rc kernel.
>>>
>>
>> Sorry to say, but this did not work:
>>
>> # echo 1 > /sys/bus/scsi/devices/0:0:0:0/power/state
>> bash: echo: write error: Invalid argument
>> # ll !$
>> ll /sys/bus/scsi/devices/0:0:0:0/power/state
>> -rw-r--r-- 1 root root 0 Aug  1 20:00 
>> /sys/bus/scsi/devices/0:0:0:0/power/state
>> # cat !$
>> cat /sys/bus/scsi/devices/0:0:0:0/power/state
>> 0
>> # uname -a
>> Linux bugs 2.6.18-rc3 #2 PREEMPT Sun Jul 30 16:26:22 CEST 2006 i686 
>> GNU/Linux
> 
> You probably should do 'echo -n 1', the parsing function is pretty picky.
> 
Given that the data from the "cat" of state returned a zero with 
newline, perhaps unreasonably picky. On a Fedora kernel it just doesn't 
seem to work for SATA drives, sample size = 1.

-- 
Bill Davidsen <davidsen@tmr.com>
   Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

Hi Pavel,

Pavel Machek wrote:
> 
> How do you manage to spindown SATA disks? I tried hdparm -y, but that
> did not work iirc.
> 							Pavel
> 

"hdparm -S"?

Please note that standard 3.5" disks are not made to spin
down every 5 minutes. This feature can reduce the lifespan
of your disk, AFAIK.


Regards

Harri



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 414 bytes --]

Tejun Heo wrote:
> Harald Dunkel wrote:
>>
>> Sorry to say, but this did not work:
>>
>> # echo 1 > /sys/bus/scsi/devices/0:0:0:0/power/state
>> bash: echo: write error: Invalid argument
> 
> You probably should do 'echo -n 1', the parsing function is pretty picky.
> 

# echo -n 1 > /sys/bus/scsi/devices/0:0:0:0/power/state
bash: echo: write error: Invalid argument

???


Regards

Harri



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Pavel Machek]

Hi!

> >ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 
> >0x2 frozen
> >ata1.00: (BMDMA stat 0x20)
> >ata1.00: tag 0 cmd 0xca Emask 0x4 stat 0x40 err 0x0 
> >(timeout)
> >ata1: port is slow to respond, please be patient
> >ata1: port failed to respond (30 secs)
> >ata1: soft resetting port
> >ata1.00: configured for UDMA/133
> >ata1: EH complete
> >SCSI device sda: 312581808 512-byte hdwr sectors 
> >(160042 MB)
> >sda: Write Protect is off
> >sda: Mode Sense: 00 3a 00 00
> >SCSI device sda: drive cache: write back
> >
> >The disk is a SAMSUNG SP1614C.
> >
> >On another machine (with a SAMSUNG SP2504C inside) 
> >there is no
> >such problem: The disk is back after just a few seconds.
> 
> In standby mode, the drive's interface and state 
> machines stay online and are supposed to spin up and 
> process the command when it receives one.  The above 
> message is printed because an IO command hasn't finished 
> in 30 secs meaning that it didn't wake up when it should 
> have.  The drive seems to act incorrectly.
> 
> >Is there some trick to wake up the disk a little bit 
> >faster?
> 
> Can you try the following instead of hdparm?
> 
> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state

Really? I thought power/state takes 0/3 (for D0 and D3)

							Pavel
-- 
Thanks for all the (sleeping) penguins.

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Tejun Heo]

Pavel Machek wrote:
>> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state
> 
> Really? I thought power/state takes 0/3 (for D0 and D3)

Yes, of course.  My mistake.  Sorry about the confusion.  The correct 
command is 'echo -n 3 > /sys/bus/scsi/devices/x:y:z:w/power/state'.

-- 
tejun

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

Tejun Heo wrote:
> Pavel Machek wrote:
>>> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state
>>
>> Really? I thought power/state takes 0/3 (for D0 and D3)
> 
> Yes, of course.  My mistake.  Sorry about the confusion.  The correct
> command is 'echo -n 3 > /sys/bus/scsi/devices/x:y:z:w/power/state'.
> 

(Sure?  :-)

Now this did not work at all. The '-n 3' was probably
correct, but when I tried to access the disk, then it
did not spin up again (I waited for 5 minutes). There
was no message on the console, either.

But I could not reproduce this problem.

How do I monitor that the disk spins down and up?


Regards

Harri



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Tejun Heo]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]

Harald Dunkel wrote:
> Tejun Heo wrote:
>> Pavel Machek wrote:
>>>> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state
>>> Really? I thought power/state takes 0/3 (for D0 and D3)
>> Yes, of course.  My mistake.  Sorry about the confusion.  The correct
>> command is 'echo -n 3 > /sys/bus/scsi/devices/x:y:z:w/power/state'.
>>
> 
> (Sure?  :-)

The sleeping part is correct.  That will make libata put the disk to sleep.

> Now this did not work at all. The '-n 3' was probably
> correct, but when I tried to access the disk, then it
> did not spin up again (I waited for 5 minutes). There
> was no message on the console, either.
> 
> But I could not reproduce this problem.
> 
> How do I monitor that the disk spins down and up?

But the waking up part isn't.  You need to issue wake up explicitly by 
doing 'echo -n 0 > /sys/...'  I've been a complete idiot in this thread. 
  Please excuse me.  :-(

I think the solution to your problem is adjusting command timeout to 
more reasonable values which should make the problem more bearable. 
It'll take some time to figure out how to make timeouts more intelligent 
without breaking support for slow devices.  I'll work on that.

I'm attaching a temporary patch for the time being.

-- 
tejun

[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 987 bytes --]

diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index 98bd3aa..5676388 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -99,7 +99,7 @@ #define SD_MAX_DISKS	(((26 * 26) + 26 + 
 /*
  * Time out in seconds for disks and Magneto-opticals (which are slower).
  */
-#define SD_TIMEOUT		(30 * HZ)
+#define SD_TIMEOUT		(7 * HZ)
 #define SD_MOD_TIMEOUT		(75 * HZ)
 
 /*
diff --git a/include/linux/libata.h b/include/linux/libata.h
index b941670..45686f9 100644
--- a/include/linux/libata.h
+++ b/include/linux/libata.h
@@ -200,9 +200,9 @@ enum {
 	ATA_HOST_SIMPLEX	= (1 << 0),	/* Host is simplex, one DMA channel per host_set only */
 	
 	/* various lengths of time */
-	ATA_TMOUT_BOOT		= 30 * HZ,	/* heuristic */
-	ATA_TMOUT_BOOT_QUICK	= 7 * HZ,	/* heuristic */
-	ATA_TMOUT_INTERNAL	= 30 * HZ,
+	ATA_TMOUT_BOOT		= 10 * HZ,	/* heuristic */
+	ATA_TMOUT_BOOT_QUICK	= 5 * HZ,	/* heuristic */
+	ATA_TMOUT_INTERNAL	= 10 * HZ,
 	ATA_TMOUT_INTERNAL_QUICK = 5 * HZ,
 
 	/* ATA bus states */

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Bill Davidsen]

Tejun Heo wrote:

> Harald Dunkel wrote:
>
>> Tejun Heo wrote:
>>
>>> Pavel Machek wrote:
>>>
>>>>> echo 1 > /sys/bus/scsi/devices/1:0:0:0/power/state
>>>>
>>>> Really? I thought power/state takes 0/3 (for D0 and D3)
>>>
>>> Yes, of course.  My mistake.  Sorry about the confusion.  The correct
>>> command is 'echo -n 3 > /sys/bus/scsi/devices/x:y:z:w/power/state'.
>>>
>>
>> (Sure?  :-)
>
>
> The sleeping part is correct.  That will make libata put the disk to 
> sleep.
>
>> Now this did not work at all. The '-n 3' was probably
>> correct, but when I tried to access the disk, then it
>> did not spin up again (I waited for 5 minutes). There
>> was no message on the console, either.
>>
>> But I could not reproduce this problem.
>>
>> How do I monitor that the disk spins down and up?
>
>
> But the waking up part isn't.  You need to issue wake up explicitly by 
> doing 'echo -n 0 > /sys/...'  I've been a complete idiot in this 
> thread.  Please excuse me.  :-(
>
> I think the solution to your problem is adjusting command timeout to 
> more reasonable values which should make the problem more bearable. 
> It'll take some time to figure out how to make timeouts more 
> intelligent without breaking support for slow devices.  I'll work on 
> that. 

Tejun, would it be possible and sensible to either let the user tune 
this per-drive, or to have the kernel note how long {something} takes 
and auto-tune to that? As you said, the issue is not breaking slow devices.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: 2.6.18-rc2, problem to wake up spinned down drive?

[Tejun Heo]

Bill Davidsen wrote:
>> I think the solution to your problem is adjusting command timeout to 
>> more reasonable values which should make the problem more bearable. 
>> It'll take some time to figure out how to make timeouts more 
>> intelligent without breaking support for slow devices.  I'll work on 
>> that. 
> 
> Tejun, would it be possible and sensible to either let the user tune 
> this per-drive, or to have the kernel note how long {something} takes 
> and auto-tune to that? As you said, the issue is not breaking slow devices.

I think the driver can be made to have sufficient static intelligence to 
not require user or auto tuning.  !BUSY wait in pre/postreset which are 
often cause of unnecessary 30s delay during recovery can be avoided by...

1. for !hotplug, waiting for BSY before reset doesn't make sense in the 
first place (why would we be resetting the device if it can clear BSY?)

2. for hotplug, we can make things much more intelligent.  e.g. try 
prereset waiting and softreset from 0-5s, then hardresets 5-10s, 10-15s, 
15-30s and 30s-60s, which will guarantee 1. slow device is given full 
idle 30s to get ready eventually  2. recovery reset is complete in 60s, 
while giving fast devices several chances to be fast.

And, for IO command timeouts, some operating system is said to use 7s 
timeout for ATA IO commands and simply adopting that value would be good 
enough.  We also can choose more agressive timeouts for some EH commands 
(IDNTIFY, SET_FEATURES...).

With all above combined, EH recovery should be pretty snappy and 
recovery time well-bound.

-- 
tejun