SCSI dynamic power management

SCSI dynamic power management

[Alan Stern]

Oliver (or anybody else):

Adding dynamic (AKA runtime) power management to the SCSI core is
looking a little difficult.  (Actually, since as far as I know the SCSI
specification takes no heed of power management, perhaps this should be
called "idle-device management".)

Imagine a SCSI disk has been idle long enough that we want to spin it
down.  When I/O requests do start to arrive later on, we want to delay
them in the request queue until the disk has a chance to spin back up.  
Or even to fail them, if the user has done a "manual" suspend.

These are conflicting requirements.  How can we send the START-STOP 
UNIT commands to spin the disk up/down through the request queue while 
delaying or failing all others?  What happens if one of the 
START-STOP UNIT commands encounters a problem and the error handler 
needs to send some commands of its own?

Any suggestions or ideas are welcome.

Alan Stern

Re: SCSI dynamic power management

[Matthew Wilcox]

On Mon, Nov 19, 2007 at 10:36:19AM -0500, Alan Stern wrote:
> These are conflicting requirements.  How can we send the START-STOP 
> UNIT commands to spin the disk up/down through the request queue while 
> delaying or failing all others?

You can insert commands at the head of a request queue.

> What happens if one of the 
> START-STOP UNIT commands encounters a problem and the error handler 
> needs to send some commands of its own?

It'll do a reset and recover through the normal paths?

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: SCSI dynamic power management

[Alan Stern]

On Mon, 19 Nov 2007, Matthew Wilcox wrote:

> On Mon, Nov 19, 2007 at 10:36:19AM -0500, Alan Stern wrote:
> > These are conflicting requirements.  How can we send the START-STOP 
> > UNIT commands to spin the disk up/down through the request queue while 
> > delaying or failing all others?
> 
> You can insert commands at the head of a request queue.

Sure.  But that won't do any good if the requests get held on the queue
(or failed immediately) because the disk is supposedly "suspended".
Somehow those requests have to be allowed to proceed while all others 
are forced to wait (or to fail).

> > What happens if one of the 
> > START-STOP UNIT commands encounters a problem and the error handler 
> > needs to send some commands of its own?
> 
> It'll do a reset and recover through the normal paths?

Not if those normal paths can't send commands through to the device.

Alan Stern

RE: SCSI dynamic power management

[Salyzyn, Mark]

Alan Stern sez:
> Sure.  But that won't do any good if the requests get held on 
> the queue
> (or failed immediately) because the disk is supposedly "suspended".
> Somehow those requests have to be allowed to proceed while all others 
> are forced to wait (or to fail).

Not a failure. Not ready is reported back in a check condition on a
media based request, a spin-up request is issued, then a subsequent loop
sits there probing every second with a Test Unit ready to wait for the
drive to spin back up. There is ample time provided in this path for the
drive to spin-up.

Sincerely -- Mark Salyzyn

RE: SCSI dynamic power management

[Alan Stern]

On Mon, 19 Nov 2007, Salyzyn, Mark wrote:

> Alan Stern sez:
> > Sure.  But that won't do any good if the requests get held on 
> > the queue
> > (or failed immediately) because the disk is supposedly "suspended".
> > Somehow those requests have to be allowed to proceed while all others 
> > are forced to wait (or to fail).
> 
> Not a failure. Not ready is reported back in a check condition on a
> media based request, a spin-up request is issued, then a subsequent loop
> sits there probing every second with a Test Unit ready to wait for the
> drive to spin back up. There is ample time provided in this path for the
> drive to spin-up.

You're talking about what the stack does now, but I'm talking about 
what it will do once the dynamic power management code is in place.  
Commands won't even get sent to the low-level driver, never mind coming 
back with a check condition status.

Here's how it will work: scsi_prep_state_check() will see that the
device is in a suspended state (probably a substate of SDEV_QUIESCE).  
The return value will depend on the type of suspend:

	For manual suspend or system suspend, the routine simply 
	returns BLKPREP_KILL.

	For autosuspend, the routine will submit a workqueue request
	to autoresume the device and will return BLKPREP_DEFER.

The trick is somehow to allow the START-STOP UNIT commands get past 
this filter.  Is this what the REQ_PREEMPT flag is intended for?

Alan Stern

RE: SCSI dynamic power management

[James Bottomley]

On Mon, 2007-11-19 at 11:46 -0500, Alan Stern wrote:
> On Mon, 19 Nov 2007, Salyzyn, Mark wrote:
> 
> > Alan Stern sez:
> > > Sure.  But that won't do any good if the requests get held on 
> > > the queue
> > > (or failed immediately) because the disk is supposedly "suspended".
> > > Somehow those requests have to be allowed to proceed while all others 
> > > are forced to wait (or to fail).
> > 
> > Not a failure. Not ready is reported back in a check condition on a
> > media based request, a spin-up request is issued, then a subsequent loop
> > sits there probing every second with a Test Unit ready to wait for the
> > drive to spin back up. There is ample time provided in this path for the
> > drive to spin-up.
> 
> You're talking about what the stack does now, but I'm talking about 
> what it will do once the dynamic power management code is in place.  
> Commands won't even get sent to the low-level driver, never mind coming 
> back with a check condition status.
> 
> Here's how it will work: scsi_prep_state_check() will see that the
> device is in a suspended state (probably a substate of SDEV_QUIESCE).  
> The return value will depend on the type of suspend:
> 
> 	For manual suspend or system suspend, the routine simply 
> 	returns BLKPREP_KILL.
> 
> 	For autosuspend, the routine will submit a workqueue request
> 	to autoresume the device and will return BLKPREP_DEFER.
> 
> The trick is somehow to allow the START-STOP UNIT commands get past 
> this filter.  Is this what the REQ_PREEMPT flag is intended for?

Um, that model is pretty host centric ... we wouldn't be able to use
that for power management of shared devices like SAS/SATA devices in an
expander or FC devices.  The basic problem with power management of
external devices (whether shared or not) is that it's not just what the
host did to them, it's also what the environment or administrator did to
them.

James

RE: SCSI dynamic power management

[Alan Stern]

On Mon, 19 Nov 2007, James Bottomley wrote:

> > Here's how it will work: scsi_prep_state_check() will see that the
> > device is in a suspended state (probably a substate of SDEV_QUIESCE).  
> > The return value will depend on the type of suspend:
> > 
> > 	For manual suspend or system suspend, the routine simply 
> > 	returns BLKPREP_KILL.
> > 
> > 	For autosuspend, the routine will submit a workqueue request
> > 	to autoresume the device and will return BLKPREP_DEFER.
> > 
> > The trick is somehow to allow the START-STOP UNIT commands get past 
> > this filter.  Is this what the REQ_PREEMPT flag is intended for?
> 
> Um, that model is pretty host centric ... we wouldn't be able to use
> that for power management of shared devices like SAS/SATA devices in an
> expander or FC devices.  The basic problem with power management of
> external devices (whether shared or not) is that it's not just what the
> host did to them, it's also what the environment or administrator did to
> them.

Well, think of this as "idle-device" management rather than power 
management.

It's no more host-centric than the existing code in sd_suspend().  
When I mentioned sending START-STOP UNIT commands above, that was a bit
sloppy.  What the code would actually do is call the high-level
driver's suspend method, which would then send the appropriate commands
to the device.  (Or maybe the code would simply call scsi_bus_suspend.)  
If sending START-STOP UNIT during a suspend is host-centric then
sd_suspend() is already guilty.


Regarding this thread's original question, the best idea I've come up
with so far is to store an extra flag in the scsi_device structure
indicating that a suspend or resume transition is underway.  When the
flag is set, commands with REQ_PREEMPT would be accepted.  If the
device is "suspended" and the flag is clear, then commands would be
delayed or killed in the prep function regardless of REQ_PREEMPT.  This 
scheme could potentially let unwanted commands go through, but I 
haven't been able to think of anything more suitable.

Alan Stern

Re: SCSI dynamic power management

[Matthew Wilcox]

On Mon, Nov 19, 2007 at 02:16:12PM -0500, Alan Stern wrote:
> Regarding this thread's original question, the best idea I've come up
> with so far is to store an extra flag in the scsi_device structure
> indicating that a suspend or resume transition is underway.  When the
> flag is set, commands with REQ_PREEMPT would be accepted.  If the
> device is "suspended" and the flag is clear, then commands would be
> delayed or killed in the prep function regardless of REQ_PREEMPT.  This 
> scheme could potentially let unwanted commands go through, but I 
> haven't been able to think of anything more suitable.

You could allow only START_STOP and TUR to go through?

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: SCSI dynamic power management

[Alan Stern]

On Mon, 19 Nov 2007, Matthew Wilcox wrote:

> On Mon, Nov 19, 2007 at 02:16:12PM -0500, Alan Stern wrote:
> > Regarding this thread's original question, the best idea I've come up
> > with so far is to store an extra flag in the scsi_device structure
> > indicating that a suspend or resume transition is underway.  When the
> > flag is set, commands with REQ_PREEMPT would be accepted.  If the
> > device is "suspended" and the flag is clear, then commands would be
> > delayed or killed in the prep function regardless of REQ_PREEMPT.  This 
> > scheme could potentially let unwanted commands go through, but I 
> > haven't been able to think of anything more suitable.
> 
> You could allow only START_STOP and TUR to go through?

That's a pretty good idea...

There will be other problems to sort out.  For example, if someone is 
using a SCSI cdrom drive to play an audio CD, it will appear to the 
system that the drive is idle because no commands will flow.  But that 
sort of thing can be handled later.

Alan Stern

RE: SCSI dynamic power management

[James Bottomley]

On Mon, 2007-11-19 at 14:16 -0500, Alan Stern wrote:
> On Mon, 19 Nov 2007, James Bottomley wrote:
> 
> > > Here's how it will work: scsi_prep_state_check() will see that the
> > > device is in a suspended state (probably a substate of SDEV_QUIESCE).  
> > > The return value will depend on the type of suspend:
> > > 
> > > 	For manual suspend or system suspend, the routine simply 
> > > 	returns BLKPREP_KILL.
> > > 
> > > 	For autosuspend, the routine will submit a workqueue request
> > > 	to autoresume the device and will return BLKPREP_DEFER.
> > > 
> > > The trick is somehow to allow the START-STOP UNIT commands get past 
> > > this filter.  Is this what the REQ_PREEMPT flag is intended for?
> > 
> > Um, that model is pretty host centric ... we wouldn't be able to use
> > that for power management of shared devices like SAS/SATA devices in an
> > expander or FC devices.  The basic problem with power management of
> > external devices (whether shared or not) is that it's not just what the
> > host did to them, it's also what the environment or administrator did to
> > them.
> 
> Well, think of this as "idle-device" management rather than power 
> management.

Sure ... but surely that's a subset of proper power management?

> It's no more host-centric than the existing code in sd_suspend().  
> When I mentioned sending START-STOP UNIT commands above, that was a bit
> sloppy.  What the code would actually do is call the high-level
> driver's suspend method, which would then send the appropriate commands
> to the device.  (Or maybe the code would simply call scsi_bus_suspend.)  
> If sending START-STOP UNIT during a suspend is host-centric then
> sd_suspend() is already guilty.

Ah, but if you look, sd_suspend by default doesn't do anything ... it
has to be enabled on a per device basis by the manage_start_stop
attribute ... that's precisely because we can't have the sd ULD idling a
SAN device when others are trying to use it.

> Regarding this thread's original question, the best idea I've come up
> with so far is to store an extra flag in the scsi_device structure
> indicating that a suspend or resume transition is underway.  When the
> flag is set, commands with REQ_PREEMPT would be accepted.  If the
> device is "suspended" and the flag is clear, then commands would be
> delayed or killed in the prep function regardless of REQ_PREEMPT.  This 
> scheme could potentially let unwanted commands go through, but I 
> haven't been able to think of anything more suitable.

I don't understand why you want to do this.  Power management is a
layered issue on SCSI, divided (as always) into host, device and
transport.  The idle you're talking about is a pure device thing, so it
can be managed by the ULD (and currently is).  When a unit is stopped,
it automatically returns correct NOT_READY sense to most commands.  The
sd_done post processing could decide to wake the device and retry or
return an error without ever troubling the mid layer or even block for
anything.    All the hooks are already in sd, why does this now need to
go into block (unless there's some longer scheme for incorporating the
transport and other elements into this?)

James

Re: SCSI dynamic power management

[Oliver Neukum]

Am Dienstag 20 November 2007 schrieb James Bottomley:
> I don't understand why you want to do this.  Power management is a
> layered issue on SCSI, divided (as always) into host, device and
> transport.  The idle you're talking about is a pure device thing, so it
> can be managed by the ULD (and currently is).  When a unit is stopped,

The lower layers don't know how to correctly suspend a device. sd_suspend()
may know how to do it. It would also mean the LLD having to detect idleness.

	Regards
		Oliver

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SCSI dynamic power management

[James Bottomley]

On Tue, 2007-11-20 at 16:07 +0100, Oliver Neukum wrote:
> Am Dienstag 20 November 2007 schrieb James Bottomley:
> > I don't understand why you want to do this.  Power management is a
> > layered issue on SCSI, divided (as always) into host, device and
> > transport.  The idle you're talking about is a pure device thing, so it
> > can be managed by the ULD (and currently is).  When a unit is stopped,
> 
> The lower layers don't know how to correctly suspend a device. sd_suspend()
> may know how to do it. It would also mean the LLD having to detect idleness.

You really mean you want to involve the transport as well, right?  So
ipso facto this is more than simple idleness management.  Thus, you
really need to look into the full solution (host, transport and ULD).

James

Re: SCSI dynamic power management

[Oliver Neukum]

Am Dienstag 20 November 2007 schrieb James Bottomley:
> 
> On Tue, 2007-11-20 at 16:07 +0100, Oliver Neukum wrote:
> > Am Dienstag 20 November 2007 schrieb James Bottomley:
> > > I don't understand why you want to do this.  Power management is a
> > > layered issue on SCSI, divided (as always) into host, device and
> > > transport.  The idle you're talking about is a pure device thing, so it
> > > can be managed by the ULD (and currently is).  When a unit is stopped,
> > 
> > The lower layers don't know how to correctly suspend a device. sd_suspend()
> > may know how to do it. It would also mean the LLD having to detect idleness.
> 
> You really mean you want to involve the transport as well, right?  So

Yes, we cannot avoid that. Some device drop their caches unless they are
flushed.

> ipso facto this is more than simple idleness management.  Thus, you
> really need to look into the full solution (host, transport and ULD).

Only detecting idleness without doing anything with that knowledge would
be pointless :-)
What is the right kind of sequence?

	Regards
		Oliver

Re: SCSI dynamic power management

[James Bottomley]

On Tue, 2007-11-20 at 16:36 +0100, Oliver Neukum wrote:
> Am Dienstag 20 November 2007 schrieb James Bottomley:
> > 
> > On Tue, 2007-11-20 at 16:07 +0100, Oliver Neukum wrote:
> > > Am Dienstag 20 November 2007 schrieb James Bottomley:
> > > > I don't understand why you want to do this.  Power management is a
> > > > layered issue on SCSI, divided (as always) into host, device and
> > > > transport.  The idle you're talking about is a pure device thing, so it
> > > > can be managed by the ULD (and currently is).  When a unit is stopped,
> > > 
> > > The lower layers don't know how to correctly suspend a device. sd_suspend()
> > > may know how to do it. It would also mean the LLD having to detect idleness.
> > 
> > You really mean you want to involve the transport as well, right?  So
> 
> Yes, we cannot avoid that. Some device drop their caches unless they are
> flushed.

So that would be fixed by having the ULD send a flush before START STOP
UNIT?

> > ipso facto this is more than simple idleness management.  Thus, you
> > really need to look into the full solution (host, transport and ULD).
> 
> Only detecting idleness without doing anything with that knowledge would
> be pointless :-)
> What is the right kind of sequence?

I don't really know ... I don't have a clear idea of what you're trying
to do.  I know Alan said it was something simple, but from what you're
saying it sounds like you need a full blown power management
infrastructure in all three places.

James

Re: SCSI dynamic power management

[Alan Stern]

On Tue, 20 Nov 2007, James Bottomley wrote:

> I don't really know ... I don't have a clear idea of what you're trying
> to do.  I know Alan said it was something simple, but from what you're
> saying it sounds like you need a full blown power management
> infrastructure in all three places.

Basically that's right.

The higher-level drivers already have some PM infrastructure; all we
would have to add would be idle-timeout detection.  The midlayer has to
be involved in order to coordinate between the high-level and the
low-level drivers.  The key addition is to get the low-level drivers 
involved.

The main point I'm aiming for is to have the midlayer to inform the LLD 
when all the devices on a particular bus are "idle".  At that point the 
LLD is free to suspend or power down the transport.  Conversely, if the 
transport is suspended and an I/O request arrives, the LLD has to be 
told to reactivate the transport.  Without some such form of 
communication, there's no way to know when the tranport can safely be 
suspended.

Does that clarify things?

Alan Stern

Re: SCSI dynamic power management

[James Bottomley]

On Tue, 2007-11-20 at 11:02 -0500, Alan Stern wrote:
> On Tue, 20 Nov 2007, James Bottomley wrote:
> 
> > I don't really know ... I don't have a clear idea of what you're trying
> > to do.  I know Alan said it was something simple, but from what you're
> > saying it sounds like you need a full blown power management
> > infrastructure in all three places.
> 
> Basically that's right.
> 
> The higher-level drivers already have some PM infrastructure; all we
> would have to add would be idle-timeout detection.  The midlayer has to
> be involved in order to coordinate between the high-level and the
> low-level drivers.  The key addition is to get the low-level drivers 
> involved.
> 
> The main point I'm aiming for is to have the midlayer to inform the LLD 
> when all the devices on a particular bus are "idle".

This is the wrong assumption ... the mid layer would provide an API to
inform the transport.  Transport and host are separate things.  an 8
port SAS card could suspend a single phy and device while still
processing the others.  So what it sounds like you need is a transport
API saying a given device has been suspended or is requested to be
resumed?  What than means for the host would be transport class policy,
I think.

>   At that point the 
> LLD is free to suspend or power down the transport.  Conversely, if the 
> transport is suspended and an I/O request arrives, the LLD has to be 
> told to reactivate the transport.  Without some such form of 
> communication, there's no way to know when the tranport can safely be 
> suspended.
> 
> Does that clarify things?

A little ... I think your wake on command idea above is policy rather
than actual implementation ... so it would probably have to be
selectable (either error or wake).

James

Re: SCSI dynamic power management

[Alan Stern]

On Tue, 20 Nov 2007, James Bottomley wrote:

> > The main point I'm aiming for is to have the midlayer to inform the LLD 
> > when all the devices on a particular bus are "idle".
> 
> This is the wrong assumption ... the mid layer would provide an API to
> inform the transport.  Transport and host are separate things.  an 8
> port SAS card could suspend a single phy and device while still
> processing the others.  So what it sounds like you need is a transport
> API saying a given device has been suspended or is requested to be
> resumed?  What than means for the host would be transport class policy,
> I think.

Okay.  I know little about the intricacies of these newer, more exotic
transports; I'm pretty firmly rooted in the old SPI technology.

Anyway, yes, I want an API to inform the transport and/or host adapter 
driver that either a given device or all devices on a bus has been 
suspended or requests to be resumed.  For an 8-port SAS card the driver 
might want to know about each individual device; for a plain old SPI 
card the driver would care only about the entire bus.

> A little ... I think your wake on command idea above is policy rather
> than actual implementation ... so it would probably have to be
> selectable (either error or wake).

Yes.  That is already part of the USB dynamic PM implementation, and I
would copy it for the SCSI implementation.  There is a sysfs file
(/sys/devices/.../power/level) to which the user can write three
possible values:

	"on" means the device is permanently at full power;

	"auto" means the device is subject to autosuspend after
	an idle-timeout (the timeout value is stored in a separate
	sysfs file) with autoresume at the next I/O request;

	"suspend" means the device is permanently suspended
	and I/O requests will fail immediately.

My intention was to add a couple of method pointers to the SCSI
host_template structure, one for suspend notifications and one for
resume notifications.  Do you think this is the right way to do it, or
is something more complex called for?

Alan Stern

Re: SCSI dynamic power management

[Jens Axboe]

On Mon, Nov 19 2007, Alan Stern wrote:
> On Mon, 19 Nov 2007, Matthew Wilcox wrote:
> 
> > On Mon, Nov 19, 2007 at 10:36:19AM -0500, Alan Stern wrote:
> > > These are conflicting requirements.  How can we send the START-STOP 
> > > UNIT commands to spin the disk up/down through the request queue while 
> > > delaying or failing all others?
> > 
> > You can insert commands at the head of a request queue.
> 
> Sure.  But that won't do any good if the requests get held on the queue
> (or failed immediately) because the disk is supposedly "suspended".
> Somehow those requests have to be allowed to proceed while all others 
> are forced to wait (or to fail).

Yeah, head-of-queue or not has no relevance. But the SCSI layer has had
the notion of 'allow some commands, disallow others' for quite some time
already - grep for REQ_PREEMPT in scsi/

-- 
Jens Axboe

Re: SCSI dynamic power management

[Douglas Gilbert]

Alan Stern wrote:
> Oliver (or anybody else):
> 
> Adding dynamic (AKA runtime) power management to the SCSI core is
> looking a little difficult.  (Actually, since as far as I know the SCSI
> specification takes no heed of power management, perhaps this should be
> called "idle-device management".)
> 
> Imagine a SCSI disk has been idle long enough that we want to spin it
> down.  When I/O requests do start to arrive later on, we want to delay
> them in the request queue until the disk has a chance to spin back up.  
> Or even to fail them, if the user has done a "manual" suspend.
> 
> These are conflicting requirements.  How can we send the START-STOP 
> UNIT commands to spin the disk up/down through the request queue while 
> delaying or failing all others?  What happens if one of the 
> START-STOP UNIT commands encounters a problem and the error handler 
> needs to send some commands of its own?
> 
> Any suggestions or ideas are welcome.

There should have always been at least two types of "block":
   a) block media access commands
   b) block all commands because the transport has told us
      (or we know that we are about to remove some component)
      that commands cannot be delivered to the device (logical
      unit)

For case a) all SCSI commands apart from those that explicitly
access the media (e.g. READ, WRITE, VERIFY) should be sent to
the device. Some commands, such as MODE SENSE, may fail since
they might depend on data held on the media. Such failures
will be fast with appropriate sense_key/asc/ascq codes returned.


IMO if a transport tells us that a SCSI device (i.e. target
device and/or logical unit) is accessible and the SCSI subsystem
stops us sending an INQUIRY to that device then the SCSI
subsystem is broken.

Doug Gilbert

Re: SCSI dynamic power management

[Alan Stern]

On Tue, 20 Nov 2007, Douglas Gilbert wrote:

> There should have always been at least two types of "block":
>    a) block media access commands
>    b) block all commands because the transport has told us
>       (or we know that we are about to remove some component)
>       that commands cannot be delivered to the device (logical
>       unit)

That sounds reasonable.  I'm talking about some other kinds of "block":

     c) Temporarily block all commands other than START-STOP UNIT or 
	TEST UNIT READY because the device has been idle for N seconds
	and we want to prepare it prior to powering-down the
	transport.

     d) Temporarily block all commands because we have told the
	transport that it can power down; after we ask the transport
	to resume normal operation the commands will be delivered.

     e) Fail all commands immediately because the user has told us
	that no I/O to the device should be permitted and the device 
	must remain "idle".

> For case a) all SCSI commands apart from those that explicitly
> access the media (e.g. READ, WRITE, VERIFY) should be sent to
> the device. Some commands, such as MODE SENSE, may fail since
> they might depend on data held on the media. Such failures
> will be fast with appropriate sense_key/asc/ascq codes returned.
> 
> 
> IMO if a transport tells us that a SCSI device (i.e. target
> device and/or logical unit) is accessible and the SCSI subsystem
> stops us sending an INQUIRY to that device then the SCSI
> subsystem is broken.

What if the SCSI subsystem prevents you from sending an INQUIRY (or any 
other command) because the user/administrator has told it to do so?

Alan Stern

Re: SCSI dynamic power management

[Alan Stern]

On Mon, 19 Nov 2007, James Smart wrote:

> Also, don't forget that you need to sequence spin-up. A jbod tower with
> multiple drives all spinning up concurrently can overload the power
> supply. Most towers avoid this at power-on by having drive jumpers
> sequence the drives. This is a hard nut to crack as it requires some
> notion of physical packaging. When I worked on this in the past, the
> only reasonable approach was to sequence based on target id.

What happens during an ordinary system resume?  Don't the drives all 
get spun-up concurrently then?  Or does the sd_start_stop_device() 
routine wait until the disk is fully spun-up before returning?

Alan Stern