suspending I/Os to a device

suspending I/Os to a device

[Infante, Jon]

Is there a standard, or generally accepted way, for a driver to request the
scsi layer temporarily suspend all I/Os to a particular Target or
Target/Lun. Conversely, I would want to resume I/Os at a driver determined
time without the scsi layer timing them out. 

--
Jon Infante
Emulex

RE: suspending I/Os to a device

[Ely, Paul]

Goodafternoon Jon,

What about the use of scsi_block_requests and scsi_unblock_requests?

There doesn't appear to be any timer event used by the OS and the
only way to restart commands is for the driver to call
scsi_unblock_requests.

We have two routines for this:  lpfc_block_requests and
lpfc_unblock_requests.

Paul


-----Original Message-----
From: Infante, Jon 
Sent: Friday, April 23, 2004 12:00 PM
To: Linux SCSI Reflector
Subject: suspending I/Os to a device



Is there a standard, or generally accepted way, for a driver to request the
scsi layer temporarily suspend all I/Os to a particular Target or
Target/Lun. Conversely, I would want to resume I/Os at a driver determined
time without the scsi layer timing them out. 

--
Jon Infante
Emulex
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: suspending I/Os to a device

[Infante, Jon]

Here is the senerio I am trying to handle within our Fibre Channel driver.
In Fibre Channel, it is possible for the Target, or HBA, to be disconnected
from one port on a switch and plugged into another port on the same (or even
a different) switch. This transitition can happen with live traffic and the
scsi subsystem / driver is supposed to recover gracefully, without failing
all I/Os back up to applications that are running. In order to accomplish
this, there is a configurable timer, that tells the driver how long to wait
(at the Fibre Channel level), when a device disappears, to give it a chance
to come back. Some OEMs / environments set this to a couple seconds, some a
to couple of minutes.

The interface scsi_block_request() cannot block based on specific targets.
scsi_block_when_processing errors() does not look like a routine a HBA
driver should be calling as it is primarily used by sd / sg /st. 

Since the HBA driver is not supposed to internally queue requests / change
the scsi timeout imposed on the I/O, its needs a mechanism to tell the scsi
layer to 
1) stop / start sending I/Os to a specific Target
2) do not start any scsi timeouts on I/Os being held for the target while
I/O is suspended.

Are there any other drivers out there that require, or have implemented,
this functionality and/or is there a preferred way to accomplish this?

--
Jon Infante
Emulex


 


-----Original Message-----
From: Infante, Jon 
Sent: Friday, April 23, 2004 12:00 PM
To: Linux SCSI Reflector
Subject: suspending I/Os to a device



Is there a standard, or generally accepted way, for a driver to request the
scsi layer temporarily suspend all I/Os to a particular Target or
Target/Lun. Conversely, I would want to resume I/Os at a driver determined
time without the scsi layer timing them out. 

--
Jon Infante
Emulex
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: suspending I/Os to a device

[James Bottomley]

On Fri, 2004-04-23 at 18:12, Infante, Jon wrote:
> Here is the senerio I am trying to handle within our Fibre Channel driver.
> In Fibre Channel, it is possible for the Target, or HBA, to be disconnected
> from one port on a switch and plugged into another port on the same (or even
> a different) switch. This transitition can happen with live traffic and the
> scsi subsystem / driver is supposed to recover gracefully, without failing
> all I/Os back up to applications that are running. In order to accomplish
> this, there is a configurable timer, that tells the driver how long to wait
> (at the Fibre Channel level), when a device disappears, to give it a chance
> to come back. Some OEMs / environments set this to a couple seconds, some a
> to couple of minutes.

Actually, there is no interface because no-one has had any prior
requirements for this.

This is effectively a FC specific thing, and I believe you know because
you get a lip event telling you the target has gone, so it sounds like
the ideal thing to handle in the FC transport class (since it would
apply to all FC HBAs) The target reconnect timeout would then be a
transport class property and we'd start failing I/Os when it expired.  I
would guess there needs to be an additional state in the device state
model like SUSPEND (QUIESCE doesn't quite fit because it allows the
pending queue to drain and the error handler to operate), but we can
accomodate that.

By default, I would think the reconnect timeout should be zero because
most FC operates in a multi-path environment and you want path failure
notifications as fast as possible, so users requiring this feature would
need to set the timeout.

James

Re: suspending I/Os to a device

[Patrick Mansfield]

On Fri, Apr 23, 2004 at 07:28:30PM -0400, James Bottomley wrote:

> Actually, there is no interface because no-one has had any prior
> requirements for this.
> 
> This is effectively a FC specific thing, and I believe you know because
> you get a lip event telling you the target has gone, so it sounds like
> the ideal thing to handle in the FC transport class (since it would
> apply to all FC HBAs) The target reconnect timeout would then be a
> transport class property and we'd start failing I/Os when it expired.  I
> would guess there needs to be an additional state in the device state
> model like SUSPEND (QUIESCE doesn't quite fit because it allows the
> pending queue to drain and the error handler to operate), but we can
> accomodate that.

Attaching your USB or firewire device to another port is not much
different.

We should someday allow removal and re-attachment of any disk, or the
replacement of the transport (like hotplug pci card, or the cables),
without having to close or unmount the disk.

-- Patrick Mansfield

RE: suspending I/Os to a device

[Infante, Jon]

So it may be better to make this a hotplug feature since it is not
necessarily FC specific? When the FC driver gets a LIP or RSCN event, there
is potential for a device to be removed, or a new device added. In addition
to add / remove if there was a suspended state the device could put in,
assuming it would transition to added or removed shortly afterwards, that
would also meet the requirements for FC.

--
Jon Infante
Emulex


-----Original Message-----
From: Patrick Mansfield [mailto:patmans@us.ibm.com]
Sent: Monday, April 26, 2004 9:40 AM
To: James Bottomley
Cc: Infante, Jon; Linux SCSI Reflector
Subject: Re: suspending I/Os to a device


On Fri, Apr 23, 2004 at 07:28:30PM -0400, James Bottomley wrote:

> Actually, there is no interface because no-one has had any prior
> requirements for this.
> 
> This is effectively a FC specific thing, and I believe you know because
> you get a lip event telling you the target has gone, so it sounds like
> the ideal thing to handle in the FC transport class (since it would
> apply to all FC HBAs) The target reconnect timeout would then be a
> transport class property and we'd start failing I/Os when it expired.  I
> would guess there needs to be an additional state in the device state
> model like SUSPEND (QUIESCE doesn't quite fit because it allows the
> pending queue to drain and the error handler to operate), but we can
> accomodate that.

Attaching your USB or firewire device to another port is not much
different.

We should someday allow removal and re-attachment of any disk, or the
replacement of the transport (like hotplug pci card, or the cables),
without having to close or unmount the disk.

-- Patrick Mansfield

Re: suspending I/Os to a device

['Patrick Mansfield']

On Mon, Apr 26, 2004 at 10:12:18AM -0700, Infante, Jon wrote:
> 
> So it may be better to make this a hotplug feature since it is not
> necessarily FC specific? When the FC driver gets a LIP or RSCN event, there
> is potential for a device to be removed, or a new device added. In addition
> to add / remove if there was a suspended state the device could put in,
> assuming it would transition to added or removed shortly afterwards, that
> would also meet the requirements for FC.

I didn't mean to imply use of hotplug, just that the feature might be
useful for transports other than FCP.

If it were hotplug based that would be really nice, but currently linux
scsi can't handle moving a device to a different port. Generally, we can't
change the host/chan/id/lun for a given scsi_device, instead you must
remove and add the device (either physically if the hardware notices
the addition and removal, or manually via scsi sysfs interfaces).

Current hotplug could handle the addition of a new target (or LUN), we've
discussed this before. If the adapter generates a hotplug event for a new
target, user code could scan the new target.

-- Patrick Mansfield

RE: suspending I/Os to a device

[Smart, James]

James,

So I'm not clear on how this gets implemented...

I'm assuming that this would mirror the scsi_block/unblock_requests()
functions - but now on a scsi-device basis. The SUSPEND state would be an
internal state in the midlayer.  The driver would use the new attribute to
know when to call the unblock function.   And, wouldn't it make sense to
have a midlayer function that could be pointed at a scsi_host/scsi target id
value and have it recurse the block/unblock calls to all the related scsi
devices ?

How does this move forward ?  If we want to have this functionality, must we
put together a patch ? Who modifies the mid layer ?  What I'd like to avoid
is adding error queues that implement the suspend function in our driver.

-- James S



> -----Original Message-----
> From: James Bottomley [mailto:James.Bottomley@steeleye.com]
> Sent: Friday, April 23, 2004 7:29 PM
> To: Infante, Jon
> Cc: Linux SCSI Reflector
> Subject: RE: suspending I/Os to a device
> 
> 
> On Fri, 2004-04-23 at 18:12, Infante, Jon wrote:
> > Here is the senerio I am trying to handle within our Fibre 
> Channel driver.
> > In Fibre Channel, it is possible for the Target, or HBA, to 
> be disconnected
> > from one port on a switch and plugged into another port on 
> the same (or even
> > a different) switch. This transitition can happen with live 
> traffic and the
> > scsi subsystem / driver is supposed to recover gracefully, 
> without failing
> > all I/Os back up to applications that are running. In order 
> to accomplish
> > this, there is a configurable timer, that tells the driver 
> how long to wait
> > (at the Fibre Channel level), when a device disappears, to 
> give it a chance
> > to come back. Some OEMs / environments set this to a couple 
> seconds, some a
> > to couple of minutes.
> 
> Actually, there is no interface because no-one has had any prior
> requirements for this.
> 
> This is effectively a FC specific thing, and I believe you 
> know because
> you get a lip event telling you the target has gone, so it sounds like
> the ideal thing to handle in the FC transport class (since it would
> apply to all FC HBAs) The target reconnect timeout would then be a
> transport class property and we'd start failing I/Os when it 
> expired.  I
> would guess there needs to be an additional state in the device state
> model like SUSPEND (QUIESCE doesn't quite fit because it allows the
> pending queue to drain and the error handler to operate), but we can
> accomodate that.
> 
> By default, I would think the reconnect timeout should be zero because
> most FC operates in a multi-path environment and you want path failure
> notifications as fast as possible, so users requiring this 
> feature would
> need to set the timeout.
> 
> James
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: suspending I/Os to a device

[Ely, Paul]

All,

As James Smart indicated at the bottom of this thread
entry:

http://marc.theaimsgroup.com/?l=linux-scsi&m=108940122907934&w=2

Here is the continuing conversation regarding storage devices that
disappear and then return.  Please reference this link for
Jon Infante's and Patrick Mansfield's discussion of this feature.

http://marc.theaimsgroup.com/?l=linux-scsi&m=108378376227530&w=2


I have been looking through the midlayer and block level code
to come up with enough detail to start a discussion on a design that
could address this requirement and has a nice fit in the scsi 
midlayer.  

I used the 2.6.7 kernel sources for the initial research.

Some scsi LLDs need a way to suspend all scsi commands sent by
the midlayer, including retries, during the time that a connected
device has disappeared.  The goal of this requirement is to 
prevent premature IO failures back to the application before 
the scsi LLD has a chance to recover the device.  The connected
device may not be recoverable making it necessary for the 
midlayer to detect a bad device.  Should the device be 
recovered by the scsi LLD, then traffic would resume normally
after the scsi LLD informs the midlayer of such an event.

The proposal:
	o A new device state called SDEV_SUSPEND
		- The device's current state must be SDEV_CREATED or
SDEV_RUNNING.
		- Once in SDEV_SUSPEND, the device state is allowed to
transition
		  to SDEV_RUNNING or SDEV_CANCEL.

	o A new routine called scsi_device_suspend to enter the SDEV_SUSPEND
	  state.
		- Called by scsi LLDs only.
		- scsi LLD must not hold any locks, including the host lock,
		  when calling this routine.
		- Routine can be called from interrupt context or normal
kernel
		  context.
		- This routine calls into the scsi_device_set_state
		  function to handle the state transition.  Other
functionality
		  is TBD.

	o A routine that the scsi LLD calls to successfully transition from
	  SDEV_SUSPEND to SDEV_RUNNING state and restart scsi commands.  I
am
	  proposing some alternatives.
		- Reuse scsi_device_resume.
			+ Overloads quiesce and suspend.  This may work
since
			  the state machine will not allow transitions
to/from
			  SDEV_QUIESCE and SDEV_SUSPEND.

		OR

		- Provide a new routine called scsi_device_continue 
			+ Analogous functionality as scsi_device_resume, but
is
			  called specifically to transition out of
SDEV_SUSPEND.

	  In either case, the function would transition the device back to
	  SDEV_RUNNING and goose the device queue.

	o Provide TMO control that limits the amount of time a device can 
	  disappear.  The timeout handling would clean up pending
	  commands on the device queue.  I am proposing some
	  alternatives here.
		- Require the scsi LLD to provide tmo functionality.
Timeout 
		  value would be a driver default, but also settable via a
sysfs
		  class transport attribute.  For example, Fibre Channel
drivers
		  would use a FC transport value in sysfs.

		OR

		- Require the scsi midlayer to provide tmo functionality.
		  Timeout value would be a default, but also settable via
		  a sysfs class transport attribute by the scsi LLD or an
		  admin.

		OR

		- Don't require a tmo on the continue and require the scsi
LLD to 
		  transition out of suspend just like scsi_block_requests
and 
		  scsi_unblock_requests.

Newly discovered devices or devices that get removed without suspend
requirements
require the scsi LLD to call scsi_add_device and scsi_remove_device and so
are
not addressed by this proposal.

Clearly there is more to this issue, but this seems like a good
place to start.

----------
Paul Ely
Emulex


-----Original Message-----
From: Smart, James 
Sent: Wednesday, May 05, 2004 3:01 PM
To: 'James Bottomley'; Infante, Jon
Cc: Linux SCSI Reflector
Subject: RE: suspending I/Os to a device


James,

So I'm not clear on how this gets implemented...

I'm assuming that this would mirror the scsi_block/unblock_requests()
functions - but now on a scsi-device basis. The SUSPEND state would be an
internal state in the midlayer.  The driver would use the new attribute to
know when to call the unblock function.   And, wouldn't it make sense to
have a midlayer function that could be pointed at a scsi_host/scsi target id
value and have it recurse the block/unblock calls to all the related scsi
devices ?

How does this move forward ?  If we want to have this functionality, must we
put together a patch ? Who modifies the mid layer ?  What I'd like to avoid
is adding error queues that implement the suspend function in our driver.

-- James S



> -----Original Message-----
> From: James Bottomley [mailto:James.Bottomley@steeleye.com]
> Sent: Friday, April 23, 2004 7:29 PM
> To: Infante, Jon
> Cc: Linux SCSI Reflector
> Subject: RE: suspending I/Os to a device
> 
> 
> On Fri, 2004-04-23 at 18:12, Infante, Jon wrote:
> > Here is the senerio I am trying to handle within our Fibre 
> Channel driver.
> > In Fibre Channel, it is possible for the Target, or HBA, to 
> be disconnected
> > from one port on a switch and plugged into another port on 
> the same (or even
> > a different) switch. This transitition can happen with live 
> traffic and the
> > scsi subsystem / driver is supposed to recover gracefully, 
> without failing
> > all I/Os back up to applications that are running. In order 
> to accomplish
> > this, there is a configurable timer, that tells the driver 
> how long to wait
> > (at the Fibre Channel level), when a device disappears, to 
> give it a chance
> > to come back. Some OEMs / environments set this to a couple 
> seconds, some a
> > to couple of minutes.
> 
> Actually, there is no interface because no-one has had any prior
> requirements for this.
> 
> This is effectively a FC specific thing, and I believe you 
> know because
> you get a lip event telling you the target has gone, so it sounds like
> the ideal thing to handle in the FC transport class (since it would
> apply to all FC HBAs) The target reconnect timeout would then be a
> transport class property and we'd start failing I/Os when it 
> expired.  I
> would guess there needs to be an additional state in the device state
> model like SUSPEND (QUIESCE doesn't quite fit because it allows the
> pending queue to drain and the error handler to operate), but we can
> accomodate that.
> 
> By default, I would think the reconnect timeout should be zero because
> most FC operates in a multi-path environment and you want path failure
> notifications as fast as possible, so users requiring this 
> feature would
> need to set the timeout.
> 
> James
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: suspending I/Os to a device

[Christoph Hellwig]

On Mon, Jul 12, 2004 at 04:53:34PM -0400, Ely, Paul wrote:
> The proposal:
> 	o A new device state called SDEV_SUSPEND
> 		- The device's current state must be SDEV_CREATED or
> SDEV_RUNNING.
> 		- Once in SDEV_SUSPEND, the device state is allowed to
> transition
> 		  to SDEV_RUNNING or SDEV_CANCEL.
> 
> 	o A new routine called scsi_device_suspend to enter the SDEV_SUSPEND
> 	  state.
> 		- Called by scsi LLDs only.
> 		- scsi LLD must not hold any locks, including the host lock,
> 		  when calling this routine.
> 		- Routine can be called from interrupt context or normal
> kernel
> 		  context.
> 		- This routine calls into the scsi_device_set_state
> 		  function to handle the state transition.  Other
> functionality
> 		  is TBD.
> 
> 	o A routine that the scsi LLD calls to successfully transition from
> 	  SDEV_SUSPEND to SDEV_RUNNING state and restart scsi commands.  I
> am
> 	  proposing some alternatives.

This sounds fine to me.

> 		- Reuse scsi_device_resume.
> 			+ Overloads quiesce and suspend.  This may work
> since
> 			  the state machine will not allow transitions
> to/from
> 			  SDEV_QUIESCE and SDEV_SUSPEND.
> 
> 		OR
> 
> 		- Provide a new routine called scsi_device_continue 
> 			+ Analogous functionality as scsi_device_resume, but
> is
> 			  called specifically to transition out of
> SDEV_SUSPEND.

Please use separate functions.

> 	o Provide TMO control that limits the amount of time a device can 
> 	  disappear.  The timeout handling would clean up pending
> 	  commands on the device queue.  I am proposing some
> 	  alternatives here.
> 		- Require the scsi LLD to provide tmo functionality.
> Timeout 
> 		  value would be a driver default, but also settable via a
> sysfs
> 		  class transport attribute.  For example, Fibre Channel
> drivers
> 		  would use a FC transport value in sysfs.

Not sure if I parsed that correctly.  You want a timeout after which the
device gets unblocked again?  In that case the scsi_device should get a timer
that calls the unblock routine, sounds fine to me and can be done in the
scsi midlayer.

RE: suspending I/Os to a device

[Ely, Paul]

Christoph,
 
Thanks for the feedback.

I was not clear enough on the TMO handling.  Yes, I 
do think a TMO value is necessary to prevent the scsi
lld from suspending the device indefinitely - if the
scsi lld could do this, the suspend functionality is
just block/unblock.  The intention of suspend is a
device loss that is transient.  Beyond the TMO value,
it is treated as a device removal.
 
I agree that the scsi device should get a timer and
call scsi_device_continue to transition back to 
SDEV_RUNNING and that it all belongs in the midlayer.
 
There is a condition that I have to work through for
scsi llds.  Suppose a scsi lld calls scsi_device_suspend,
and sets some internal state.  After the TMO value has
expired, the scsi lld has not called the scsi_device_continue.
The midlayer times out the suspend and starts queueing
scsi commands again.   There are a few scenerios:
 
 	1. The device is recovered in nearly the same time.  In
 	   this case, the scsi lld calls continue and the device
 	   state machine ignores the transition since it would be
 	   SDEV_RUNNING transition from current state SDEV_RUNNING,
 	   and the device would recover.
 
 	2. The device recovers within the midlayer's retry count.
 
 	3. The device recovers post-TMO.  Too late.  The midlayer believes
	   transitioned to SDEV_CANCEL.  

	4. The device never recovers.  Similar to scenerio 3.  In both 
	   cases, the driver needs a way to recover - possibly a hint
	   via scsi_device_continue failing.
 
I plan on submitting a patch for the work discussed so far this
week.
 
 Thanks,
 
 Paul

> -----Original Message-----
> From: Christoph Hellwig [mailto:hch@infradead.org]
> Sent: Saturday, July 17, 2004 8:25 AM
> To: Ely, Paul
> Cc: Smart, James; 'James Bottomley'; Linux SCSI Reflector
> Subject: Re: suspending I/Os to a device
> 
> 
> On Mon, Jul 12, 2004 at 04:53:34PM -0400, Ely, Paul wrote:
> > The proposal:
> > 	o A new device state called SDEV_SUSPEND
> > 		- The device's current state must be SDEV_CREATED or
> > SDEV_RUNNING.
> > 		- Once in SDEV_SUSPEND, the device state is allowed to
> > transition
> > 		  to SDEV_RUNNING or SDEV_CANCEL.
> > 
> > 	o A new routine called scsi_device_suspend to enter the 
> SDEV_SUSPEND
> > 	  state.
> > 		- Called by scsi LLDs only.
> > 		- scsi LLD must not hold any locks, including 
> the host lock,
> > 		  when calling this routine.
> > 		- Routine can be called from interrupt context or normal
> > kernel
> > 		  context.
> > 		- This routine calls into the scsi_device_set_state
> > 		  function to handle the state transition.  Other
> > functionality
> > 		  is TBD.
> > 
> > 	o A routine that the scsi LLD calls to successfully 
> transition from
> > 	  SDEV_SUSPEND to SDEV_RUNNING state and restart scsi 
> commands.  I
> > am
> > 	  proposing some alternatives.
> 
> This sounds fine to me.
> 
> > 		- Reuse scsi_device_resume.
> > 			+ Overloads quiesce and suspend.  This may work
> > since
> > 			  the state machine will not allow transitions
> > to/from
> > 			  SDEV_QUIESCE and SDEV_SUSPEND.
> > 
> > 		OR
> > 
> > 		- Provide a new routine called scsi_device_continue 
> > 			+ Analogous functionality as 
> scsi_device_resume, but
> > is
> > 			  called specifically to transition out of
> > SDEV_SUSPEND.
> 
> Please use separate functions.
> 
> > 	o Provide TMO control that limits the amount of time a 
> device can 
> > 	  disappear.  The timeout handling would clean up pending
> > 	  commands on the device queue.  I am proposing some
> > 	  alternatives here.
> > 		- Require the scsi LLD to provide tmo functionality.
> > Timeout 
> > 		  value would be a driver default, but also 
> settable via a
> > sysfs
> > 		  class transport attribute.  For example, Fibre Channel
> > drivers
> > 		  would use a FC transport value in sysfs.
> 
> Not sure if I parsed that correctly.  You want a timeout 
> after which the
> device gets unblocked again?  In that case the scsi_device 
> should get a timer
> that calls the unblock routine, sounds fine to me and can be 
> done in the
> scsi midlayer.
> 

RE: suspending I/Os to a device

[Ely, Paul]

All,


The following patch provides an implementation of
the proposal contained in this mail thread.  The
patch does not provide a complete solution such
as:
	nodevice tmo handling by the scsi midlayer
	scsi_device structure member for nodevice tmo
	handling of SDEV_SUSPEND state by midlayer

Subsequent patches to address remaining implementation.


Thanks,

Paul



diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_lib.c
changed_kernel_2_6_7/drivers/scsi/scsi_lib.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_lib.c	2004-07-19
13:04:44.873200000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_lib.c	2004-07-22
16:03:49.423920000 -0400
@@ -1581,6 +1581,7 @@
 		case SDEV_CREATED:
 		case SDEV_OFFLINE:
 		case SDEV_QUIESCE:
+		case SDEV_SUSPEND:
 			break;
 		default:
 			goto illegal;
@@ -1608,11 +1609,22 @@
 		}
 		break;
 
+	case SDEV_SUSPEND:
+		switch (oldstate) {
+		case SDEV_CREATED:
+		case SDEV_RUNNING:
+			break;
+		default:
+			goto illegal;
+		}
+		break;
+
 	case SDEV_CANCEL:
 		switch (oldstate) {
 		case SDEV_CREATED:
 		case SDEV_RUNNING:
 		case SDEV_OFFLINE:
+		case SDEV_SUSPEND:
 			break;
 		default:
 			goto illegal;
@@ -1627,7 +1639,6 @@
 			goto illegal;
 		}
 		break;
-
 	}
 	sdev->sdev_state = state;
 	return 0;
@@ -1691,3 +1702,92 @@
 }
 EXPORT_SYMBOL(scsi_device_resume);
 
+/**
+ * Function:	scsi_device_suspend()
+ *
+ * Purpose:	Suspend request made by scsi lld's to temporarily stop all
+ *		scsi commands on the specified device.  Called from
interrupt
+ *		or normal process context.
+ *
+ * Arguments:	sdev:	scsi device to suspend
+ *
+ * Lock status:	Assumed that host lock is not held upon entry.
+ *
+ * Returns:	zero if successful 
+ *		an error otherwise (for example, state transition illegal)
+ *		  
+ *
+ * Notes:       
+ *	This routine transitions the device to the SDEV_SUSPEND state
+ *	(which must be a legal transition).  When the device is in this
+ *	state, all commands are deferred until the scsi lld reenables
+ *	the device with scsi_device_continue or device_suspend_tmo fires.
+ **/
+int
+scsi_device_suspend(struct scsi_device *sdev)
+{
+	request_queue_t *q;
+	int err;
+	unsigned long flags;
+
+	q = sdev->request_queue;
+	err = scsi_device_set_state(sdev, SDEV_SUSPEND);
+	if (err)
+		return err;
+
+	/* 
+	 * The device has transitioned to SDEV_SUSPEND.  Stop the
+	 * block layer from calling the midlayer with this device's
+	 * request queue.  
+	 */
+	spin_lock_irqsave(q->queue_lock, flags);
+	blk_stop_queue(q);
+	spin_unlock_irqrestore(q->queue_lock, flags);
+
+	return 0;
+}
+EXPORT_SYMBOL(scsi_device_suspend);
+
+/**
+ * Function:	scsi_device_continue()
+ *
+ * Purpose:	Called by scsi lld's or the midlayer to restart the device
queue
+ *		for the previously suspended scsi device.  Called from
+ *		interrupt or normal process context.
+ *
+ * Arguments:	sdev:	scsi device to restart
+ *
+ * Lock status:	Assumed that host lock is not held upon entry.
+ *
+ * Returns:	zero if successful 
+ *		an error otherwise (for example, state transition illegal)
+ *		  
+ *
+ * Notes:       
+ *	This routine transitions the device to the SDEV_RUNNING state
+ *	(which must be a legal transition) allowing the midlayer to
+ *	goose the queue for this device.
+ **/
+int
+scsi_device_continue(struct scsi_device *sdev)
+{
+	request_queue_t *q;
+	int err;
+	unsigned long flags;
+
+	q = sdev->request_queue;
+	err = scsi_device_set_state(sdev, SDEV_RUNNING);
+	if (err)
+		return err;
+
+	/* 
+	 * The device has transitioned to SDEV_RUNNING.  Restart
+	 * this device's request by calling the block to resume
+	 * calling the scsi_request_fn.
+	 */
+	spin_lock_irqsave(q->queue_lock, flags);
+	blk_start_queue(q);
+	spin_unlock_irqrestore(q->queue_lock, flags);
+	return 0;
+}
+EXPORT_SYMBOL(scsi_device_continue);
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_sysfs.c
changed_kernel_2_6_7/drivers/scsi/scsi_sysfs.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_sysfs.c	2004-07-19
13:04:43.644200000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_sysfs.c	2004-07-19
12:46:50.470200000 -0400
@@ -29,6 +29,7 @@
 	{ SDEV_DEL, "deleted" },
 	{ SDEV_QUIESCE, "quiesce" },
 	{ SDEV_OFFLINE,	"offline" },
+	{ SDEV_SUSPEND,	"suspend" },
 };
 
 const char *scsi_device_state_name(enum scsi_device_state state)
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c
changed_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c	2004-07-19
13:04:47.270200000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c	2004-07-22
15:52:22.224920000 -0400
@@ -28,7 +28,7 @@
 
 static void transport_class_release(struct class_device *class_dev);
 
-#define FC_NUM_ATTRS 	3	/* increase this if you add attributes */
+#define FC_NUM_ATTRS 	4	/* increase this if you add attributes */
 #define FC_OTHER_ATTRS 	0	/* increase this if you add "always
on"
 				 * attributes */
 
@@ -62,10 +62,12 @@
 static int fc_setup_transport_attrs(struct scsi_device *sdev)
 {
 	/* I'm not sure what values are invalid.  We should pick some
invalid
-	 * values for the defaults */
+	 * values for the defaults.  Set the no device timeout to 60 seconds
+	 * as the default.*/
 	fc_node_name(sdev) = -1;
 	fc_port_name(sdev) = -1;
 	fc_port_id(sdev) = -1;
+	fc_nodevice_tmo(sdev) = 60;
 
 	return 0;
 }
@@ -125,6 +127,7 @@
 fc_transport_rd_attr_cast(node_name, "0x%llx\n", unsigned long long);
 fc_transport_rd_attr_cast(port_name, "0x%llx\n", unsigned long long);
 fc_transport_rd_attr(port_id, "0x%06x\n");
+fc_transport_rw_attr(nodevice_tmo, "%d\n");
 
 #define SETUP_ATTRIBUTE_RD(field)				\
 	i->private_attrs[count] = class_device_attr_##field;	\
@@ -165,6 +168,7 @@
 	SETUP_ATTRIBUTE_RD(port_id);
 	SETUP_ATTRIBUTE_RD(port_name);
 	SETUP_ATTRIBUTE_RD(node_name);
+	SETUP_ATTRIBUTE_RW(nodevice_tmo);
 
 	BUG_ON(count > FC_NUM_ATTRS);
 
diff -uNr clean_kernel_2_6_7/include/scsi/scsi_device.h
changed_kernel_2_6_7/include/scsi/scsi_device.h
--- clean_kernel_2_6_7/include/scsi/scsi_device.h	2004-07-19
13:05:49.284200000 -0400
+++ changed_kernel_2_6_7/include/scsi/scsi_device.h	2004-07-19
12:46:39.286200000 -0400
@@ -30,6 +30,9 @@
 				 * originate in the mid-layer) */
 	SDEV_OFFLINE,		/* Device offlined (by error handling or
 				 * user request */
+	SDEV_SUSPEND,		/* Device suspended by scsi lld.  No block 
+				 * commands from user and midlayer should be
issued
+				 * to the scsi lld. */
 };
 
 struct scsi_device {
diff -uNr clean_kernel_2_6_7/include/scsi/scsi_transport_fc.h
changed_kernel_2_6_7/include/scsi/scsi_transport_fc.h
--- clean_kernel_2_6_7/include/scsi/scsi_transport_fc.h	2004-07-19
13:05:49.377200000 -0400
+++ changed_kernel_2_6_7/include/scsi/scsi_transport_fc.h	2004-07-22
14:00:43.519924000 -0400
@@ -28,18 +28,23 @@
 	int port_id;
 	uint64_t node_name;
 	uint64_t port_name;
+	uint32_t nodevice_tmo;
 };
 
 /* accessor functions */
 #define fc_port_id(x)	(((struct fc_transport_attrs
*)&(x)->transport_data)->port_id)
 #define fc_node_name(x)	(((struct fc_transport_attrs
*)&(x)->transport_data)->node_name)
 #define fc_port_name(x)	(((struct fc_transport_attrs
*)&(x)->transport_data)->port_name)
+#define fc_nodevice_tmo(x) (((struct fc_transport_attrs
*)&(x)->transport_data)->nodevice_tmo)
 
 /* The functions by which the transport class and the driver communicate */
 struct fc_function_template {
 	void 	(*get_port_id)(struct scsi_device *);
 	void	(*get_node_name)(struct scsi_device *);
 	void	(*get_port_name)(struct scsi_device *);
+	void    (*get_nodevice_tmo)(struct scsi_device *);
+	void	(*set_nodevice_tmo)(struct scsi_device *, int);
+
 	/* The driver sets these to tell the transport class it
 	 * wants the attributes displayed in sysfs.  If the show_ flag
 	 * is not set, the attribute will be private to the transport
@@ -47,6 +52,7 @@
 	unsigned long	show_port_id:1;
 	unsigned long	show_node_name:1;
 	unsigned long	show_port_name:1;
+	unsigned long   show_nodevice_tmo:1;
 	/* Private Attributes */
 };
 
@@ -54,3 +60,4 @@
 void fc_release_transport(struct scsi_transport_template *);
 
 #endif /* SCSI_TRANSPORT_FC_H */
+
diff -uNr clean_kernel_2_6_7/Makefile changed_kernel_2_6_7/Makefile
--- clean_kernel_2_6_7/Makefile	2004-07-19 13:03:26.385200000 -0400
+++ changed_kernel_2_6_7/Makefile	2004-07-19 15:17:14.137200000 -0400
@@ -1,8 +1,8 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 7
-EXTRAVERSION =
-NAME=Zonked Quokka
+EXTRAVERSION = -pae1.358.4.3smp
+NAME=pae
 
 # *DOCUMENTATION*
 # To see a list of typical targets execute "make help"

> -----Original Message-----
> From: Christoph Hellwig [mailto:hch@infradead.org]
> Sent: Saturday, July 17, 2004 8:25 AM
> To: Ely, Paul
> Cc: Smart, James; 'James Bottomley'; Linux SCSI Reflector
> Subject: Re: suspending I/Os to a device
> 
> 
> On Mon, Jul 12, 2004 at 04:53:34PM -0400, Ely, Paul wrote:
> > The proposal:
> > 	o A new device state called SDEV_SUSPEND
> > 		- The device's current state must be SDEV_CREATED or
> > SDEV_RUNNING.
> > 		- Once in SDEV_SUSPEND, the device state is allowed to
> > transition
> > 		  to SDEV_RUNNING or SDEV_CANCEL.
> > 
> > 	o A new routine called scsi_device_suspend to enter the 
> SDEV_SUSPEND
> > 	  state.
> > 		- Called by scsi LLDs only.
> > 		- scsi LLD must not hold any locks, including 
> the host lock,
> > 		  when calling this routine.
> > 		- Routine can be called from interrupt context or normal
> > kernel
> > 		  context.
> > 		- This routine calls into the scsi_device_set_state
> > 		  function to handle the state transition.  Other
> > functionality
> > 		  is TBD.
> > 
> > 	o A routine that the scsi LLD calls to successfully 
> transition from
> > 	  SDEV_SUSPEND to SDEV_RUNNING state and restart scsi 
> commands.  I
> > am
> > 	  proposing some alternatives.
> 
> This sounds fine to me.
> 
> > 		- Reuse scsi_device_resume.
> > 			+ Overloads quiesce and suspend.  This may work
> > since
> > 			  the state machine will not allow transitions
> > to/from
> > 			  SDEV_QUIESCE and SDEV_SUSPEND.
> > 
> > 		OR
> > 
> > 		- Provide a new routine called scsi_device_continue 
> > 			+ Analogous functionality as 
> scsi_device_resume, but
> > is
> > 			  called specifically to transition out of
> > SDEV_SUSPEND.
> 
> Please use separate functions.
> 
> > 	o Provide TMO control that limits the amount of time a 
> device can 
> > 	  disappear.  The timeout handling would clean up pending
> > 	  commands on the device queue.  I am proposing some
> > 	  alternatives here.
> > 		- Require the scsi LLD to provide tmo functionality.
> > Timeout 
> > 		  value would be a driver default, but also 
> settable via a
> > sysfs
> > 		  class transport attribute.  For example, Fibre Channel
> > drivers
> > 		  would use a FC transport value in sysfs.
> 
> Not sure if I parsed that correctly.  You want a timeout 
> after which the
> device gets unblocked again?  In that case the scsi_device 
> should get a timer
> that calls the unblock routine, sounds fine to me and can be 
> done in the
> scsi midlayer.
> 

Re: suspending I/Os to a device

[Nathan Bryant]

Ely, Paul wrote:

>All,
>
>
>The following patch provides an implementation of
>the proposal contained in this mail thread.  The
>patch does not provide a complete solution such
>as:
>	nodevice tmo handling by the scsi midlayer
>	scsi_device structure member for nodevice tmo
>	handling of SDEV_SUSPEND state by midlayer
>
>Subsequent patches to address remaining implementation.
>
I'm a little late to the party. Is this intended to facilitate the PCI 
power management (queue quiesce) issues that were discussed this week? 
Will it be sufficient to "just call" these from pcidev->suspend(), 
pcidev->resume()?

Duh, like thanks and stuff. I think.
Nathan

RE: suspending I/Os to a device

[Ely, Paul]

Nathan,

This proposal is specific to those scsi lld's that
manage storage devices that have to gracefully deal
with devices temporarily disappearing and reappearing -
such as Fibre Channel device drivers - but could have 
other applications such as iSCSI lld's or 

This proposal is not intended to deal with PCI 
power management.  Additionally, the existing
scsi_device_quiesce and resume functions and
device state are insufficient for this feature because 
for device suspend, the scsi lld does not want the midlayer
to have special's access to the scsi lld and in
particular retries.

HTH,

Paul

> -----Original Message-----
> From: Nathan Bryant [mailto:nbryant@optonline.net]
> Sent: Friday, July 23, 2004 12:36 AM
> To: Ely, Paul
> Cc: 'Christoph Hellwig'; Smart, James; 'James Bottomley'; Linux SCSI
> Reflector
> Subject: Re: suspending I/Os to a device
> 
> 
> Ely, Paul wrote:
> 
> >All,
> >
> >
> >The following patch provides an implementation of
> >the proposal contained in this mail thread.  The
> >patch does not provide a complete solution such
> >as:
> >	nodevice tmo handling by the scsi midlayer
> >	scsi_device structure member for nodevice tmo
> >	handling of SDEV_SUSPEND state by midlayer
> >
> >Subsequent patches to address remaining implementation.
> >
> I'm a little late to the party. Is this intended to 
> facilitate the PCI 
> power management (queue quiesce) issues that were discussed 
> this week? 
> Will it be sufficient to "just call" these from pcidev->suspend(), 
> pcidev->resume()?
> 
> Duh, like thanks and stuff. I think.
> Nathan
> 

RE: suspending I/Os to a device

[Ely, Paul]

I think the sysfs implementation for the nodevice_tmo
attribute needs to be rethought.

The nodevice_tmo really is a scsi device-specific attribute
and although it could apply to Fibre Channel, iSCSI,
or usb-connected storage, defining it in the scsi_transport_fc.c/h
files doesn't really work since:
	1. The midlayer doesn't interface with the transport
	   interface directly.
	2. Each transport type would need to replicate the
	   transport_fc at least for the nodevice_tmo.
	3. Implementing this is messy.

I am working on a modification to move the nodevice_timeout
value and timeout error handler into the scsi_device structure
since it really is a scsi device attribute that needs to be
managed by the midlayer.

Thoughts?

Paul


> -----Original Message-----
> From: Ely, Paul 
> Sent: Thursday, July 22, 2004 4:16 PM
> To: 'Christoph Hellwig'
> Cc: Smart, James; 'James Bottomley'; Linux SCSI Reflector
> Subject: RE: suspending I/Os to a device
> 
> 
> All,
> 
> 
> The following patch provides an implementation of
> the proposal contained in this mail thread.  The
> patch does not provide a complete solution such
> as:
> 	nodevice tmo handling by the scsi midlayer
> 	scsi_device structure member for nodevice tmo
> 	handling of SDEV_SUSPEND state by midlayer
> 
> Subsequent patches to address remaining implementation.
> 
> 
> Thanks,
> 
> Paul
> 
> 
> 
> diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_lib.c
> changed_kernel_2_6_7/drivers/scsi/scsi_lib.c
> --- clean_kernel_2_6_7/drivers/scsi/scsi_lib.c	2004-07-19
> 13:04:44.873200000 -0400
> +++ changed_kernel_2_6_7/drivers/scsi/scsi_lib.c	2004-07-22
> 16:03:49.423920000 -0400
> @@ -1581,6 +1581,7 @@
>  		case SDEV_CREATED:
>  		case SDEV_OFFLINE:
>  		case SDEV_QUIESCE:
> +		case SDEV_SUSPEND:
>  			break;
>  		default:
>  			goto illegal;
> @@ -1608,11 +1609,22 @@
>  		}
>  		break;
>  
> +	case SDEV_SUSPEND:
> +		switch (oldstate) {
> +		case SDEV_CREATED:
> +		case SDEV_RUNNING:
> +			break;
> +		default:
> +			goto illegal;
> +		}
> +		break;
> +
>  	case SDEV_CANCEL:
>  		switch (oldstate) {
>  		case SDEV_CREATED:
>  		case SDEV_RUNNING:
>  		case SDEV_OFFLINE:
> +		case SDEV_SUSPEND:
>  			break;
>  		default:
>  			goto illegal;
> @@ -1627,7 +1639,6 @@
>  			goto illegal;
>  		}
>  		break;
> -
>  	}
>  	sdev->sdev_state = state;
>  	return 0;
> @@ -1691,3 +1702,92 @@
>  }
>  EXPORT_SYMBOL(scsi_device_resume);
>  
> +/**
> + * Function:	scsi_device_suspend()
> + *
> + * Purpose:	Suspend request made by scsi lld's to 
> temporarily stop all
> + *		scsi commands on the specified device.  Called from
> interrupt
> + *		or normal process context.
> + *
> + * Arguments:	sdev:	scsi device to suspend
> + *
> + * Lock status:	Assumed that host lock is not held upon entry.
> + *
> + * Returns:	zero if successful 
> + *		an error otherwise (for example, state 
> transition illegal)
> + *		  
> + *
> + * Notes:       
> + *	This routine transitions the device to the SDEV_SUSPEND state
> + *	(which must be a legal transition).  When the device is in this
> + *	state, all commands are deferred until the scsi lld reenables
> + *	the device with scsi_device_continue or 
> device_suspend_tmo fires.
> + **/
> +int
> +scsi_device_suspend(struct scsi_device *sdev)
> +{
> +	request_queue_t *q;
> +	int err;
> +	unsigned long flags;
> +
> +	q = sdev->request_queue;
> +	err = scsi_device_set_state(sdev, SDEV_SUSPEND);
> +	if (err)
> +		return err;
> +
> +	/* 
> +	 * The device has transitioned to SDEV_SUSPEND.  Stop the
> +	 * block layer from calling the midlayer with this device's
> +	 * request queue.  
> +	 */
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	blk_stop_queue(q);
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL(scsi_device_suspend);
> +
> +/**
> + * Function:	scsi_device_continue()
> + *
> + * Purpose:	Called by scsi lld's or the midlayer to restart 
> the device
> queue
> + *		for the previously suspended scsi device.  Called from
> + *		interrupt or normal process context.
> + *
> + * Arguments:	sdev:	scsi device to restart
> + *
> + * Lock status:	Assumed that host lock is not held upon entry.
> + *
> + * Returns:	zero if successful 
> + *		an error otherwise (for example, state 
> transition illegal)
> + *		  
> + *
> + * Notes:       
> + *	This routine transitions the device to the SDEV_RUNNING state
> + *	(which must be a legal transition) allowing the midlayer to
> + *	goose the queue for this device.
> + **/
> +int
> +scsi_device_continue(struct scsi_device *sdev)
> +{
> +	request_queue_t *q;
> +	int err;
> +	unsigned long flags;
> +
> +	q = sdev->request_queue;
> +	err = scsi_device_set_state(sdev, SDEV_RUNNING);
> +	if (err)
> +		return err;
> +
> +	/* 
> +	 * The device has transitioned to SDEV_RUNNING.  Restart
> +	 * this device's request by calling the block to resume
> +	 * calling the scsi_request_fn.
> +	 */
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	blk_start_queue(q);
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +	return 0;
> +}
> +EXPORT_SYMBOL(scsi_device_continue);
> diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_sysfs.c
> changed_kernel_2_6_7/drivers/scsi/scsi_sysfs.c
> --- clean_kernel_2_6_7/drivers/scsi/scsi_sysfs.c	2004-07-19
> 13:04:43.644200000 -0400
> +++ changed_kernel_2_6_7/drivers/scsi/scsi_sysfs.c	2004-07-19
> 12:46:50.470200000 -0400
> @@ -29,6 +29,7 @@
>  	{ SDEV_DEL, "deleted" },
>  	{ SDEV_QUIESCE, "quiesce" },
>  	{ SDEV_OFFLINE,	"offline" },
> +	{ SDEV_SUSPEND,	"suspend" },
>  };
>  
>  const char *scsi_device_state_name(enum scsi_device_state state)
> diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c
> changed_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c
> --- clean_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c	
> 2004-07-19
> 13:04:47.270200000 -0400
> +++ changed_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c	
> 2004-07-22
> 15:52:22.224920000 -0400
> @@ -28,7 +28,7 @@
>  
>  static void transport_class_release(struct class_device *class_dev);
>  
> -#define FC_NUM_ATTRS 	3	/* increase this if you 
> add attributes */
> +#define FC_NUM_ATTRS 	4	/* increase this if you 
> add attributes */
>  #define FC_OTHER_ATTRS 	0	/* increase this if you 
> add "always
> on"
>  				 * attributes */
>  
> @@ -62,10 +62,12 @@
>  static int fc_setup_transport_attrs(struct scsi_device *sdev)
>  {
>  	/* I'm not sure what values are invalid.  We should pick some
> invalid
> -	 * values for the defaults */
> +	 * values for the defaults.  Set the no device timeout 
> to 60 seconds
> +	 * as the default.*/
>  	fc_node_name(sdev) = -1;
>  	fc_port_name(sdev) = -1;
>  	fc_port_id(sdev) = -1;
> +	fc_nodevice_tmo(sdev) = 60;
>  
>  	return 0;
>  }
> @@ -125,6 +127,7 @@
>  fc_transport_rd_attr_cast(node_name, "0x%llx\n", unsigned long long);
>  fc_transport_rd_attr_cast(port_name, "0x%llx\n", unsigned long long);
>  fc_transport_rd_attr(port_id, "0x%06x\n");
> +fc_transport_rw_attr(nodevice_tmo, "%d\n");
>  
>  #define SETUP_ATTRIBUTE_RD(field)				\
>  	i->private_attrs[count] = class_device_attr_##field;	\
> @@ -165,6 +168,7 @@
>  	SETUP_ATTRIBUTE_RD(port_id);
>  	SETUP_ATTRIBUTE_RD(port_name);
>  	SETUP_ATTRIBUTE_RD(node_name);
> +	SETUP_ATTRIBUTE_RW(nodevice_tmo);
>  
>  	BUG_ON(count > FC_NUM_ATTRS);
>  
> diff -uNr clean_kernel_2_6_7/include/scsi/scsi_device.h
> changed_kernel_2_6_7/include/scsi/scsi_device.h
> --- clean_kernel_2_6_7/include/scsi/scsi_device.h	2004-07-19
> 13:05:49.284200000 -0400
> +++ changed_kernel_2_6_7/include/scsi/scsi_device.h	2004-07-19
> 12:46:39.286200000 -0400
> @@ -30,6 +30,9 @@
>  				 * originate in the mid-layer) */
>  	SDEV_OFFLINE,		/* Device offlined (by error handling or
>  				 * user request */
> +	SDEV_SUSPEND,		/* Device suspended by scsi 
> lld.  No block 
> +				 * commands from user and 
> midlayer should be
> issued
> +				 * to the scsi lld. */
>  };
>  
>  struct scsi_device {
> diff -uNr clean_kernel_2_6_7/include/scsi/scsi_transport_fc.h
> changed_kernel_2_6_7/include/scsi/scsi_transport_fc.h
> --- clean_kernel_2_6_7/include/scsi/scsi_transport_fc.h	
> 2004-07-19
> 13:05:49.377200000 -0400
> +++ changed_kernel_2_6_7/include/scsi/scsi_transport_fc.h	
> 2004-07-22
> 14:00:43.519924000 -0400
> @@ -28,18 +28,23 @@
>  	int port_id;
>  	uint64_t node_name;
>  	uint64_t port_name;
> +	uint32_t nodevice_tmo;
>  };
>  
>  /* accessor functions */
>  #define fc_port_id(x)	(((struct fc_transport_attrs
> *)&(x)->transport_data)->port_id)
>  #define fc_node_name(x)	(((struct fc_transport_attrs
> *)&(x)->transport_data)->node_name)
>  #define fc_port_name(x)	(((struct fc_transport_attrs
> *)&(x)->transport_data)->port_name)
> +#define fc_nodevice_tmo(x) (((struct fc_transport_attrs
> *)&(x)->transport_data)->nodevice_tmo)
>  
>  /* The functions by which the transport class and the driver 
> communicate */
>  struct fc_function_template {
>  	void 	(*get_port_id)(struct scsi_device *);
>  	void	(*get_node_name)(struct scsi_device *);
>  	void	(*get_port_name)(struct scsi_device *);
> +	void    (*get_nodevice_tmo)(struct scsi_device *);
> +	void	(*set_nodevice_tmo)(struct scsi_device *, int);
> +
>  	/* The driver sets these to tell the transport class it
>  	 * wants the attributes displayed in sysfs.  If the show_ flag
>  	 * is not set, the attribute will be private to the transport
> @@ -47,6 +52,7 @@
>  	unsigned long	show_port_id:1;
>  	unsigned long	show_node_name:1;
>  	unsigned long	show_port_name:1;
> +	unsigned long   show_nodevice_tmo:1;
>  	/* Private Attributes */
>  };
>  
> @@ -54,3 +60,4 @@
>  void fc_release_transport(struct scsi_transport_template *);
>  
>  #endif /* SCSI_TRANSPORT_FC_H */
> +
> diff -uNr clean_kernel_2_6_7/Makefile changed_kernel_2_6_7/Makefile
> --- clean_kernel_2_6_7/Makefile	2004-07-19 
> 13:03:26.385200000 -0400
> +++ changed_kernel_2_6_7/Makefile	2004-07-19 
> 15:17:14.137200000 -0400
> @@ -1,8 +1,8 @@
>  VERSION = 2
>  PATCHLEVEL = 6
>  SUBLEVEL = 7
> -EXTRAVERSION =
> -NAME=Zonked Quokka
> +EXTRAVERSION = -pae1.358.4.3smp
> +NAME=pae
>  
>  # *DOCUMENTATION*
>  # To see a list of typical targets execute "make help"
> 
> > -----Original Message-----
> > From: Christoph Hellwig [mailto:hch@infradead.org]
> > Sent: Saturday, July 17, 2004 8:25 AM
> > To: Ely, Paul
> > Cc: Smart, James; 'James Bottomley'; Linux SCSI Reflector
> > Subject: Re: suspending I/Os to a device
> > 
> > 
> > On Mon, Jul 12, 2004 at 04:53:34PM -0400, Ely, Paul wrote:
> > > The proposal:
> > > 	o A new device state called SDEV_SUSPEND
> > > 		- The device's current state must be SDEV_CREATED or
> > > SDEV_RUNNING.
> > > 		- Once in SDEV_SUSPEND, the device state is allowed to
> > > transition
> > > 		  to SDEV_RUNNING or SDEV_CANCEL.
> > > 
> > > 	o A new routine called scsi_device_suspend to enter the 
> > SDEV_SUSPEND
> > > 	  state.
> > > 		- Called by scsi LLDs only.
> > > 		- scsi LLD must not hold any locks, including 
> > the host lock,
> > > 		  when calling this routine.
> > > 		- Routine can be called from interrupt context or normal
> > > kernel
> > > 		  context.
> > > 		- This routine calls into the scsi_device_set_state
> > > 		  function to handle the state transition.  Other
> > > functionality
> > > 		  is TBD.
> > > 
> > > 	o A routine that the scsi LLD calls to successfully 
> > transition from
> > > 	  SDEV_SUSPEND to SDEV_RUNNING state and restart scsi 
> > commands.  I
> > > am
> > > 	  proposing some alternatives.
> > 
> > This sounds fine to me.
> > 
> > > 		- Reuse scsi_device_resume.
> > > 			+ Overloads quiesce and suspend.  This may work
> > > since
> > > 			  the state machine will not allow transitions
> > > to/from
> > > 			  SDEV_QUIESCE and SDEV_SUSPEND.
> > > 
> > > 		OR
> > > 
> > > 		- Provide a new routine called scsi_device_continue 
> > > 			+ Analogous functionality as 
> > scsi_device_resume, but
> > > is
> > > 			  called specifically to transition out of
> > > SDEV_SUSPEND.
> > 
> > Please use separate functions.
> > 
> > > 	o Provide TMO control that limits the amount of time a 
> > device can 
> > > 	  disappear.  The timeout handling would clean up pending
> > > 	  commands on the device queue.  I am proposing some
> > > 	  alternatives here.
> > > 		- Require the scsi LLD to provide tmo functionality.
> > > Timeout 
> > > 		  value would be a driver default, but also 
> > settable via a
> > > sysfs
> > > 		  class transport attribute.  For example, Fibre Channel
> > > drivers
> > > 		  would use a FC transport value in sysfs.
> > 
> > Not sure if I parsed that correctly.  You want a timeout 
> > after which the
> > device gets unblocked again?  In that case the scsi_device 
> > should get a timer
> > that calls the unblock routine, sounds fine to me and can be 
> > done in the
> > scsi midlayer.
> > 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: suspending I/Os to a device

[James Bottomley]

On Mon, 2004-07-26 at 15:11, Ely, Paul wrote: 
> I think the sysfs implementation for the nodevice_tmo
> attribute needs to be rethought.
> 
> The nodevice_tmo really is a scsi device-specific attribute
> and although it could apply to Fibre Channel, iSCSI,
> or usb-connected storage, defining it in the scsi_transport_fc.c/h
> files doesn't really work since:
> 	1. The midlayer doesn't interface with the transport
> 	   interface directly.
> 	2. Each transport type would need to replicate the
> 	   transport_fc at least for the nodevice_tmo.
> 	3. Implementing this is messy.

Surely not if the implementation is done in scsi_lib.c?  The idea being
that you provide the storage area in the transport attributes but the
routines for handling it are all in scsi_lib so any transport that needs
it can hook in easily and transports that don't care don't bother and
don't waste the storage space.

This concept applies to a few other attributes we might do, which is why
I'm interested in seeing how it could work.

> I am working on a modification to move the nodevice_timeout
> value and timeout error handler into the scsi_device structure
> since it really is a scsi device attribute that needs to be
> managed by the midlayer.

It's a device attribute, but since it's use should be optional I'm not
convinced it needs to be in every SCSI device.

James

RE: suspending I/Os to a device

[Ely, Paul]

> -----Original Message-----
> From: James Bottomley [mailto:James.Bottomley@SteelEye.com]
> Sent: Wednesday, July 28, 2004 10:42 AM
> To: Ely, Paul
> Cc: 'Christoph Hellwig'; Smart, James; Linux SCSI Reflector
> Subject: RE: suspending I/Os to a device
> 
> 
> On Mon, 2004-07-26 at 15:11, Ely, Paul wrote: 
> > I think the sysfs implementation for the nodevice_tmo
> > attribute needs to be rethought.
> > 
> > The nodevice_tmo really is a scsi device-specific attribute
> > and although it could apply to Fibre Channel, iSCSI,
> > or usb-connected storage, defining it in the scsi_transport_fc.c/h
> > files doesn't really work since:
> > 	1. The midlayer doesn't interface with the transport
> > 	   interface directly.
> > 	2. Each transport type would need to replicate the
> > 	   transport_fc at least for the nodevice_tmo.
> > 	3. Implementing this is messy.
> 
> Surely not if the implementation is done in scsi_lib.c?  The 
> idea being
> that you provide the storage area in the transport attributes but the
> routines for handling it are all in scsi_lib so any transport 
> that needs
> it can hook in easily and transports that don't care don't bother and
> don't waste the storage space.
> 
> This concept applies to a few other attributes we might do, 
> which is why
> I'm interested in seeing how it could work.
> 
OK.  I can work the implementation some more.  I was concerned about
imparting transport-specific knowledge in the scsi midlayer with a
dereference of the transportt pointer, but can certainly work on some
access functions.  I plan on releasing another patch this week with the
error handling and state handling in the midlayer making the patch
pretty complete - I think.

> > I am working on a modification to move the nodevice_timeout
> > value and timeout error handler into the scsi_device structure
> > since it really is a scsi device attribute that needs to be
> > managed by the midlayer.
> 
> It's a device attribute, but since it's use should be optional I'm not
> convinced it needs to be in every SCSI device.
> 

I agree with this and is why I drove it into scsi_transport_fc.c 
initially.  

> James
> 
> 

Thanks,

Paul

RE: suspending I/Os to a device

[Ely, Paul]

Hello James,

I wanted some additional feedback or at least
to get a better feel for what you had in mind
for getting transport data into the midlayer.

Do you see scsi_lib.c including say scsi_transport_fc.h
or did you hope that a more transparent mechanism could
be implemented such as calling into scsi_transport_fc.c
directly based on the class name?

Also, what are your thoughts about per-attribute retrieval
versus transport datastructure retrieval?

I have an implementation going that implements the former
idea and admittedly I am struggling with how to glue
this in.

Thanks,

Paul

> -----Original Message-----
> From: James Bottomley [mailto:James.Bottomley@SteelEye.com]
> Sent: Wednesday, July 28, 2004 10:42 AM
> To: Ely, Paul
> Cc: 'Christoph Hellwig'; Smart, James; Linux SCSI Reflector
> Subject: RE: suspending I/Os to a device
> 
> 
> On Mon, 2004-07-26 at 15:11, Ely, Paul wrote: 
> > I think the sysfs implementation for the nodevice_tmo
> > attribute needs to be rethought.
> > 
> > The nodevice_tmo really is a scsi device-specific attribute
> > and although it could apply to Fibre Channel, iSCSI,
> > or usb-connected storage, defining it in the scsi_transport_fc.c/h
> > files doesn't really work since:
> > 	1. The midlayer doesn't interface with the transport
> > 	   interface directly.
> > 	2. Each transport type would need to replicate the
> > 	   transport_fc at least for the nodevice_tmo.
> > 	3. Implementing this is messy.
> 
> Surely not if the implementation is done in scsi_lib.c?  The 
> idea being
> that you provide the storage area in the transport attributes but the
> routines for handling it are all in scsi_lib so any transport 
> that needs
> it can hook in easily and transports that don't care don't bother and
> don't waste the storage space.
> 
> This concept applies to a few other attributes we might do, 
> which is why
> I'm interested in seeing how it could work.
> 
> > I am working on a modification to move the nodevice_timeout
> > value and timeout error handler into the scsi_device structure
> > since it really is a scsi device attribute that needs to be
> > managed by the midlayer.
> 
> It's a device attribute, but since it's use should be optional I'm not
> convinced it needs to be in every SCSI device.
> 
> James
> 
> 

RE: suspending I/Os to a device

[Paul.Ely]

Here is the completed patch for the scsi device suspend/continue
functionality.  It includes a method for get transport data in
a non-transport specific way and builds clean in the 2.6.7 
kernel source.

I have not started testing the patch yet, but will next week.

Comments welcome.

Paul

====================================


diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi.c changed_kernel_2_6_7/drivers/scsi/scsi.c
--- clean_kernel_2_6_7/drivers/scsi/scsi.c	2004-07-19 13:04:44.898201000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi.c	2004-08-04 11:57:16.665960000 -0400
@@ -512,6 +512,19 @@
 		/* return 0 (because the command has been processed) */
 		goto out;
 	}
+
+	/* Check to see if the scsi lld put this device into suspend. */
+	if (unlikely(cmd->device->sdev_state == SDEV_SUSPEND)) {
+		/* in SDEV_SUSPEND, the command is just put back on the device
+		 * queue.  The suspend state has already blocked the queue so
+		 * future requests should not occur until the device 
+		 * transitions out of the suspend state.
+		 */
+		scsi_queue_insert(cmd, SCSI_MLQUEUE_DEVICE_BUSY);
+		SCSI_LOG_MLQUEUE(3, printk("queuecommand : request rejected\n"));
+		goto out;
+	}
+
 	/* Assign a unique nonzero serial_number. */
 	/* XXX(hch): this is racy */
 	if (++serial_number == 0)
@@ -1088,8 +1101,8 @@
 
 /**
  * scsi_device_cancel - cancel outstanding IO to this device
- * @sdev:	pointer to struct scsi_device
- * @data:	pointer to cancel value.
+ * @sdev:	Pointer to struct scsi_device
+ * @recovery:	Boolean instructing function to recover device or not.
  *
  **/
 int scsi_device_cancel(struct scsi_device *sdev, int recovery)
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_error.c changed_kernel_2_6_7/drivers/scsi/scsi_error.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_error.c	2004-07-23 15:46:15.061800000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_error.c	2004-08-06 15:31:06.804920000 -0400
@@ -24,6 +24,7 @@
 #include <linux/blkdev.h>
 #include <linux/smp_lock.h>
 #include <scsi/scsi_ioctl.h>
+#include <scsi/scsi_transport.h>
 
 #include "scsi.h"
 #include "hosts.h"
@@ -155,6 +156,81 @@
 }
 
 /**
+ * scsi_add_sdev_timer - Start a timer for a scsi device that
+ *			 has been suspened by the scsi lld.
+ * @scmd:	scsi device to start timing.
+ * @timeout:	amount of time this device has to recover.
+ * @complete:	timeout function to call if timer isn't cancelled.
+ *
+ * Notes:
+ *	The scsi device is given a limited amount of time to
+ *	call scsi_device_continue and cancel this timer.  If the
+ *	scsi lld does not call in time, the device transitions 
+ *	to the cancel state.
+ *
+ * Returns:
+ *	None.
+ **/
+void scsi_add_sdev_timer(struct scsi_device *sdev, int timeout,
+			 void (*complete)(struct scsi_device *))
+{
+	/*
+	 * If the clock was already running for this command, then
+	 * first delete the timer.  The timer handling code gets rather
+	 * confused if we don't do this.
+	 */
+	if (sdev->device_recovert)
+		if (sdev->device_recovert->eh_nodev_timeout.function)
+			del_timer(&sdev->device_recovert->eh_nodev_timeout);
+
+	sdev->device_recovert->eh_nodev_timeout.data = (unsigned long) sdev;
+	sdev->device_recovert->eh_nodev_timeout.expires = jiffies + timeout;
+	sdev->device_recovert->eh_nodev_timeout.function = (void (*)(unsigned long)) complete;
+
+	SCSI_LOG_ERROR_RECOVERY(5, printk("%s: sdev: %p, time:"
+				" %d, (%p)\n", __FUNCTION__,
+				 sdev, timeout, complete));
+
+	add_timer(&sdev->device_recovert->eh_nodev_timeout);
+}
+
+/**
+ * scsi_delete_sdev_timer - Cancel timer for a given scsi device that was
+ *			    previously suspended.
+ *
+ * @sdev:	Scsi device on which to cancel the timer.
+ *
+ * Notes:
+ *	The scsi device is guaranteed to have recovered if this call
+ *	executes.  However, the scsi device could have recovered too
+ *	late and this routine needs to detect this scenerio.
+ *
+ * Return value:
+ *	0 if the timer function has already started to run.
+ *	1 if the timer was successfully detached.
+ *
+ **/
+int scsi_delete_sdev_timer(struct scsi_device *sdev)
+{
+	int rtn;
+
+	if (!sdev->device_recovert)
+		return -EACCES;
+
+	rtn = del_timer(&sdev->device_recovert->eh_nodev_timeout);
+
+	SCSI_LOG_ERROR_RECOVERY(5, printk("%s: sdev: %p,"
+					 " rtn: %d\n", __FUNCTION__,
+					 sdev, rtn));
+
+	sdev->device_recovert->eh_nodev_timeout.data = (unsigned long)NULL;
+	sdev->device_recovert->eh_nodev_timeout.function = NULL;
+
+	return rtn;
+}
+
+
+/**
  * scsi_times_out - Timeout function for normal scsi commands.
  * @scmd:	Cmd that is timing out.
  *
@@ -174,6 +250,31 @@
 }
 
 /**
+ * scsi_timeout_suspended_sdev - Timeout handler for suspended scsi devices that fail
+ *				 to recover in the alloted time.
+ * @sdev:	scsi device that failed to reappear in the alloted time.
+ **/
+void scsi_timeout_suspended_sdev(struct scsi_device *sdev)
+{
+	int rtn;
+
+	/* 
+	 * Cancel all IO to this device and do not attempt recovery.
+	 * There should not be any outstanding IO to the scsi lld
+	 * managing this device since the lld should have completed
+	 * all IO before transitioning to device state SDEV_SUSPEND.
+	 * The return value from scsi_device_cancel never fails and
+	 * is ignored here.
+	 */
+	rtn = scsi_device_cancel(sdev, 0);
+	if (rtn) {
+		SCSI_LOG_ERROR_RECOVERY(5, printk("%s: rtn: %d\n", __FUNCTION__,
+					  rtn));
+	}
+}
+
+
+/**
  * scsi_block_when_processing_errors - Prevent cmds from being queued.
  * @sdev:	Device on which we are performing recovery.
  *
@@ -1196,6 +1297,7 @@
 	return;
 }
 
+
 /**
  * scsi_sleep_done - timer function for scsi_sleep
  * @sem:	semphore to signal
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_lib.c changed_kernel_2_6_7/drivers/scsi/scsi_lib.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_lib.c	2004-07-19 13:04:44.873200000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_lib.c	2004-08-06 15:29:15.911920000 -0400
@@ -18,8 +18,8 @@
 
 #include <scsi/scsi_driver.h>
 #include <scsi/scsi_host.h>
+#include <scsi/scsi_transport.h>
 #include "scsi.h"
-
 #include "scsi_priv.h"
 #include "scsi_logging.h"
 
@@ -59,6 +59,7 @@
 #undef SP
 
 
+
 /*
  * Function:    scsi_insert_special_req()
  *
@@ -1581,6 +1582,12 @@
 		case SDEV_CREATED:
 		case SDEV_OFFLINE:
 		case SDEV_QUIESCE:
+		case SDEV_SUSPEND:
+			printk(KERN_WARNING "scsi_device_set_state: transitioning"
+				"to SDEV_RUNNING from current state %d"
+				"on host %d channel %d id %d lun %d\n", oldstate,
+				sdev->host->host_no, sdev->channel, sdev->id, 
+				sdev->lun);
 			break;
 		default:
 			goto illegal;
@@ -1608,11 +1615,32 @@
 		}
 		break;
 
+	case SDEV_SUSPEND:
+		switch (oldstate) {
+		case SDEV_CREATED:
+		case SDEV_RUNNING:
+			printk(KERN_WARNING "scsi_device_set_state: transitioning"
+				"to SDEV_SUSPEND from current state %d"
+				"on host %d channel %d id %d lun %d\n", oldstate,
+				sdev->host->host_no, sdev->channel, sdev->id, 
+				sdev->lun);
+			break;
+		default:
+			goto illegal;
+		}
+		break;
+
 	case SDEV_CANCEL:
 		switch (oldstate) {
 		case SDEV_CREATED:
 		case SDEV_RUNNING:
 		case SDEV_OFFLINE:
+		case SDEV_SUSPEND:
+			printk(KERN_WARNING "scsi_device_set_state: transitioning to"
+				"SDEV_CANCEL from current state %d on host %d "
+				"channel %d id %d lun %d\n", oldstate, 
+				sdev->host->host_no, sdev->channel, sdev->id, 
+				sdev->lun);
 			break;
 		default:
 			goto illegal;
@@ -1627,7 +1655,6 @@
 			goto illegal;
 		}
 		break;
-
 	}
 	sdev->sdev_state = state;
 	return 0;
@@ -1691,3 +1718,134 @@
 }
 EXPORT_SYMBOL(scsi_device_resume);
 
+/**
+ * Function:	scsi_device_suspend()
+ *
+ * Purpose:	Suspend request made by scsi lld's to temporarily stop all
+ *		scsi commands on the specified device.  Called from interrupt
+ *		or normal process context.
+ *
+ * Arguments:	sdev:	scsi device to suspend
+ *
+ * Returns:	 0	if successful 
+ *		-1	if error
+ *		  
+ *
+ * Notes:       
+ *	This routine transitions the device to the SDEV_SUSPEND state
+ *	(which must be a legal transition).  When the device is in this
+ *	state, all commands are deferred until the scsi lld reenables
+ *	the device with scsi_device_continue or device_suspend_tmo fires.
+ *	This routine assumes no locks are held upon entry.
+ **/
+int
+scsi_device_suspend(struct scsi_device *sdev)
+{
+	request_queue_t *q = sdev->request_queue;
+	unsigned long flags;
+	int  nodev_timeout = 0;
+
+	/* Before setting the new device state, make sure the host managing this
+	 * device has a recovery attribute for the suspend timeout.
+	 */
+	if (sdev->device_recovert) {
+		if (sdev->device_recovert->nodev_timeout < 1) {
+			printk(KERN_WARNING "scsi_device_suspend: nodev_timeout"
+				"value invalid on host %d channel %d id %d" 
+				"lun %d nodev_timeout=%d\n",
+				sdev->host->host_no, sdev->channel, sdev->id, 
+				sdev->lun, sdev->device_recovert->nodev_timeout);
+		return -1;
+		}
+
+		nodev_timeout = sdev->device_recovert->nodev_timeout;
+	} else {
+		printk(KERN_WARNING "scsi_device_suspend: host %d channel %d"
+			"id %d lun %d has no recovery data available.\n",
+			sdev->host->host_no, sdev->channel, sdev->id, 
+			sdev->lun);
+		return -1;
+	}
+
+	if (scsi_device_set_state(sdev, SDEV_SUSPEND) != 0) {
+		printk(KERN_WARNING "scsi_device_suspend: state set to suspend failed"
+			"on host %d channel %d id %d lun %d state=%d\n",
+			sdev->host->host_no, sdev->channel, sdev->id, sdev->lun,
+			sdev->sdev_state);
+		return -1;
+	}
+
+	/* 
+	 * The device has transitioned to SDEV_SUSPEND.  Stop the
+	 * block layer from calling the midlayer with this device's
+	 * request queue.  
+	 */
+	spin_lock_irqsave(q->queue_lock, flags);
+	blk_stop_queue(q);
+	spin_unlock_irqrestore(q->queue_lock, flags);
+
+	/*
+	 * Start timing the suspend.  The scsi lld is only allowed to suspend for
+	 * the nodev_timeout value in seconds. 
+	 */
+	scsi_add_sdev_timer(sdev, nodev_timeout, scsi_timeout_suspended_sdev);
+	return 0;
+}
+EXPORT_SYMBOL(scsi_device_suspend);
+
+/**
+ * Function:	scsi_device_continue()
+ *
+ * Purpose:	Called by scsi lld's or the midlayer to restart the device queue
+ *		for the previously suspended scsi device.  Called from
+ *		interrupt or normal process context.
+ *
+ * Arguments:	sdev:	scsi device to restart
+ *
+ * Returns:	 0	if successful 
+ *		-1	if error
+ *		  
+ *
+ * Notes:       
+ *	This routine transitions the device to the SDEV_RUNNING state
+ *	(which must be a legal transition) allowing the midlayer to
+ *	goose the queue for this device.  This routine assumes no locks
+ *	are held upon entry.
+ **/
+int
+scsi_device_continue(struct scsi_device *sdev)
+{
+	request_queue_t *q = sdev->request_queue; 
+	int err;
+	unsigned long flags;
+	
+	/* Stop the scsi device timer first. Take no action on the del_timer
+	 * failure as the state machine state change will validate the
+	 * transaction.
+	 */
+	err = scsi_delete_sdev_timer(sdev);
+	if (!err) {
+		printk(KERN_WARNING "scsi_device_continue: no device timeout on" 
+			"host %d channel %d id %d lun %d state %d\n",
+			sdev->host->host_no, sdev->channel, sdev->id, sdev->lun,
+			sdev->sdev_state);
+	} 
+
+	/* Try to transition the scsi device to SDEV_RUNNING
+	 * and goose the device queue if successful.  
+	 */
+	err = scsi_device_set_state(sdev, SDEV_RUNNING);
+	if (err) {
+		printk(KERN_WARNING "scsi_device_continue: state set to running"
+			"failed on host %d channel %d id %d lun %d state=%d\n",
+			sdev->host->host_no, sdev->channel, sdev->id, sdev->lun,
+			sdev->sdev_state);
+		return err;
+	}
+
+	spin_lock_irqsave(q->queue_lock, flags);
+	blk_start_queue(q);
+	spin_unlock_irqrestore(q->queue_lock, flags);
+	return 0;
+}
+EXPORT_SYMBOL(scsi_device_continue);
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_priv.h changed_kernel_2_6_7/drivers/scsi/scsi_priv.h
--- clean_kernel_2_6_7/drivers/scsi/scsi_priv.h	2004-07-19 13:04:45.468202000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_priv.h	2004-07-30 17:32:15.779440000 -0400
@@ -100,6 +100,7 @@
 
 /* scsi_error.c */
 extern void scsi_times_out(struct scsi_cmnd *cmd);
+extern void scsi_timeout_suspended_sdev(struct scsi_device *sdev);
 extern int scsi_error_handler(void *host);
 extern int scsi_decide_disposition(struct scsi_cmnd *cmd);
 extern void scsi_eh_wakeup(struct Scsi_Host *shost);
@@ -116,6 +117,7 @@
 extern void scsi_free_queue(struct request_queue *q);
 extern int scsi_init_queue(void);
 extern void scsi_exit_queue(void);
+extern int scsi_get_transport_info(void);
 
 /* scsi_proc.c */
 #ifdef CONFIG_SCSI_PROC_FS
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_syms.c changed_kernel_2_6_7/drivers/scsi/scsi_syms.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_syms.c	2004-07-19 13:04:44.842200000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_syms.c	2004-07-27 12:56:37.106600000 -0400
@@ -86,6 +86,8 @@
 EXPORT_SYMBOL(scsi_add_device);
 EXPORT_SYMBOL(scsi_remove_device);
 EXPORT_SYMBOL(scsi_device_cancel);
+EXPORT_SYMBOL(scsi_device_suspend);
+EXPORT_SYMBOL(scsi_device_continue);
 
 EXPORT_SYMBOL(__scsi_mode_sense);
 EXPORT_SYMBOL(scsi_mode_sense);
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_sysfs.c changed_kernel_2_6_7/drivers/scsi/scsi_sysfs.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_sysfs.c	2004-07-19 13:04:43.644200000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_sysfs.c	2004-08-05 09:41:49.706920000 -0400
@@ -29,6 +29,7 @@
 	{ SDEV_DEL, "deleted" },
 	{ SDEV_QUIESCE, "quiesce" },
 	{ SDEV_OFFLINE,	"offline" },
+	{ SDEV_SUSPEND,	"suspend" },
 };
 
 const char *scsi_device_state_name(enum scsi_device_state state)
diff -uNr clean_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c changed_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c
--- clean_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c	2004-07-19 13:04:47.270200000 -0400
+++ changed_kernel_2_6_7/drivers/scsi/scsi_transport_fc.c	2004-07-29 12:39:05.466520000 -0400
@@ -28,7 +28,7 @@
 
 static void transport_class_release(struct class_device *class_dev);
 
-#define FC_NUM_ATTRS 	3	/* increase this if you add attributes */
+#define FC_NUM_ATTRS 	4	/* increase this if you add attributes */
 #define FC_OTHER_ATTRS 	0	/* increase this if you add "always on"
 				 * attributes */
 
@@ -61,11 +61,14 @@
 
 static int fc_setup_transport_attrs(struct scsi_device *sdev)
 {
-	/* I'm not sure what values are invalid.  We should pick some invalid
-	 * values for the defaults */
+	/*
+	 * I'm not sure what values are invalid.  We should pick some invalid
+	 * values for the defaults.
+	 */
 	fc_node_name(sdev) = -1;
 	fc_port_name(sdev) = -1;
 	fc_port_id(sdev) = -1;
+	fc_nodevice_tmo(sdev) = -1;
 
 	return 0;
 }
@@ -125,6 +128,7 @@
 fc_transport_rd_attr_cast(node_name, "0x%llx\n", unsigned long long);
 fc_transport_rd_attr_cast(port_name, "0x%llx\n", unsigned long long);
 fc_transport_rd_attr(port_id, "0x%06x\n");
+fc_transport_rw_attr(nodevice_tmo, "%d\n");
 
 #define SETUP_ATTRIBUTE_RD(field)				\
 	i->private_attrs[count] = class_device_attr_##field;	\
@@ -165,6 +169,7 @@
 	SETUP_ATTRIBUTE_RD(port_id);
 	SETUP_ATTRIBUTE_RD(port_name);
 	SETUP_ATTRIBUTE_RD(node_name);
+	SETUP_ATTRIBUTE_RW(nodevice_tmo);
 
 	BUG_ON(count > FC_NUM_ATTRS);
 
diff -uNr clean_kernel_2_6_7/include/scsi/scsi_device.h changed_kernel_2_6_7/include/scsi/scsi_device.h
--- clean_kernel_2_6_7/include/scsi/scsi_device.h	2004-07-19 13:05:49.284200000 -0400
+++ changed_kernel_2_6_7/include/scsi/scsi_device.h	2004-08-04 13:36:20.854920000 -0400
@@ -30,6 +30,9 @@
 				 * originate in the mid-layer) */
 	SDEV_OFFLINE,		/* Device offlined (by error handling or
 				 * user request */
+	SDEV_SUSPEND,		/* Device suspended by scsi lld.  No block 
+				 * commands from user and midlayer should be issued
+				 * to the scsi lld. */
 };
 
 struct scsi_device {
@@ -99,17 +102,16 @@
 				 * this device */
 	unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN
 				     * because we did a bus reset. */
-	unsigned use_10_for_rw:1; /* first try 10-byte read / write */
-	unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */
+	unsigned use_10_for_rw:1;	/* first try 10-byte read / write */
+	unsigned use_10_for_ms:1;	/* first try 10-byte mode sense/select */
 	unsigned skip_ms_page_8:1;	/* do not use MODE SENSE page 0x08 */
 	unsigned skip_ms_page_3f:1;	/* do not use MODE SENSE page 0x3f */
-	unsigned use_192_bytes_for_3f:1; /* ask for 192 bytes from page 0x3f */
+	unsigned use_192_bytes_for_3f:1;/* ask for 192 bytes from page 0x3f */
 	unsigned no_start_on_add:1;	/* do not issue start on add */
-	unsigned allow_restart:1; /* issue START_UNIT in error handler */
+	unsigned allow_restart:1;	/* issue START_UNIT in error handler */
 
 	unsigned int device_blocked;	/* Device returned QUEUE_FULL. */
-
-	unsigned int max_device_blocked; /* what device_blocked counts down from  */
+	unsigned int max_device_blocked;   /* what device_blocked counts down from */
 #define SCSI_DEFAULT_DEVICE_BLOCKED	3
 
 	int timeout;
@@ -118,10 +120,11 @@
 	struct class_device	sdev_classdev;
 
 	struct class_device	transport_classdev;
-
+	struct device_recover_template	*device_recovert; /* Device recovery template. */
 	enum scsi_device_state sdev_state;
 	unsigned long		transport_data[0];
 } __attribute__((aligned(sizeof(unsigned long))));
+
 #define	to_scsi_device(d)	\
 	container_of(d, struct scsi_device, sdev_gendev)
 #define	class_to_sdev(d)	\
@@ -133,7 +136,8 @@
 		uint, uint, uint);
 extern void scsi_remove_device(struct scsi_device *);
 extern int scsi_device_cancel(struct scsi_device *, int);
-
+int scsi_device_suspend(struct scsi_device *);
+int scsi_device_continue(struct scsi_device *);
 extern int scsi_device_get(struct scsi_device *);
 extern void scsi_device_put(struct scsi_device *);
 extern struct scsi_device *scsi_device_lookup(struct Scsi_Host *,
diff -uNr clean_kernel_2_6_7/include/scsi/scsi_eh.h changed_kernel_2_6_7/include/scsi/scsi_eh.h
--- clean_kernel_2_6_7/include/scsi/scsi_eh.h	2004-07-19 13:05:49.340200000 -0400
+++ changed_kernel_2_6_7/include/scsi/scsi_eh.h	2004-07-27 12:54:27.578603000 -0400
@@ -4,6 +4,11 @@
 extern void scsi_add_timer(struct scsi_cmnd *, int,
 			   void (*)(struct scsi_cmnd *));
 extern int scsi_delete_timer(struct scsi_cmnd *);
+
+extern void scsi_add_sdev_timer(struct scsi_device *, int,
+				void (*complete)(struct scsi_device *));
+extern int scsi_delete_sdev_timer(struct scsi_device *sdev);
+
 extern void scsi_report_bus_reset(struct Scsi_Host *, int);
 extern void scsi_report_device_reset(struct Scsi_Host *, int, int);
 extern int scsi_block_when_processing_errors(struct scsi_device *);
diff -uNr clean_kernel_2_6_7/include/scsi/scsi_transport_fc.h changed_kernel_2_6_7/include/scsi/scsi_transport_fc.h
--- clean_kernel_2_6_7/include/scsi/scsi_transport_fc.h	2004-07-19 13:05:49.377200000 -0400
+++ changed_kernel_2_6_7/include/scsi/scsi_transport_fc.h	2004-07-29 12:39:58.053520000 -0400
@@ -26,6 +26,7 @@
 
 struct fc_transport_attrs {
 	int port_id;
+	int nodevice_tmo;
 	uint64_t node_name;
 	uint64_t port_name;
 };
@@ -34,12 +35,16 @@
 #define fc_port_id(x)	(((struct fc_transport_attrs *)&(x)->transport_data)->port_id)
 #define fc_node_name(x)	(((struct fc_transport_attrs *)&(x)->transport_data)->node_name)
 #define fc_port_name(x)	(((struct fc_transport_attrs *)&(x)->transport_data)->port_name)
+#define fc_nodevice_tmo(x) (((struct fc_transport_attrs *)&(x)->transport_data)->nodevice_tmo)
 
 /* The functions by which the transport class and the driver communicate */
 struct fc_function_template {
 	void 	(*get_port_id)(struct scsi_device *);
 	void	(*get_node_name)(struct scsi_device *);
 	void	(*get_port_name)(struct scsi_device *);
+	void    (*get_nodevice_tmo)(struct scsi_device *);
+	void	(*set_nodevice_tmo)(struct scsi_device *, uint32_t);
+
 	/* The driver sets these to tell the transport class it
 	 * wants the attributes displayed in sysfs.  If the show_ flag
 	 * is not set, the attribute will be private to the transport
@@ -47,6 +52,7 @@
 	unsigned long	show_port_id:1;
 	unsigned long	show_node_name:1;
 	unsigned long	show_port_name:1;
+	unsigned long   show_nodevice_tmo:1;
 	/* Private Attributes */
 };
 
@@ -54,3 +60,4 @@
 void fc_release_transport(struct scsi_transport_template *);
 
 #endif /* SCSI_TRANSPORT_FC_H */
+
diff -uNr clean_kernel_2_6_7/include/scsi/scsi_transport.h changed_kernel_2_6_7/include/scsi/scsi_transport.h
--- clean_kernel_2_6_7/include/scsi/scsi_transport.h	2004-07-19 13:05:49.363200000 -0400
+++ changed_kernel_2_6_7/include/scsi/scsi_transport.h	2004-08-05 10:20:04.110923000 -0400
@@ -38,4 +38,22 @@
 	int	size;
 };
 
+/* device_recover_template:  
+ *	Scsi llds that want to take advantage of the suspend/continue device
+ *	feature allocate an instance of this data type during slave alloc.
+ *	The scsi lld sets the nodev_timeout value during slave_configure and
+ *	frees this resource in slave_destroy.
+ *
+ *	This feature is optional to scsi llds.  The midlayer detects the
+ *	presence or absence of this feature by a scsi lld by checking the
+ *	device_recovert pointer in the scsi_device instance. 
+ */
+struct device_recover_template {
+	/* Device recovery template. */
+	unsigned int nodev_timeout;		/* Device suspend timeout value
+						 * in seconds. */
+	struct timer_list eh_nodev_timeout;	/* Used to timeout the suspend 
+						 * state. */
+};
+
 #endif /* SCSI_TRANSPORT_H */

> -----Original Message-----
> From: Ely, Paul 
> Sent: Monday, August 02, 2004 5:23 PM
> To: 'James Bottomley'
> Cc: 'Christoph Hellwig'; Smart, James; Linux SCSI Reflector
> Subject: RE: suspending I/Os to a device
> 
> 
> Hello James,
> 
> I wanted some additional feedback or at least
> to get a better feel for what you had in mind
> for getting transport data into the midlayer.
> 
> Do you see scsi_lib.c including say scsi_transport_fc.h
> or did you hope that a more transparent mechanism could
> be implemented such as calling into scsi_transport_fc.c
> directly based on the class name?
> 
> Also, what are your thoughts about per-attribute retrieval
> versus transport datastructure retrieval?
> 
> I have an implementation going that implements the former
> idea and admittedly I am struggling with how to glue
> this in.
> 
> Thanks,
> 
> Paul
> 
> > -----Original Message-----
> > From: James Bottomley [mailto:James.Bottomley@SteelEye.com]
> > Sent: Wednesday, July 28, 2004 10:42 AM
> > To: Ely, Paul
> > Cc: 'Christoph Hellwig'; Smart, James; Linux SCSI Reflector
> > Subject: RE: suspending I/Os to a device
> > 
> > 
> > On Mon, 2004-07-26 at 15:11, Ely, Paul wrote: 
> > > I think the sysfs implementation for the nodevice_tmo
> > > attribute needs to be rethought.
> > > 
> > > The nodevice_tmo really is a scsi device-specific attribute
> > > and although it could apply to Fibre Channel, iSCSI,
> > > or usb-connected storage, defining it in the scsi_transport_fc.c/h
> > > files doesn't really work since:
> > > 	1. The midlayer doesn't interface with the transport
> > > 	   interface directly.
> > > 	2. Each transport type would need to replicate the
> > > 	   transport_fc at least for the nodevice_tmo.
> > > 	3. Implementing this is messy.
> > 
> > Surely not if the implementation is done in scsi_lib.c?  The 
> > idea being
> > that you provide the storage area in the transport 
> attributes but the
> > routines for handling it are all in scsi_lib so any transport 
> > that needs
> > it can hook in easily and transports that don't care don't 
> bother and
> > don't waste the storage space.
> > 
> > This concept applies to a few other attributes we might do, 
> > which is why
> > I'm interested in seeing how it could work.
> > 
> > > I am working on a modification to move the nodevice_timeout
> > > value and timeout error handler into the scsi_device structure
> > > since it really is a scsi device attribute that needs to be
> > > managed by the midlayer.
> > 
> > It's a device attribute, but since it's use should be 
> optional I'm not
> > convinced it needs to be in every SCSI device.
> > 
> > James
> > 
> > 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>