RE: A different look at block device hotswap in the Linux kernel

RE: A different look at block device hotswap in the Linux kernel

[Cress, Andrew R]

My $.02:

The comparing of a saved device list snapshot with the current device should
be the responsibility of a user-space daemon, provided that the kernel
exposes enough information  to uniquely identify the devices (like serial
numbers, or some other UID if no ser num exists).  

The kernel would assume that the device is new (not the same) unless told so
by a daemon that is watching.  

Andy

-----Original Message-----
From: Matthew Jacob [mailto:mjacob@feral.com] 
Sent: Friday, January 24, 2003 10:26 AM
To: David Brownell
Cc: Oliver Neukum; Steven Dake; Luben Tuikov; Alan Stern; Matthew Dharm;
Mike Anderson; Greg KH; linux-usb-devel@lists.sourceforge.net; Linux SCSI
list
Subject: Re: A different look at block device hotswap in the Linux kernel



> >>...
> >
> >
> > Could this time limit be fixed (or parameterized) known to all LLDDs?
> > This would allow one to try and avoid flooding SCSI with detach/reattach
> > events for the 'same' device.
>
> And what exactly is the "same" device?  And who's keeping history
> about devices that have previously been attached?  And, says the guy
> who's full of questions, didn't Linus want to get rid of such history?

Hrmm. That's a damned good point. I was going to say things like "the
FC HBA driver knows that device XYX left the fabric and now has
returned", but if XYZ left the fabric, why am I keeping track of it
still? Once gone, it's gone. I had convinced myself that if an FC device
(re)appears, it's not up to the HBA to say it's the same (the content
may have been changed even if the container tag is the same).

Hrm.


-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: A different look at block device hotswap in the Linux kernel

[Bryan Henderson]

>The comparing of a saved device list snapshot with the current device
should
>be the responsibility of

>From a usability standpoint, I don't think any such comparing should be
done by anyone.  When I unplug a device and then plug it in again, I want a
total reset.  I'm willing to take my lumps if I unplug something that isn't
in a state to be safely unplugged.

It's like when I pull the power plug because my system is totally hosed and
I want to start over.  I know I can cause damage by doing that, but I would
be upset if the new system booted back to the broken state it was in when I
unplugged it.

RE: A different look at block device hotswap in the Linux kernel

[Matthew Jacob]

>
> It's like when I pull the power plug because my system is totally hosed and
> I want to start over.  I know I can cause damage by doing that, but I would
> be upset if the new system booted back to the broken state it was in when I
> unplugged it.

I had this conversation with doug offlist- this is a policy choice. You
may want your device to reattach as totally new. You may, on the other
hand, want your device to resume where you left off. I can see valid
reasons for wanting either behaviour (but it can't/shouldn't be deduced
by the OS).

Re: [linux-usb-devel] RE: A different look at block device hotswap in the Linux kernel

[Andries Brouwer]

On Fri, Jan 24, 2003 at 10:09:32AM -0800, Matthew Jacob wrote:

> > It's like when I pull the power plug because my system is totally hosed and
> > I want to start over.  I know I can cause damage by doing that, but I would
> > be upset if the new system booted back to the broken state it was in when I
> > unplugged it.
> 
> You may want your device to reattach as totally new. You may, on the other
> hand, want your device to resume where you left off. I can see valid
> reasons for wanting either behaviour (but it can't/shouldn't be deduced
> by the OS).

The kernel does not remember this device once it is gone.
If user space thinks it recognizes it, it must have a means to
tell the kernel what it should do. What kind of info?
The only thing that I can think of right now is the name.
User space may tell the kernel under what name a device should
be accessible.

[Today user space can tell the kernel the numbering of the partitions.]

More generally, the question of what "same device" means
depends on the application. Is /dev/fd0 the same device
as /dev/fd0 also when the floppy is different?
Yes, the name is the same. The contents not.

In all cases user space does not tell the kernel "it is the same as ..."
since the kernel does not remember anyway. User space just instructs
the kernel to do certain definite things.

Andries

Re: [linux-usb-devel] Re: [PATCH] USB changes for 2.5.58

[Alan Stern]

Here's another question to add to your discussion.

When a device is unplugged, the system's representation of that device
can't be removed immediately; there may be open fd's, mounts, pointers,
and so on.  Until the time comes when all these handles are released, all
interaction with the device has to fail, one way or another.

Whose responsibility is it to fail these interactions?

For something simple, like a USB serial port, it might turn out that the 
low-level device driver gets all these requests and then fails them.  That 
means the driver has to keep track of the fact that the device is no 
longer connected until some reference count goes to 0.

For SCSI and emulated SCSI devices, it might be the one of the SCSI layers 
that keeps track of the fact that the device has disconnected.  Or it 
might be somewhere else in the kernel.

It would nice to have some sort of coherent plan for how to handle this.  
In fact, it ought to be part of the device-driver model that underlies 
sysfs.  But so far as I am aware, there is currently nothing in the sysfs 
documentation to address the problem.

Alan Stern

Re: [linux-usb-devel] Re: [PATCH] USB changes for 2.5.58

[Oliver Neukum]

Am Donnerstag, 23. Januar 2003 18:46 schrieb Luben Tuikov:
> Oliver Neukum wrote:
> > Not all the world is a SAN. USB has no possibility to even try an
> > interaction after the device is gone. We have to handle this flexibly.
>
> Thus the example in the original post.  I.e. for simple transports whose
> portals get notified when a device is plugged off (USB), the LLDD
> can notify SCSI Core, by setting a state variable in scsi_device.
> In which case SCSI Core can answer with the proper TARGET error code.
> (This was outlined before, scsi_command->online:1 ...)

Very well, so you agree that the SCSI layer should export to the LLDD
a function to set devices offline?

> > In fact, if a device
> > can vanish without a LLDD knowing about it, this is purely a problem of
> > the SCSI layer.
>
> No, of course not.  (Think of IP.)  When a device vanishes and LLDD doesn't
> know about it (more complicated transports), the CDB will return with
> the proper Service Response, since the transport(s) won't be able to
> deliver it. This will bubble up through SCSI Core and the error returned
> will have to be the same as that of the simpler transports, as outlined
> above.

Yes, sorry. To be precise, this means that the LLDD has to do nothing
special, as it has to implement checking for a failing command anyway.
But it's not entirely the same. If a command cannot be delivered it may or may
not be appropriate to start error recovery. After the LLDD has told
the SCSI layer that it has noticed a device going away, there must be no
error recovery.

> > That means that we have to have a way to ensure that no more commands
> > will reach the LLDD which can be triggered without any commands to be
> > executed at all. This functionality has to come from the scsi mid layer.
>
> For simple transports yes; for more complicated ones, the CDB will
> not be able to be delivered, and will return with error.

Good.
So the first thing a LLDD has to do after it has learned about a device
being removed is to have the device block.
1. set device offline
But commands may still be in flight.IMHO it is not right to assume that
all commands now in flight to a device have failed, as some may have
completed successfully in time, or failed for other reasons than unplugging.
So it should be the LLDD's responsibility to finish the outstanding commands.
Furthermore, there's a window for commands already having passed the check
for offline but not yet being noticed by the LLDD. The simplest solution is to
use a waiting primitive from RCU. So we are at:

1. set device offline
2. synchronize the kernel
3. finish all pending commands

So far with me?
The LLDD could now forget about the device and be done with it.
However there's a problem left. The device may come back.
What happens if a device with the same ID is reconnected?

	Regards
		Oliver

A different look at block device hotswap in the Linux kernel

[Steven Dake]

Oliver and others,

In regards to hotswap, any real operating system should be _told_ that a 
block device is going to be removed from the top.  There are several 
reasons.

1) File mounts should be removed from the filesystem layer
2) files accessing block devices directly should be terminated
3) raid members using that block device should be hot removed
4) I'm sure you can think of others :)

The key is that the removal request should come from the top, not the 
bottom.  If someone is stupid enough to surprise remove a device (ie: 
unplug their USB SCSI device while the device is in use by the OS), they 
get what they deserve (I/O errors, dirty OS data, queued up requests 
which never shut down).  If they tell the OS that the device is going to 
be removed, so it may flush the device and shut down I/O to the device, 
the request should be granted on all accounts (expected removal).

The device driver should not be responsible for managing hotswap in any 
regard.  Its only purpose should be to tell the block device removal 
layer that a surprise extraction was initiated such that the block 
device removal code can ask the mid layer drivers to shut down error 
correction routines to the device and dump its pending I/O queue and 
clean up after the device.  The main advantage of this technique is 
simplicity (the LLDD's don't have to have repetative logic for each 
device driver), genericity (the block device removal code can be 
maintained in one place and be guaranteed to ensure the OS is in a 
stable state after a device is removed either surprise or expected and 
finally it solves the in-flight I/O problem by stopping new I/O to the 
device, shutting down I/O to the device, flushing the pending I/O 
queues, and killing all references in the OS of the device.

If you think about what your suggesting, your suggesting that the LLDD 
tells the scsi layer that the device is gone, that then times out errors 
and leaves the filesystem and sys_open/close file tables, and RAID 
layers in a state of disarray.  We don't want the LLDD knowing about the 
RAID system and whether it should tell the RAID layer to hot remove, do we?

I've developed code to do exactly what I have described here (surprise 
and expected extractions genericized into one file with one simple call 
from userland and a method for lower layers to indicate a surprise 
extraction if they have detected one.  I'll post as soon as I have time 
to make a patch against 2.5 .

Thanks
-steve


Oliver Neukum wrote:

>Am Donnerstag, 23. Januar 2003 18:46 schrieb Luben Tuikov:
>  
>
>>Oliver Neukum wrote:
>>    
>>
>>>Not all the world is a SAN. USB has no possibility to even try an
>>>interaction after the device is gone. We have to handle this flexibly.
>>>      
>>>
>>Thus the example in the original post.  I.e. for simple transports whose
>>portals get notified when a device is plugged off (USB), the LLDD
>>can notify SCSI Core, by setting a state variable in scsi_device.
>>In which case SCSI Core can answer with the proper TARGET error code.
>>(This was outlined before, scsi_command->online:1 ...)
>>    
>>
>
>Very well, so you agree that the SCSI layer should export to the LLDD
>a function to set devices offline?
>
>  
>
>>>In fact, if a device
>>>can vanish without a LLDD knowing about it, this is purely a problem of
>>>the SCSI layer.
>>>      
>>>
>>No, of course not.  (Think of IP.)  When a device vanishes and LLDD doesn't
>>know about it (more complicated transports), the CDB will return with
>>the proper Service Response, since the transport(s) won't be able to
>>deliver it. This will bubble up through SCSI Core and the error returned
>>will have to be the same as that of the simpler transports, as outlined
>>above.
>>    
>>
>
>Yes, sorry. To be precise, this means that the LLDD has to do nothing
>special, as it has to implement checking for a failing command anyway.
>But it's not entirely the same. If a command cannot be delivered it may or may
>not be appropriate to start error recovery. After the LLDD has told
>the SCSI layer that it has noticed a device going away, there must be no
>error recovery.
>
>  
>
>>>That means that we have to have a way to ensure that no more commands
>>>will reach the LLDD which can be triggered without any commands to be
>>>executed at all. This functionality has to come from the scsi mid layer.
>>>      
>>>
>>For simple transports yes; for more complicated ones, the CDB will
>>not be able to be delivered, and will return with error.
>>    
>>
>
>Good.
>So the first thing a LLDD has to do after it has learned about a device
>being removed is to have the device block.
>1. set device offline
>But commands may still be in flight.IMHO it is not right to assume that
>all commands now in flight to a device have failed, as some may have
>completed successfully in time, or failed for other reasons than unplugging.
>So it should be the LLDD's responsibility to finish the outstanding commands.
>Furthermore, there's a window for commands already having passed the check
>for offline but not yet being noticed by the LLDD. The simplest solution is to
>use a waiting primitive from RCU. So we are at:
>
>1. set device offline
>2. synchronize the kernel
>3. finish all pending commands
>
>So far with me?
>The LLDD could now forget about the device and be done with it.
>However there's a problem left. The device may come back.
>What happens if a device with the same ID is reconnected?
>
>	Regards
>		Oliver
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
>  
>

Re: A different look at block device hotswap in the Linux kernel

[Oliver Neukum]

Am Donnerstag, 23. Januar 2003 21:41 schrieb Steven Dake:
> Oliver and others,
>
> In regards to hotswap, any real operating system should be _told_ that a
> block device is going to be removed from the top.  There are several
> reasons.

Users don't do what they should. It is as simple as that.
The hotplugging busses are supposed to handle that.

> 1) File mounts should be removed from the filesystem layer
> 2) files accessing block devices directly should be terminated
> 3) raid members using that block device should be hot removed
> 4) I'm sure you can think of others :)
>
> The key is that the removal request should come from the top, not the
> bottom.  If someone is stupid enough to surprise remove a device (ie:

No! You have to be able to handle a sudden failure. If you don't do this
you are already buggy. Hardware doesn't send advance notification before
failing. Data loss will occur. It's unavoidable. Anything else must not happen.
And a failure of hardware can only be recognised at the layer closest to
the hardware in the generic case.

> The device driver should not be responsible for managing hotswap in any
> regard.  Its only purpose should be to tell the block device removal

Yes.

> layer that a surprise extraction was initiated such that the block
> device removal code can ask the mid layer drivers to shut down error
> correction routines to the device and dump its pending I/O queue and
> clean up after the device.  The main advantage of this technique is

Yes. But not ask. Demand. There's no asking here. Do or die.

> If you think about what your suggesting, your suggesting that the LLDD
> tells the scsi layer that the device is gone, that then times out errors
> and leaves the filesystem and sys_open/close file tables, and RAID
> layers in a state of disarray.  We don't want the LLDD knowing about the
> RAID system and whether it should tell the RAID layer to hot remove, do we?

I want:
LLDD to SCSI: device is gone
SCSI to LLDD: Ok. I'll handle from here on.
LLDD: OK. I am gone. And won't have any contact until the next device is
plugged in.

The process can be somewhat more complicated, under some conditions:
- it never fails
- it is done within a finite, bounded, reasonable time

	Regards
		Oliver



-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com
_______________________________________________
linux-usb-devel@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

Re: A different look at block device hotswap in the Linux kernel

[Steven Dake]

>I want:
>LLDD to SCSI: device is gone
>SCSI to LLDD: Ok. I'll handle from here on.
>LLDD: OK. I am gone. And won't have any contact until the next device is
>plugged in.
>  
>
The downside of this approach is that the LLDD must now be able to 
detect insertions and removals when it may not be able to do so.  If it 
is able to do so, then fine, it can tell upper layers about it, but the 
actual control of removal of a device should occur higher up to fix 
several problems with the approach of having the LLDD manage the hotswap 
state of the device.

>The process can be somewhat more complicated, under some conditions:
>- it never fails
>- it is done within a finite, bounded, reasonable time
>
>	Regards
>		Oliver
>
>
>
>  
>

Re: [linux-usb-devel] Re: A different look at block device hotswap in the Linux kernel

[Matthew Dharm]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

On Thu, Jan 23, 2003 at 05:54:57PM -0700, Steven Dake wrote:
> >I want:
> >LLDD to SCSI: device is gone
> >SCSI to LLDD: Ok. I'll handle from here on.
> >LLDD: OK. I am gone. And won't have any contact until the next device is
> >plugged in.
> >  
> >
> The downside of this approach is that the LLDD must now be able to 
> detect insertions and removals when it may not be able to do so.  If it 
> is able to do so, then fine, it can tell upper layers about it, but the 
> actual control of removal of a device should occur higher up to fix 
> several problems with the approach of having the LLDD manage the hotswap 
> state of the device.

Huh?

Aren't we talking about a hotplug scenario?  How can you talk about the
'LLDD must now be able to detect... when it may not be able to do so.'?

Oh... I see.  We keep talking about devices.  I'm trying to hotswap an
entire host, which is mapped to a single USB device.

But the theory is the same, really.

In the end, you can only hotswap something that is hotswapable.  That means
that the driver has to support the hotswap system, whatever it is.

If you can't support hotswap detection, then this entire scenario is
reduced to 'what happens if I blow a FET on my HD', because it's the exact
same thing.  Recovering from fatal error is a separate discussion.

Matt

-- 
Matthew Dharm                              Home: mdharm-usb@one-eyed-alien.net 
Maintainer, Linux USB Mass Storage Driver

A:  The most ironic oxymoron wins ...
DP: "Microsoft Works"
A:  Uh, okay, you win.
					-- A.J. & Dust Puppy
User Friendly, 1/18/1998

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]