[RFC] SCSI Hotplug for lk 2.5

[RFC] SCSI Hotplug for lk 2.5

[Douglas Gilbert]

                 SCSI Hotplug for lk 2.5

The direction the lk 2.5 series is moving is to do things like
device scanning, persistent mapping and other administrative
tasks in the user space, where practical.
To this end we need driverfs to:
	- work out a device naming strategy
	- add its class hierarchy
	- allow drivers to set and show parameters (and perhaps
	  state)
	- settle down
While this is happening perhaps we can discuss the design of
the SCSI Hotplug logic.

The hotplug subsystem provides an established mechanism for
drivers in the kernel to inform user space programs primarily
that a new device has been "hot plugged" or "hot unplugged". It
is used extensively by the USB subsystem in the lk 2.4 series.
This idea might be extended to provide user space alerts for
important state changes in a device (e.g. removable media
inserted).

Here are five categories of SCSI events that are candidates for
hotplug alerts:
	1) scsi (lower level) driver addition or removal
	2) scsi host (initiator) addition or removal
	3) scsi device (target/lun) addition or removal
	4) removable media addition or removal
	5) unit attention or "asynchronous event notification"
I believe hotplug alerts for 2) and 3) are required; for 1)
is questionable and for 4) + 5) would be "nice to have".

The existing /sbin/hotplug subsystem would be called from the
kernel space with a first "command line" parameter of "scsi"
with the following environment variables:
	ACTION		add | remove [ | event {for UA or AEN} ]
	CATEGORY	DRIVER | HOST | TARGET | MEDIA | EVENT
	TARGET_PATH	{for categories TARGET, MEDIA and EVENT}
	HOST_PATH	{for categories MEDIA, TARGET, and HOST}
	DRIVER_PATH	{for all categories}
	PERIPHERAL_TYPE	{for categories MEDIA and TARGET}
	SCSI_LEVEL	{for categories MEDIA and TARGET}
	SCSI_PROTOCOL	SPI | FC | USB_MASS_STORAGE | SBP | SAS ...
	.....
	VENDOR
	PRODUCT
	REVISION

There is a trade-off between how much information is passed to
/sbin/hotplug as environment variables and how much is available
from driverfs and other sources. Timing of the alerts is important:
the ACTION=add should be as late as possible so that all driverfs
information is available (and upper level drivers have attached
to the device ?). 

ACTION=remove (apart from media) comes with no additional
driverfs (or other kernel) information because the 
host/device is gone :-) In the scsimon driver (see note c) ) there 
is the notion of ADDITION_SEQNO and REMOVAL_SEQNO to track attaches 
and detaches (in this case of hosts and targets). Sequence numbers are
easy (and SMP safe) to implement in the kernel and could act as a
key for user space programs that are tracking the state of host
and device attachments.

Comments?


Notes:
  a)	SCSI host and device hotplug alerts could be added relatively
	easily to the scsi mid level
  b)	The "..._PATH" environment variables would be relative
	paths into the driverfs tree. For example:
		TARGET_PATH="root/pci0/00:0c.0/scsi1/1:0:0:0"
		DRIVER_PATH="bus/scsi/drivers/aic7xxx"
  c)	scsimon is an optional upper level scsi subsystem driver
	described at http://www.torque.net/scsi/scsimon.html
	Linus has made it clear that he thinks this driver does too
	much in the kernel space (e.g. holding trailing device state).
	It has an example of generating hotplug alerts, see 
	sm_call_policy() [which should use snprintf() in lk 2.5]
  d)	James Bottomley would like to do as much scanning and
	device identification as possible in the user space. I still
	think the mid level needs to do a 36 byte INQUIRY (the LLDDs
	may want to see a longer inquiry) and perhaps looks for
	INQUIRY VPD page=0x83
  e)	for multipath devices I would expect a hotplug alert for
	each path a device is discovered on (perhaps this is not
	practical)	
  f)	this document was written when lk 2.5.29 was current.
	lk 2.5.30 has just been released with driverfs patches
	that address some of the issues mentioned above


Doug Gilbert
1st August 2002

Re: [RFC] SCSI Hotplug for lk 2.5

[Oliver Neukum]

> Here are five categories of SCSI events that are candidates for
> hotplug alerts:
> 	1) scsi (lower level) driver addition or removal
> 	2) scsi host (initiator) addition or removal
> 	3) scsi device (target/lun) addition or removal
> 	4) removable media addition or removal
> 	5) unit attention or "asynchronous event notification"
> I believe hotplug alerts for 2) and 3) are required; for 1)
> is questionable and for 4) + 5) would be "nice to have".

How about new paths to a device? It seems that the FibreChannel
people need that.

> The existing /sbin/hotplug subsystem would be called from the
> kernel space with a first "command line" parameter of "scsi"
> with the following environment variables:
> 	ACTION		add | remove [ | event {for UA or AEN} ]
> 	CATEGORY	DRIVER | HOST | TARGET | MEDIA | EVENT
> 	TARGET_PATH	{for categories TARGET, MEDIA and EVENT}
> 	HOST_PATH	{for categories MEDIA, TARGET, and HOST}
> 	DRIVER_PATH	{for all categories}
> 	PERIPHERAL_TYPE	{for categories MEDIA and TARGET}

Merge this with CATEGORY ?

> 	SCSI_LEVEL	{for categories MEDIA and TARGET}
> 	SCSI_PROTOCOL	SPI | FC | USB_MASS_STORAGE | SBP | SAS ...
> 	.....
> 	VENDOR
> 	PRODUCT
> 	REVISION
>
> There is a trade-off between how much information is passed to
> /sbin/hotplug as environment variables and how much is available
> from driverfs and other sources. Timing of the alerts is important:

Ideally you'd want reduced functionality without driverfs mounted.

> the ACTION=add should be as late as possible so that all driverfs
> information is available (and upper level drivers have attached
> to the device ?).

Yes, after that.

> ACTION=remove (apart from media) comes with no additional
> driverfs (or other kernel) information because the
> host/device is gone :-) In the scsimon driver (see note c) ) there

This is very bad. You may very well have to change permissions
of a device node.

	Regards
		Oliver

Re: [RFC] SCSI Hotplug for lk 2.5

[Patrick Mansfield]

On Fri, Aug 02, 2002 at 11:40:29AM +0200, Oliver Neukum wrote:
> 
> How about new paths to a device? It seems that the FibreChannel
> people need that.

It is not only fibrechannel - dual initiated busses, dual ported
devices, maybe iSCSI and maybe other low level drivers can have
mutiple paths to the actual device.

Did you see Doug's:

  e)	for multipath devices I would expect a hotplug alert for
	each path a device is discovered on (perhaps this is not
	practical)	

With my current multi-path code, a hotplug per path is possible, and
not difficult to add. The device alert (as Doug describes) already includes
a path, we probably would also need some flag that says this is the first
path; on removal, you would want a flag saying if this was the last path.

User level code would have to notice (for example) that a device was
already found and mounted, and that this is just another path to the device.

-- Patrick Mansfield

Re: [RFC] SCSI Hotplug for lk 2.5

[Matthew Dharm]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On Thu, Aug 01, 2002 at 06:57:31PM -0400, Douglas Gilbert wrote:
>                  SCSI Hotplug for lk 2.5

On a related note....

It would be really nice if a low-level driver could signal, somehow, to the
VFS layer that it was a hot-unpluggable device, and thus precautions should
be taken (i.e. perfer mounting with -o sync, etc.)

Matt

-- 
Matthew Dharm                              Home: mdharm-usb@one-eyed-alien.net 
Maintainer, Linux USB Mass Storage Driver

C:  They kicked your ass, didn't they?
S:  They were cheating!
					-- The Chief and Stef
User Friendly, 11/19/1997

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [RFC] SCSI Hotplug for lk 2.5

[Matthew Jacob]

> It would be really nice if a low-level driver could signal, somehow, to the
> VFS layer that it was a hot-unpluggable device, and thus precautions should
> be taken (i.e. perfer mounting with -o sync, etc.)
> 

I don't think this s wise. By definition all SAN devices are
'hot-unpluggable', and you sure don't want to mount them sync only.

Re: [RFC] SCSI Hotplug for lk 2.5

[Patrick Mansfield]

On Thu, Aug 01, 2002 at 06:57:31PM -0400, Douglas Gilbert wrote:
>                  SCSI Hotplug for lk 2.5

Douglas -

I'm still trying to figure out what events we might need for user versus
kernel scanning or some combination. I'm re-reading James original OLS
paper about user scanning, and I've been thinking about how to scan from
user level. It might be best to do work on user level scanning after we
have better driverfs support, and always have initrd. 

I've been trying to re-code scsi_scan.c with user level scanning in mind
(not such a big deal, since we already have proc scanning), especially
so that we can scan a single target, rather than a single LUN via the
proc interface. We probably need a target hotplug event, this would be best
handled by adding a device_register() for each target; this is not so easy
for multi-path, since target id's can be different for the same target 
(even if the linux scsi id was the FC ID), so it would be easier to just
generate a hotplug event that could kick off scanning.

I haven't done anything with the device_list[] (in terms of hotplug or
moving it to user space).

> Here are five categories of SCSI events that are candidates for
> hotplug alerts:
> 	1) scsi (lower level) driver addition or removal
> 	2) scsi host (initiator) addition or removal
> 	3) scsi device (target/lun) addition or removal
> 	4) removable media addition or removal
> 	5) unit attention or "asynchronous event notification"
> I believe hotplug alerts for 2) and 3) are required; for 1)
> is questionable and for 4) + 5) would be "nice to have".

My understanding is that driverfs will automatically generate hotplug
events on each device_register() call. These have been added to scsi,
so the two main events are already handled, and we need not make other
changes.

Multi-path will also hook into driverfs, and driverfs would then
generate hotplug events for them.

-- Patrick Mansfield

Re: [RFC] SCSI Hotplug for lk 2.5

[Matthew Jacob]

On Sat, 3 Aug 2002, Oliver Neukum wrote:

> Am Freitag, 2. August 2002 23:21 schrieb Matthew Jacob:
> > > It would be really nice if a low-level driver could signal, somehow, to
> > > the VFS layer that it was a hot-unpluggable device, and thus precautions
> > > should be taken (i.e. perfer mounting with -o sync, etc.)
> >
> > I don't think this s wise. By definition all SAN devices are
> > 'hot-unpluggable', and you sure don't want to mount them sync only.
> 
> Other measures, eg preperation to handle replugging could so be shifted
> up. There's nothing lost be exporting it and some to gain.

You don't necessarily have predictive control over hotplugging.

> OTOH this might be better placed in driverfs than in hotplug.

I'd been given this some thought for a while, but I don't have a good
answer. Perhaps an 'involuntary hotplug' event, as opposed to a 'voluntary'
one is needed. I dunno.

RE: [RFC] SCSI Hotplug for lk 2.5

[Cress, Andrew R]

Doug/Patrick,

I've been working on a scan daemon in user-space for hot-plug disks (case 3
below).  It uses sg to access scsi devices, and also to access access any
SAF-TE HSCs.  It then invokes sfdisk, mdadm, etc. to auto-remove or
auto-insert the disk in a software raid like a hardware raid would.  It
works nicely if the device nodes already exist (replace a disk), but has
enumeration problems for new devices.  I'm currently using kernel 2.4.18,
but I'm interested in the discussion on this topic for 2.5.

I'm thinking that the kernel changes that would help would be to do a
rescan_scsis function, slightly different than scan_scsis, so that existing
devices are left alone, but new devices are initialized and attached to
device nodes.  This rescan_scsis could be triggered by a scsi reset (and
maybe by a user directive, which might be needed if the user-space daemon
detected a SAF-TE insertion, for example).  

What do you think?  Should this kind of thing be done with driverfs rather
than an sg device node?  

If you are interested, the source to what I'm working on is at
http://cvs.carrierlinux.org/viewcvs/viewcvs.cgi/components/scsirastools/src/
sgraidmon.c

Andy Cress

-----Original Message-----
From: Patrick Mansfield [mailto:patmans@us.ibm.com] 
Sent: Friday, August 02, 2002 9:28 PM
To: Douglas Gilbert
Cc: linux-scsi@vger.kernel.org
Subject: Re: [RFC] SCSI Hotplug for lk 2.5

On Thu, Aug 01, 2002 at 06:57:31PM -0400, Douglas Gilbert wrote:
>                  SCSI Hotplug for lk 2.5

Douglas -

I'm still trying to figure out what events we might need for user versus
kernel scanning or some combination. I'm re-reading James original OLS
paper about user scanning, and I've been thinking about how to scan from
user level. It might be best to do work on user level scanning after we
have better driverfs support, and always have initrd. 

I've been trying to re-code scsi_scan.c with user level scanning in mind
(not such a big deal, since we already have proc scanning), especially
so that we can scan a single target, rather than a single LUN via the
proc interface. We probably need a target hotplug event, this would be best
handled by adding a device_register() for each target; this is not so easy
for multi-path, since target id's can be different for the same target 
(even if the linux scsi id was the FC ID), so it would be easier to just
generate a hotplug event that could kick off scanning.

I haven't done anything with the device_list[] (in terms of hotplug or
moving it to user space).

> Here are five categories of SCSI events that are candidates for
> hotplug alerts:
> 	1) scsi (lower level) driver addition or removal
> 	2) scsi host (initiator) addition or removal
> 	3) scsi device (target/lun) addition or removal
> 	4) removable media addition or removal
> 	5) unit attention or "asynchronous event notification"
> I believe hotplug alerts for 2) and 3) are required; for 1)
> is questionable and for 4) + 5) would be "nice to have".

My understanding is that driverfs will automatically generate hotplug
events on each device_register() call. These have been added to scsi,
so the two main events are already handled, and we need not make other
changes.

Multi-path will also hook into driverfs, and driverfs would then
generate hotplug events for them.

-- Patrick Mansfield
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] SCSI Hotplug for lk 2.5

[Patrick Mansfield]

Andy -

On Mon, Aug 05, 2002 at 08:09:34AM -0700, Cress, Andrew R wrote:
> Doug/Patrick,
> 
> I've been working on a scan daemon in user-space for hot-plug disks (case 3
> below).  It uses sg to access scsi devices, and also to access access any
> SAF-TE HSCs.  It then invokes sfdisk, mdadm, etc. to auto-remove or
> auto-insert the disk in a software raid like a hardware raid would.  It
> works nicely if the device nodes already exist (replace a disk), but has
> enumeration problems for new devices.  I'm currently using kernel 2.4.18,
> but I'm interested in the discussion on this topic for 2.5.

I'm not very familiar with SAF-TE or the software raid tools.

Your program polls a special interface to the SAF-TE to figure out
what LUNs (or targets) are attached, and if a new LUN or target shows
up, you scan it? Does mdadm do the scan for you? 

Does it always generate a bus reset when a device is added or removed?

> I'm thinking that the kernel changes that would help would be to do a
> rescan_scsis function, slightly different than scan_scsis, so that existing
> devices are left alone, but new devices are initialized and attached to
> device nodes.  This rescan_scsis could be triggered by a scsi reset (and
> maybe by a user directive, which might be needed if the user-space daemon
> detected a SAF-TE insertion, for example).  
> 
> What do you think?  Should this kind of thing be done with driverfs rather
> than an sg device node?  

I don't have any specific answers.

There should probably be a hotplug event (via driverfs) for a new target
showing up - not part of the mid-layer or scanning, something the low
level driver might trigger (and not one of the 5 events Douglas listed,
unless his item 3 target/lun means target *or* lun on a target). 

Triggering off of a scsi reset would not be good, since the error handler
can send a reset, and not all SPI busses (enclosures) will generate a bus
reset on device addition or removal. Triggering a rescan via a daemon (like
your SAF-TE) would be ok, especially a rescan of a single LUN.

Generally, an entire rescan would be too much when we want to only scan
a target (or maybe even a single LUN). We also want an un-scan
or something to remove a target (and all of its LUNs), like the 
scsi proc remove-single-device (plus a way to remove a single Scsi_Host,
a problem with pci hotplug removing an adapter).

For FCP, adding a new target will generate an SCN (or a LIP), and there
could then be a hotplug event (via driverfs) for the new target (or the
removed target); removal generates an SCN (but not always a LIP).

For iSCSI there could be user level commands to add or remove connections,
generating a hotplug event (via driverfs), I don't know much about iSCSI.

When a new LUN is made available on a target, you could get a unit
attention (SPC spec shows a ASC/ASCQ 3F/0E REPORTED LUNS DATA HAS CHANGED),
but this would come back to every initiator on that target for every
LUN (not sure if it is really for every LUN, that could be specific to the
hardware). If it's every LUN, it might be execessive or difficult to
figure out if a rescan of the target is needed (imagine 100 LUNs on one
target, some of them seldom used, and getting a hotplug event flood).
Letting an admin manually rescan might be safer in such cases.

The current proc interface is good enough for selective scanning of
a specific LUN; it is not SMP safe right now - if the error handler
is running, or someone else is trying to access the host_queue list
without the BKL (outside of the proc interface) it is not SMP safe.
A Scsi_Device list lock is needed to protect removal/insertion of
a Scsi_Device.

-- Patrick Mansfield

> > Here are five categories of SCSI events that are candidates for
> > hotplug alerts:
> > 	1) scsi (lower level) driver addition or removal
> > 	2) scsi host (initiator) addition or removal
> > 	3) scsi device (target/lun) addition or removal
> > 	4) removable media addition or removal
> > 	5) unit attention or "asynchronous event notification"
> > I believe hotplug alerts for 2) and 3) are required; for 1)
> > is questionable and for 4) + 5) would be "nice to have".

Re: [RFC] SCSI Hotplug for lk 2.5

[Matthew Jacob]

> I'm not very familiar with SAF-TE or the software raid tools.
> 
> Your program polls a special interface to the SAF-TE to figure out
> what LUNs (or targets) are attached, and if a new LUN or target shows
> up, you scan it? Does mdadm do the scan for you? 
> 
> Does it always generate a bus reset when a device is added or removed?

Nope. SAF-TE is similar to SES. It's a SCSI protocol that you use to
identify components and status in enclosures. One of these objects
are 'disk slots' showing presence/absence.

RE: [RFC] SCSI Hotplug for lk 2.5

[Cress, Andrew R]

The SCSI SAF-TE interface to hot-swap controllers reports the number of
insertion events by slot number.  If there is a SAF-TE HSC, detecting the
insertion is easy, but if not, the user-space daemon has to have help to
notice that the insertion has happened.  

The two types of disk enclosures I have do generate resets when a new disk
is inserted, but I believe that there are disk units/enclosures that just
bring the disk online.  Perhaps even these have Unit Attention, or Start
Unit, or something to detect.  

TODO:
- Detect the disk insertion, which I think has to be done in kernel space.
  If there is HSC:  Use SAF-TE (or SES) to detect the insertion.
  If there is no HSC:
  Reset is one entry point to trigger looking for new SCSI disks, but we  
  probably need to trigger it at Unit Attention too as you suggest.
  Since the idea is to make this automatic, rather than require user 
  intervention, we don't know which target and/or lun to look for, so I
  think we still have to scan at the whole channel with Inquiry commands 
  (like scan_scsis does).  
- Create and attach a device node for the new disk.
  This logic would be similar to what scan_scsis does, but would be more
  surgical if the HSC is present.

Andy

-----Original Message-----
From: Patrick Mansfield [mailto:patmans@us.ibm.com] 
Sent: Tuesday, August 06, 2002 6:43 PM
Subject: Re: [RFC] SCSI Hotplug for lk 2.5

[...]
Your program polls a special interface to the SAF-TE to figure out
what LUNs (or targets) are attached, and if a new LUN or target shows
up, you scan it? Does mdadm do the scan for you? 

Does it always generate a bus reset when a device is added or removed?

> [...]  This rescan_scsis could be triggered by a scsi reset (and
> maybe by a user directive, which might be needed if the user-space daemon
> detected a SAF-TE insertion, for example).  
[...]

Triggering off of a scsi reset would not be good, since the error handler
can send a reset, and not all SPI busses (enclosures) will generate a bus
reset on device addition or removal. Triggering a rescan via a daemon (like
your SAF-TE) would be ok, especially a rescan of a single LUN.

[...]
When a new LUN is made available on a target, you could get a unit
attention (SPC spec shows a ASC/ASCQ 3F/0E REPORTED LUNS DATA HAS CHANGED),
but this would come back to every initiator on that target for every
LUN (not sure if it is really for every LUN, that could be specific to the
hardware). If it's every LUN, it might be execessive or difficult to
figure out if a rescan of the target is needed (imagine 100 LUNs on one
target, some of them seldom used, and getting a hotplug event flood).
Letting an admin manually rescan might be safer in such cases.

[...]