PCI hotplug question.

PCI hotplug question.

[Cameron, Steve]

Hi,

We have gotten PCI hot-plug with the cciss driver nominally 
working, (Maybe I abuse the word "working" here.) with 
2.4.21pre5 kernel, and a patch from Torben, and some
changes we've made to cciss (mainly in the initialization
code, waiting for the board's firmware to become ready, etc.)

That is, we can press the pushbutton to power off the slot, and
the driver's remove_one function will be called, and powering
back up the slot, the driver's init_one function will be called.

However, once the button is pressed, the driver has no choice
but to remove the instance.  This means (as currently implemented)
the per-adapter structure is deallocated.  In essence, the driver
is presuming that when remove_one is called, it is because rmmod
was invoked.

In particular, the driver is not expecting any more i/o requests
to come in.  If this expectation is met, things are ok.  i/o can
be done prior to hot-unplug, and after hot-re-plug, but may not
be attempted during or between these events.

However, with PCI hot-plug, this expectation is not guaranteed (or even
all that likely) to be met.  For example, if the md driver is using the cciss 
driver (for multipath typically, since cciss is for hardware RAID), the md
driver expects one of two things to happen with its requests.  
1) they succeed or 2) they at least return with failure status.  

It does not expect that they will 
make the driver dereference a pointer which is now NULL.  In particular
PCI hot-unplugging the driver does not in any way tell the md driver not
to attempt any more i/o to that device.

So, the question is this.  For HBA drivers, block storage drivers in
particular, are there plans to make hot-unplugging behave similarly to
rmmod (checking that nobody has the device open before kicking
the driver out and so on) or, is it the driver's duty to defend 
against incoming i/o's to devices which may be yanked out from
underneath it whenever somebody presses the pushbutton?

Thanks for any thoughts.

-- steve

Re: PCI hotplug question.

[Greg KH]

On Tue, Mar 04, 2003 at 05:10:16PM -0600, Cameron, Steve wrote:
> 
> So, the question is this.  For HBA drivers, block storage drivers in
> particular, are there plans to make hot-unplugging behave similarly to
> rmmod (checking that nobody has the device open before kicking
> the driver out and so on) or, is it the driver's duty to defend 
> against incoming i/o's to devices which may be yanked out from
> underneath it whenever somebody presses the pushbutton?

I can't find my copy of the PCI Hotplug spec right now (and it's not
free for download anymore...), but I think that once a user presses the
latch button, the OS _has_ to power down that slot within a reasonable
amount of time.  So there's no way that a driver could return an error
to the remove() callback and have a chance to still be around in a
moment or so.  And some systems (ACPI controlled PCI Hotplug), we don't
have a choice, as the BIOS is about to do the powerdown anyway, and we
can't stop it.

So no, we can't rely on the same module count type mechanism that rmmod
can use, so the driver has to do everything it possibly can to shutdown
that device, and clean up after itself when remove() is called.

I'll try to find my copy of the spec to verify this, but it might take a
while to dig it up.  Anyone else with access to it right now?

thanks,

greg k-h

Re: PCI hotplug question.

[Russell King]

On Tue, Mar 04, 2003 at 10:15:20PM -0800, Greg KH wrote:
> I can't find my copy of the PCI Hotplug spec right now (and it's not
> free for download anymore...), but I think that once a user presses the
> latch button, the OS _has_ to power down that slot within a reasonable
> amount of time.  So there's no way that a driver could return an error
> to the remove() callback and have a chance to still be around in a
> moment or so.  And some systems (ACPI controlled PCI Hotplug), we don't
> have a choice, as the BIOS is about to do the powerdown anyway, and we
> can't stop it.

Not only that, but with Cardbus-based PCI expansion systems, a complete
PCI bus tree could have been removed and be gone before the drivers get
notified, if the user pulls the cardbus card from the slot.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html