Re: unclean yanking out of device?

[linas@austin.ibm.com]

On Wed, Jan 14, 2004 at 04:19:39PM -0800, Greg KH wrote:
> 
> What kind of hardware base is this?  (PPC64, ia64, etc.)?

ppc64

> What specific drivers?

pcnet32 ethernet, some gigabit ethernet cards, some fibre-channel cards,
and the symbios-2 scsi driver.

> > Hmm.  Let me point out that other operating systems already do this
> > (and have been doing so for a while).   The chipset I'm working with
> > has been shipping for, I dunno, a couple of years or something like 
> > that.
> 
> What chipset?

The one in the ppc64 mainframes.  It shows up in the ppc64 arch code 
under the name of "phb", and its eeh.c that deals with configing the
support I'm talking about.  Right now, eeh.c calls panic() when things
go kaboom, and my goal is to replace the panic with a graceful recovery.

> > They've already gone through a four or five engineering upgrades
> > on the feature set.  So when you say you're sorry, did you mean 
> > "its a bad idea, I will personally oppose it",  or do you mean 
> > "you'll have to implement all of this by your lonesome"?
> 
> I (as the Linux kernel PCI and PCI Hotplug maintainer) oppose it.  :)

oh boy.  Well, its reassuring to know I'm talking to the right person.

> I think you just need to change the way you are thinking about this.
> Linux already has a pci hotplug framework.  Just implement your specific
> pci controller into this framework.  That way you will not have to
> modify any specific PCI card device drivers.  If you do this, then all
> of the existing userspace tools will work for your hardware.

I really really wasn't joking about the cosmic ray.  It hits. There
is an unrecoverable parity error.  What do I do?  

Lets assume the adapter is in the middle of a dma when this happens.
Do I pass the known-bad garbled data to the device driver, possibly
corupting the file system, etc?  Suppose the bad data was something
that trashed a pointer; some million or billion cpu cycles later
the kernel oopses for some mystery reason.   Instead of allowing
this kind of 'silent corruption' to occur, the current ppc64
code for this simply halts the system, it calls panic (in eeh.c), 
and that's that.

Suppose the error occured while the driver was PIO'ing some status
word to the adapter. Is it safe to continue on, and assume the adpater 
is in the state that the driver thinks its in?  Again, millions of
cycles later, your adapter either hangs for some mystery reason,
or maybe you've silently corrupted some data, e.g. some financial
data, which is the kind of stuff that these kinds of machines handle.

The error might have been an address error, then what? I have 
a pio/dma to a known-corrupted addresss.  What do I do with it?

What the chipset currently does is to bar all further i/o to that
slot until is re-enabled again.  I see two scenarios for recovering:

1) shut down the device driver, then re-enable i/o, then re-init the 
   device driver, or

2) tell the device driver to device-remove, then scurry off to 
   re-enable i/o for the duration of the remove, and then hope 
   the driver is able to successfully complete the remove (and 
   not garbage anything further in the process).

I'm thinking that 2) fits in your paradigm a whole lot better 
than 1) does.  It might be enough, I'd have to think about it 
and consult a bit.  But solution 1) really does make me sleep 
better at night.  

Here's why I say that: the failure may be one-shot (the cosmic ray)
or it may be repeatable: overheating, vibration, dust, humidity,
sysadmin accidentally dropped a screwdriver thing.  If its the latter,
then re-enabling i/o is going to do nothing except make matters
worse. 

This raises a third possibility:
3) keep i/o disabled, let the device driver keep going until it 
   realizes that the hardware is hung, and let it deal with that.

But that's kind of stupid, since I already *know* its hung. 
I may as well tell the device driver about it.

So what I *really* want to do is to tell the device driver,
"your hardware is hung. Deal with it."   This is pretty darn 
indistinguishable from the "stupid sysadmin unplugged before 
doing hotplug-remove" scenario.  So I figured if I solve the 
stupid-sysadmin scenario, then I get the vibration/dust/humidity
scenario for free.

Note that the machines currently in the field can support
thousands of pci slots, and that future machines will do even more.
So even if any one pci slot/adapter goes bad once every few years,
with a thousand slots, that works out to one or two failures a day.
If we panic'ed on each one, that would be a reboot a day. 

(Actually, these machines do stay up longer than that, which
means that the failure rate per slot is a whole lot less than that,
but I think this illustrates the point).

--linas



-------------------------------------------------------
This SF.net email is sponsored by: Perforce Software.
Perforce is the Fast Software Configuration Management System offering
advanced branching capabilities and atomic changes on 50+ platforms.
Free Eval! http://www.perforce.com/perforce/loadprog.html
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel